Chapter 5: Approaching the end

Introduction

With the first deployments in December 2020 and the SmartyCore development, the Director proposed a change in the focus of this Master Thesis. Instead of the e-commerce being the main use case, we would focus on SmartyCore and leave the e-commerce as a use case.

Today I want to post about the last changes in the SmartyCore plugin & tools. There was no development in January and early February due to exams.

SmartyCore

SmartyCore was conceived as a helpful tool to optimise the microservices’ code so they do not need to know the complex infrastructure below and a developer only needs to develop business logic on top of the SmartyCore abstractions. As the needs kept growing, the plugin also grew and became a bigger component of the Master Thesis

SmartyCore configuration

As SmartyCore grew, more features were added to it, so I had to update the configuration properties to allow that flexibility, without forgetting the main reason for this plugin is to allow microservices to be agnostic of the platform it runs into or the infrastructure if communicates with.

coredeps {
    microservice = <true/false>,
    library = <true/false>,
    api = <true/false>,
    kafka = <true/false>,
    mongo = <true/false>,
    elastic = <true/false>,
    mainClass = <string>,
    monitoringProperties = [
        <string>: <object>
    ]
}

The code above illustrates the simplicity of the plugin configuration, allowing a lot of magic to happen beneath, as we will discuss in the subsections below. SmartyCore configuration only needs to know what services each Microservice/library project needs (Microservice, library, Kafka, mongo, API, elastic) and where the application entry point is.

Another interesting configuration property is “monitoringProperty”. SmartyCore will expose three endpoints for monitoring purposes (Actuator Info, Actuator Prometheus and Actuator Health). This configuration property affects the first, as it allows the developer to expose more information in that endpoint, on top of the application basic information, dependencies versions, etc… added by SmartyCore automatically.

Gradle plugins

The first feature implemented in SmartyCore was a Gradle plugin to generate AVRO schemas from Java POJOs. This later evolved into a set of three plugins:

• AvroGenerator: Gradle plugin that generates .avsc schemas from Java classes and converts those AVSC to Kafka SpecificRecords out-of-the-box. This feature only requires the developer to annotate the Java class so the plugin recognises it. The plugin will also check the Java classes extend from a specific parent so they inherit common fields, such as event type, metadata, tracing…
• SmartyCore Config (SC Config): Gradle plugin to parse the above configuration.
• SmartyCore: Main Gradle plugin that performs the magic. This plugin will receive the configuration from the SC Config and inject all needed artefacts.

Automatic configuration of dependencies and repositories

In the previous paragraph, I said that SmartyCore can inject all needed artefacts. By artefacts, I mean repositories, dependencies, other Gradle plugins (e.g. AvroGenerator), etc… The following list states most of the artefacts SmartyCore configures automatically to the developer.

• Repositories & dependencies: Knowing what type of project the developer marked in the configuration, we can inject the dependencies the project needs and the Maven/Gradle repositories those dependencies reside.
• Gradle Plugins: Many of the plugins used by all the projects are almost the same with the difference in configuration values, so SmartyCore can also inject and enable only the plugins each project needs:
  • Libraries: Java Library, AvroPlugin, AvroGenerator Gradle Plugin (see above)
  • Microservice: Java Plugin
  • Common plugins to all projects: SpringBoot, DependencyManagement, Jacoco, Sonarqube…
• Gradle Tasks: As with the Gradle Plugins, SmartyCore also configures some common tasks or creates some useful ones:
  • Libraries: Jar enable, BootJar disable, AvroPlugin config, AvroGenerator config
  • Microservice: Java Plugin config (Enable task and configure main class)
  • Kafka: Configure DependencyManagement Plugin with the Spring Cloud BOM
  • Common tasks for all projects: Jacoco and JUnit configuration
  • Custom tasks for all projects: Print all configured repositories, validate SmartyCore configuration (e.g. Main class is available in classpath), generate Kubernetes deployment artefacts (service, deployment, ingress, autoscaler…)
• Application utils: To avoid unnecessary configuration in each project, SmartyCore will include default application.yml configuration based on the environment the application is deployed. It will also include support for additional topics such as automatic testing to have small snippets of code that contain the exact business test and do not need to configure the Kafka broker, Schema Registry, MongoDB database… The project will not know what infrastructure is below, only its kind (event broker, database…).
• Monitoring and security: The last developed topic is the automatic configuration of the monitoring (using Prometheus & Grafana) and security based on OAuth2 scopes & roles. The projects should not reinvent the wheel, so these topics are centralised in SmartyCore, which I will show in the following posts.

Many more features in-the-making

These last months I have developed many more SmartyCore features than the ones described here, such as:

• Automatic security design by default
• Automatic Kubernetes configuration and deployment
• Automatic infrastructure artefact deployment (Kafka Schema CRUD, topic CRUD…)
• Observability: Monitoring with Prometheus and Grafana

In an effort to keep this post shorter, the features of the above list will be published in a few days in a separate post.

In this next post, I will also include snapshots of the current status of the Grafana dashboards, the frontend of the system and the status of the e-commerce development (including deployment diagrams).

Future steps

With the current status of the project, we can see two groups of features that still need development. On one hand, the technical ones, such as observability (log aggregation & retrieval, trace analysis and load testing).

On the other hand, there are a few business domains still to be analysed & developed, such as the checkout, better uses of the Elasticsearch indices (including indexing more information and smarter facetted searches) or improvements in the user account domain. I will write about the user account domain in the next post as this was a requirement for the “Automatic security design by default“.

What is the “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.

Chapter 4: First functional deployments

After installing the infrastructure and configuring all the elements (Kafka, MongoDB, Kubernetes, ElasticSearch…) to be highly available and resilient, the first domains have been deployed and the next ones are becoming ready to be developed. The project is scheduled to have 3 major milestones: 

• Read-only platform (prices, products, searcher, promotions…)
• Checkout platform (user accounts, cart and checkout process)
• Final project (technical tweaks and non-functional testing – performance, resiliency)

Frontend

The project has two frontend components: one for the administrative back office and another one for the actual e-commerce platform. Only the last one will be visible to the user when purchasing items.

Right now, only the e-commerce one is being developed. I expect the back office to be ready for the last milestone. With that being said, the e-commerce frontend is up-to-date with the backend. Although I am now being both a frontend and backend developer, both projects live in isolation.

To achieve this isolation, I created a mock of all available endpoints in the backend. As long as the API contract is the same in both the backend and the mocked one, the only integration problems that shall occur are if the URIs are not valid or the backend does not answer.

SmartyCore

After developing multiple microservices, all of them can be characterised in a small subset of types. These types correspond to the underlying technologies the microservice needs (Kafka, API, ElasticSearch, library…). All microservices is compounded by one or more of these types and each type follows the same pattern for its dependencies, the code components it needs, etc…

How I realised this was a fact was after building my 4th Kafka-MongoDB Microservice and having copied the same code from one to another and having to fix issues in all of them at once. The copy-paste technique is valid if the code is in a few code projects, but in my case was no longer an option. 

So, how I proceeded? I analysed how the code was in all of them and extracted the common aspects, including the dependencies of that code, into a middleware between the code and between the infrastructure and Spring. The extracted code ended in a Gradle project, with a submodule for each type (Kafka, API, elastic…). That Gradle Project also has a submodule for a Gradle Plugin that adds all the dependencies, including the extracted code.

smartycore {
    api = true
    microservice = true
    mongodb = true
}

So, now a microservice only needs to add a Gradle Plugin and provide its types with a snippet similar to the one above. With that, all dependencies and helpers get imported magically. The first clear impact is the performance, now a new microservice is developed and deployed much, MUCH faster.

Backend

Finished domains

I started with the easier domains so I can handle the errors in a proper way, rather than in the most complex domains. So, right now, the system can handle products and prices, which are stored in MongoDB. The products are also indexed in an ElasticSearch.
The products and prices are received via Kafka, then processed and stored in the database. After it is stored, the API can provide the data and the frontend display it.

Just-designed domains

The first complex domain is the promotion domain, as it needs information from both products and prices. I did an initial design and refined it to fix the design issues. After that and being in a dead-end, I started from scratch and now I have two designs. I am analysing both designs and picking the best elements from each one.

Product image storing

The product images were also a headache, not because of the difficulty, but because of the many available options to store them: a database, a stateful set of K8S, NFS… 

After writing the advantages and disadvantages of each model, I decided to store the images in GridFS inside MongoDB. Why? First, because I already had a MongoDB cluster. Secondly, because I needed resilience and high availability and how I set up MongoDB this is straightforward.

Next steps

Now, I have near 10 microservices deployed in the staging environment. The next steps will be the promotion domain and its integration in the frontend and deploying everything in the preproduction environment so I can perform the performance tests.

What is the “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.

Chapter 2: The Microservice Architecture

It has been a long time since my first post of this series. However, now I have lots of changes that are set and defined. In this post, we will discuss the architecture and the data model.

In the last post, I stated the main guidelines of the Master Thesis, the development and deployment of an e-commerce platform on a Cloud Microservice Architecture. After more than 2 months, we have defined the main requirements of the platform and a first approach to the architecture and the data layer. Also, the first fully functional version is expected to be finished next July 2021 only with the happy path.

This first analysis has resulted in between 17 and 19 microservices across the 9 domains, not taking into account the frontend and back office of the application. Each domain also has at least one persistent storage unit, which is either a database (relational or NoSQL) or an ElasticSearch pod. These last days, I also finished the design of all entities each domain is responsible for.

Next steps
After this second milestone, the next steps will be generating a dummy project for each microservice with the needed libraries to represent the data model and deploy all of them in the 2-node Kubernetes Cluster.

What is “Diary of a Master Thesis”?
Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis that is supposed to be defended in September 2022.