Blog – lAngelP

Chapter 5: Approaching the end

6 April 20226 April 2022
by lAngelP

Introduction

With the first deployments in December 2020 and the SmartyCore development, the Director proposed a change in the focus of this Master Thesis. Instead of the e-commerce being the main use case, we would focus on SmartyCore and leave the e-commerce as a use case.

Today I want to post about the last changes in the SmartyCore plugin & tools. There was no development in January and early February due to exams.

SmartyCore

SmartyCore was conceived as a helpful tool to optimise the microservices’ code so they do not need to know the complex infrastructure below and a developer only needs to develop business logic on top of the SmartyCore abstractions. As the needs kept growing, the plugin also grew and became a bigger component of the Master Thesis

SmartyCore configuration

As SmartyCore grew, more features were added to it, so I had to update the configuration properties to allow that flexibility, without forgetting the main reason for this plugin is to allow microservices to be agnostic of the platform it runs into or the infrastructure if communicates with.

coredeps {
    microservice = <true/false>,
    library = <true/false>,
    api = <true/false>,
    kafka = <true/false>,
    mongo = <true/false>,
    elastic = <true/false>,
    mainClass = <string>,
    monitoringProperties = [
        <string>: <object>
    ]
}

The code above illustrates the simplicity of the plugin configuration, allowing a lot of magic to happen beneath, as we will discuss in the subsections below. SmartyCore configuration only needs to know what services each Microservice/library project needs (Microservice, library, Kafka, mongo, API, elastic) and where the application entry point is.

Another interesting configuration property is “monitoringProperty”. SmartyCore will expose three endpoints for monitoring purposes (Actuator Info, Actuator Prometheus and Actuator Health). This configuration property affects the first, as it allows the developer to expose more information in that endpoint, on top of the application basic information, dependencies versions, etc… added by SmartyCore automatically.

Gradle plugins

The first feature implemented in SmartyCore was a Gradle plugin to generate AVRO schemas from Java POJOs. This later evolved into a set of three plugins:

AvroGenerator: Gradle plugin that generates .avsc schemas from Java classes and converts those AVSC to Kafka SpecificRecords out-of-the-box. This feature only requires the developer to annotate the Java class so the plugin recognises it. The plugin will also check the Java classes extend from a specific parent so they inherit common fields, such as event type, metadata, tracing…
SmartyCore Config (SC Config): Gradle plugin to parse the above configuration.
SmartyCore: Main Gradle plugin that performs the magic. This plugin will receive the configuration from the SC Config and inject all needed artefacts.

Automatic configuration of dependencies and repositories

In the previous paragraph, I said that SmartyCore can inject all needed artefacts. By artefacts, I mean repositories, dependencies, other Gradle plugins (e.g. AvroGenerator), etc… The following list states most of the artefacts SmartyCore configures automatically to the developer.

Repositories & dependencies: Knowing what type of project the developer marked in the configuration, we can inject the dependencies the project needs and the Maven/Gradle repositories those dependencies reside.
Gradle Plugins: Many of the plugins used by all the projects are almost the same with the difference in configuration values, so SmartyCore can also inject and enable only the plugins each project needs:
- Libraries: Java Library, AvroPlugin, AvroGenerator Gradle Plugin (see above)
- Microservice: Java Plugin
- Common plugins to all projects: SpringBoot, DependencyManagement, Jacoco, Sonarqube…
Gradle Tasks: As with the Gradle Plugins, SmartyCore also configures some common tasks or creates some useful ones:
- Libraries: Jar enable, BootJar disable, AvroPlugin config, AvroGenerator config
- Microservice: Java Plugin config (Enable task and configure main class)
- Kafka: Configure DependencyManagement Plugin with the Spring Cloud BOM
- Common tasks for all projects: Jacoco and JUnit configuration
- Custom tasks for all projects: Print all configured repositories, validate SmartyCore configuration (e.g. Main class is available in classpath), generate Kubernetes deployment artefacts (service, deployment, ingress, autoscaler…)
Application utils: To avoid unnecessary configuration in each project, SmartyCore will include default application.yml configuration based on the environment the application is deployed. It will also include support for additional topics such as automatic testing to have small snippets of code that contain the exact business test and do not need to configure the Kafka broker, Schema Registry, MongoDB database… The project will not know what infrastructure is below, only its kind (event broker, database…).
Monitoring and security: The last developed topic is the automatic configuration of the monitoring (using Prometheus & Grafana) and security based on OAuth2 scopes & roles. The projects should not reinvent the wheel, so these topics are centralised in SmartyCore, which I will show in the following posts.

Many more features in-the-making

These last months I have developed many more SmartyCore features than the ones described here, such as:

Automatic security design by default
Automatic Kubernetes configuration and deployment
Automatic infrastructure artefact deployment (Kafka Schema CRUD, topic CRUD…)
Observability: Monitoring with Prometheus and Grafana

In an effort to keep this post shorter, the features of the above list will be published in a few days in a separate post.

In this next post, I will also include snapshots of the current status of the Grafana dashboards, the frontend of the system and the status of the e-commerce development (including deployment diagrams).

Future steps

With the current status of the project, we can see two groups of features that still need development. On one hand, the technical ones, such as observability (log aggregation & retrieval, trace analysis and load testing).

On the other hand, there are a few business domains still to be analysed & developed, such as the checkout, better uses of the Elasticsearch indices (including indexing more information and smarter facetted searches) or improvements in the user account domain. I will write about the user account domain in the next post as this was a requirement for the “Automatic security design by default“.

What is the “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.

Diary of a Master Thesis

Chapter 4: First functional deployments

10 November 20216 April 2022
by lAngelP

After installing the infrastructure and configuring all the elements (Kafka, MongoDB, Kubernetes, ElasticSearch…) to be highly available and resilient, the first domains have been deployed and the next ones are becoming ready to be developed. The project is scheduled to have 3 major milestones:

Read-only platform (prices, products, searcher, promotions…)
Checkout platform (user accounts, cart and checkout process)
Final project (technical tweaks and non-functional testing – performance, resiliency)

Frontend

The project has two frontend components: one for the administrative back office and another one for the actual e-commerce platform. Only the last one will be visible to the user when purchasing items.

Right now, only the e-commerce one is being developed. I expect the back office to be ready for the last milestone. With that being said, the e-commerce frontend is up-to-date with the backend. Although I am now being both a frontend and backend developer, both projects live in isolation.

To achieve this isolation, I created a mock of all available endpoints in the backend. As long as the API contract is the same in both the backend and the mocked one, the only integration problems that shall occur are if the URIs are not valid or the backend does not answer.

SmartyCore

After developing multiple microservices, all of them can be characterised in a small subset of types. These types correspond to the underlying technologies the microservice needs (Kafka, API, ElasticSearch, library…). All microservices is compounded by one or more of these types and each type follows the same pattern for its dependencies, the code components it needs, etc…

How I realised this was a fact was after building my 4th Kafka-MongoDB Microservice and having copied the same code from one to another and having to fix issues in all of them at once. The copy-paste technique is valid if the code is in a few code projects, but in my case was no longer an option.

So, how I proceeded? I analysed how the code was in all of them and extracted the common aspects, including the dependencies of that code, into a middleware between the code and between the infrastructure and Spring. The extracted code ended in a Gradle project, with a submodule for each type (Kafka, API, elastic…). That Gradle Project also has a submodule for a Gradle Plugin that adds all the dependencies, including the extracted code.

smartycore {
    api = true
    microservice = true
    mongodb = true
}

So, now a microservice only needs to add a Gradle Plugin and provide its types with a snippet similar to the one above. With that, all dependencies and helpers get imported magically. The first clear impact is the performance, now a new microservice is developed and deployed much, MUCH faster.

Backend

Finished domains

I started with the easier domains so I can handle the errors in a proper way, rather than in the most complex domains. So, right now, the system can handle products and prices, which are stored in MongoDB. The products are also indexed in an ElasticSearch.
The products and prices are received via Kafka, then processed and stored in the database. After it is stored, the API can provide the data and the frontend display it.

Just-designed domains

The first complex domain is the promotion domain, as it needs information from both products and prices. I did an initial design and refined it to fix the design issues. After that and being in a dead-end, I started from scratch and now I have two designs. I am analysing both designs and picking the best elements from each one.

Product image storing

The product images were also a headache, not because of the difficulty, but because of the many available options to store them: a database, a stateful set of K8S, NFS…

After writing the advantages and disadvantages of each model, I decided to store the images in GridFS inside MongoDB. Why? First, because I already had a MongoDB cluster. Secondly, because I needed resilience and high availability and how I set up MongoDB this is straightforward.

Next steps

Now, I have near 10 microservices deployed in the staging environment. The next steps will be the promotion domain and its integration in the frontend and deploying everything in the preproduction environment so I can perform the performance tests.

What is the “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.

Diary of a Master Thesis

Chapter 3: Installing the Infrastructure

25 August 202110 November 2021
by lAngelP

After finishing the Spring Semester’s exams, I have been able to continue the Project between trips and visiting relatives who live far apart. These two months I have been focused on deploying a robust infrastructure so I need not think about it more than necessary.

Automated elements

The infrastructure has the following elements:

CICD: Atlassian Bitbucket, Jenkins (tried Atlassian Bamboo but it is too limited) and SonarQube.
Issue manager and documentation: Atlassian JIRA and Atlassian Confluence.
Deployments: Nexus (Docker containers and libraries) and Kubernetes.
Additional tools: Elasticsearch, Apache Kafka and MongoDB.

One assumption all the infrastructure has is that there are two environments, one with unstable code (staging – develop) and one with stable code (preproduction – master). As this project focuses on having a preproduction product be deployed to production, there is no productive environment.

The first thing I set up was the Atlassian ecosystem (JIRA, Confluence and BitBucket). With these elements in place, the next part was Jenkins. This was my first time installing and configuring Jenkins, not just using it, so the magic I was used to in other projects needed to be demolished.

Jenkins Templating Engine (JTE)

On one side, I wanted to just define the type of project and let Jenkins decide the programmed stages to build, test and deploy. On the other side, I needed to define which programmed steps were available. After researching a lot on how is Jenkins at its core, I found what seemed to be my solution: Jenkins Templating Engine (JTE).

JTE seemed so easy and to be the perfect match for my requirements. Oh boy, was I wrong! Although it was exactly what I needed, it was not easy at all. First, JTE lacks proper documentation. Second, the syntax is quite similar to the declarative pipelines (DP) and not all elements of the DP are supported yet.

Finally, after spending almost two weeks of daily work, I managed to define all the steps I needed and implement most of them. The only JTE code all git projects have to set was a one-line file to load either the library JTE libraries or the microservice JTE libraries.

Jenkins pipeline

After all the templatizing made with JTE, the basic stages are live and work:

Build phase: Checkout code from SCM and build the project.
Test & Quality phase: Run all tests and check if the code complies with the Sonar Quality Gates. All branches and PRs are analized.
Deploy phase: Only run in case of master and develop branches. If the project is a library, it is only deployed to Nexus Sonatype in snapshots or releases. In the case of a microservice, the docker container is built and then deployed to Nexus Sonatype and to Kubernetes (K8S) after.
Slack notification: After each Jenkins pipeline run, notify Slack channels (one is for successfull builds and another for failing ones.

Kubernetes (K8S)

K8S was another tool I had only used, not managed. In this case, the installation and basic usage were easier than expected. I could deploy some dummy containers to test what Deployments and Services are. As there is more than one node in the cluster, I had to set up an HAProxy to balance the requests across all nodes.

I had to choose what type of API Manager I wanted to use in the project. The two main options were Spring Cloud Eureka and K8S Ingress Controller, the native approach of Kubernetes. After writing the pros and cons of each the decision was the native approach with NGINX.

After so much trouble because the requests did not reach the pod, reinstalling the Ingress Controller and trying multiple configurations I realized everything was running under HTTPS/SSL with a custom CA except the HAProxy. Adding the certificates to the latter solved all my problems and the APIs could be accessed from outside the cluster.

When K8S was completely stable, the deployment was automated via Jenkins and, when a deployment is to be performed, Jenkins connects to K8S to create the Deployment, Service and Ingress of a microservice. If the branch being deployed is “develop”, all K8S resources are run against the develop K8S namespace (staging environment). If the branch is “master”, the K8S resources go to the master K8S namespace (preproduction environment).

What still needs to be automated?

Most of the CICD & Infrastructure components are ready and have been tested with a dummy Spring Boot Application, but there are a few things left to be automated:

Creation of Kafka Topics
Elasticsearch index management.
MongoDB database collections management.

When all resource provisioning tasks are automated, using a Bitbucket repository linked to a Jenkins pipeline should allow the management of all pending automation.

Future steps

With most of the infrastructure deployed and working, the next step is fast-forwarding. As the project is split into 3 versions and the first is the ingest stages, next month I will be developing the ingest of all business domains, the indexations and basic search in Elasticsearch and a frontend to browse products.

What is “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.

Diary of a Master Thesis

Chapter 2: The Microservice Architecture

12 April 202114 April 2021
by lAngelP

It has been a long time since my first post of this series. However, now I have lots of changes that are set and defined. In this post, we will discuss the architecture and the data model.

In the last post, I stated the main guidelines of the Master Thesis, the development and deployment of an e-commerce platform on a Cloud Microservice Architecture. After more than 2 months, we have defined the main requirements of the platform and a first approach to the architecture and the data layer. Also, the first fully functional version is expected to be finished next July 2021 only with the happy path.

This first analysis has resulted in between 17 and 19 microservices across the 9 domains, not taking into account the frontend and back office of the application. Each domain also has at least one persistent storage unit, which is either a database (relational or NoSQL) or an ElasticSearch pod. These last days, I also finished the design of all entities each domain is responsible for.

Next steps
After this second milestone, the next steps will be generating a dummy project for each microservice with the needed libraries to represent the data model and deploy all of them in the 2-node Kubernetes Cluster.

What is “Diary of a Master Thesis”?
Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis that is supposed to be defended in September 2022.

Diary of a Master Thesis

Chapter 1: Everything has a starting point

28 December 202012 April 2021
by lAngelP

I have decided to collect in posts the different steps I will be performing from the Master Thesis beginning until I hand it in and defend. At this point, after a brief kickoff, my director and I have reached the first approach in the analysis phase: the development and deployment of an e-commerce platform using Cloud Computing and Microservices technologies.

My director and I have set the Business lines the project needs and we have defined the most important Distributed System properties that will impact the application directly. We do not want to deploy an e-commerce platform and finish the project, we want to see where the limits of this technology are and what is the current state-of-the-art, to show important improvements to the Industry.

Right now, the long-distance race of this Thesis has just started, and will not finish until mid-2022 with the defence of a Master Thesis I truly enjoy, and the earning of the Master Degree Diploma. In the midpoint, the various subjects of this degree will help to build and add even better technical knowledge to this Project.

The next milestone is the Analysis phase: write the requirements and define the boundaries.