Chapter 3: Installing the Infrastructure

After finishing the Spring Semester’s exams, I have been able to continue the Project between trips and visiting relatives who live far apart. These two months I have been focused on deploying a robust infrastructure so I need not think about it more than necessary.

Automated elements

The infrastructure has the following elements:

• CICD: Atlassian Bitbucket, Jenkins (tried Atlassian Bamboo but it is too limited) and SonarQube.
• Issue manager and documentation: Atlassian JIRA and Atlassian Confluence.
• Deployments: Nexus (Docker containers and libraries) and Kubernetes.
• Additional tools: Elasticsearch, Apache Kafka and MongoDB.

One assumption all the infrastructure has is that there are two environments, one with unstable code (staging – develop) and one with stable code (preproduction – master). As this project focuses on having a preproduction product be deployed to production, there is no productive environment.

The first thing I set up was the Atlassian ecosystem (JIRA, Confluence and BitBucket). With these elements in place, the next part was Jenkins. This was my first time installing and configuring Jenkins, not just using it, so the magic I was used to in other projects needed to be demolished.

Jenkins Templating Engine (JTE)

On one side, I wanted to just define the type of project and let Jenkins decide the programmed stages to build, test and deploy. On the other side, I needed to define which programmed steps were available. After researching a lot on how is Jenkins at its core, I found what seemed to be my solution: Jenkins Templating Engine (JTE).

JTE seemed so easy and to be the perfect match for my requirements. Oh boy, was I wrong! Although it was exactly what I needed, it was not easy at all. First, JTE lacks proper documentation. Second, the syntax is quite similar to the declarative pipelines (DP) and not all elements of the DP are supported yet.

Finally, after spending almost two weeks of daily work, I managed to define all the steps I needed and implement most of them. The only JTE code all git projects have to set was a one-line file to load either the library JTE libraries or the microservice JTE libraries.

Jenkins pipeline

After all the templatizing made with JTE, the basic stages are live and work:

• Build phase: Checkout code from SCM and build the project.
• Test & Quality phase: Run all tests and check if the code complies with the Sonar Quality Gates. All branches and PRs are analized.
• Deploy phase: Only run in case of master and develop branches. If the project is a library, it is only deployed to Nexus Sonatype in snapshots or releases. In the case of a microservice, the docker container is built and then deployed to Nexus Sonatype and to Kubernetes (K8S) after.
• Slack notification: After each Jenkins pipeline run, notify Slack channels (one is for successfull builds and another for failing ones.

Kubernetes (K8S)

K8S was another tool I had only used, not managed. In this case, the installation and basic usage were easier than expected. I could deploy some dummy containers to test what Deployments and Services are. As there is more than one node in the cluster, I had to set up an HAProxy to balance the requests across all nodes.

I had to choose what type of API Manager I wanted to use in the project. The two main options were Spring Cloud Eureka and K8S Ingress Controller, the native approach of Kubernetes. After writing the pros and cons of each the decision was the native approach with NGINX.

After so much trouble because the requests did not reach the pod, reinstalling the Ingress Controller and trying multiple configurations I realized everything was running under HTTPS/SSL with a custom CA except the HAProxy. Adding the certificates to the latter solved all my problems and the APIs could be accessed from outside the cluster.

When K8S was completely stable, the deployment was automated via Jenkins and, when a deployment is to be performed, Jenkins connects to K8S to create the Deployment, Service and Ingress of a microservice. If the branch being deployed is “develop”, all K8S resources are run against the develop K8S namespace (staging environment). If the branch is “master”, the K8S resources go to the master K8S namespace (preproduction environment).

What still needs to be automated?

Most of the CICD & Infrastructure components are ready and have been tested with a dummy Spring Boot Application, but there are a few things left to be automated:

• Creation of Kafka Topics
• Elasticsearch index management.
• MongoDB database collections management.

When all resource provisioning tasks are automated, using a Bitbucket repository linked to a Jenkins pipeline should allow the management of all pending automation.

Future steps

With most of the infrastructure deployed and working, the next step is fast-forwarding. As the project is split into 3 versions and the first is the ingest stages, next month I will be developing the ingest of all business domains, the indexations and basic search in Elasticsearch and a frontend to browse products.

What is “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.