Introduction

With the first deployments in December 2020 and the SmartyCore development, the Director proposed a change in the focus of this Master Thesis. Instead of the e-commerce being the main use case, we would focus on SmartyCore and leave the e-commerce as a use case.

Today I want to post about the last changes in the SmartyCore plugin & tools. There was no development in January and early February due to exams.

SmartyCore

SmartyCore was conceived as a helpful tool to optimise the microservices’ code so they do not need to know the complex infrastructure below and a developer only needs to develop business logic on top of the SmartyCore abstractions. As the needs kept growing, the plugin also grew and became a bigger component of the Master Thesis

SmartyCore configuration

As SmartyCore grew, more features were added to it, so I had to update the configuration properties to allow that flexibility, without forgetting the main reason for this plugin is to allow microservices to be agnostic of the platform it runs into or the infrastructure if communicates with.

coredeps {
    microservice = <true/false>,
    library = <true/false>,
    api = <true/false>,
    kafka = <true/false>,
    mongo = <true/false>,
    elastic = <true/false>,
    mainClass = <string>,
    monitoringProperties = [
        <string>: <object>
    ]
}

The code above illustrates the simplicity of the plugin configuration, allowing a lot of magic to happen beneath, as we will discuss in the subsections below. SmartyCore configuration only needs to know what services each Microservice/library project needs (Microservice, library, Kafka, mongo, API, elastic) and where the application entry point is.

Another interesting configuration property is “monitoringProperty”. SmartyCore will expose three endpoints for monitoring purposes (Actuator Info, Actuator Prometheus and Actuator Health). This configuration property affects the first, as it allows the developer to expose more information in that endpoint, on top of the application basic information, dependencies versions, etc… added by SmartyCore automatically.

Gradle plugins

The first feature implemented in SmartyCore was a Gradle plugin to generate AVRO schemas from Java POJOs. This later evolved into a set of three plugins:

• AvroGenerator: Gradle plugin that generates .avsc schemas from Java classes and converts those AVSC to Kafka SpecificRecords out-of-the-box. This feature only requires the developer to annotate the Java class so the plugin recognises it. The plugin will also check the Java classes extend from a specific parent so they inherit common fields, such as event type, metadata, tracing…
• SmartyCore Config (SC Config): Gradle plugin to parse the above configuration.
• SmartyCore: Main Gradle plugin that performs the magic. This plugin will receive the configuration from the SC Config and inject all needed artefacts.

Automatic configuration of dependencies and repositories

In the previous paragraph, I said that SmartyCore can inject all needed artefacts. By artefacts, I mean repositories, dependencies, other Gradle plugins (e.g. AvroGenerator), etc… The following list states most of the artefacts SmartyCore configures automatically to the developer.

• Repositories & dependencies: Knowing what type of project the developer marked in the configuration, we can inject the dependencies the project needs and the Maven/Gradle repositories those dependencies reside.
• Gradle Plugins: Many of the plugins used by all the projects are almost the same with the difference in configuration values, so SmartyCore can also inject and enable only the plugins each project needs:
  • Libraries: Java Library, AvroPlugin, AvroGenerator Gradle Plugin (see above)
  • Microservice: Java Plugin
  • Common plugins to all projects: SpringBoot, DependencyManagement, Jacoco, Sonarqube…
• Gradle Tasks: As with the Gradle Plugins, SmartyCore also configures some common tasks or creates some useful ones:
  • Libraries: Jar enable, BootJar disable, AvroPlugin config, AvroGenerator config
  • Microservice: Java Plugin config (Enable task and configure main class)
  • Kafka: Configure DependencyManagement Plugin with the Spring Cloud BOM
  • Common tasks for all projects: Jacoco and JUnit configuration
  • Custom tasks for all projects: Print all configured repositories, validate SmartyCore configuration (e.g. Main class is available in classpath), generate Kubernetes deployment artefacts (service, deployment, ingress, autoscaler…)
• Application utils: To avoid unnecessary configuration in each project, SmartyCore will include default application.yml configuration based on the environment the application is deployed. It will also include support for additional topics such as automatic testing to have small snippets of code that contain the exact business test and do not need to configure the Kafka broker, Schema Registry, MongoDB database… The project will not know what infrastructure is below, only its kind (event broker, database…).
• Monitoring and security: The last developed topic is the automatic configuration of the monitoring (using Prometheus & Grafana) and security based on OAuth2 scopes & roles. The projects should not reinvent the wheel, so these topics are centralised in SmartyCore, which I will show in the following posts.

Many more features in-the-making

These last months I have developed many more SmartyCore features than the ones described here, such as:

• Automatic security design by default
• Automatic Kubernetes configuration and deployment
• Automatic infrastructure artefact deployment (Kafka Schema CRUD, topic CRUD…)
• Observability: Monitoring with Prometheus and Grafana

In an effort to keep this post shorter, the features of the above list will be published in a few days in a separate post.

In this next post, I will also include snapshots of the current status of the Grafana dashboards, the frontend of the system and the status of the e-commerce development (including deployment diagrams).

Future steps

With the current status of the project, we can see two groups of features that still need development. On one hand, the technical ones, such as observability (log aggregation & retrieval, trace analysis and load testing).

On the other hand, there are a few business domains still to be analysed & developed, such as the checkout, better uses of the Elasticsearch indices (including indexing more information and smarter facetted searches) or improvements in the user account domain. I will write about the user account domain in the next post as this was a requirement for the “Automatic security design by default“.

What is the “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.