After installing the infrastructure and configuring all the elements (Kafka, MongoDB, Kubernetes, ElasticSearch…) to be highly available and resilient, the first domains have been deployed and the next ones are becoming ready to be developed. The project is scheduled to have 3 major milestones: 

• Read-only platform (prices, products, searcher, promotions…)
• Checkout platform (user accounts, cart and checkout process)
• Final project (technical tweaks and non-functional testing – performance, resiliency)

Frontend

The project has two frontend components: one for the administrative back office and another one for the actual e-commerce platform. Only the last one will be visible to the user when purchasing items.

Right now, only the e-commerce one is being developed. I expect the back office to be ready for the last milestone. With that being said, the e-commerce frontend is up-to-date with the backend. Although I am now being both a frontend and backend developer, both projects live in isolation.

To achieve this isolation, I created a mock of all available endpoints in the backend. As long as the API contract is the same in both the backend and the mocked one, the only integration problems that shall occur are if the URIs are not valid or the backend does not answer.

SmartyCore

After developing multiple microservices, all of them can be characterised in a small subset of types. These types correspond to the underlying technologies the microservice needs (Kafka, API, ElasticSearch, library…). All microservices is compounded by one or more of these types and each type follows the same pattern for its dependencies, the code components it needs, etc…

How I realised this was a fact was after building my 4th Kafka-MongoDB Microservice and having copied the same code from one to another and having to fix issues in all of them at once. The copy-paste technique is valid if the code is in a few code projects, but in my case was no longer an option. 

So, how I proceeded? I analysed how the code was in all of them and extracted the common aspects, including the dependencies of that code, into a middleware between the code and between the infrastructure and Spring. The extracted code ended in a Gradle project, with a submodule for each type (Kafka, API, elastic…). That Gradle Project also has a submodule for a Gradle Plugin that adds all the dependencies, including the extracted code.

smartycore {
    api = true
    microservice = true
    mongodb = true
}

So, now a microservice only needs to add a Gradle Plugin and provide its types with a snippet similar to the one above. With that, all dependencies and helpers get imported magically. The first clear impact is the performance, now a new microservice is developed and deployed much, MUCH faster.

Backend

Finished domains

I started with the easier domains so I can handle the errors in a proper way, rather than in the most complex domains. So, right now, the system can handle products and prices, which are stored in MongoDB. The products are also indexed in an ElasticSearch.
The products and prices are received via Kafka, then processed and stored in the database. After it is stored, the API can provide the data and the frontend display it.

Just-designed domains

The first complex domain is the promotion domain, as it needs information from both products and prices. I did an initial design and refined it to fix the design issues. After that and being in a dead-end, I started from scratch and now I have two designs. I am analysing both designs and picking the best elements from each one.

Product image storing

The product images were also a headache, not because of the difficulty, but because of the many available options to store them: a database, a stateful set of K8S, NFS… 

After writing the advantages and disadvantages of each model, I decided to store the images in GridFS inside MongoDB. Why? First, because I already had a MongoDB cluster. Secondly, because I needed resilience and high availability and how I set up MongoDB this is straightforward.

Next steps

Now, I have near 10 microservices deployed in the staging environment. The next steps will be the promotion domain and its integration in the frontend and deploying everything in the preproduction environment so I can perform the performance tests.

What is the “Diary of a Master Thesis”?

Starting December 2020, I decided to collect in a series of posts on LinkedIn my progress in the Master Thesis, a distributed e-commerce platform. It is supposed to be defended in September 2022.