Smarketer

NFQ software developers took a challenge to build a platform that could successfully handle growing amounts of data (at the start of June 2019, there were 1 million products, now, in 2021, there are around 20 million) and to do it in 6 months before the marketplace's launch. Thus Smarketer was born.

Smarketer.Shopping is a well-known German price comparison website. Our team, programming in the Kotlin language on an Elasticsearch search engine, built a marketplace on AWS, suited for rapid growth where end-users can quickly and effectively find desired items and compare their prices across different suppliers.

And it worked. The system's unique visitors are growing by an average of 66% every 6 months and continue to grow despite the COVID-19 situation in the market during 2020-2021.

"Smarketer was built with a considerable emphasis on stability and scalability. In the not-so-distant future, we seek to boost the platform's monetization capabilities and make it possible to increase the total product count radically. The level we aim to reach eventually is 100 million products," says Vainius Venclovas, Team Lead at NFQ.

Technological challenge

While developing the Smarketer.Shopping platform, we tackled an important problem. The large file sizes of high-resolution product images were slowing down the load speed of the web platform. Considering that the planned number of products was huge, image processing had to be managed delicately. To import millions of product images every day, the NFQ's engineers took advantage of Amazon Web Services. 

More specifically: when product images are imported, their size is reduced in real-time using the AWS Lambda compute service. Once processed, images are stored in the AWS S3 data warehouse.

And that's just a single small process in a well-designed ecosystem. To mention just a few others:

• Product files are loaded into the system from the admin module. During import, product data in the server is processed and loaded/updated to an Elasticsearch search engine. For the front end of the admin dashboard, CoreUI on the React.js framework is used.
• The open-source language TypeScript is also used to develop the system’s Front-end (an online marketplace portal) with the React.js and Elastic UI frameworks and ReactiveSearch library components.
• To develop the back-end and handle the system`s traffic and logic, our team codes in Kotlin on the Ktor framework.

"In developing the project, the biggest challenge for me personally has been learning how complex the mechanics can be under what from the outside may look like something very simple. And thinking through all the details and situations that could occur," – says Paul-Willem Thum, Senior Team Lead at Smarketer.