About Concil
Concil was founded in 1993, with the objective of helping companies of all sizes and segments to have access to effective financial management for their business, providing results that are more efficient and reliable. Its purpose is to transform financial management into something simple, unified, and smart. To do this, it uses technology, unified information, and strategic partnerships to provide secure decisions, time savings, and financial results.
Currently, the company has more than 6,000 customers, through its General Ledger Reconciliation (Concil Contábil) and Card Reconciliation (Concil Card) solutions, making financial management easier and faster, providing more efficient and reliable results.
The Challenge
With a wide variety of customers, Concil processes more than 50 million transactions monthly, generating more than 300 million lines of data to be analyzed, applying complex business rules to deliver financial data reconciliation results.
Concil wanted to increase the number of customers. However, a major challenge was scaling up data processing capacity at the speed required for the data reconciliation process, while keeping prices under control.
The solution originally used was based on data processing directly in an Oracle database, hosted in the Oracle Cloud, and the complex transformations performed overloaded the database, slowing down analytics and hindering the addition of new customers.
The Solution
To overcome its challenges, Concil counted on Solvimm’s (an e-Core company) support to develop a scalable and event-driven solution on AWS, where a premise was the processing of files using independent resources, based on best practices of microservices architecture. In other words, each file would be processed in most of the data stream using computing resources only while it is being processed, minimizing the amount of idle resources, reflecting directly on cost optimization.
The strategy to improve data management was to use the Lakehouse concepts, enabling unified governance for easy data movement, as well as Apache Hudi that efficiently manages the business requirements. This improved data quality, in addition to promoting ACID properties for Amazon S3 data.
Throughout the process, data processing was performed in Apache Spark on batch EMR Clusters and the data queried by analytical processes via Amazon Redshift.
The Results
Concil can now process thousands of files in minutes, enabling it to expand its business without the infrastructure being a bottleneck for processing. With the implemented solution, it was possible to process 10 times more files per minute, which was one of the main bottlenecks of the previous model, increasing by more than 20 times the capacity to receive new customers, while facilitating maintenance by reducing the complexity of the database schemas.
Moreover, another key item was the easy maintenance of the solution, reducing the number of database schemas by 600 times.
* Originally published by AWS here
