About Navi
Navi is an investment fund manager specialized in shares of Brazilian companies, with approximately R$5 billion in assets under management. Founded in 2018 by Kondor Invest’s equities team after a successful ten-year track record, the firm is known for its strong research culture: its teams seek to deeply understand the companies in which they invest.
The Challenge
Navi relentlessly pursues high return investments for its clients, combining deep knowledge and experience about the Brazilian market with efficient data analytics. With an excellent track record of more than ten years, Navi believes that robust data processes contribute significantly to increasing the return from its investments. Thus, in 2019, it created the Data Science area to improve its stock analysis process.
The Brazilian Stock Exchange has more than 300 stocks listed across 11 sectors, so processing large volumes of information was one of the pivotal challenges of the new sector. In addition, with a lean and trained staff, the members of the Data Science department needed to create robust and automated Data Pipelines.
“Covering virtually all sectors of the stock exchange, our challenge is to automate all our data processes so that our team can focus on building new projects and studies,” says Antonio Lobato, partner and head of the Data Science team. “We needed to create systems that would work without the constant need for human intervention,” concludes Antonio.
To face this challenge, the use of cloud computing was the best option. “There are many hidden operating costs in infrastructure management. The use of managed services is a fundamental component of our department’s way of working,” adds Carlos Domingues, head of Infrastructure and DevOps.
The Solution
After studying the challenge, the Data Science department decided that managed components would be an essential part of the data streams to be built. “We evaluated other platforms and concluded that AWS has the best offering of services related to the analysis and processing of large amounts of data,” recalls Carlos.
To create reproducible and performant pipelines, the team decided to build a Data Lake and Data Warehouse with Multi-Hop architecture, in which raw data is stored together with semi-structured and structured data. This architecture was made possible through the use of multiple serverless components from AWS, such as Amazon S3, AWS Glue Data Catalog, Amazon Aurora Serverless, and Amazon Athena. Using AWS managed services also makes it simpler to handle the security and observability of the developed projects.
In addition, the team decided to build microservices-oriented pipelines, encapsulating operations in pre-defined contracts, and streamlining communication between different projects. In this architecture, Navi is able to have significant cost reductions through the use of Spot Instances and Auto Scaling Groups, since the data streams do not have significant high-availability SLA requirements.
The image shows the architecture used in more detail. Microservices run in containers, which use Hashicorp Nomad as orchestrator and EC2 Spot Instances in an Auto Scaling Group as workers. Microservices coordinate interactions with the Data Lake, which is implemented through S3 buckets and metadata cataloged by the Glue Data Catalog. Data processing is done by Databricks clusters running on AWS and the processed data is queried using AWS Athena. Finally, aggregated data is stored in a Data Warehouse that is implemented using Amazon Aurora Serverless.
The Brazilian Stock Exchange has more than 300 stocks listed across 11 sectors, so processing large volumes of information was one of the pivotal challenges of the new sector. In addition, with a lean and trained staff, the members of the Data Science department needed to create robust and automated Data Pipelines.
“Covering virtually all sectors of the stock exchange, our challenge is to automate all our data processes so that our team can focus on building new projects and studies,” says Antonio Lobato, partner and head of the Data Science team. “We needed to create systems that would work without the constant need for human intervention,” concludes Antonio.
To face this challenge, the use of cloud computing was the best option. “There are many hidden operating costs in infrastructure management. The use of managed services is a fundamental component of our department’s way of working,” adds Carlos Domingues, head of Infrastructure and DevOps.
Navi is already reaping the benefits of the architecture running on AWS. “Using AWS services and a microservices-based architecture, we were able to create an environment where our data scientists spend most of their time creating new projects and adding intelligence to our analytics,” Antonio celebrates.
To ensure that the whole environment complies with the best practices and principles of the Well Architected Framework, Navi turned to Solvimm (an e-Core company), an AWS consulting partner with experience in several projects, to support the Data Science team in implementing the framework and using the AWS components following security best practices, as shown in the diagram above. Using AWS Config enabled automating the auditing and notification of resources that are non-compliant with internal security policies. Moreover, AWS Guard Duty constantly monitors network flows and usage logs for detecting potential misuse or attacks. The Security Hub is used to provide a unified view of the status of each environment. Finally, an identity integration process and single sign-on integrated with the Enterprise Directory through AWS SSO was implemented. “In this work with Solvimm we were able to implement excellent security practices without the bureaucracy of traditional processes,” says Carlos.
The Results
In 2019, the Long Only strategy outperformed its benchmark, the Ibovespa index, by 9.74%. “This new way of organizing and analyzing data has enabled more informed discussions with a level of detail that previously would not have been possible in an efficient manner,” says Felipe Campos, partner and member of Navi’s management team.
Navi currently has more than 25 Data Science projects in production at AWS. “Even with a lean team, we can deliver high value to business analytics processes. This is because we have mature delivery pipelines and spend little time on operational and maintenance activities,” says Antonio. The average development time for a project is about two weeks.
Another benefit found in using AWS services was the convenience of managing costs using an architecture with serverless and scalable elements. “When we started the department we decided to commit to a budget. Thanks to all the savings and optimizations made possible by AWS, we only spent this budget more than a year later,” says Carlos.
The implementation of the Well Architected Framework also brought several benefits in other pillars besides security. “AWS has many different services, and new services are launched all the time. The support of our partner, Solvimm, is key to keep our architecture always up to date, using the cutting edge of what AWS is able to offer,” adds Carlos.
Next Steps
With increasing demands, Navi will continue to expand the number of Data Science projects and improve the quality of the existing infrastructure. “The data team is always working so that Navi continues to invest in good ideas,” Antonio points out. By extending their studies of the Well Architected Framework, Carlos and Antonio intend to further increase their use of AWS managed services, always with the help of their Solvimm partners. A next step is to optimize the health analysis of each of the running projects and services through Amazon Cloudwatch and AWS App Mesh services. These services allow continuous monitoring of the runtime of each analysis and of occasional errors and failures, enabling a quick and effective action by the application team.
Solvimm (an e-Core company) is an Advanced Consulting Partner of the AWS Partner Network. In addition to being part of the Well Architected Program, they are also part of the Public Sector Program and feature AWS Lambda and AWS CloudFormation expertise.
Originally published in The AWS Blog on 03 SEP 2020
![[Blogpost] Webinar FinO_ Practical Solutions
for Cloud Cost Efficiency_](https://www.e-core.com/wp-content/uploads/2024/04/Blogpost-Webinar-FinO_-Practical-Solutions
for-Cloud-Cost-Efficiency_-768x297.webp)
