ONS builds Data Lake on AWS to support Brazil’s electric load forecasting

Published: May 5, 2023

About ONS

ONS, the National Electricity System Operator in Brazil is the agency responsible for coordinating and controlling the operation of electricity generation and transmission facilities in the National Interconnected System (SIN) and for planning the operation of isolated systems in the country, under the supervision and regulation of the Brazilian Electricity Regulatory Agency (Aneel). To this end, ONS develops a series of studies and actions conducted over the SIN and its subsidiary agents to manage the different sources of energy and the transmission grid, in order to ensure the continuity, cost-effectiveness and security of supply throughout the country. To provide even greater accuracy to the electric load forecasting processes for operation planning and scheduling, in 2018, the institution defined its new data architecture, which included a Data Lake on AWS. This strategy allowed data to be centralized in a single repository and, at the same time, technically enabled the business areas to create their own statistical and machine learning models, as well as visualization dashboards, according to a vision of democratizing access to data. 

It was exactly what we needed to make information available to our engineers and internal processes, and AWS offered us their ecosystem of data and innovation solutions.”

Sergio Mafra

ONS Data Architecture Specialist

ONS logo

The Challenge

The National Electricity System Operator (ONS) is the agency responsible for coordinating and controlling the operation of electricity generation and transmission facilities throughout the country. The engineer responsible for the new data architecture project, Sérgio Mafra, explains that the agency’s work enables coordinating the entire Brazilian electricity system, which is totally interconnected. “We are able to take large blocks of energy generated in one part of Brazil and transport it to another place that needs it. This gives us flexibility to handle climate issues in the different regions,” he explains.

He recalls that ONS was born from the restructuring process of the electricity sector, in 1998, following a model that would meet private and public interests for SIN’s forecasting and operations, a role previously performed by Eletrobras. “We are responsible for the engineering part. We look at Brazil and plan how to meet the energy demand, according to a plan that starts from the next day up to five years ahead. Studies that involve several processes in the organization and that include both electrical and energy studies are performed daily, considering the generation of all energy sources: hydroelectric, thermal, nuclear, photovoltaic, and wind power. In these processes there is a series of elements with their peculiarities,” he says.

From these studies, ONS establishes how much energy should be generated, and which power plants should be considered for dispatch in order to optimize resources and minimize costs. These factors are analyzed using mathematical models, to define the priority order in which each power source will be used. Once the plan is defined, it is sent to the four control centers in Brazil to carry out the effective operations of the Brazilian electricity system. These centers are located in Rio de Janeiro (Southeast), Recife (Northeast), Brasília (North/Central West and Brazil vision) and Florianópolis (South) and, based on this information, will organize the distribution of energy generation and transmission throughout Brazil.

“To make it all work, we utilize data we get from the power companies and external providers. We process this data and continuously make the necessary decisions to keep the system running. We are an intelligence and data-driven company. We have been working for a long time to incorporate this data-driven culture, one of the pillars of our journey towards Digital Transformation, which is part of the ONS Strategic Planning,” explains Mafra.

The engineer also reminds us that, to implement the visions of the future, the agency works with mathematical models that receive a series of inputs, such as economic forecasts, weather forecasts, status of the power plants, internal consumption, etc. All these data points are processed to create the scenarios with which the agency will work, also including the interaction with Aneel (Brazilian Electricity Regulatory Agency) for bidding actions to increase generation and transmission capacity, based on reinforcement studies also conducted by ONS.

Centralizing this data as well as streamlining and speeding up its analysis and processing was one of the great challenges faced by ONS. “We had a lot of data scattered around the organization and still have many challenges. We wanted to take the data spread across several silos in our business areas, centralize it, and offer it as products for consumption,” recalls Sérgio Mafra.

The organization decided to build a cloud Data Lake that would allow, in addition to centralizing and storing large volumes, the massive processing of this data. According to Mafra, this process began in 2018, when the architecture team decided to revise the current data architecture, with the goal of creating models more suitable to the Big Data challenge, in accordance with the company’s strategic planning. “We realized that we needed to have a decoupled view of storage and processing, thus enabling us to meet, with flexibility and scalability, the corporate data processing requirements in real time,” he explains.

A multidisciplinary IT team was then assembled to deliver the solution for calculating the electric load, with data coming from the agents and ONS. The challenge was to build an application system that could exchange data with an information bus, a Data Lake, an FTP server that acquires external data, and an integrator that captures data from the corporate databases. “The complexity was to integrate all these components and orchestrate them, ensuring the delivery and processing of data accurately and at the right time,” says Elvis Galiza, project manager of the Electric Load Verification System – SACG.

Considering the multiple data providers for the system – often with information of the same nature at different times – the traceability and validity of each record became extremely necessary. The impossibility of updating and physically deleting records in the Data Lake was another point that had to be circumvented by the modeling, as well as the fact that the data structure to be assembled would be for corporate use and not exclusive to the SACG project. “We had to evolve our original documentation method and perform some adjustments to the conventional modeling, so we could also have all the metadata of the information stored in the Data Lake properly registered and integrated into our data catalog,” says Carlos Alberto Rodrigues Alves, ONS Data Modeling Administrator.

The data environment built will allow statistical and machine learning models to be developed and implemented by professionals who have the business expertise and are authorized by the IT area. “The Load Forecasting models were key elements in the use of Amazon Elastic Container Service (Amazon ECS) and Amazon SageMaker at the organization, using the Data Lake as storage for inputs and results of the models executed via Amazon ECS, allowing us to more quickly meet the requirements of our electric system operation planning cycle,” comments Gabriel Gonçalves, Forecasting and Load Monitoring Engineer. 

Why AWS

Mafra points out that in addition to the need for aggregating all the data, the solution should have a vast storage capacity, very fast processing, and the support of specialized professionals to ensure proper implementation. “It was exactly what we needed to make information available to our professionals and internal processes, and AWS offered us their ecosystem of data and innovation solutions,” he says.

The project started to be implemented in 2020 and today ONS has an environment where all this information is processed. Building the Data Lake has also enabled the machine learning cycle, within the model recommended by Gartner, called Citizens Data Science. “Today we offer our knowledge to leverage the business areas to build their models. With Amazon SageMaker, for example, these areas will be able to make their own models, so that in the end we will have greater expertise and a large and organized model library following the best practices,” he predicts.

Mafra explains that the architecture developed by the team and implemented with the support of AWS Professional Services created a mechanism where incoming data is stored where it makes the most sense, respecting its purpose. “It is an architectural process that we have built and are evolving, allowing consumption at the ends to be better organized, whether using a data warehouse, graph data, time series, or key-value structure,” he says. According to the engineer, the preparation process for the internal teams involves holding webinars with the business areas, so that everyone involved understands how to use the Data Lake, which today has the following architecture:

Data Architecture View (Cloud)

Data Architecture View (Cloud)

Benefits

According to Mafra, the available data will enable all areas at ONS to make decisions more quickly. “The implementation of this environment was completed recently with the support of AWS Professional Services and its partners Solvimm (a company owned by e-Core) and Tenbu. In 2021, we will have an hourly pricing model, which is making our organization clock faster, using a greater volume of information in a much shorter time cycle,” he says.

The solution adopted at AWS allows integration, flexibility, capacity, and, most importantly, constant innovation in its services, abstracting from the complexity and operation of cutting-edge technologies, at an affordable price. 

Next Steps

With support from AWS and Professional Services technologies as well as using the Citizen Data Science vision, ONS is creating a new environment using electric load through the Data Lake and making it available for use in different ML models through Amazon SageMaker, providing the engineering team with data and technology to enable the new models required to implement the information chain that is essential for ONS to perform its function, placing ONS at a new level as an innovative company and a protagonist in the Brazilian electricity sector.

The team is also working on the Power Sector Technological Development Plan (PDTE) with improvements to the mathematical models and data modeling with best practices.

About ONS

ONS, the National Electricity System Operator in Brazil is the agency responsible for coordinating and controlling the operation of electricity generation and transmission facilities in the National Interconnected System (SIN) and for planning the operation of isolated systems in the country, under the supervision and regulation of the Brazilian Electricity Regulatory Agency (Aneel). Established as a private legal entity, in the form of a non-profit civil association, ONS was created on August 26, 1998, by Law No. 9,648, amended by Law No. 10,848/2004 and regulated by Decree No. 5,081/2004.

To perform its legal attributions and fulfill its institutional mission, ONS develops a series of studies and actions conducted over the system and its subsidiary agents to manage the different energy sources and the transmission grid, in order to ensure the security of the continuous supply throughout the country. ONS is composed of associate members and participating members, which are the generation, transmission, and distribution companies, free consumers, energy importers and exporters. The Brazilian Ministry of Mines and Energy (MME) and representatives of the Consumer Councils also participate in the organization.

*Content originally published on the AWS website

Benefits with AWS

  • Greater speed in decision making;
  • Simplification of cutting-edge technologies;
  • Reduced operating costs;
  • Greater integration and flexibility in the architecture.

AWS Services Used

Amazon ECS

Amazon ECS is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications.

Learn More >

Amazon SageMaker

Amazon SageMaker helps data scientists and developers quickly prepare, create, train, and deploy high-quality machine learning (ML) models by bringing together a comprehensive set of features built specifically for ML.

Learn More >

AWS Lambda

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes.

Learn More >

Amazon Redshift

With Redshift, you can query and combine exabytes of structured and semi-structured data across data warehouses, operational databases, and your data lake using standard SQL.

Learn More >

Get started now
Companies of all sizes and in all industries are transforming their business every day using AWS.
Contact our experts and start your cloud journey today.