In many enterprise ETL (Extract, Transform, Load) pipelines, the biggest bottleneck lies not in data processing, but in the time spent waiting for external responses.

Database queries, third-party API calls, reading internal services, and scraping processes often consume far more time waiting for I/O than utilizing CPU. Executing these operations sequentially leads to unavoidable resource waste.

This exact scenario motivated our adoption of an asynchronous architecture using asyncio and Quart.

Why Asyncio?

Asyncio allows a single application to manage hundreds or even thousands of input and output operations concurrently without needing to create a thread for each request.

While a connection waits for an API response, the event loop continues executing other tasks. In practice, this means much better hardware utilization and a significant reduction in overall execution time.

This model is especially efficient for applications with high I/O loads, such as:

  1. Enterprise system integrations;
  2. Data collection across multiple APIs;
  3. Scrapers running in parallel;
  4. Queue processing;
  5. Automation services;
  6. Integration gateways.

The Role of Quart

For our microservices, we chose Quart, a framework compatible with the Flask API, but fully based on ASGI.

This compatibility made the migration of existing services straightforward, allowing us to maintain a familiar structure while leveraging all the benefits of asynchronous programming.

With Quart, we created endpoints capable of handling hundreds of concurrent requests without blocking the application during external calls.

Concurrency in Practice

A simple example demonstrates how multiple HTTP requests can be executed concurrently using aiohttp and asyncio.


import asyncio
import aiohttp

async def fetch_api(session, url):
    async with session.get(url) as response:
        return await response.json()

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [
            fetch_api(session, f"https://api.example.com/data/${i}")
            for i in range(10)
        ]

        results = await asyncio.gather(*tasks)

        print(f"Successfully processed ${len(results)} records!")

asyncio.run(main())

Even though this example only uses ten requests, the same pattern can be applied to hundreds of concurrent calls, respecting the rate limiting of the APIs involved.

Lessons Learned

Migrating to an asynchronous model brought benefits that go far beyond speed.

The code became more scalable, reduced the need for running multiple processes in parallel, and made managing large volumes of simultaneous integrations much simpler.

Another key benefit was the reduction in resource consumption. Instead of keeping dozens of threads blocked waiting for network responses, the event loop uses this time to run other tasks, significantly increasing the application's overall efficiency.

Naturally, asynchronous programming also requires caution. Blocking libraries can compromise the entire system's performance, making it essential to use asyncio-compatible tools, such as aiohttp, asyncpg, and other asynchronous drivers.

Results

After migrating several legacy microservices to a fully non-blocking architecture, we observed a reduction of approximately 70% in the overall execution time of our operational pipelines.

In addition to the performance improvement, the architecture became highly prepared to scale horizontally, supporting a much larger number of concurrent integrations without a proportional increase in computing resources.

For applications where the bottleneck is in network operations and external service access, the combination of Python, Asyncio, and Quart proved to be an extremely efficient solution, maintaining the simplicity of the language without sacrificing scalability and high performance.