Netflix rewrites Conductor for 420 million workflows a month

Netflix reworked Conductor, its distributed workflow orchestration engine, to handle growth that's edging into uncomfortable territory: roughly 200,000 workflow definitions spread across 150 applications, running approximately 420 million workflows per month. With version 4.0, the company increased the maximum supported workflow size from about 2,500 to 30,000 tasks (a jump of more than 10x) and cut p99 workflow evaluation latency by roughly 40%.

For anyone building orchestration systems, data pipelines, or anything that coordinates long-running tasks across services, what's interesting here isn't Netflix's giant number, it's the nature of the bottleneck they hit and how they solved it. It's a case study on what breaks when the size of each unit of work grows, not just the volume.

Where the engine got stuck

Conductor orchestrates distributed flows across Netflix's Content and Studio Engineering, Ads, and Games areas. The stack had already gone through several part swaps over the years: migrating execution data from Dynomite to Cassandra, moving large task inputs and outputs to Amazon S3, replacing DynoQueues with Timestone, and bringing in Kafka to decouple indexing from the execution path, with Elasticsearch and Iceberg handling indexing and long-term storage.

Even so, one bottleneck remained: workflow evaluation. In earlier versions, Conductor loaded the entire workflow state into memory on every evaluation. That works fine when flows are small, but it becomes a problem as they grow, because memory pressure scales together with the size of each workflow.

This wasn't a theoretical problem. Old community discussions had already exposed the symptom. In 2022, a user reported that 55,745 workflows and 310,000 tasks were pushing JVM heap usage to 5 GB. Aravind Ramkumar, who maintains Conductor at Netflix, explained at the time that the engine loaded the entire running workflow into memory to evaluate it, and that loading only the necessary parts was on the roadmap. Another discussion described a 1.7 MB workflow with nearly 5,000 task entries: just storing the definition took close to two minutes, and repeatedly reloading it from the database affected execution time.

The core architecture decision

The fundamental change in Conductor 4.0 is separating workflow metadata from task and user data, storing tasks independently. Instead of loading the entire state, the evaluator now works with a lightweight workflow blueprint and loads only the task data needed to make the next decision.

It's the difference between opening the entire file to read one line and having an index that points straight to the piece that matters. When a workflow has 30,000 tasks, loading only what the next decision requires instead of everything is what makes that size viable without blowing up memory.

Taking the lock and evaluation out of the synchronous path

Two other moves attack contention. The first removes locking from task state coordination: pending and terminal states are now stored separately and reconciled at the application layer, with the terminal state taking precedence. The second takes workflow evaluation out of the synchronous request path, placing updates into dedicated Timestone queues for asynchronous, sequential processing.

The reported result is straightforward: failed lock acquisition attempts, which reached about 2,700 per interval during contention spikes, dropped to essentially zero.

| Metric | Before | After (4.0) | |---|---|---| | Maximum workflow size | ~2,500 tasks | ~30,000 tasks | | p99 evaluation latency | baseline | ~40% lower | | Lock acquisition failures per interval | ~2,700 | ~0 | | Workflows executed per month | | ~420 million |

Scaling is horizontal, not more aggressive polling

One recurring lesson from community discussions is worth highlighting because it goes against a common instinct. A user reported between 25,000 and 30,000 workflows running with HTTP task queues piling up. The natural temptation is to increase worker polling frequency. According to the source's account, Aravind Ramkumar recommended the opposite: increasing polling counts wouldn't improve processing and could overload the system, and scaling horizontally was preferable.

It's the kind of trap any team operating workers falls into: when the queue grows, stepping on the polling accelerator tends to make contention worse, not resolve it. Adding more workers in parallel is the way to go.

What comes in the 4.0 package

Beyond the architecture changes, Conductor 4.0 adds native concurrency controls, dynamic worker allocation, and a type-safe Java workflow SDK, already used by Netflix Ads for creative asset ingestion and Data Clean Room workflows. The engine keeps its task-based orchestration approach, with workflow definitions and a separate execution engine, targeting long-running distributed flows.

Netflix projects that workflow demand could grow up to five times as the company advances in areas like live content, games, and podcasts, which helps explain why it's investing now in supporting flows 10x larger.

The detail that changes who can use it

Here's the point of attention for Brazilian teams that considered Conductor as an orchestration foundation. Netflix discontinued maintenance of the public Conductor OSS repository in December 2023, citing the migration to its internal fork. In other words: all the improvements described here run on Netflix's internal version, and there's no guarantee they'll reach the open-source code.

What remains available are the modules and extensions contributed by the community, maintained in a separate repository. In practice, anyone depending on Conductor OSS today needs to weigh the community maintenance scenario before betting on it in production. The design decisions Netflix described (separating the blueprint from task data, taking evaluation out of the synchronous path, reconciling state at the application layer instead of locking) are replicable in any homegrown orchestration engine, and that may be where the greatest value of this account lies: not in ready-made code, but in the map of bottlenecks that appear when the size of each workflow, not just their number, grows.