Netflix Migrates 30,000 Flink Jobs to Open Source Autoscaler, Cuts Costs by Up to 58%

Netflix is migrating more than 30,000 streaming jobs, spread across multiple AWS regions, to the open source Apache Flink Autoscaler. The reason, according to the company: the in-house autoscaler it had maintained since ~2019 scaled at the cluster level, and that stopped making sense for complex stateful pipelines, where different operators within the same job have distinct processing needs.

The number that stands out in the report: one team cut annualized Flink compute spend by 58%, saving roughly $1.1 million a year. That's not a global result of the migration, it's the cited case of a single team, but it helps size up why the decision was made.

Why the Old Autoscaler Hit a Wall

Netflix has run Apache Flink since 2017 and built its first autoscaler around 2019. That system ran on top of Mantis and consumed cluster-level telemetry from Atlas: CPU, network usage, Kafka lag, ingestion rate, and consumption rate. With that, it adjusted the total number of TaskManagers and cut resource consumption by 25% to 45% across thousands of pipelines.

The problem was the unit of scale. Because the autoscaler reasoned about the whole cluster rather than individual operators, every operator in a job shared the same scaling decision. That works poorly for stateful pipelines with branches, joins, and terabytes of state, where different parts of the dataflow process very different volumes. You end up over-provisioning the entire job because of a single bottleneck, or the opposite.

How the Open Source Autoscaler Thinks by Operator

Instead of looking at cluster metrics, the Apache Flink Autoscaler uses metrics exposed by the running job itself to estimate the actual processing rate of each operator, based on throughput and busy time (the time the operator actually spends busy). From there, it walks the job graph and calculates the parallelism needed for each vertex separately.

The approach is described in FLIP-271, which deals precisely with autoscaling for heterogeneous jobs and the cost of rescaling stateful applications. Technically, it draws on research from the DS2 project. Vasiliki Kalavri, a systems researcher involved in the work, said the project started out exploring critical path analysis before adopting True Processing Rate as a simpler baseline.

It turned out that this very simple idea worked really well.

>

-- Vasiliki Kalavri, systems researcher

That idea ended up being incorporated into Flink's autoscaling work.

What Netflix Had to Adapt

Netflix didn't plug the autoscaler in directly via the Flink Kubernetes Operator. Instead, it integrated the autoscaler with its internal control plane. A Spring Boot service uses Temporal workflows to isolate autoscaling decisions for each job individually.

Along the way, the engineering team needed several changes that serve as a warning for anyone planning to run this at a similar scale:

• Modified JobManager metrics collection to support jobs with up to 3,000 subtasks;
• Added server-side metrics filtering;
• Preserved subgraphs with forward connections during scaling;
• Added handling for sink backpressure.

The forward-connection point is subtle and has already come up in Flink community discussions: changing parallelism across a FORWARD connection can require redistributing data, so Netflix's implementation keeps forward-connected operators together. There's also an open issue, FLINK-38538, about cases where busy operators can be affected by scaling decisions based on output ratio.

A Tuning Detail Worth Copying

Netflix operates with a utilization target of 0.45, below the Flink community default of 0.7. The choice is deliberate: with a more conservative target, the autoscaler rescales large stateful jobs less aggressively, avoiding the high cost of recovering terabytes of state on every rescale.

| Aspect | In-house autoscaler (2019) | Apache Flink Autoscaler (open source) | |---|---|---| | Scaling unit | Entire cluster | Operator (graph vertex) | | Metrics source | Cluster telemetry (Atlas) | Job metrics (throughput, busy time) | | Resource reduction | 25% to 45% | Cited case: 58% cost cut for one team | | Utilization target | -- | 0.45 (community default: 0.7) |

It's worth comparing this to generic autoscalers like KEDA, which scale workloads based on external metrics or events. The Flink autoscaler does something different: it reasons about the internal dataflow graph and each operator's capacity. These are tools for distinct problems, and that distinction matters when choosing one.

What Changes for Pipeline Builders Here

The practical point for Brazilian devs is that none of this is proprietary to Netflix. The autoscaler is part of the Apache Flink project; FLIP-271 and the integration with the Flink Kubernetes Operator are available to any team. Anyone already running Flink in real-time production (event ingestion, streaming ETL, feature stores for ML) can adopt the same per-operator scaling logic without rebuilding a control plane from scratch.

The takeaway: Netflix's differentiator here wasn't inventing the autoscaler, it was the integration layer and the tuning: the 0.45 target, per-job isolation via Temporal, the metrics collection adjustments for huge jobs. For smaller teams, the most direct path is to use the autoscaler through the Flink Kubernetes Operator and calibrate the utilization target according to the state cost of their jobs, starting conservative if there are heavy stateful pipelines.

What's still open: Netflix itself says it will migrate the remaining internal cases and investigate Flink 2's disaggregated state architecture to tackle the cost of state recovery during rescaling, one of the Achilles' heels of scaling stateful jobs. Anyone relying on pipelines with heavy state should keep an eye on this front before assuming that autoscaling solves everything on its own.