Lyft retired its homegrown Flink operator and migrated to the official Kubernetes Operator

Lyft moved hundreds of production Apache Flink jobs from a homegrown Kubernetes operator to the official Apache Flink Kubernetes Operator. The account appeared in a Lyft engineering post published on August 31, signed by streaming engineers Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar, and was detailed by InfoQ. The point that matters to anyone operating streaming isn't the announcement itself, it's the decision map: when a homegrown operator starts costing more than it solves, and what pain you inherit by swapping it for a community project.

Why Lyft had its own operator

In 2020, when Lyft built its operator, the open source community still didn't have a dedicated control plane for Flink on Kubernetes. The homegrown solution upgraded via dual-deployment: it spun up a second cluster, took a savepoint, canceled the old job, and restored from the savepoint. The legacy operator's BlueGreen mode kept both clusters running until engineers manually flipped the traffic.

The problem was in the details that only show up in production. The savepoint trigger had no retry and no idempotency. A timeout on a large-state job could fail the deploy or, worse, restart the job with no state at all when no recent checkpoint was available. And off-heap memory reservation sat behind a single parameter, systemMemoryFraction, basically a guess at the overhead the Python harnesses of Apache Beam needed. Too low, and TaskManagers got OOM-killed. Too high, and the entire fleet wasted memory. Every Flink upgrade required maintaining this code, and decent autoscaling and rollback were missing.

What the community operator delivered

The Apache operator treats last-state as a first-class upgrade mode: it restores from high-availability metadata or the most recent checkpoint even when the JobManager is having trouble. Instead of rewriting the Jsonnet templates, Lyft had its own deploy API translate the legacy FlinkApplication specs into FlinkDeployment resources at the boundary. This layer mapped jarName to jarURI, lifted node selectors into PodTemplateSpec objects, injected the old environment variables, and switched to last-state as the default upgrade mode. It's the kind of strategy that avoids a big-bang migration and keeps the application team oblivious to the swap.

The financial gain came from autoscaling. According to the authors, the autoscaler is "doing what we expected, which is correctly sizing a fleet that was overprovisioned by a few million dollars a year."

The price of the bill: downtime and bugs

Here's the part that doesn't show up in the release and that matters for your evaluation. The Apache operator brings up JobManagers first to manage the TaskManagers' lifecycle, and leaving the legacy operator replaced dual deployments with stop-then-start deploys. The result: 3 to 6 minutes of downtime on a typical deploy, and about 20 minutes on the largest jobs.

To solve this, Lyft adopted FlinkBlueGreenDeployment, a CRD that was still in development when they started testing it and was only released in operator version 1.14.0, on February 15, 2026. With it, new versions run alongside old ones before the cutover. Along the way, they ran into a configuration rename bug (tracked as FLINK-38548) and contributed the fix upstream. In the team's words: "When we noticed the BlueGreen bug, we opened an issue and a fix landed upstream within a few days." That's the real cost of adopting a feature still in development: you become a co-author of the fix.

The upgrade chain that autoscaling requires

Autoscaling in Flink isn't a standalone switch, it's a sequence of version dependencies, and Lyft's account makes that explicit:

• Flink 1.19 to autoscale. Version 1.17 required a restart to change parallelism, exactly what the jobs that most needed to scale couldn't afford. In-place scaling is documented starting with Flink 1.18.
• KinesisStreamsSource. Upgrading to 1.19 unlocked this source, released in flink-connector-aws 5.0.0 in November 2024, which requires Flink 1.19 or later. It emits record-backlog metrics for Kinesis streams the same way the Kafka source does, giving the autoscaler a signal that was missing on Kinesis-connected jobs.
• Autotuning versus in-place. Autotuning reclaimed the JVM overhead that Beam's Python processes still used, but it started OOM-killing the Beam fleet until Lyft moved the harness to a dedicated sidecar with its own limits.

And there's an important architectural conflict: since autotuning resizes container memory and needs to restart pods, it clashes with in-place autoscaling. The workaround was to split features by criticality. Pricing and routing jobs use in-place autoscaling without autotuning; less critical workloads accept restarts to adjust resources. When restart-based autoscaling moves pods, a custom most-allocated scheduler packs pods densely, allowing underutilized nodes to be shut down, while Karpenter provisions EC2 capacity on demand.

How this compares and what remains open

Lyft didn't invent the Beam sidecar architecture. Spotify's Flink operator, forked from Google's now-discontinued operator, already documents running the Beam Python SDK harness as sidecar containers alongside Flink's TaskManagers, exactly where Lyft ended up after the autotuning OOMs. Amazon Managed Service for Apache Flink, meanwhile, manages checkpoints and eliminates operator maintenance entirely, at the cost of losing per-pod sidecar customization.

At the other end of the scale, Netflix runs more than 30,000 Flink jobs (versus Lyft's hundreds) and chose a different path: it keeps its own autoscaler and the open source one side by side, converging on the open source autoscaler instead of adopting the Kubernetes Operator. In other words, there's no single answer, and job volume weighs heavily on the decision.

What changes for teams operating streaming in Brazil

For Brazilian teams running Flink on EKS, GKE, or their own clusters, this case works as a feasibility checklist before retiring homegrown automation. Some concrete points to bring to the architecture meeting:

• Keeping a homegrown operator costs you on every Flink upgrade. If your team spends recurring effort chasing each new version, that's the signal Lyft cites as the trigger for the migration.
• Autoscaling isn't free or immediate. It drags along an upgrade chain (1.18/1.19 at minimum) and forces decisions between in-place and restart based on job criticality. Map out which jobs can't tolerate a restart before promising savings.
• Translating specs at the boundary avoids a rewrite. The strategy of mapping FlinkApplication to FlinkDeployment within the deploy API itself is replicable and spares the application team from any change.
• A CRD still in development means becoming a contributor. Adopting FlinkBlueGreenDeployment before it stabilized produced a bug and an upstream fix. If you don't have the appetite to contribute, wait for the stable release.

The next step for Lyft's Streaming Compute team is to speed up marketplace pricing loops and autonomous vehicle telemetry, workloads that depend precisely on the autoscaling the migration unlocked.