From 2011 to 2025: How Booking.com Shut Down Hadoop and Rethought Its Data Architecture

- Reading
- 7 min
In a talk at QCon London, principal engineer Jabez Eliezer Manuel broke down two decades of data and machine learning evolution at Booking.com, from the MySQL migration to the end of Hadoop and the foundation for AI agents.
Two decades in three layers
In a talk at QCon London, Jabez Eliezer Manuel, Principal Engineer at Booking.com, reconstructed twenty years of evolution of the company's data and machine learning platform. Manuel has two decades of software engineering experience, spent the last decade working on low-latency search personalization problems and unified payment ledgers, and today leads the long-term memory infrastructure for agentic AI systems at Booking.com.
He structured the talk in three layers: data management, ML engineering, and domain intelligence. It is the data layer, and especially the end of an entire Hadoop era, that yields the most direct lessons for those who maintain production systems in Brazil.
From Postgres to a giant MySQL cluster
Booking.com started with Perl and Postgres. The switch to MySQL came early, driven by asynchronous replication. In 2005 the topology was one master and four slaves; fifteen years later, the company operated one of the largest MySQL clusters in the world, with around 7,000 instances managed by Orchestrator, a replication management tool born inside Booking itself.
The scale was only sustainable with heavy automation: every 60 days, the 7,000 instances are deprovisioned and reprovisioned, which means one infrastructure action every three minutes, a volume no human team could sustain manually.
The engineering decisions behind Booking.com's MySQL stand out for what they leave out: no specialized hardware, no stored procedures, UDFs, views, or caching layer. The trick was keeping databases small (a soft limit of 2 terabytes, a hard limit of 10 terabytes) so they would fit in the server's own fast storage, effectively using MySQL as a cache. The result: point queries under 350 microseconds, competing with in-memory engines.
Instead of maintaining an internal MySQL fork, which they had the expertise to do, the team chose to contribute upstream. GTID-based replication, now standard in MySQL, originated from a design proposal made by Booking.com itself.
When machine learning was Perl and cron
Before any ML platform, Booking.com's intelligence ran on Perl scripts with statistical libraries from CPAN, triggered by cron. The deployment strategy, in Manuel's words, was "copy the script and pray." Even so, it solved real problems: fraud detection, demand forecasting with regression, and recommendations with KNN.
Hadoop arrives in 2011, leaves only in 2025
As data grew, MySQL was no longer enough. In 2011 Booking.com set up two redundant on-premise Hadoop clusters, with the classic HDFS, YARN, and MapReduce stack, plus Oozie for orchestration, Spark for workflows, Hive for SQL, and Sqoop for moving data between MySQL and Hadoop. Data arrived via nightly MySQL jobs and via Kafka Connect.
Even real-time prediction was, in practice, disguised batch processing: for a simple device-and-country model predicting booking probability, the team enumerated every possible combination of features, pre-calculated the predictions offline, and saved the result in a MySQL table. When the request came in, the system simply looked up the table as if it were predicting in real time.
Hadoop sustained most of the pipeline for years, but it accumulated problems: a shared environment between training and production, a forced single version for all data scientists, zero GPU support, painful synchronization between two data centers, capacity operating near its limit, "noisy neighbors" capable of freezing the cluster with a poorly written query, and no discovery or ownership layer for data, even with teams depending on each other all the time.
At its peak, scale reached hundreds of petabytes, 190,000 Hadoop tables, and more than 7,000 workflows spread across hundreds of teams.
In short: Booking.com identified the Hadoop problem in 2018, but it took until 2022 to put the system into maintenance mode and three more years to shut it down for good, in 2025.
The reason for the delay explains a lot about large migrations:
- 2018: a first attempt to migrate to the cloud is interrupted; the reason, depending on who tells it, is cost or a change in strategy;
- In the following years: the team builds internal tools to ease the pain, such as a data catalog alongside Hadoop itself and Kubernetes clusters to run Spark;
- Later: a modernization project focuses on data quality, ownership, and governance, without yet migrating the infrastructure;
- During this period: a second attempt with another cloud provider shows promising results, but not enough to justify a full migration;
- 2023: leadership decides to go all-in, with no plan B, and the entire migration happens in a year and a half.
How to migrate without freezing the company
The 2023 turnaround worked thanks to a specific method. First, the team mapped every table and every workflow dependency into a single graph, what Manuel called "mapping the universe." Then, they cross-referenced that graph with real usage analysis to find out what was actually critical to the business.
The best architectural tool is the Delete key.
Jabez Eliezer Manuel, Principal Engineer at Booking.com
The phrase sums up the result: 92% of workflows failed the "does anyone care about this" test and were simply deleted. Only after this cleanup did the team apply graph analysis to prioritize what remained by centrality and impact, migrating in batches and gradually shutting down the legacy system.
A central piece of the process was a unified command dashboard that concentrated the dependency graph: any engineer migrating a workflow could see there when their dependencies would be migrated and plan their own work accordingly. This replaced control spreadsheets and the tribal knowledge that had until then dictated the pace of the migration.
The four lessons Booking.com would send to its own past
Manuel summed up the learning in four recommendations, valid for any team that accumulates data and workflows over the years:
- Catalog every data asset and workflow, with no exceptions;
- Establish clear ownership, both business and technical, for each piece of data;
- Build quality gates before sharing data between teams;
- Track end-to-end lineage.
From batch prediction to billions of inferences per day
The ML layer followed a similar pace of maturation. After Perl with MySQL, Hadoop brought big data analysis; Oozie with Python and scikit-learn kicked off real ML; Spark sped up processing; H2O made it possible to train faster and, for the first time, run models directly in any Java environment. In 2017, deep learning entered through the door of an internal hackathon to improve image quality, and from there it spread across the company. In 2023 came the wave of LLMs and generative products; in 2025, the first foundations of agentic systems.
2015 was a pivotal year, according to Manuel, because it finally solved the real-time prediction problem. In a hackathon, the team built a small Scala service backed by storage to serve models. Data scientists uploaded the model as a CSV file with the features used, their types, and the weights. In production, a backend service collected the necessary features and used a Perl SDK to call the Scala service.
That service evolved into today's model serving platform, now supporting more than 12 different model formats. It runs on multiple clusters: some general-purpose and multi-tenant, others dedicated to critical workloads, and others geared toward batch processing. The clusters both host their own models and talk to externally hosted models, such as on SageMaker, in addition to accessing a central feature store. A custom proxy hides this complexity from consuming teams, and every prediction made is logged to the data lake and to the monitoring platforms. According to the talk's own summary, this infrastructure today sustains up to 400 billion predictions per day.
What remains open
The material available from the talk covers the data layer and the beginning of the ML layer in detail, but it is cut off before the part about personalization, interleaved experimentation, and Booking.com's new agentic AI foundation, topics the talk's official summary promises but without technical detail accessible here. It's worth watching the full video on InfoQ for anyone who wants that part of the story.
What this means for those scaling systems in Brazil
Booking.com's experimentation culture is the most replicable point outside the big tech context: the company runs more than 1,000 simultaneous experiments at any given time, has surpassed 150,000 experiments in total, and fewer than 25% of them succeed. The stated goal was never to get it right the first time, but to learn fast, which eliminates endless opinion debates and replaces them with testing.
For teams still carrying a Hadoop, a data lake, or any legacy stack with no defined owner, Booking.com's playbook is almost a manual: map dependencies before migrating anything, measure actual usage before deciding what's worth keeping, and accept that deleting 92% of what exists can be more productive than modernizing everything. The decision to contribute upstream to MySQL instead of maintaining an internal fork is also a direct reminder: an infrastructure tool that becomes an open source project spreads the maintenance cost, instead of locking it inside a single team.
Translated from the Brazilian Portuguese original · Read the original
Ruby flaw exposes default Rails apps to memory corruption; RCE not demonstrated
An advisory from Veria Labs describes a one-byte heap overflow in the Ruby interpreter, reachable from Rails' default configuration and present in GitLab, GitHub Enterprise, Mastodon, Discourse, and Chatwoot, but the company has not demonstrated remote code execution.






