How to run distributed Spring Batch without a message broker, using only a database

The problem the article tackles

Spring Batch has been the standard framework for batch processing on the JVM for more than a decade: jobs built from steps, with a metadata repository that makes executions restartable. It works well on a single machine, until the day the data volume blows past the processing window.

The framework's own established answer for this scenario is remote partitioning: a manager step divides the work into partitions and distributes them to workers via a message broker, whether Apache Kafka, RabbitMQ, or a JMS provider, using Spring Integration. It works, but it requires standing up, securing, and monitoring an entire messaging layer just for this purpose.

An article published on October 21, 2026 on InfoQ, written by Janardhan Reddy Chejarla and reviewed by Michael Redlich, proposes an alternative for the common case: teams that already maintain a well-operated relational database (Spring Batch requires one for its job repository anyway) and have no appetite for running a messaging cluster just to fan out batch partitions. The full text is on InfoQ.

The four questions any coordination needs to answer

Reduced to its essentials, distributed batch always has to solve four questions, broker or no broker:

• How the work is divided, and across how many workers, since dividing too much creates overhead and dividing too little leaves nodes idle.
• Who is responsible for a specific execution, dividing the work, tracking it to completion, and finalizing the job.
• How a worker takes ownership of a partition without a second worker running the same partition at the same time.
• What happens when a node dies mid-execution, and how to recover the work in progress without violating the previous rule.

A broker solves a good part of this for free: its queue delivers each message to a single consumer and decouples production from execution. Removing the broker means answering each of these questions explicitly in SQL, against a database designed for durable shared state, not for message dispatch.

Three tables play the broker's role

The design uses the database that Spring Batch already requires as the single coordination plane, with no broker and no coordination service like Apache ZooKeeper. All cluster state lives in ordinary SQL tables, inspectable with a SELECT.

Three tables complement the Spring Batch schema. BATCH_NODES is the node registry, with status (active or unreachable), the timestamp of the last heartbeat, and current load. BATCH_PARTITIONS tracks the lifecycle of each partition, indexed by that partition's step_execution_id (each partition is itself a Spring Batch StepExecution), with assigned node, status (PENDING, CLAIMED, COMPLETED, or FAILED), and whether it can be transferred to another node in case of failure. BATCH_JOB_COORDINATION ties each job execution to its master node and to the manager step.

Each partition's payload remains a Spring Batch ExecutionContext, written by the master and read back by whoever executes it. This means the shared, durable repository remains a requirement of remote partitioning, broker or no broker: the difference is that, in the traditional design, the broker carries only a pointer to the partition, never the data itself. What this pattern removes is the broker, not the repository.

There is no master elected for the entire cluster. The node that launches a job becomes the master of that specific execution, and only that one. Two jobs launched on two different nodes have two distinct masters running at the same time, which avoids reintroducing the leader-election machinery and the single point of failure that removing the broker was supposed to eliminate.

What changes in the code: almost nothing

For those who already write partitioned steps in Spring Batch, the practical question is where the code changes. The answer, according to the article, is almost no lines at all, because the hook uses an extension point the framework has always had: the Partitioner decides how to divide the work, and the PartitionHandler decides how to execute each partition.

The built-in TaskExecutorPartitionHandler runs everything on local threads; traditional remote partitioning swaps it for a MessageChannelPartitionHandler that sends via the broker. The manager step is still built the same way:

new StepBuilder("reconcileStep.manager", jobRepository)
 .partitioner("reconcileStep", clusterAwarePartitioner) // how the work is divided
 .partitionHandler(clusterAwarePartitionHandler) // how partitions are distributed
 .step(reconcileStep) // the common worker step
 .aggregator(aggregator)
 .build();

The .partitionHandler(...) line is the entire extension point: the pattern supplies its own PartitionHandler, which writes partition rows to the database and monitors them in the spot where a broker-based handler would act. The worker step remains an ordinary chunk-oriented step or tasklet, results are aggregated by the standard StepExecutionAggregator, and restart works through the usual JobRepository. Turning a deployable into a cluster member is configuration, not code: a flag switches on the behavior, with auto-configuration registering the node, starting the heartbeat and polling, with no broker client or extra beans.

How the master drives the execution

The master step is an ordinary Spring Batch partitioned step, running on the node that launched the job. What makes it the master is where the PartitionHandler executes: when Spring Batch calls the handler's handle(...) method, it blocks on that node until the entire distributed execution finishes.

Inside that call, the master does four things in order: registers the execution in the coordination table as STARTED, writes a PENDING row for each partition, runs a monitoring loop that reassigns orphaned partitions from removed nodes and checks whether anything is still pending, and finally marks the job as COMPLETED or fails the step if any partition ended as FAILED. Workers never call the master and the master never calls the workers: every decision passes through reading and writing these tables, which explains why losing a master affects only that job, never the whole cluster.

Sizing the work to the live cluster

Traditional remote partitioning divides the work before knowing how much capacity exists to execute it, usually with a fixed grid size. Without a broker, the master queries the database to find out how many nodes are alive at that instant and passes that count to the Partitioner, which then decides how many partitions to create (one or two per live node, for example) and which parameters go into each ExecutionContext.

An assignment strategy then maps partitions to nodes: round-robin for even distribution, fixed-node-count to confine a job to n nodes, or least-loaded, which directs each partition to the node with the lowest reported load. The assignment is fixed from that point on, meaning it's placement, not real-time rebalancing: a node that finishes early doesn't pull partitions from a slower peer, and only a node failure moves work. That's exactly the rebalancing advantage a broker's pull semantics give for free, and that this design sacrifices.

How duplicate execution is avoided

The critical point of correctness is the transition from PENDING to CLAIMED, made with a single conditional UPDATE that also serves the following transitions:

-- claim, then complete or fail: a single guarded UPDATE
UPDATE batch_partitions
SET status = ?, -- CLAIMED, then COMPLETED or FAILED
 last_updated = CURRENT_TIMESTAMP
WHERE step_execution_id = ?
 AND assigned_node = ?
 AND status IN ('PENDING', 'CLAIMED');

The WHERE clause only accepts a row that is still PENDING or CLAIMED: once a partition reaches a terminal state (COMPLETED or FAILED) or is reassigned, no late write can move it. This isn't a contended queue with SELECT ... FOR UPDATE SKIP LOCKED: since partitions already come pre-assigned, the guarded update governs the lifecycle, not the dispatch. The one-node-per-partition guarantee that a broker gives through its queue comes here from the row-by-row state transition.

Recovering from node failure

Each node updates a heartbeat timestamp at a configurable interval, compared against the database's own clock, which avoids clock-skew problems between different machines. Detection and removal are decentralized: every node runs the cleanup sweep, so the death of a single monitor doesn't leave the cluster blind.

Recovery is a two-phase process, with production defaults of around ten seconds to mark a node as unreachable and thirty seconds to actually remove it and reassign its incomplete, transferable partitions. The correctness contract is explicit: transferable work runs at least once and must be idempotent; non-transferable work runs at most once and is marked as failed rather than moved. A partition is never silently lost.

Where this pattern makes sense, and where it doesn't

The article itself draws the ceiling of the approach: it's meant for small-to-medium clusters, where operational simplicity and cost matter more than extreme latency. Past a certain point, with too much database contention, very large clusters, or the need for sub-second latency, a broker is still the right tool.

For those running overnight financial reconciliation pipelines, mid-sized ETL, or data validation on dynamically provisioned cloud nodes, and who already operate a well-maintained relational database, the pattern eliminates an entire operational layer without giving up automatic failover. The original InfoQ article is worth reading for anyone planning to implement it: the full text also details the observability metrics exposed and the test scenarios used to validate failover behavior.