How to turn Postgres into a durable workflow orchestrator without Temporal

Adding an external orchestrator like Temporal or AWS Step Functions is usually the standard advice for anyone who needs durable execution (that idea of continuously checkpointing state, like a videogame autosave for backend code). But an article published on InfoQ by Raman Varma, from the Kestrel Workflows team, argues that if you already run Postgres as your system of record, you don't need anything else for that. The thesis is direct: durability is fundamentally a matter of writing state to a durable database, so why not make Postgres itself the orchestrator?

The scenario that drove the decision is familiar to any infra team: a workflow that fires when a Kubernetes workload fails, runs an AI-based root cause analysis, generates a fix, waits for human approval on Slack, and opens a pull request via GitOps. If the pod gets rescheduled in the middle of that (and in Kubernetes it will), losing the execution isn't an option.

Why the team gave up the external orchestrator

The article lists the real cost of plugging in a dedicated orchestrator, and this is where the discussion gets concrete for builders:

• One more stateful system to deploy, secure, monitor, and update, sitting on the critical path of every workflow (in other words, a single point of failure).
• A new security and audit surface: in Kestrel's case, step payloads include infrastructure topology, source code, and logs, which would start flowing to an external system.
• A proprietary data model, usually a key-value store optimized for workflow state, which makes ad-hoc analysis less straightforward than a relational query.

The author's conclusion is that the team had already invested heavily in making Postgres perform at scale, so adding an orchestrator would mean paying twice for the same durability guarantee.

The clause that holds it all together: SKIP LOCKED

In Kestrel, there's no orchestrator process. Each application server runs an embedded workflow library and talks directly to Postgres. Every trigger (from PagerDuty alerts to GitHub webhooks) inserts a row into a workflow_executions table, and servers poll that table to claim work.

The trick that makes this safe is a single clause:

SELECT id, input
FROM workflow_executions
WHERE status = 'enqueued'
ORDER BY created_at
LIMIT 1
FOR UPDATE SKIP LOCKED;

FOR UPDATE SKIP LOCKED locks the rows a worker grabs and tells other workers to skip them instead of blocking. Two servers can scan the same table at the same time and never hand the same workflow to two executors. No broker, no leader election, no Redis locks with a TTL. It's worth noting that the pattern isn't exclusive to Postgres: MySQL 8.0+, MariaDB 10.6+, and Oracle also support SKIP LOCKED, while Db2 and SQL Server offer equivalent mechanisms.

The article makes three honest caveats for production: the claim transaction has to be tiny (just lock, change the status, and commit, never hold the transaction while the step runs); LIMIT can be higher than 1 to grab batches, but that increases the blast radius if the worker dies; and SKIP LOCKED doesn't preserve FIFO under contention, so anyone who needs strict ordering shouldn't use this pattern.

Idempotency as a constraint, not as code

Each step's checkpoints go into an operation_outputs table with primary key (execution_id, step_id). The worker writes the result with an upsert that never overwrites:

INSERT INTO operation_outputs (execution_id, step_id, output)
VALUES ($1, $2, $3)
ON CONFLICT (execution_id, step_id) DO NOTHING
RETURNING output;

If a recovering worker re-executes a step that already committed, the uniqueness constraint prevents the duplicate checkpoint. Instead of running the side effect again, the worker reads the existing checkpoint and returns the previous result. It's the database, not the application code, that guarantees idempotency. It's a nice example of expressing a distributed system invariant as a relational constraint instead of writing the bookkeeping by hand.

Recovery with leases and a sweeper

The interesting failure is when a worker claims an execution, sets it to running, and dies (OOM kill, node drain, pod eviction). The row stays stuck at running with no live owner. The solution is a set of lease columns: the worker that owns an execution sends a heartbeat at an interval, pushing lease_expires forward. A periodic sweeper returns everything that expired back to the queue:

UPDATE workflow_executions
SET status = 'enqueued', owner_id = NULL
WHERE status = 'running'
AND lease_expires < now();

Lease duration is an explicit tradeoff: too short and a slow (but alive) worker gets its work stolen; too long and recovery after a crash is delayed. Here, checkpoint idempotency is what gives the freedom to use aggressive leases, because a false-positive sweep would at most cause duplicate work, not a doubled side effect.

Sleeps and human approvals that survive a restart

Waiting for human approval on Slack, which can take anywhere from two hours to two weeks, is durable too. Instead of a goroutine praying for the pod to stay alive long enough, it's just a row in a workflow_waits table with a wake_at timestamp. The execution sets itself to waiting, and a sweeper re-enqueues whatever has already come due. It doesn't matter how long the wait window is: the cost is always the same, one row.

Observability and scale: where the math checks out

Since every workflow and every step is a row, monitoring becomes plain SQL. Want all the executions that errored out this month? SELECT * FROM workflow_executions WHERE created_at > NOW() - INTERVAL '1 month' AND status = 'error'. You can go further and join executions with steps and approvals to answer "which fixes got rejected, and at which step?" With external orchestrators, this kind of relational query requires exporting the key-value store's state to a separate analytics system.

On scale, the author states that a single instance handled tens of thousands of workflows per second without read replicas or Citus, scaling throughput with stateless workers. And he cites the real points of concern: connection pressure (the recommendation is PgBouncer in transaction pooling mode in front, with backoff and jitter in the polling) and dead tuples/bloat in a high-churn queue table (autovacuum tuning, partial indexes, and partitioning completed executions away from the live queue).

The article itself is honest about the limit: if you need fan-out to thousands of workers and sub-millisecond dispatch latency, Temporal is still the better architectural choice. Kestrel runs I/O-bound automation pipelines that last from seconds to hours and need modest concurrency, and for that profile, Postgres isn't a compromise, it's a fit.

What changes for those building in Brazil

For startups and small teams, the practical takeaway is that a capability many people assume requires an external service can come from a database you already operate, pay for, and know how to maintain. One less stateful system on the critical path means less infrastructure to provision, a smaller audit surface (LGPD, Brazil's data protection law, will thank you: no workflow payload leaves the database), and a cloud bill that doesn't gain another component. Reliability and security collapse into a single dependency, because if Postgres goes down, the rest of the product has already gone down with it.

The honest starting point before replicating this pattern: assess your workload's profile. If your workflows are I/O-bound automation with moderate concurrency, the design described here is reproducible today with SKIP LOCKED, constraints, and a sweeper. If you need strict ordering, massive fan-out, or sub-millisecond dispatch, the authors themselves point to the dedicated orchestrator as the way out.