Leases and failure

macbatch assumes workers die. The lease is the mechanism that makes that survivable.

Task state machinepending · leased · done · failed

The lease

When a worker leases a task, three things happen atomically:

1. status becomes leased
2. lease_until is set to now + 600s
3. attempts is incremented

That third point matters: the attempt is counted on lease, not on failure. A worker that dies without ever reporting anything still burns an attempt.

The update is guarded by WHERE id = ? AND status = 'pending', so two workers polling simultaneously cannot both take the same task — the second one's update affects zero rows and is skipped.

Reclaiming

A task whose lease_until has passed is assumed abandoned: it goes back to pending, its assigned_worker is cleared, and a lease_expired event is recorded.

This runs opportunistically inside request handlers — lease, heartbeat, job read, worker list — not on a timer. An idle control plane reclaims nothing until something asks it a question, which is harmless because the next poll triggers it.

Force it manually:

curl -X POST <url>/v1/admin/reclaim

The count of lease_expired events for a job is your flakiness metric. It appears as lease_expired_events in GET /v1/jobs/{id} and in every benchmark summary. Across every published run it is zero.

Errors and retries

When a worker reports an error:

• Fewer than 3 attempts — the task returns to pending with the error recorded, and another worker picks it up.
• 3 or more attempts — the task becomes failed, its result is cleared, and the worker's tasks_failed counter increments.

There is no backoff between retries and no dead-letter queue. A task that fails deterministically — malformed input, a model the pool does not have — burns three attempts quickly and then stops.

At-least-once, not exactly-once

A task can run more than once. Two ways:

1. A worker finishes a shard, but its complete_batch call fails. The lease expires and the shard is re-run elsewhere.
2. A worker is slow enough that its lease expires mid-shard. The shard is reassigned and both machines finish it.

The control plane handles the duplicate — _apply_complete returns early if the task is already done, so the first result wins and later ones are discarded. But your side must be idempotent: keying results by the item id you submitted is enough.

Tuning for reliability

Symptom	Lever
Shards keep expiring mid-run	Lower shard_size so each task finishes well inside 600s
Losing lots of work when a laptop closes	Lower shard_size and lease_limit
Too much control-plane chatter	Raise shard_size
One worker hoarding everything	Raise shard_size; scheduling fairness is not implemented

The 600-second lease is a module constant (LEASE_SECONDS in macbatch.control.db), not configurable at runtime.

Heartbeats

Separate from leases. A worker posts a heartbeat each cycle with its hostname, job types, models, and concurrency settings. It counts as alive if that heartbeat is under 60 seconds old and it has not reported itself offline.

Liveness affects reporting only. A stale worker's tasks are not reclaimed early — the lease still has to expire on its own.