Jobs, shards and tasks

Three words that are easy to confuse, and the tuning knob that matters most.

• A job is what you submit: a list of items, a type, and a model.
• A shard is a contiguous slice of those items, sized by shard_size.
• A task is a shard's row in the queue — the unit that gets leased, retried, and reassigned.

One job of 500 items with shard_size=32 becomes 16 tasks.

Job to result500 items → 16 tasks → flattened results

Submitting

{
  "type": "embed",
  "model": "nomic-embed-text",
  "shard_size": 32,
  "items": [
    { "id": "doc-1", "text": "first chunk" },
    { "id": "doc-2", "text": "second chunk" }
  ],
  "meta": { "anything": "you want echoed back" }
}

shard_size accepts 1–2048 and defaults to 32. items must be non-empty.

The response tells you how the job was split:

{ "job_id": "…", "n_items": 500, "n_tasks": 16, "n_shards": 16, "shard_size": 32, "status": "running" }

Choosing shard_size

This is the highest-leverage number in the system, and the two forces pull opposite ways.

Small shards	Large shards
One HTTP round trip per few items	Round-trip cost amortised across many items
A dead worker loses almost nothing	A dead worker loses the whole shard's work
More rows in tasks, more SQLite writes	Fewer, larger rows
Fine-grained reassignment	Coarse reassignment

The measured effect of getting this wrong is large. The same 500-item job on the same machine:

Effect of shard_sizeidentical hardware, identical corpus

Unsharded (shard_size=1) spends most of its wall time on HTTP, not inference. Packing 32 items per task cut the job from 20.2s to 7.1s — a 2.8× speedup from batching alone.

A practical starting point: pick a shard that takes each worker 30 seconds to a few minutes. Long enough that HTTP overhead disappears, short enough that a closed laptop lid does not cost much.

Results

GET /v1/jobs/{job_id}/results returns both views:

{
  "results": [ { "task_id": "…", "status": "done", "worker_id": "…", "payload": {…}, "result": {…} } ],
  "items":   [ { "task_id": "…", "worker_id": "…", "id": "doc-1", "embedding": [...], "dim": 768 } ],
  "n_shards": 16,
  "n_items": 500
}

Read items. It is the reduce step — every shard's inner items flattened into one list, each tagged with the task and worker that produced it. The results array is the per-shard view, useful for debugging distribution and errors.

Job status

Derived on read, not maintained by a background process:

Status	Meaning
running	Tasks are still pending or leased
completed	No open tasks, nothing failed
partial	No open tasks, at least one failed permanently

GET /v1/jobs/{id}/stats adds a throughput block. Note that items_done_est is an estimate — it scales n_items by the fraction of shards completed, so it is only exact when every shard is the same size.

Item shapes by job type

Type	Item fields	Runs
embed	id, text	POST /api/embeddings per item
ocr	id, image_b64 or image_path, optional prompt	POST /api/generate with images
classify, generate	id, prompt or text	POST /api/generate

embed also still accepts a legacy single-item payload ({id, text} with no items array) from before sharding existed.