Control plane API

FastAPI app at macbatch.control.app:app. JSON in, JSON out.

Unauthenticated

No endpoint requires credentials, including job submission, result retrieval, and /v1/admin/reclaim. Anyone with the URL has full access. Put your own auth layer in front of anything exposed.

GET /health

{ "status": "ok", "service": "macbatch-control", "ts": 1754212345.6, "version": "0.2.0" }

POST /v1/jobs

Submit a job. Items are packed into shards; each shard becomes one task.

{
  "type": "embed",
  "model": "nomic-embed-text",
  "shard_size": 32,
  "items": [{ "id": "doc-1", "text": "…" }],
  "meta": {}
}

Field	Type	Notes
type	embed | ocr | classify | generate	Default embed
model	string	Default nomic-embed-text. Must already be pulled on the workers.
items	array	Required, non-empty
shard_size	int 1–2048	Default 32
meta	object	Echoed back on the job record

Response:

{ "job_id": "…", "n_items": 500, "n_tasks": 16, "n_shards": 16, "shard_size": 32, "status": "running" }

GET /v1/jobs/

Reclaims expired leases, then reports current state. Also promotes the job to completed or partial when no tasks remain open.

{
  "job_id": "…",
  "type": "embed",
  "status": "completed",
  "counts": { "pending": 0, "leased": 0, "done": 16, "failed": 0, "total": 16 },
  "by_worker": [{ "worker_id": "…", "done": 16, "avg_duration_ms": 674.48, "sum_duration_ms": 10791.68 }],
  "lease_expired_events": 0,
  "wall_sec": 7.118,
  "meta": {}
}

404 if the job does not exist.

GET /v1/jobs/{job_id}/results

{
  "job_id": "…",
  "results": [{ "task_id": "…", "status": "done", "worker_id": "…", "duration_ms": 674.5,
                "payload": {}, "result": {}, "error": null }],
  "items": [{ "task_id": "…", "worker_id": "…", "id": "doc-1", "embedding": [0.1], "dim": 768 }],
  "n_shards": 16,
  "n_items": 500
}

results is per shard. items is the flattened per-item view and is what you normally want. n_items is null when no shard produced an inner items array.

GET /v1/jobs/{job_id}/stats

Everything from the job endpoint plus a throughput block.

{
  "throughput": { "shards_done": 16, "items_done_est": 500, "n_items": 500,
                  "items_per_sec": 70.19, "items_per_hour": 252685.63 },
  "workers_active": 1
}

items_done_est is an estimate — shards done divided by total shards, times n_items. Exact only when all shards are the same size.

POST /v1/tasks/lease

Called by workers.

{ "worker_id": "mac-abc123", "limit": 4, "job_types": ["embed"] }

Reclaims expired leases, upserts the worker as online, then atomically claims up to limit pending tasks in rowid order. job_types filters by job type; omit for any.

{ "tasks": [{ "task_id": "…", "job_id": "…", "type": "embed", "model": "nomic-embed-text",
              "payload": { "shard_index": 0, "shard_size": 32, "items": [] },
              "lease_until": 1754212945.6, "attempts": 1 }] }

Leasing sets lease_until to now + 600s and increments attempts.

POST /v1/tasks/complete_batch

One call for many completions. Preferred over the per-task endpoint.

{
  "worker_id": "mac-abc123",
  "items": [{ "task_id": "…", "result": {}, "error": null, "duration_ms": 674.5 }]
}

{ "ok": true, "n": 2, "statuses": [{ "task_id": "…", "status": "done" }] }

Per-item status is done, requeued, failed, or missing.

POST /v1/tasks/{task_id}/complete

Single-task equivalent. Workers fall back to this when a control plane returns 404 for the batch endpoint.

{ "worker_id": "mac-abc123", "result": {}, "error": null, "duration_ms": 674.5 }

A non-null error requeues the task, or fails it permanently once attempts reaches 3. Completing an already-done task is a no-op — the first result wins.

POST /v1/workers/heartbeat

{ "worker_id": "mac-abc123", "status": "online",
  "info": { "host": "…", "ollama": "…", "jobs": ["embed"], "models": ["nomic-embed-text"] } }

Upserts the worker and reclaims expired leases. Returns { "ok": true, "server_time": … }.

GET /v1/workers

{ "workers": [{ "id": "…", "status": "online", "last_heartbeat": 1754212345.6,
                "alive": true, "tasks_done": 16, "tasks_failed": 0, "info": {} }] }

alive means heartbeated within 60 seconds and not self-reported offline. tasks_done and tasks_failed are cumulative across all jobs — for per-job attribution use by_worker on the job endpoint.

POST /v1/admin/reclaim

Force-expire stale leases immediately.

{ "reclaimed": 3 }