open source · apple silicon · MIT
A job queue, lease scheduler, and worker CLI for latency-tolerant work — embedding, OCR, classification. Every worker runs a whole model and takes a shard of the queue. Machines are assumed unreliable; tasks are reassigned when they vanish.
npm i -g macbatchNot a live inference API. If you need low latency, use a hosted provider — the cost model shows where this does and does not pay off.
You have Macs sitting idle. macbatch turns them into a pool that chews through work where latency does not matter — re-embedding a corpus after a model change, OCR on a document backlog, classifying a table of records overnight.
It is three moving parts: a control plane that holds the queue, worker processes that lease shards of it, and Ollama on each worker actually running the model.
A live inference API. There is no streaming, no per-request SLA, and a task can be retried on a different machine at any moment. If you need a completion in under a second, use a hosted provider — that is a different product with different physics.
The largest measured win so far is not distribution, it is batching: packing items into shards cut the same 500-item job from 20.2s to 7.1s on one machine. Adding a second Mac over a tunnel gave only 1.18×, because greedy leasing lets the fast local worker drain the queue first.
Looking for the hosted pool with signup, dashboards, and managed machines? That is a separate product — see macbatch cloud.