Methodology

What macbatch bench measures, and what it deliberately does not.

Running one

macbatch bench embed --n 500 --mode solo --shard-size 32

Flag	Meaning
--n	Number of items
--mode	A label only — solo or multi. It does not change behaviour; it tags the run for comparison.
--shard-size	Items packed per task
--model	Ollama model, default nomic-embed-text
--corpus	Optional JSONL path. Without it, a synthetic corpus is generated.
--control-url	Override the configured control plane

What the harness does

1. Reads GET /v1/workers and records which workers are alive before submitting. If none are, it aborts — benchmarking an empty pool measures the polling loop.
2. Opens a run record with the full environment as metadata.
3. Submits all items as one job and starts a wall clock.
4. Polls GET /v1/jobs/{id}/stats until no tasks are pending or leased.
5. Writes a summary, an event timeline, and appends one comparison line.

Artifacts:

benchmarks/runs/<run_id>/summary.json    every stat, plus environment
benchmarks/runs/<run_id>/events.jsonl    timeline of the run
benchmarks/runs/<run_id>/results_meta.jsonl
benchmarks/runs/embed_compare.jsonl      one line per run
benchmarks/runs/latest.json              pointer to the most recent run

In a source checkout these land in the repo. Installed from npm or pip, they land in ~/.macbatch/benchmarks/ instead.

What is measured

Metric	Definition
wall_sec_client	Submit to last completion, measured by the client
wall_sec_server	The control plane's own view of the same interval
items_per_sec / items_per_hour	Items divided by wall time
shards_done / failed	Task-level outcome counts
by_worker	Tasks completed and average duration, per worker
lease_expired_events	Reclaimed leases — the flakiness signal
n_workers_used	Workers that completed at least one task

Known limitations

Synthetic corpus. The generated items are template sentences of ~26 tokens. Real documents are longer and slower per item. Any throughput number here is an upper bound for real work — this is exactly why the cost model scales throughput by document length.

Cold vs warm state is uncontrolled. The harness does not restart Ollama or clear the model from memory between runs. The first run of a session pays model load cost that later runs do not. One early two-machine run came in at 8.06s against a 17.11s clean re-run; the difference is unexplained variance and the slower, repeatable number is the one published.

No isolation. Runs happen on a machine that is also running the control plane, a browser, and everything else. There is no attempt to quiesce the system.

mode is not verified. Nothing checks that a run labelled multi actually had two workers. Read n_workers_used in the summary rather than trusting the label.

Item completion is estimated. items_done_est in the stats endpoint scales n_items by the fraction of shards completed, so it is only exact when all shards are equal size — i.e. when n_items divides evenly by shard_size.

Single sample. Every published figure is one run. There are no repeats, no error bars, and no confidence intervals.

Comparing runs

cat benchmarks/runs/embed_compare.jsonl

One JSON object per run with run_id, mode, n, shard_size, wall_sec, items_per_hour, n_workers_used, and failure counts. Since mode is a free-text label, keep it meaningful when you are comparing configurations.

Bringing your own corpus

macbatch corpus --n 5000 --out mycorpus.jsonl     # generate a synthetic one
macbatch bench embed --n 5000 --corpus ./mycorpus.jsonl

The file is JSONL with id and text per line; content is accepted as an alias for text, and a missing id is filled with the line index. Benchmarking your own documents is the only way to get a number that predicts your workload.