Submit a run

Use this manual route for vLLM, LM Studio, custom llama.cpp rigs, or any server you already launched yourself. It is the publishable fallback when the catalog quant lacks artifact pins or a pasted repo run is local-only.

A. Fetch the frozen suite

localbench fetch-suite \
  --site https://local-bench.ai \
  --suite suite-v1-full-exec-6axis-v1 \
  --accept-suite-terms

suite-v1-full-exec-6axis-v1 is the current ranked suite: it measures six axes; five are weighted in the Index, and tool-calling is reported as an unweighted diagnostic.

This downloads the sha256-pinned item sets, verifies them against the release manifest, and caches them locally. --accept-suite-terms acknowledges the upstream benchmark licenses listed under benchmark sources & licenses.

B. Prepare your model's tokenizer

Ranked runs pass --hf-model-idso the harness can introspect your model's chat template. That introspection is deliberately offline (the run never phones home mid-benchmark). When online, the CLI checks the local cache first and automatically downloads only the required tokenizer/template files when they are missing, records the resolved revision, then retries introspection from the local cache. To pre-warm the cache, or before using --offline, run:

localbench cache-tokenizer <the-model's-HF-repo>

This downloads exactly what introspection needs, verifies the tokenizer loads offline, and prints the resolved revision and chat-template hash. Use the tokenizer and tokenizer_config files from that Hugging Face snapshot in step 4. Use the original model's repo (the transformers-format one), not the GGUF repo. Gated repos (for example google/gemma-*) additionally need a one-time hf auth login after accepting the license on huggingface.co — or use an ungated mirror such as the unsloth/ upload of the same model. No exact non-GGUF repo exists for your model? Skip this step and pass --gguf-repo-only instead of --hf-model-id in step 4 — the run is then labeled basic identity (tokenizer/template digests null).

C. Run against your server

Already serving the model yourself (LM Studio, ollama, vLLM, anything OpenAI-compatible)? Use run instead:

localbench run \
  --endpoint http://localhost:8080/v1 \
  --model MaziyarPanahi/Qwen3-8B-GGUF:Q4_K_M \
  --hf-model-id Qwen/Qwen3-8B \
  --lane bounded-final-v2 \
  --profile auto \
  --tier standard \
  --publishable \
  --sampler-temperature 0 \
  --sampler-top-k 1 \
  --sampler-seed 1234 \
  --determinism-policy gpu-greedy-single-slot-v1 \
  --model-file <path-to-qwen3-8b-q4-k-m.gguf> \
  --model-family Qwen3 \
  --quant-label Q4_K_M \
  --model-format gguf \
  --tokenizer-file ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B/snapshots/<revision>/tokenizer.json \
  --chat-template-file ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B/snapshots/<revision>/tokenizer_config.json \
  --runtime-name llama.cpp \
  --runtime-version <llama.cpp-build> \
  --kv-cache-quant f16 \
  --ctx-len-configured 32768 \
  --parallel-slots 1 \
  --out runs/qwen3-8b-q4-k-m.json

The ranked board uses the bounded-final protocol: every model gets the same generated-token budget per item, and --profile auto reads your model's own chat template to decide whether it thinks (bounded) or answers directly. A run must pin its sampler settings to be publishable (--publishable requires temperature 0, top-k 1, and a seed); the CLI warns up front — before any GPU time is spent — if your flags make the run unpublishable. Keep the .json extension on --out — the campaign directory is derived from it. Want these pre-filled for your VRAM and model? The recipe builder on the home page generates this exact sequence.

suite-v1-full-exec-6axis-v1 is the current ranked suite. suite-v1-static-exec-5axis-v1 and suite-v1-static-core-diag-v1 are static or diagnostic suites; they preserve evidence but do not produce rankable rows.

D. Submit manually

localbench submit run --run runs/qwen3-8b-q4-k-m.json

One command takes a finished run all the way in: it packs the signed bundle, requests a submission ticket (signing a proof-of-possession challenge with your key), uploads the bundle, completes the submission, and prints your submission id and status. The suite is auto-resolved from your fetch-suite cache; --suite-dir overrides it. --run accepts the run JSON or its campaign directory; add --display-name once to set your credit line (remembered in ~/.localbench/submit.json).

Bundles are content-addressed by sha256, so duplicates are detected: re-running submit run with a bundle that is already in tells you its existing submission id instead of creating a new row, and a bundle whose payload matches an earlier submission is flagged for the reviewer. If a ticket expires mid-flight, a fresh one is minted automatically. The public status page is local-bench.ai/submission?id=<submission_id>. You can also check from the CLI:

localbench submit status <submission_id> --site https://local-bench.ai