Blackwellboy / Tools
Checks to run before blaming the model.
The toolbox follows the order I actually debug in: establish whether the endpoint is sane, inspect what the model received, check the stack-specific traps, then reproduce the lane before changing bigger things.
Start with this path
Four steps that turn “the model looks wrong” into a falsifiable problem.
Diagnose
Registry
model-serving-minefield
A registry of serving and configuration failures that can make a healthy model look broken or make a bad measurement look convincing. It currently contains 168 entries across 9 categories: evaluation, memory, quantization, reasoning, routing, runtime, template, tools, versioning.
Every entry starts with the symptom, then the mechanism, the cheapest useful check and the fix. Evidence status is explicit.
Fastest triage
The Core 12
The short list for when you do not yet know whether the failure lives in the model, template, runtime, quant, harness or client. These are cheap to test and easy to misread as model behaviour.
Endpoint preflight
minefield-doctor
A stdlib-only OpenAI-compatible endpoint preflight with 70 read-only checks around reasoning fields, templates, thinking control, tool parsing and output ceilings. Anything it cannot test is reported as not checked.
Prompt path
preflight_template.py
Renders a marked multi-turn conversation through your own path so you can inspect the actual prompt at turn N. This catches history stripping, role mutations and template branches that an ordinary API smoke test cannot see.
Model lookup
Traps by model
A model-family map that links the traps observed on each tested or reported checkpoint. It also says when evidence is upstream or contributor-supplied so absence from the table never becomes a fake safety claim.
Stack lookup
Stack pages
11 stack pages covering hf-transformers, llama.cpp, lm-studio, MLX, Ollama, sglang, tabbyapi, tensorrt-llm, text-generation-inference, text-generation-webui and vLLM. Each page gives the traps most likely to bite there and the checks to run first.
Reproduce & operate
When the diagnosis is no longer enough and you need a known-good lane or an ordered procedure.
Operational playbooks
Minefield playbooks
7 ordered playbooks for jobs such as publishing an A/B, debugging reasoning loss, porting a harness, investigating long context and reading a soak. They turn the registry into procedures rather than a list of bugs.
Deployment recipes
Large-model bring-up repos
Kimi K3 on SparkInfer, Hy3-295B on dual Spark, MiniMax-M3 on the stock driver, plus the Laguna lab. These are the public runnable surfaces behind the deployment cards on Work.
Worked example
Laguna S 2.1 operators guide
A model-specific worked example of the method end to end: serving configuration, thinking control, tool format, quantisation floor and verification checks, with every claim linked to the study directory that supports it.
Verify the research
The tools that keep a result from becoming a story that drifts away from its evidence.
Self-checking claims
Minefield integrity layer
A claim ledger records corrections and retractions. Propagation checks search public surfaces for superseded wording, while mutation tests deliberately reintroduce known failures to prove the checks can still catch them.
Published raw evidence
model-serving-minefield-evidence
Public scrubbed evidence bundles for studies where publishing the full raw material is appropriate, including long-soak manifests and immutable study indexes. Sensitive prose is hashed or redacted rather than quietly omitted.
Corrections
What changed
The public correction surface for claims that were retracted, narrowed or re-scoped after better evidence arrived. The correction is part of the research artifact, not an embarrassing footnote.