Every night, Bitdepth asks the same questions to every company hosting a popular AI model, grades the answers, and writes down what came back. No jargon needed. This page explains it from the ground up.
Five minute read. Skip to the glossary if a word trips you up.
They all sell "grandma's chocolate cake" from the same recipe. Same name on every sign.
Some quietly use less chocolate. They charge the same. From the outside, you can't tell.
Bitdepth buys a slice from every bakery, every night, and writes down which ones still taste like the recipe.
Illustration only. Not real results.
Some AI models are open: anyone can download the files and run them. DeepSeek, Llama and gpt-oss are examples. So when an app asks for "DeepSeek," the request can land at any of dozens of hosting companies. Each one says it is serving the same model. They don't all behave the same.
One September 2026 analysis found a single model served by 33 providers, priced up to 11 times apart.
Providers describe their own setup. Some run a compressed, cheaper version. Nobody outside checks the label.
A cut-off or worse answer still comes back as "success." Users notice before anyone measures it.
People blame the app they pay, not the host they never picked. Contracts promise speed and uptime, never the model.
Source for the 33-provider figure: moksh on dev.to, Sept 2026.
Publishes the open files. The "recipe."
Dozens of companies. The "bakeries."
Usually on price and speed. Not on taste.
You see the answer. Not who served it.
A fixed set of prompts goes to every host serving a model, through the same door any customer uses.
Did it get the checkable answers right? Does it agree with itself when asked twice? Did it change since last week?
Each host gets a dated record. The method is public, so anyone can rerun it and get the same grade.
What each host returned, compared to the reference and to itself.
Why it differs. We measure behavior, not motives. The cause is the provider's to explain.
An AI model is billions of numbers. Bit depth is how much detail each number keeps.
Fewer bits makes a model cheaper to run, and a little blurrier. Like saving a photo as a smaller file. Usually fine. Sometimes it isn't. Hosts rarely say which they used.
Bar length is detail kept per number. Illustration only. More on this in fp8 vs fp4 vs bf16.
A benchmark tells you which model is smartest, once. Bitdepth tells you whether the host you are using tonight is still serving what it claims. Every night, every host.
We don't look inside. We look at the answers. If a host answers differently from the reference, by more than normal randomness, that difference is the finding.
Same risk any mystery shopper faces. Our requests arrive like any customer's. Rotating the question set is how we keep it honest as grades go public.
Anyone paying for AI answers who needs to know what they got. Apps, gateways, and teams buying AI at scale. The public scorecard is free to read.
Yes. The public scorecard lists every endpoint by name. Each provider gets a review window before publication, and the method is public.
No. Today, plain code does the grading: right or wrong against a known answer, same or different from last time.