Why Do LLM Visibility Tools Disagree?

A prospect sent us two dashboards last month. Same brand, same week, same category. One LLM visibility tool put their AI presence at 34%. The other put it at 61%.

They wanted to know which one was lying.

Neither was. Both numbers were produced correctly by instruments that had been pointed at slightly different things, and nobody had told them that was possible.

Every page on the first search results page for this category is a roundup of fifteen products. The most careful of them, Omniscient Digital's tool guide, concedes in a single sentence that methodologies "produce significantly different numbers for the same brand," then moves on to the next vendor without saying how different, or why.

Here is the how much and the why.

Why do two LLM visibility tools disagree?

Two LLM visibility tools disagree because each one runs a different prompt set, against a different mix of engines, a different number of times, through a different interface, and scores the result with a different counting rule. Change any one of those five settings and the number moves without your brand moving. There is no reference measurement to check either against.

That is the whole answer. The rest is what each setting is worth, and how to hold them still.

You are not buying accuracy. You are buying an instrument, and instruments are judged on repeatability, not truth.

There is no ground truth, so no tool can be the accurate one

Start with the thing that makes this category different from rank tracking. A Google result page exists. Two rank trackers checking position 4 can both be checked against the page. An AI answer is generated at request time and then it is gone.

So when two tools disagree there is no third thing to consult.

The variance floor exists before any vendor touches it

Thinking Machines Lab ran a clean version of this experiment. They sent one prompt to one model a thousand times with sampling switched off, at temperature 0, the setting that is supposed to make output deterministic.

They got 80 unique completions. The most common appeared 78 times. All thousand agreed through "Feynman was born on May 11, 1918, in" and then split at token 103: 992 continued "Queens, New York" and 8 continued "New York City."

The cause is other customers, not your brand

The reason is a property called batch invariance, or the lack of it. Inference kernels change how they split a numerical reduction as the batch size changes, and the batch size depends on how many other requests hit that endpoint in the same millisecond.

Thinking Machines put it plainly: the other concurrent users are "not an 'input' to the system but rather a nondeterministic property of the system."

Your vendor cannot set that. Neither can the model provider, without deliberately giving up throughput. SGLang's deterministic mode and vLLM's batch-invariance flag exist and they work, at a real speed cost, and no consumer AI interface runs them.

What buyers ask a vendor:

• •Which engines do you cover?
• •How many prompts do I get?
• •What does it cost per month?
• •Do you integrate with our warehouse?

What actually decides the number:

• •Which engines, weighted how?
• •Which prompts, written by whom?
• •How many runs per prompt, per day?
• •Which interface, with grounding on or off?
• •Does a mention count the same as a citation?

Every vendor answers the first list on its pricing page. The second list decides what you are looking at, and most of it is undocumented.

5 settings that move your number before your work does

None of these are defects. Each is a legitimate design choice, and reasonable engineers land on different ones. The problem is that the choices are invisible in the output, so two defensible instruments produce two numbers and the buyer is left refereeing.

Setting #1: Engine mix is worth 18.3 points on its own

This is the one we can price from our own data rather than argue about.

Our concluded CITE Index study ran 500 buyer prompts nightly through ChatGPT, Gemini and Google AI Mode for 63 days, collecting 90,132 AI answers across 10 categories. The share of answers carrying at least one source ran 97.4% on Google AI Mode, 92.5% on ChatGPT and 79.1% on Gemini.

That is an 18.3-point spread between the most and least citation-dense engine we measured.

A tool weighted toward Google AI Mode will report a healthier brand than one weighted toward Gemini. Same brand, same day, same pages. The difference belongs to the sampling config.

Setting #2: Two vendors seeded with your category will not write the same prompts

Prompt sets are generated, not standardised. Give two products the same brand and category and they produce different lists, usually 30 to 60 prompts each, with maybe a third in common.

Prompt wording decides which competitors appear in the answer at all. That changes your denominator before the counting starts. We worked through how to build a set you control in how to select prompts for LLM tracking.

Setting #3: Runs per prompt sets your noise floor, and it is rarely published

One run per prompt per day samples a distribution exactly once. Given the 80-unique-completions result above, a single run is a coin flip dressed as a measurement.

Averaging four or five runs narrows the interval considerably. Both approaches render as one confident percentage on a dashboard. The run budget arithmetic is the part worth asking about before you compare quotes, because it is what you are actually paying for.

Setting #4: API, grounded API and consumer app are three different populations

An API call with web search off queries the model's parametric memory. The same call with grounding enabled queries a retrieval layer. A scrape of the consumer interface gets the product your buyer actually uses, including personalisation and whatever the app is testing that week.

These sample three different things. Only the third resembles the surface you are trying to win, and it is the hardest and most fragile to collect.

Setting #5: Mention, citation and recommendation are three separate events

Being named in prose, being linked as a source, and being recommended as the pick are different outcomes with different commercial value.

Some products score all three as presence and report one blended percentage. Others count only linked citations. A brand named warmly in ten answers without a single link scores near the top on one instrument and near zero on the other.

Ask which event the headline number counts. If the answer takes more than one sentence, that number cannot be compared to anyone else's.

What each setting costs you if you get it wrong

Setting	What goes wrong when it is unknown	What to ask for
Engine mix	Your trend line tracks the vendor's engine weighting, not your work. Adding a citation-dense engine looks like a win.	The per-engine breakdown, and the weighting used in the blended score.
Prompt set	You cannot compare quarters, because a refreshed prompt list resets the baseline silently.	The full prompt list as text, plus a written change policy.
Runs per prompt	Weekly churn gets read as performance. Teams reallocate budget against sampling noise.	Runs per prompt per day, and whether the figure shown is a single run or a mean.
Interface and grounding	You optimise for a surface your buyers never touch, and the wins do not reproduce.	Which route produced each reading: raw API, grounded API, or consumer interface.
Counting rule	Two dashboards disagree by 20 points and both are right, which destroys trust in the whole program.	The definition, in one sentence, of what increments the headline metric.
None of the above	Nothing, if you freeze all five and only compare a tool to itself.	A settings sheet you keep beside the dashboard.

The six jobs to score any of these products against sit in the AI visibility platform buyer's guide, and the shortlist itself is in which AI visibility tools B2B teams use.

Run a bake-off instead of reading another roundup

The diagnostic half is done. If you are choosing between two products, or trying to reconcile two you already own, this is the sequence we run. It takes about three weeks and it settles the argument.

Step 1: Write the prompt set yourself and give both tools the identical list

Do not let either vendor generate it. Write 30 prompts your buyers actually type, paste the same list into both, and refuse any product that will not accept an imported set.

This alone removes the largest source of disagreement, and some vendors will decline. That is a useful answer.

Step 2: Force both tools onto the same engines, or read them per engine

If one product covers nine engines and the other covers four, do not compare the blended scores. Compare only the engines both cover, engine by engine.

Given the 18.3-point spread we measured across engines, a blended-to-blended comparison is not a comparison.

Step 3: Run a no-change period for 14 days before you judge anything

Ship nothing. Publish nothing. Change no page.

Whatever both dashboards do over those two weeks is your noise floor, and it is the only number that tells you how large a movement has to be before it means something. Most teams discover their threshold is far higher than the movements they have been reporting.

Step 4: Ask each vendor to name the counting rule in one sentence

Send the same question to both: what exactly increments the headline metric, a mention, a linked citation, or a recommendation?

Then re-read both dashboards knowing the answer. A large share of the gap usually closes right here, without either tool being wrong.

Step 5: Reconcile the remainder against a surface you control

Take the engines both cover and check a handful of prompts by hand, in the consumer app, on the same day. You will not resolve the percentages, and that is not the goal.

The goal is to learn which instrument's picture matches what a buyer sees. That is the one to standardise on, and then you never compare it to anyone else's number again.

What no setting on either tool reaches

There is a limit that applies to every product in this category, and the bake-off above will not move it.

Across those 90,132 answers, 8 of the 12 most-cited domains were not brand-owned. Reddit alone drew 14,698 citations and appeared in 13.6% of all answers we collected. The average category leader held 78.2% of its category's answers, and the leader flipped on only 18.7% of day pairs. In four of the ten categories it never changed once. The full corpus sits in the final report.

Read those together and the picture is uncomfortable for the whole tooling category. Most of what decides your visibility is incumbency inside a source pool you do not own, and it barely moves week to week.

A tool tells you where you stand in the source pool. It cannot put you in it.

That second half is earned: comparison placements, review-site position, documentation the models can actually parse, community presence. It is why a managed GEO agency sits beside tooling rather than inside it, and why the weekly dashboard argument matters less than most teams think. We covered the drift itself in why your AI visibility changes weekly.

FAQ

What is an LLM visibility tool?

An LLM visibility tool runs a set of buyer prompts through AI engines like ChatGPT, Gemini, Perplexity and Google AI Mode on a schedule, then reports whether each answer named your brand, linked your site, or recommended a competitor instead. It turns "are we showing up in AI search" into a tracked percentage. What it does not do is measure a fact that exists independently of the tool: the answer it scored was generated at request time and cannot be re-checked, so the number is a property of the instrument's settings as much as of your brand.

How accurate is an AI visibility tracker?

The question has no clean answer, because there is no reference measurement to be accurate against. A rank tracker can be checked against a Google result page that exists; an AI visibility tracker scores an answer that was generated once and is gone. The useful property is repeatability rather than accuracy. Judge a tracker on whether it returns the same number when nothing has changed, which you establish by running a 14-day no-change period and watching the noise floor.

What is LLM brand visibility?

LLM brand visibility is how often, and in what terms, large language models name your brand when someone asks a question in your category. It splits into three events worth measuring separately: being mentioned in the prose, being cited as a linked source, and being recommended as the pick. Those carry different commercial value and different tools blend them differently, which is the single most common reason two dashboards disagree about the same company.

How much do LLM monitoring tools cost?

Published list prices in this category run from roughly $29 a month at the entry tier to about $999 a month for the top self-serve plans, with enterprise products quoting privately and starting far higher. Price alone compares poorly across vendors, because tiers meter different things: prompts, engines, runs per prompt, or credits that mix all three. Normalise to cost per thousand AI answers before comparing, and get the monthly answer volume at your tier in writing.

Can I do LLM visibility tracking without a tool?

Yes, for a baseline. Pick 20 buyer prompts, run each one three times in the consumer interfaces on the same morning, and record whether you were named, linked, or recommended. A spreadsheet gets you a defensible starting point and teaches you your own category's volatility. What it will not do is scale to weekly cadence across several engines, which is the actual thing the products sell you.

The bottom line

The two dashboards our prospect sent us were 27 points apart. When we put both vendors' settings side by side, the gap came down to three things: one weighted Google AI Mode heavily and the other did not cover it, one counted unlinked mentions and the other did not, and one ran each prompt once a day while the other averaged four runs.

None of that was in either interface. All of it was answerable by email in a day.

Pick the instrument whose settings match how your buyers actually search, write those settings down, freeze them, and compare that tool only to itself. Then spend the argument time on the source pool instead, because that is the half the number was pointing at all along.