Compare a complete repeated-run panel
Use one frozen prompt, one AI surface, and comparable conditions per panel. Paste one URL per line and leave a blank line—or a line containing ---—between runs.
What the calculator measures
| Output | Calculation | Question answered |
|---|---|---|
| Per-URL persistence | Runs containing an exact URL ÷ all valid runs | How often did this source appear in the repeated panel? |
| Pairwise Jaccard | Unique URLs shared by two runs ÷ their combined unique URLs | How much did the source membership overlap for this pair? |
| Mean pairwise Jaccard | Average of every run-pair Jaccard value | How much membership overlap did this completed panel observe overall? |
| Exact-order repeat pair | Same URL count, strings, duplicates, and positions | Did the entire displayed sequence repeat exactly? |
| One-run-only URL | Exact URL appears in one valid run | Which sources were observed only once in this panel? |
Jaccard treats each run as a set, so duplicates inside a run count once for membership. Exact-order comparison keeps duplicates and position. The calculator trims surrounding whitespace but does not lowercase, resolve redirects, strip tracking parameters, or merge URLs by domain.
Keep the runs comparable
- Freeze one natural question. Do not mix prompt variants in the same calculation.
- Freeze the surface and conditions. Record engine, mode, location, login state, date, and time separately.
- Paste complete lists. A missing result must mean the source was absent, not that collection stopped early.
- Retain null runs. A valid answer with no citations is an empty run and should remain in the evidence log. This text interface cannot represent an empty block unambiguously, so record that run in the citation response log and label the panel limitation.
- Compare like with like. Build separate panels for different prompts, engines, countries, modes, or cohorts.
Interpret overlap as an observation, not a universal score
A mean Jaccard of 0.40 means the completed run pairs shared 40% of their combined unique URL membership on average. It does not mean a source has a 40% future probability, that an answer was 40% accurate, or that one platform is 40% reliable.
Frequently asked questions
How do you calculate AI citation stability across repeated searches?
Calculate per-URL persistence as the number of valid runs containing that exact URL divided by all valid runs. Calculate source-set stability separately with pairwise Jaccard overlap: shared unique URLs divided by the union for each pair of runs.
What is a good AI citation stability score?
There is no universal good score. Compare the same frozen prompt, surface, and conditions over time, retain every valid run, and interpret persistence and overlap as descriptive observations rather than a platform-wide probability.
Why show exact order as well as Jaccard?
Two runs can contain the same unique URLs and therefore have Jaccard 1.00 while displaying them in different positions. Exact-order comparison catches that difference and also preserves duplicate lines.
Does this calculator send pasted citation URLs anywhere?
No. Calculation and CSV generation happen in your browser. The page does not upload, transmit, or store the pasted run lists.