Deviations log — UK AI Visibility Report for Solicitors, July 2026
Report TAI-R-2026-001
What this document is
This report is based on a single collection run: 68 prompts across 17 UK cities, put to ChatGPT and Perplexity, producing 1,360 answer runs and 12,279 recorded citations, collected on 19 July 2026.
The prompt panel, the city selection rule and the classification approach were all fixed before any collection began. This document records every departure from that plan, including corrections to our own errors. Each entry gives the date, what went wrong, what we did about it, and whether it affected the published figures.
We publish this because a study you cannot check is not worth citing. Four things went wrong during collection and analysis. All four were found and corrected before any results were reported.
18/07/2026 — API quota outage mid-collection
What happened. The collection run hit an API quota limit partway through. ChatGPT calls started returning rate-limit errors, leaving some prompts with fewer than the target number of clean runs.
What we did. Paused the run. After the quota reset, resumed it. The collection script counts existing clean runs per prompt and platform and tops up to the target, so no duplicates were created and every prompt reached the full target.
Effect on published figures. None. All runs were collected before any figures were calculated.
18/07/2026 — Configuration file lost during a server deployment
What happened. A routine server deployment replaced the running container partway through collection, wiping the generated configuration file, which had not yet been committed to version control. The collection process was killed.
What we did. Regenerated the configuration from the committed source data. Generation is deterministic given the same inputs, so the regenerated file was identical to the original. Resumed collection. Committed the configuration to version control so it cannot be lost this way again.
Effect on published figures. None. No collected data was lost; incomplete prompts were topped up.
18–19/07/2026 — Firm-name matching returned false negatives
What happened. The configuration generator left the firm-name field empty for every tracked firm. The matching function that checks whether a firm is named in an AI response therefore returned “not mentioned” for every firm, regardless of what the response actually said. 874 records were wrongly flagged.
How we found it. Manual inspection of stored responses. Several responses clearly named tracked firms while every corresponding flag read “not mentioned”.
What we did. Fixed the configuration generator to populate firm names from the source register data. Built a new name-matching library with normalised matching — stripping common suffixes, treating “&” and “and” as equivalent, requiring multi-word names to appear as contiguous phrases, and requiring single-word names to appear in a list context to prevent false positives. It carries 24 unit tests built on real stored responses. Rebuilt every flag from the stored response text.
Effect on published figures. None. The correction was applied before any figures were calculated.
19/07/2026 — Citation URL pattern mismatch
What happened. The configuration recorded our own profile page URLs using an old path pattern that no longer matched the live site. Citation matching compared AI-returned URLs against those wrong URLs, so nothing ever matched. The result was an apparent zero citation count, which was an artefact of the mismatch rather than a real measurement.
How we found it. An audit script showed Perplexity citing one of our profile pages nine times while the corresponding record showed no citations at all.
What we did. Corrected the URL pattern against the live routing configuration. Wrote a script to re-match every stored citation URL against the corrected targets. Two problems surfaced while doing so:
- The first version loaded all 1,360 runs into memory at once and crashed, leaving partially written flags. Rewritten to stream records and write in batches.
- The recomputation scripts overwrote each other’s results, because each rebuilt the full record while calculating only its own field. Both were rewritten to update only the field they own.
All flags were then rebuilt from the stored raw responses and citation URLs. Raw responses were never affected at any point.
We also added a check that spot-tests a sample of target URLs and fails if any return a 404, so a wrong URL pattern cannot ship silently again.
Effect on published figures. None. All corrections were applied before any figures were calculated.
Note on the classification method
The domain classification used in this report is not the first version we wrote. An earlier version used a residual rule — anything not matched to another category was treated as a firm website — which placed several small directories in the firm bucket and several media and reference sites in the directory bucket. That version produced materially different figures and was discarded before publication in favour of a five-bucket classification with suffix-safe domain matching and a manual review of the most-cited domains.
The classification list used to produce the published figures is available alongside this report.
Published by TendorAI, 02/08/2026. Reproduction permitted with attribution.