Deviations log — Most UK Solicitors Are Never Recommended by AI, August 2026
Report TAI-R-2026-002
What this document is
This report is based on two collection waves against the same fixed panel: 68 prompts across 17 UK cities, ten repeats per prompt. Wave 1 opened 18 July 2026 and covered Perplexity and ChatGPT. Wave 2 completed 24 August 2026, Perplexity only.
The prompt panel, the city selection rule and the treatment and control assignments were all fixed before any collection began. This document records every departure from that plan, including corrections to our own errors. Each entry gives the date, what went wrong, what we did about it, and whether it affected the published figures.
We publish this because a study you cannot check is not worth citing.
19/08/2026 — Wave 2 first launch abandoned
Wave 2 was launched on 19 August 2026 and terminated after 240 of an expected 1,360 runs, Perplexity only. No runs errored; the process was stopped by a database storage limit. The 240 completed runs were discarded rather than topped up: resuming would have produced a wave collected across two windows six days apart, spanning possible model and index changes. Wave 1 was verified intact at 1,360 runs before deletion.
24/08/2026 — Wave 2 relaunched clean
Wave 2 was relaunched on 24 August 2026 with the same prompt panel, platforms and configuration, and completed 680 Perplexity runs.
25/08/2026 — ChatGPT arm could not be collected
Every ChatGPT attempt in wave 2 failed: 2,040 attempts across 68 prompts. OpenAI had retired gpt-4o-mini-search-preview, the model wave 1’s ChatGPT arm was collected on. The wave 1 ChatGPT measurement cannot be reproduced on the same instrument. We did not substitute a different model, because a replacement model is a different instrument and the two figures would not be comparable. The ChatGPT series ends at wave 1; any future ChatGPT collection begins a new series.
24/08/2026 — Instrument verified unchanged
The classifier, collection script and prompt configuration were verified byte-identical between waves by MD5 hash, with no commits touching those files between the two collection dates.
Wave 2 findings are therefore Perplexity-only. This is stated in the report body and in its limitations section.