Method
How I test
Exactly how I test, and the run logs behind every scan number on this site.
Updated
The questions
Four per agency, written the way a home and auto shopper in your city would ask, and fixed when you sign up so the test stays the same every month. For example:
- “recommend a local insurance agency in [your city]”
- “best home and auto insurance agent in [your city]”
- “I just bought a house in [your city], who can help me with home and auto insurance?”
- “should I use a local agent or buy home and auto insurance online in [your city]?”
The tools
I run your questions through ChatGPT, Google's AI and Claude. Here is exactly what runs:
- ChatGPT: OpenAI's Codex CLI (codex exec) running gpt-6.1-sol, with live web search on. The city is written into each question; no location is set.
- Google's AI, two parts, 5 runs each per question: Gemini (gemini-3.8-flash) through Google's Vertex AI, grounded with Google Search and Google Maps, with the city set as the searcher's location; and Google's AI Mode, the AI answer on Google Search, fetched through HasData's AI Mode API with the city as the search location. The two add up to one Google AI number; the raw data keeps both.
- Claude, on its own line: claude-sonnet-5-5 through Anthropic's Messages API, with Anthropic's web search tool (web_search_20260318) and the city set as the searcher's approximate location.
Every run is a fresh session: the question alone, no history. These are developer tools, not the apps people use on their phones, and AI Mode is fetched by a service, not typed by a person. The apps can answer differently, because of location, account history and settings, so the apps get a separate check, by hand, reported on its own. I report what these tools said, name the tools, and don't claim to show what an app told one person.
Every run records whether the tool actually searched the web and which pages it cited.
The runs
80 per test: 4 questions, 10 runs each on ChatGPT and on Google's AI, spread over at least 3 days. Claude gets the same 4 questions 10 times each, reported on its own line and not counted in the 80. Every run is saved: the question, the time, the tool and model, the location set, the full answer and the websites it cited.
What counts as named
Your agency's name appears anywhere in the answer. A link to your site without your name doesn't count.
Why I count, instead of ranking
Ask the same question twice and you can get a different list. In October, when I asked Google's Antigravity CLI (CLI data, below) for a local insurance agency in Grand Rapids 20 times, it named 22 different agencies (counting different spellings of one agency once). So one answer tells you very little. How often your name shows up across many runs is steadier than where it lands in any one answer.
How sure the number is
80 runs is a sample, so every count has a margin. If 20 of 80 runs name you, the true rate is somewhere around 17% to 35%. If 0 of 80 name you, it could still be as high as 4.6%. (Those are 95% Wilson intervals.)
The test behind this site's numbers
The numbers on this site so far are CLI data: October 2026 scans through two command-line developer tools, OpenAI's Codex CLI (codex exec) running gpt-6.1-sol, with live web search on, and Google's Antigravity CLI (agy) running gemini-3.8-flash-high, with its search_web tool. Neither had a location set; the city was in the question. CLI data isn't comparable to the Google AI line above.
The numbers on the home page come from a scan I ran on October 11, 2026: one question, “recommend a local insurance agency in [city]”, asked 20 times in each of two cities through each tool. Agency names are hidden because they're real businesses that didn't ask to be on a marketing site.
Question asked: “recommend a local insurance agency in Grand Rapids”, and the same for Phoenix. 20 times per city through each tool, October 11, 2026. CLI data. Agency names hidden. Download the data (CSV)
| City | Tool | Named most | Named next | Different agencies named |
|---|---|---|---|---|
| Grand Rapids, MI | OpenAI Codex CLI | Agency A: 19 of 20 runs | Agency B: 16 of 20 runs | 3 in 20 runs |
| Grand Rapids, MI | Google Antigravity CLI | Agency A: 18 of 20 runs | Agency B: 12 of 20 runs | 22 in 20 runs |
| Phoenix, AZ | OpenAI Codex CLI | Agency C: 15 of 20 runs | Agency D: 4 of 20 runs | 3 in 20 runs |
| Phoenix, AZ | Google Antigravity CLI | Agency D: 17 of 20 runs | Agency E: 14 of 20 runs | 17 in 20 runs |
In those scans, the Codex CLI searched the web in all 72 of its runs. The Antigravity CLI searched in all 40 runs of the 20-run test, but in only 15 of its 32 shopping-question runs.
The same day I asked 4 shopping questions once each in 8 metros through both tools (64 answers). On “cheapest car insurance in [city]”, a local independent agency came up in 1 of 16 answers. On “I just bought a house in [city], who can help me with home and auto insurance?”, one came up in 15 of 16.
The logs
Ask me and I'll send you the prompts, times and full answers behind any number on this site: jer@plumbton.com. I keep them for as long as the number is on the site, plus 4 years.
How this page was made
Written with help from an AI writing tool. The numbers are generated from the scan logs by a script, not typed in, and the build stops if the logs and the page disagree.
Changes
- : first version.
- : Claude added on its own line; Google's AI (Gemini through Vertex AI with the city set, plus AI Mode) replaces the Antigravity CLI; the October numbers labeled CLI data. All files: Data.