The AEO Score Is a Vanity Metric Until You Ask the Engine Directly
If you run social for a brand — or you’re an indie founder who publishes content across Instagram, TikTok, YouTube, X, LinkedIn, Facebook, Threads, and Pinterest — you’ve probably noticed a new KPI creeping into client decks and investor updates: “AI visibility.” The pitch is seductive. Get cited by ChatGPT, show up in Gemini answers, own your category inside the chatbot. But most of the tools selling you that promise are measuring the wrong thing. They read your HTML, score your schema, and hand you a number that has almost nothing to do with what the engines actually say when a real buyer asks a real question. That’s the gap Oogwai Beacon is trying to close, and it’s worth paying attention to because it reframes the whole conversation — from “can you be cited” to “are you being cited, by whom, and against whom.”
The problem with scoring your own homework
The maker, Reem Saied, tells a story that should make every SEO and social strategist uncomfortable. The team behind Oogw.ai in Bengaluru ran a popular “AEO score” tool against a leading business news site. The site scored poorly. Then they asked ChatGPT and Gemini actual questions about business news, and that same site was cited again and again. The score and the engine behavior “had almost nothing to do with each other.”
That’s not a bug in one tool. It’s a structural flaw in the entire category. Most AEO tools only read your HTML — your robots.txt, your schema markup, your headings, your sitemap — and from that infer whether you could be cited. But citation is a retrieval problem, not a markup problem. The engine has to crawl you, index you, trust you, and then choose you over a competitor when a buyer asks a question that doesn’t mention your brand. No amount of clean schema fixes a missing Bing index entry or a citation-source gap.
This matters more for social-first brands than most people realize. If your content lives primarily on Instagram or TikTok, you have almost no HTML for these tools to read in the first place. Your “AI visibility” is being determined by whatever third-party coverage, review sites, and aggregators the engines lean on when someone asks “what’s the best [your category] tool” — and you have zero visibility into that. Beacon at least points a flashlight at it.
What Beacon actually does, and how it differs from the incumbents
Beacon splits its audit into two distinct halves, and the split is the smartest thing about it.
The first half is readability — an instant, no-sign-up check of whether GPTBot, ClaudeBot, and the other AI crawlers can reach and understand your pages. That covers robots.txt, schema, llms.txt, headings, and sitemaps. If you’ve used Ahrefs or Semrush for technical SEO audits, this will feel familiar — it’s the AI-crawler equivalent of a crawl health report. The difference is the audience: instead of optimizing for Googlebot, you’re optimizing for the bots that feed ChatGPT, Claude, and Perplexity.
The second half is citation — and this is where Beacon diverges from every AEO tool I’ve tested. It puts buyer questions about your category to ChatGPT and Gemini, and critically, none of those questions mention your brand. You see how often you’re named, who gets named instead, and which sources the engines lean on. The full audit adds Claude, which the team describes as “by far the toughest grader.”
Compare that to what’s on the market. Profound and Athena have been building in the AI-search-visibility space, and HubSpot’s AEO Grader and Semrush’s AI visibility features are chasing the same buyer. Most of them lean heavily on the readability/technical side because it’s cheap to compute and easy to score. Beacon’s bet is that the citation side — the expensive, live, search-grounded calls to real engines — is the half that actually tells you whether you have a problem.
Why the disagreement between engines is the feature, not the bug
A commenter, Dipanshu Kushwaha, asked the obvious question: how does Beacon handle different answers from ChatGPT and Gemini? The maker’s answer is the most operationally useful thing in the whole thread, and it’s worth quoting the mechanics directly.
Beacon doesn’t try to reconcile the engines. Each runs independently — ChatGPT through the Responses API on Bing-backed search, Gemini through generateContent with Google Search grounding, Claude on Brave search. Each gets the same brand-free buyer questions, and each answer is scored separately for whether the brand is named, where it ranks, which competitors show up, and how it’s framed. Nothing is cross-checked or averaged at the answer level. The spread is shown, not smoothed. Per-engine counts sit next to the combined pillar scores, and the composite — Brand Visibility, Share of Voice, Average Position, Sentiment — pools results across engines by weighted average into a headline score.
Why this matters: “Gemini names you, ChatGPT doesn’t” is usually a Bing indexing or citation-source gap. That’s a concrete, fixable thing. If you’d averaged the two engines into one number, you’d have hidden the exact diagnostic signal you need. In my experience running content ops, this is the difference between a dashboard and a to-do list.
What social media operators and creators can actually borrow from this
Here’s the uncomfortable truth for anyone whose primary distribution is social: AI engines don’t cite your Instagram captions. They cite the web. When ChatGPT answers “what’s the best CRM for freelancers,” it’s pulling from review sites, comparison posts, Reddit threads, and published articles — not your Reels. So if your brand’s only web presence is a thin landing page and a Linktree, you are structurally invisible to the citation layer, no matter how good your content is.
I’d bet most social-first brands have never once asked an engine what it says about their category. That’s a free experiment you can run this week, and Beacon’s free readability check is a reasonable starting point. But the more interesting borrowed insight is the methodology: ask brand-free buyer questions, run them across multiple engines, and treat divergence as signal rather than noise.
Why TikTok creators should care more than LinkedIn ones
This sounds counterintuitive, so let me explain. LinkedIn-native B2B creators already publish long-form text on a platform that Google indexes and that AI crawlers can reach. Their content has a fighting chance of being retrieved and cited. TikTok creators, by contrast, publish into a walled garden. The video is discoverable inside TikTok’s algorithm but essentially invisible to ChatGPT and Gemini. If a TikTok creator wants AI visibility, they have to build a parallel web presence — a blog, a newsletter, a YouTube description with real text — that the engines can actually read. The creators who figure this out first will own their category’s AI answers before anyone else bothers.
Where the math breaks
I want to be careful here, because the maker is transparent that every citation report makes “live, search-grounded calls to real AI engines, so each run costs us real money.” That’s an honest constraint, and it explains the work-email requirement — the report is about a company’s position in its category, so it goes to a company address. The readability check stays free.
But it also means the citation side is inherently a snapshot, not a continuous monitor. AI answers drift. A citation report from last month may not reflect this month’s index. If you’re used to Buffer or Metricool giving you always-on analytics, this is a different rhythm — periodic audits rather than live dashboards. That’s not a knock on Beacon specifically; it’s a structural reality of how expensive live engine queries are. Just don’t expect a real-time feed.
Where my judgment says it falls short
A few honest limitations, because you should never adopt a tool on a launch thread alone.
Pricing is not disclosed. The source says the readability check is free and the citation report requires a work email, but there’s no public pricing tier visible in the launch material. For an indie founder or a small social team, that’s a real unknown. Ask before you commit.
The engine set is partial. Beacon covers ChatGPT, Gemini, and (in the full audit) Claude. It does not, per the source, cover Perplexity or Copilot — both of which matter increasingly for buyer research. That’s a gap worth flagging.
The crawler-log question is unanswered. A commenter from UTTER IN, Ali Almoosawi, raised the sharpest technical point in the thread: when they checked server logs on their own WordPress sites, GPTBot, ClaudeBot, and PerplexityBot were fetching pages daily — often hitting 404s nobody knew about — and no audit score showed it. He asked whether Beacon would let a site connect its own crawler logs, so a missing citation could be traced to a crawl problem versus a content problem. The maker did not answer that question in the source. That’s an open item, and it’s the single feature that would make Beacon genuinely differentiated from every HTML-reading competitor.
Who it’s NOT for: solo creators with no web presence beyond social, anyone who needs real-time monitoring rather than periodic audits, and teams that need Perplexity or Copilot coverage today. If you’re a pure TikTok or Instagram operator with no blog and no press coverage, Beacon will mostly tell you what you already suspect — that you’re invisible to engines — without giving you the content infrastructure to fix it.
What I’d watch / test next
Three concrete moves for this week.
First, run the free readability check on your own domain and see whether GPTBot and ClaudeBot can actually reach your key pages. It costs nothing and takes minutes.
Second, do the manual version of Beacon’s citation test yourself: open ChatGPT and Gemini, ask five brand-free buyer questions in your category, and write down who gets named instead of you. That list of competitors and cited sources is your content roadmap for the next quarter.
Third — and this is the one most social teams skip — start building the web layer your social content doesn’t have. A blog post, a comparison page, a newsletter archive. Something the engines can retrieve. Because the citation game is won on the open web, not inside the feed.
I’ll be watching whether Beacon answers the crawler-log question, adds Perplexity, and publishes pricing. Those three things will determine whether it becomes a real category contender or stays a clever audit tool. Either way, the reframe — ask the engine, don’t score the HTML — is the lesson worth stealing regardless of which tool you end up using.
