The creator economy’s next analytics gap isn’t reach — it’s AI fluency
Every social media operator I know is now shipping more content with AI in the loop than without it. The problem is that nobody — not the platform dashboards, not the scheduling tools, not the agency reporting decks — can actually tell you how the AI is being used, only that it was. Reach, watch time, engagement rate: all downstream. The upstream question — did the person prompting actually know what they were doing — is invisible. That’s why the launch of Pheebs caught my attention even though it’s built for engineering teams, not creators. The measurement model it proposes is the one social teams will need to copy within eighteen months, because the same FOMO, the same vulnerability, and the same “adoption metrics lie” problem is already live in every content org I’ve worked with.
What Pheebs actually measures — and why the framing matters more than the tool
The maker, Davi Prates, opens the launch with an admission that most SaaS founders won’t make: “being vulnerable about your AI skills is not easy.” That line is doing more work than any feature list. The core argument in the launch post is that DAU/WAU and token counts tell you who has access and who shows up, but almost nothing about what happens inside a session. Which models got picked? Were they the right fit? Did anyone verify the output before committing? Did the team spend consciously, or just default to the biggest model for everything?
Pheebs answers that by recording the shape of AI sessions and surfacing two composite dimensions through what the team calls an AI Proficiency Model:
- Repertoire — which harness capabilities show up across six competencies: models, artifacts, MCP, evals, context management, and orchestration.
- Judgement — what happens to AI output before a PR is created and merged. Verification, challenge, refinement, and whether the model chosen matched the task. The maker explicitly credits the Discernment competency from Anthropic’s AI Fluency Index as inspiration.
Engineers get a private view of their own work, separate from the team report. The product is open source, and pricing is not disclosed in the launch material.
My take: the interesting move here isn’t the dashboard. It’s the refusal to reduce “AI proficiency” to a single number. Most tools in this space — and most creator-side AI analytics I’ve tested — collapse everything into a usage score, which is exactly the metric that gets gamed the fastest.
Why the two-axis model is the part worth stealing
Repertoire without judgement is a power user who ships garbage fast. Judgement without repertoire is a careful operator who never gets leverage. Most teams I’ve audited sit in one of those two failure modes and don’t know which. The split is what makes the model diagnostic rather than decorative.
The same measurement hole exists on the social side — and it’s wider
Here’s where I stop reporting and start arguing. If you run social for a brand, an agency, or your own creator business, you have the exact problem Pheebs describes, just with different nouns. Your Buffer or Later dashboard tells you what published and when. Your Metricool or native Instagram Insights tells you what performed. Almost nothing in your stack tells you whether the AI that drafted the caption, cut the CapCut variant, or wrote the YouTube hook was used well — or whether your team is quietly running the equivalent of “largest model for everything” because nobody set a standard.
When I scheduled 30 posts across 5 platforms last month for a client, the post-mortem I actually wanted wasn’t “which format won.” It was: which drafts got meaningfully rewritten before publishing, which got shipped raw, and which prompts produced hooks that survived editing. That’s the creator-side version of Judgement. Nobody sells it. Not Hootsuite, not Sprout Social, not SocialPilot. The closest thing is manual: a shared doc where editors mark “kept / light edit / heavy rewrite” on every AI-assisted draft before it goes into the scheduler.
Why TikTok creators should care more than LinkedIn ones
TikTok’s distribution rewards the first three seconds and punishes anything that reads as templated. If your team is running AI on autopilot, the algorithm will tell you — but only after you’ve burned the post. LinkedIn is more forgiving of generic phrasing, so the feedback loop is slower and the bad habits compound longer before anyone notices. That asymmetry is why I’d push a TikTok-first team to instrument AI usage before a LinkedIn-first team bothers. The cost of not knowing is higher on the platform that punishes sameness hardest.
What social teams can borrow from this launch this week
You don’t need Pheebs to steal its logic. Three concrete adaptations I’d run in my own workflow:
1. Build a lightweight Judgement log. For every AI-assisted post, tag it in your content calendar with one of four states: shipped raw, light edit, heavy rewrite, or rejected. After 30 days, look at the distribution. If “shipped raw” is above roughly a third, you have a calibration problem, not a speed problem. I’d bet the correlation between “shipped raw” and underperformance on Threads and X is strong enough to act on before you have perfect data.
2. Track model choice against task. Pheebs’s claim that “most teams run the largest model for everything” maps directly onto creators who use the same frontier model to write a 400-word LinkedIn essay and to generate 12 Pinterest pin titles. The second task doesn’t need it. The team claims Pheebs shows how much could be saved by running smaller models where they fit — I haven’t verified that math myself, but the principle is sound and the cost delta on a creator’s API bill is real.
3. Separate the private view from the team view. Pheebs gives engineers a private dashboard distinct from the team report. That’s a trust design choice, not a feature. If you’re rolling out AI-usage tracking to a content team, the fastest way to kill adoption is to make it feel like surveillance. Give writers and editors their own view first; aggregate later and only with their knowledge.
Where the math breaks
A caution I’d flag to anyone copying this: proficiency metrics are gameable the moment they become a performance review input. Pheebs’s maker addresses this in the comments, noting that the goal is “not to encourage teams to pushback on all outputs, but to encourage a healthy rate.” That’s the right instinct — but “healthy rate” is undefined in the launch material, and the maker concedes there’s currently no way for a manager to set a baseline. The team median is the stand-in for now, with industry benchmarks described as a future feature. For a creator team of three, a median is meaningless. Don’t pretend otherwise.
Where I think Pheebs falls short — and who it’s not for
Three honest limitations, reading the launch and the comment thread closely:
It reports; it doesn’t enforce. Jay Smith asked directly whether Pheebs creates restrictions or just reports. The maker’s answer: it captures data and provides analytics, and the reports help identify patterns — but it “currently doesn’t help creating restrictions specifically.” For a team that needs guardrails (blocking the expensive model on trivial tasks, flagging unverified commits), that’s a gap. My take: the analytics-first posture is correct for an open-source tool trying to earn trust, but it means the product is a mirror, not a leash.
The recommended-next-steps layer is a paid service, not the product. Sara Ford asked about a summary page with recommended actions. The maker confirmed that recommended steps live inside an “AI Enablement Assessment service” powered by Pheebs, not in the open-source tool itself. So the free tier shows you the pattern; the diagnosis costs money. That’s a legitimate business model, but it means the open-source release is closer to a lead magnet than a complete product.
It’s not built for creators. No social platform integrations, no content calendar, no publishing workflow. If you’re a solo creator or a two-person social team, Pheebs is not your tool — the closest you’ll get is a spreadsheet version of the Judgement log I described above. And if you’re a large enterprise with a compliance team, the open-source, self-hosted posture may be a feature, but the lack of baseline-setting and enforcement means it won’t satisfy an audit.
Pricing is not disclosed. The launch says it’s open source and invites feedback; there’s no published tier structure in the material. Anyone evaluating it for a paid team rollout should assume the commercial layer exists and ask directly.
What I’d watch / test next
Three things I’d do this week if I were running social or content ops:
- Instrument Judgement before you instrument Repertoire. Add a single column to your content calendar — “AI edit depth” — and fill it for every post for two weeks. Don’t change your workflow yet. Just measure. You’ll learn more from that column than from any new analytics subscription.
- Audit your model choices against task weight. List the last 20 AI-assisted assets you shipped. Mark which ones genuinely needed a frontier model and which would have been indistinguishable on a smaller one. If more than half fall in the second bucket, you’re paying the Pheebs tax without the Pheebs dashboard.
- Watch the Pheebs repo and the AI Proficiency Model doc. Even if you never install it, the model is the most useful artifact in this launch. When someone ports it to social — and someone will — you’ll want to have already thought about what “healthy pushback rate” means for a content team, because the first vendor to ship that metric will define the default.
The uncomfortable truth Pheebs names out loud is that adoption metrics are a comfort blanket. We measure them because they’re easy and they go up. The metrics that actually predict whether AI is making your team better are harder, more personal, and — as the maker admits — require a kind of vulnerability most orgs aren’t ready for. That’s true for engineers. It’s just as true for the people writing your captions.
