Every Agency Says They Use AI. Here's How to Tell Who Actually Does.

Brands evaluating partners tell us the same thing lately: every pitch deck has an AI slide. Every agency is AI-powered, AI-first, or AI-native, and the slides look nearly identical, which is itself a clue.

Some of those claims are real. Plenty of them describe the same work the agency was doing two years ago with a new label on it. From the outside, the two are hard to tell apart.

They don't have to be. Five questions will separate them quickly, and none of them require you to be technical. First, a short history lesson on why this keeps happening.

We Have Seen This Movie Before

On December 21, 2017, Long Island Iced Tea Corp. announced it was pivoting to blockchain and renamed itself Long Blockchain. The company made iced tea. Its stock jumped as much as 289% anyway. The plan to buy bitcoin mining equipment never materialized, and the following year Nasdaq delisted the company, saying it believed Long Blockchain had made public statements designed to mislead investors and cash in on interest in the technology.

AI has its own version of this. In March 2024, the SEC brought its first enforcement actions for what it calls "AI washing," settling charges against two investment advisers that told clients they were using AI in ways they were not.

Agencies aren't regulated like investment advisers, and nobody is going to fine a digital agency for an overconfident slide. That is exactly why the burden falls on the buyer.

We've written about AI visibility vendors selling tactics that can wreck a brand's search presence. The delivery side of the market has a quieter version of the same problem. And if your concern is whether AI-written code is safe to ship, we covered that separately, including the questions to ask about review and QA. This piece is about the question that comes before it: is the AI claim real at all?

1. Can You Walk Me Through a Ticket Your Agents Shipped?

Real process leaves evidence. Ask to see a recent piece of work from start to finish: the ticket, the pull request an agent opened, the review comments, what the human reviewer changed, and what QA caught. Redacted is fine. An agency doing this at scale produces these artifacts every day. An agency with a slide will offer a demo or a case study instead.

Listen for the level of detail. People who do this work describe the unglamorous parts without being asked: the agent that misread a requirement, the reviewer who sent it back, how long the round trip took. Marketing descriptions skip straight to the result.

2. Where Does the Efficiency Gain Go?

This is the most revealing question on the list, and the one buyers ask least.

If an agency really is producing more with AI, the gain has to land somewhere. It can go to you, as more hours back to your team or more output for the same budget. Or it can stay with the agency as margin, while you keep paying for hours that took less time to produce.

There's nothing illegal about the second option. It just isn't a partnership. Ask directly how the agency's pricing reflects the efficiency it's claiming. If nothing changed in what you pay or what you get, the AI is working for them, not for you.

Our position is that AI should be used to do more strategic work that advances our clients' business objectives. We've written about what clients can do with the hours they get back when the gain goes where it belongs.

3. What Changed in Your Numbers?

If AI changed how an agency delivers, something measurable moved. Throughput, cycle time, tickets per sprint, time from ticket to production. Ask for the number, and then ask what it's measured against.

A number without a baseline is a slogan. We measure our agentic development model against human-only delivery: a 238% efficiency improvement, with an 82% agent success rate on assigned tasks. You're free to push on how we measure it. The point is that there's a measurement to push on.

Be as suspicious of a perfect number as a missing one. Agents fail at some tasks, and that's expected. An honest success rate shows the failures, because the review and QA process exists to catch them.

4. How Long Have You Been Doing This, and What Broke?

Ask how many client engagements have run through the model, and what the agency changed after the first few.

Every team that has done this seriously has a list of things that went wrong early and the guardrail each one produced. That list is where a real process comes from. If an agency's story is all wins, either it hasn't done this long enough to hit the problems or it isn't telling you about them. Neither is who you want working in your codebase.

For reference, our model has been battle-tested across more than 15 client engagements, and it looks meaningfully different today than it did at the start. That's a feature, not an admission.

5. Who Is Actually on the Team?

Real agentic delivery changes who does the work. The classic agency pyramid, a few senior people over a wide base of junior implementers, exists because implementation is labor-intensive. When agents handle defined implementation work, the model shifts toward senior engineers who scope the work, review every change, and own the output.

Ask to see the proposed staffing plan. If it looks like the pyramid from three years ago with an AI line item added, the AI is probably a tool a few people use rather than a change in how the work gets done.

The Fair Objection

A reasonable buyer might ask why any of this matters. If the work gets done well, who cares how?

It's a fair point, and most brands don't audit their agency's tools any more than they audit which code editor the developers prefer. But this isn't really a tooling question. How the work gets done determines three things you do care about: how much of the work is strategic, how fast things ship, and who is accountable when something breaks. An agency that has changed its delivery model will look different on all three. One that has only changed its slides will look the same on all three, at the same price.

There's a second fair point worth making. Plenty of agencies use AI assistants day to day and describe that honestly. That's real and useful, and nothing on this list will trip them up. The questions are built to find the gap between how an agency works and how it describes the way it works.

Ask Us Too

Every question here is one we'd expect a smart buyer to ask us, and we'd rather you did. A real process should hold up under scrutiny. That's most of the reason to have one.

Long Island Iced Tea changed its name and nothing else. When an agency tells you it's AI-powered, your job is to find out whether anything else changed. Five questions will usually tell you.