AI Consulting Firms: A Buyer's Vetting Grid

[!NOTE] TL;DR: Most criteria buyers use to pick AI consulting firms (brand, deck, day rate) predict nothing about production outcomes. This analysis gives you a six-probe vetting grid built on production evidence, the failure numbers behind it (MIT NANDA 2025: only 5% of organizations got custom GenAI tools to production), and dated 2026 rate benchmarks. Interview for systems that ran under real traffic.

Most AI consulting engagements are decided on criteria that predict nothing. Brand, deck quality, and day rate, in roughly that order of irrelevance. My field position, after three years shipping production AI systems for an energy company and sitting on the buyer's side of the table more than once, is blunt: the only selection signal that holds up is production evidence, and you can probe for it in a 45-minute call. The rest of this article is the grid I now use, plus the numbers that justify it.

I once lost a deal to a firm whose demo crashed twice in the pitch. They had the nicer logo. That was the day I stopped trusting selection rituals.

The question behind the grid

Before anything else, the single question I want to answer before signing any AI consulting contract: how do you tell, in advance, whether a firm can carry a system past the pilot and into production?

Rate does not answer it. The US market spans USD 75 to 400 per hour for independents, depending on which 2026 guide you read (Layer3 Labs, updated Sep 2026, and Justin McKelvey, updated Oct 2026), and I have seen excellent work at USD 150 and theatrical work at USD 500. References barely answer it, because firms curate them. The pitch deck answers it least of all; I have yet to see a deck that lists a failed engagement.

Although the question looks commercial, it is actually an engineering question in disguise. What you are buying is not advice. You are buying the transfer of a capability that survives contact with real traffic, real data drift, and real users who click the wrong thing on purpose.

Demo experience against production experience

Here is the distinction that the whole grid rests on, and it deserves a name: demo experience (systems shown under curated conditions) against production experience (systems that ran, degraded, got debugged, and kept running).

A demo is a theater set. The facade is painted, the lighting is rehearsed, and nothing behind it carries load. A production system is a building that inspectors visit unannounced: the wiring is visible, the maintenance log exists, and somebody was called at 3 a.m. at least once. Ask a consultant about the 3 a.m. call and you learn more in two minutes than in two hours of architecture slides.

The technical reason the gap is so wide is compounding error. In AI Engineering (O'Reilly, 2025), Chip Huyen works through a concrete case: if each agent step is 95% accurate, accuracy falls to about 60% over 10 steps. A demo shows the single path that worked. Production is where the other paths live, and only someone who has operated a system knows how often they get taken.

There is an honest concession to make here. Although production experience is the best signal, plenty of strong consultants are early in their careers and have only shipped two systems. You are not screening for pedigree. You are screening for honesty about the boundary between what they have run and what they have only read about, and the grid below tests exactly that.

The vetting grid

I distilled my buyer-side notes into six probes. Each has a passing answer and a walk-away condition.

Two probes do most of the filtering in practice: handover and monitoring. They are also the two that sellers are most reluctant to discuss, which is precisely why they discriminate. I run agents in production at an energy company (volumes anonymized), and I can tell you the build is maybe half the job. The other half is the runbook, the alerting, and the eval suite that tells you when the model quietly started answering worse.

PoC theater

The named failure pattern I want buyers to recognize: PoC theater. A polished proof of concept, presented with maximum conviction, that was never designed to survive its first real user.

MIT Technology Review Insights (January 2026) quotes a chief data officer on the mechanism: "PoCs live inside a safe bubble", with curated data, few integrations, and the most senior team on the demo. The pilots do not fail technically. They are mis-designed from the start, because success in the bubble has no bearing on success in the traffic.

The red flags are learnable, and I would rather you learn them from me than from an invoice:

• No failure story, ever, from anyone on the team.
• A demo on data the vendor brought themselves, never on yours.
• No mention of evaluation, monitoring, or drift in the proposal.
• Handover absent from the statement of work.
• A maintenance line nowhere in the budget, despite industry guides putting ongoing costs at 15 to 25% of build cost per year (JanBask cost guide, June 2026).

One of these alone is noise. Three together form a pattern, and the pattern has a market cost I can quantify in the next section.

The numbers, dated

Start with the funnel everyone misquotes. The preliminary MIT NANDA study (The GenAI Divide, July 2025) found that 60% of organizations evaluated a custom or vendor-sold GenAI tool, 20% reached a pilot, and 5% reached production. The authors call the figures "directionally accurate" from interviews at 52 organizations, so treat them as an early signal, not a census. And the viral "95% of pilots fail" headline is a misreading: the 5% is a share of all organizations, not of pilots. A careful claim-check (ai-first.company, Oct 2026) untangles this, and citing it wrong costs you credibility with any technically literate board member.

{
  "type": "bar",
  "data": {
    "labels": ["Evaluated a custom GenAI tool", "Reached a pilot", "Reached production"],
    "datasets": [{ "label": "Share of organizations (%)", "data": [60, 20, 5], "backgroundColor": ["#3b82f6", "#60a5fa", "#93c5fd"] }]
  },
  "options": {
    "responsive": true,
    "plugins": { "title": { "display": true, "text": "MIT NANDA 2025: custom GenAI tools, evaluation to production" } },
    "scales": { "y": { "title": { "display": true, "text": "Share of organizations (%)" } } }
  }
}

The same study's interview sample found external partnerships reached deployment about 67% of the time, against 33% for internal builds. That is correlation, not proof that buying beats building, but it does suggest the vetting question is worth an hour of your week.

Now the money, because a grid without price context is moralizing. The GSA CALC+ federal rate tool (data read 10 Oct 2026) gives a rare published rate card: across 267 AI/ML labor categories from 96 vendors, the median ceiling rate is USD 183.05 per hour, with the middle half between USD 147.04 and 234.28. Private guides put US independents anywhere from USD 75 to 400 per hour, and boutique firms from USD 125 to 800 (FactoryJet summary of two US guides, read 9 Oct 2026). On the project side, Clutch (updated 21 Sep 2026) averages client-reported AI development projects at about USD 120,594 over roughly 10 months. Budget accordingly, and remember the 15 to 25% annual maintenance line above.

Limits of the grid

This grid is field-tested, not studied. Three limits I own.

First, my production references are self-selected in both directions: firms that volunteered references are the ones confident in them, and my buyer-side sample is one industry. Second, I sell AI consulting myself (my AI consultant engagements are built around exactly this handover-first structure), so read my grid with the corresponding discount applied. Third, the MIT NANDA figures are preliminary, interview-based, and small, and I would not quote the 5% to a board without the caveats attached.

This could easily be tested more rigorously: take 50 signed engagements, score them on the six probes before kickoff, and correlate with production status at month twelve. I have not run that study. Somebody should.

What survives the example

More generally, the principle that survives my three years of notes is this: in a market where anyone can demo anything, you audit the operator's scars, not their slides. Ask for the system that ran, the incident that hurt, and the runbook that exists. A consultant who answers all three without flinching is worth their rate, whatever tier it sits in. One who changes the subject is telling you something too.

FAQ: what buyers ask before signing

How do I evaluate AI consulting firms before signing? Run the six-probe call: production references, a failure story, handover terms, monitoring, evaluation method, and pricing structure. Firms that answer concretely within 45 minutes are shortlistable; firms that answer in generalities are not, regardless of brand.

What is a fair rate for an AI consultant in 2026? US independents bill roughly USD 75 to 400 per hour depending on seniority, and boutique firms USD 125 to 800 (published US guides read Oct 2026). The federal GSA rate card shows a median AI/ML ceiling of USD 183.05 per hour (data 10 Oct 2026), a useful sanity anchor for any quote.

Why do so many AI pilots never reach production? The preliminary MIT NANDA 2025 study found only 5% of organizations had custom GenAI tools in production, mostly for organizational and integration reasons rather than model quality. MIT Technology Review Insights (Jan 2026) attributes the gap to curated demo conditions that production traffic never respects.

Should we hire an external AI consulting firm or build in-house? MIT NANDA's interview sample found external partnerships reached deployment about 67% of the time versus 33% for internal builds, though this is correlation, not proof. The honest decision rule is capability transfer: hire externally only if the contract includes handover, documentation, and team enablement.

What are the red flags in an AI consulting proposal? The five I check first: no failure stories, demos on vendor-supplied data, no evaluation or monitoring plan, handover missing from the statement of work, and no maintenance budget despite the 15 to 25% annual industry norm (JanBask, June 2026).

If you are scoping an engagement and want the handover-first structure applied to your case, the fastest next step is a short scoping conversation, or a look at how I define the AI solutions architect role, boundaries, and rates. For proof-before-promise delivery models, my writeup on asynchronous AI freelance delivery covers the same principle from the seller's side.

Sources consulted: MIT Technology Review Insights, Jan 2026, MIT NANDA claim-check, ai-first.company, Oct 2026, GSA CALC+ rate analysis, aiservicescompanies.com, Oct 2026, US AI consultant cost guide, FactoryJet, Oct 2026, Chip Huyen, AI Engineering, O'Reilly 2025.