Pilot projects: proving fit before the retainer

Here's an uncomfortable truth about agency selection: the pitch tells you almost nothing. The pitch is the agency's best people performing their best work in a room where nobody has to integrate anything with your billing system. It's theatre. Good theatre, sometimes — but theatre.

A pilot is the opposite of theatre. A pilot is two to four weeks of real work on a real slice of your problem, with real constraints and real standups and at least one awkward moment where a date slips and you get to watch how the agency handles it. It's the closest thing procurement has to a test drive on an actual road.

We've been on both sides of this for a decade — running pilots, losing pilots, occasionally talking a client out of a pilot because the shape was wrong. This is the playbook we wish more buyers had before they design one.

What a pilot can prove (and what it can't)

A well-designed pilot answers four questions:

Can this team do the work? Not "does their portfolio suggest they once did adjacent work" — actually do yours, in your stack, against your constraints.
Do the senior people show up? The partner who dazzled in the pitch — is she in the Tuesday working session, or has she been replaced by a very polite junior holding her slide deck?
What does a week feel like? Cadence, communication quality, how estimates meet reality, whether the weekly demo is a ritual or a performance.
Do the teams fit? Not vibes — fit. Do disagreements resolve? Does feedback land? Does anyone on their side say "we think you're wrong about this" out loud?

A pilot cannot prove chemistry at scale, strategic depth over twelve months, or how an agency behaves in month seven when the honeymoon is over. Pilots are short. People can behave well for three weeks. What you can observe is the mechanics — the craft, the honesty under small pressure — and mechanics compound into months.

So don't ask a pilot to prove the retainer. Ask it to prove the week.

Designing a pilot that isn't theatre

The failure mode of most pilots is that buyers design them like a test, with hidden answers. Good pilots are designed like the first chapter of the real engagement. The ingredients:

A real problem, thinly sliced. Not a hypothetical ("design us a concept for an AI assistant") and not the full problem ("rebuild our checkout"). A thin, real slice: one onboarding flow redesigned and prototyped; one service area of the site rebuilt against a performance budget; one discovery sprint that ends with a scoped build. Success must be judgeable in the time available.

Access, granted in advance. The pilot dies waiting for staging credentials. Before kickoff, agree: analytics access, one interviewable customer or two, a named stakeholder who will actually attend sessions, and an engineer who can answer questions within a day. If you can't grant a pilot team access, you can't grant a real team access either — the pilot just finds out cheaper.

A decision-grade artefact. The pilot should end with something you'd use even if you never speak to the agency again: a tested prototype with findings, a scoped and estimated backlog, an audited codebase with a written report. If the deliverable is a "recommendation deck," you've paid four weeks for a longer pitch.

Both sides exposed. The client works too. You attend standups or you don't; you give feedback in 24 hours or in a week; your stakeholders contradict each other or they don't. The agency is evaluating you. Good ones want to know whether the decision-making setup is survivable before they commit a team to you.

Pricing a pilot honestly

Two rules.

First: pilots are paid, always. An unpaid pilot selects for agencies desperate enough to work free, and instructs both parties to treat it as sales activity rather than work. You want the pilot priced as work — near the agency's normal rates, with a defined team and hours.

Second: pilots are priced without margin games. The fair price for a three-week pilot is roughly the pro-rata cost of the named team, perhaps with a modest reduction because pilots help the agency win the engagement. What you should not see is a pilot priced far below cost (someone will recoup it later, in the engagement's estimate — congratulations, you're paying for the pilot twice) or padded with "strategy days" that don't map to named people's hours.

Ask the same questions you'd ask of any quote — we wrote a parallel guide on how to read an agency SOW, and most of it applies verbatim to a pilot letter. If the pilot price can't be decomposed into people times days times rate, don't sign it.

For calibration: a two-designer, one-engineer, three-week product pilot from a senior independent studio typically lands in the AU$35–60k range. Cheap enough to be a rounding error against a failed engagement; expensive enough that everyone behaves like it's real.

Name the exit criteria before you start

The most valuable artefact in a pilot isn't the prototype. It's the one-page exit-criteria memo, drafted in week zero, that both sides sign. It states:

What we'll assess — the four or five observable things ("prototype tested with five users; quality of weekly demos; estimate realism on the scoped build").
Who decides — one named human on the client side, not a committee that emerges in week four.
What "yes" looks like — the engagement you'd offer if the pilot passes, described concretely enough that "yes" means "start contracting," not "form a working group to explore next steps."
What "no" looks like — a clean exit: deliverables handed over, IP assigned to the client (yes, even for pilot work — get this in writing), one honest retrospective call, no lingering obligations.

Without this memo, pilots end in mush: the work was fine, nobody's sure what happens now, someone schedules a follow-up, and three months later both parties quietly pretend it didn't happen. Exit criteria are how a pilot becomes a decision instead of a delay.

Converting a pilot into an engagement

If the pilot passes, the temptation is to re-open everything: new negotiation, new scope, new rounds of procurement. Resist it. The whole point of the pilot was to make the engagement a continuation, not a fresh purchase. Three moves make that work:

**Negotiate the engagement's shape before the pilot ends.** The exit memo's "what yes looks like" does this. By the final demo, commercial terms for the follow-on should be drafted, not imagined.

Bank the context. The pilot team's knowledge is the asset you just paid for. The conversion conversation should name who continues, at what allocation. If the follow-on engagement proposes an entirely different team, ask why — sometimes there's a good reason, often there isn't.

Keep the pilot's honesty. Whatever the pilot surfaced — a wobbly stakeholder setup, a codebase with secrets, a decision that takes three meetings — goes into the engagement plan as a named risk with an owner. The pilot's findings are free discovery. Binning them is paying twice for the same lesson.

And if the pilot fails? Say so, cleanly, within a week. A fast, honest "no" is a gift to an agency — it lets them staff the team elsewhere and tells them exactly what to fix. We keep more goodwill from pilots we lost with grace than from some we won.

When you don't need a pilot

Pilots aren't universal. Skip one when the engagement itself is already small and bounded (the pilot is the project, doubled), when you've worked with the team before (you already have the data a pilot would generate), or when the timeline genuinely can't absorb it — in which case the honest move is a fixed-scope first project with an explicit renewal decision, not a compressed pretend pilot.

Also skip the pilot if what you actually need is a better brief. A pilot can't rescue a problem you haven't articulated; it just makes the confusion billable.

Key takeaways

The pitch is theatre; the pilot is the road test. Buy accordingly.
A pilot proves craft, cadence, staffing honesty and working fit. It does not prove month seven.
Design pilots as the first chapter of the real work: thin slice, granted access, decision-grade artefact, exposure on both sides.
Pay for pilots at honest rates, decomposed into people-times-days. Cheap pilots are recouped somewhere you're not looking.
Write the exit criteria memo in week zero: what's assessed, who decides, what yes and no each look like.
Convert by continuing, not renegotiating — bank the team, the terms and the findings.

FAQ

How long should a pilot be? Two to four weeks. Shorter than two and you only see the performance, not the work; longer than four and you're running an unpriced engagement with extra steps. Three weeks fits a design slice or a technical spike; four if the deliverable includes user research.

Should we pilot two agencies head-to-head? Generally no. Split attention halves the value on both sides, access becomes a fairness nightmare, and you'll struggle to separate team quality from starting conditions. Run them sequentially if you must run two — and tell both agencies; hiding a bake-off corrodes the trust you're supposed to be testing.

Who owns the IP from a pilot? You should, full stop, with assignment effective on payment. Any agency that wants to retain pilot IP is either planning to resell it or hasn't read their own contract. This is a table-stakes clause — our proposals include it by default.

What if the pilot goes well but the follow-on price is a shock? That means commercial terms weren't drafted before the pilot ended — the most common pilot failure, and an easy one to avoid. Ask for the engagement's shape and rate card in week one of the pilot, in writing.

Can a pilot be remote? Yes — most of ours are, and a remote pilot is arguably more informative, because it tests the working conditions you'll actually live in. What matters is synchronous overlap: if the agency can't make two or three hours of shared time work daily, that is the finding.

Ready to run yours properly? Tell us about the problem — we'll tell you honestly whether it's pilot-shaped or not.