How to Run an AI Pilot, Step by Step

Why a pilot, not a purchase

The most common mistake we see with AI projects is buying a server before anyone has proven the task actually works. A pilot flips that order. You test the task first, on real work, with a clear written standard for what counts as good enough, and you only spend money on hardware once you know the answer is worth having. This guide walks through the steps in the order we actually use them with clients, whether the task is reading invoices, answering questions from a document library, or writing meeting minutes.

A pilot is small on purpose. It should take days or a few weeks, not months, and it should use the documents or recordings your staff already produce, not a demo dataset. If a vendor offers to show you AI working on their own sample data, that tells you very little about how it will perform on your scanned contracts, your handwriting, or your mix of Lao, Thai and English. The only test that matters is the one run on your material.

Step 1: pick one task, not a department

Pick a single, specific task with a clear start and end: "read supplier invoices into our spreadsheet", not "automate accounting". "Write minutes from the weekly management meeting", not "improve our meetings". "Answer staff questions about the HR policy document", not "build a knowledge base". A narrow task is easier to test honestly, easier to explain to the people doing it, and easier to judge as a success or failure.

Favour a task that is repetitive, has a known correct answer most of the time, and currently takes a person real time. Avoid starting with your hardest or most unusual document type; prove the approach on something manageable first, then expand. If several tasks look promising, rank them by how much staff time they currently cost and how sensitive the data is, and start with the one that is valuable but not your most confidential material.

Step 2: collect 20 to 50 real samples

Gather between 20 and 50 real examples of the input: actual invoices from the last few months, actual recordings of real meetings, actual questions staff have asked before. Include the awkward ones on purpose: a blurry scan, a handwritten note in the margin, a recording with background noise, a document that mixes Lao and English on the same page. If your test set only contains clean, easy examples, the pilot will look better than the system ever performs in daily use.

Thirty examples is usually enough to see a pattern without taking weeks to prepare. Fewer than ten tells you little beyond "it worked on these ten." If the task covers more than one document type or more than one kind of question, include samples from each type rather than piling up variations of the easiest one. Keep a copy of the originals and a record of where each sample came from, since you will need to check the AI's answer against the real answer for each one.

Step 3: write the pass criteria down, before you see any results

This is the step people skip, and it is the one that prevents the pilot from becoming an argument of opinions later. Before running anything, write down, in a shared document, what a correct answer looks like for this task and what an acceptable error rate is. For an invoice reader, does the supplier name have to match exactly, and is a date off by one day acceptable? For a document assistant, does every answer need a citation to the source page, and what happens when it is not confident? For meeting minutes, who approves the final wording, and how is a missed action item counted?

Decide this with the people who will actually use the result day to day, not only with whoever is running the pilot. A procurement officer and an IT manager will weigh an error differently, and the person doing the work daily usually has the clearest view of what "good enough to trust" means. Writing the standard down before testing stops anyone, including the vendor, from moving the goalposts once the results come in.

Step 4: run it side by side with the current way

Run the AI on every sample and keep the normal, current way of doing the task running in parallel on the same samples, at least for this test period. Do not switch staff off the existing process yet. Put the AI's answer and the correct answer next to each other for every single sample, not just a few, so a pattern of errors is visible instead of hidden in an average.

This side-by-side step also reveals something proposals rarely mention: how long the whole pipeline actually takes, including scanning or uploading, waiting for a result, and a person checking it. A tool that is fast to process but slow to feed documents into, or whose output needs heavy correction, may save less staff time than it first appears to.

Step 5: review the mistakes with the staff who do the work

Sit down with the person who normally does this task and go through every sample where the AI's answer was wrong, not just the overall score. Ask them to categorise the mistakes: did it misread a number, miss an item entirely, misunderstand a Lao–Thai mixed phrase, or get confused by a document layout it had not seen before? Some mistakes are tolerable if a person catches them before anything is finalised; others are the kind that would have caused real harm if nobody had checked.

This review also tells you whether the errors are the kind that improve with better source documents (a scanner upgrade, clearer forms) or the kind that are inherent to the task as it currently works. It is also the point where staff who were worried about the tool either gain confidence in it or raise a concern worth taking seriously. Do not skip this step or compress it into a five-minute summary; it is where most of the real learning from a pilot happens.

Step 6: decide go, no-go, or narrow

With the criteria from Step 3 and the review from Step 5 in hand, make an explicit decision, written down, with a reason: go ahead, stop here, or narrow the task to the part that worked well. "Go" does not mean turning the tool loose unsupervised; it means moving to a properly staffed rollout with a person checking output, which the next stage of the project should plan for. "No-go" is a legitimate and useful outcome; a pilot that shows a task is not ready yet, while real documents were still in your own hands throughout, is a successful pilot, not a wasted one.

"Narrow" is common and often the right call: perhaps invoices from three suppliers worked well and the rest did not, so you roll out on those three and keep testing the others. Avoid a vague "mostly good, let's keep going" verdict with no specific scope; it tends to turn into reliance on a tool nobody tested properly for the cases that actually went into production.

Step 7: only now think about the server

Sizing and buying a server comes after the pilot succeeds, not before. By this point you know roughly how much data the system handles, how many people need accounts, and whether the task needs to keep working when the internet is down, which changes whether a private, on-premises setup makes sense; see our guide on private AI versus cloud AI for that decision. If a private server looks right, AI server setup and care covers what installation and the following months of upkeep involve.

The pilot itself does not need dedicated hardware bought in advance. Most pilots can run on existing computers or a modest temporary setup, which is part of why doing the pilot first, rather than buying first, protects your budget.

Common mistakes we see

Testing on a vendor's clean demo samples instead of your own messy documents. Skipping the written pass criteria and judging the result by a general impression afterward. Running the pilot alone instead of with the staff who do the task daily, so their practical objections surface only after rollout. Testing only ten samples, or only the easy ones, and missing a failure pattern that shows up at thirty or fifty. Letting a "mostly fine" result slide into full rollout without defining what "fine" meant in writing. And buying a server before any of this, which turns an avoidable loss into a visible one.

Starting point

If you are not sure which task to pilot first, our AI readiness assessment helps identify a good candidate from your own workflow and data. If the task involves Lao-language documents or recordings specifically, read what to test before trusting AI with Lao alongside this guide, since some of the sample-collection advice there overlaps with Step 2 here. And once a pilot succeeds on a document-heavy task, private AI document assistant describes what a production rollout of that kind of system typically includes.