Four operating rules
An assistant can write a report or change a repository in minutes. Checking it may take much longer. These are the four rules we teach on the course, with checks you can use on your own work.
You can use these checks with whichever assistant you already have. For the studies behind the course, including the different results on small coding tasks and real repository work, see the evidence page.
Verification
The rule: generated work is untrusted until independently verified.
Models produce polished work and persuasive explanations even when the result is wrong, and because the explanation and the work come from the same source, the explanation can't vouch for the work. So the evidence has to come from outside the conversation, whether the work is a code change, a drafted report or a summarised spreadsheet.
- Read the whole output before accepting any part of it, noting anything the original request never asked for.
- Run the relevant checks yourself. That means executing the tests, opening the cited source or recalculating the figure, because predicting a result is not the same as producing it.
- Check every named reference against something outside the conversation, whether that's the documentation, the runtime or the original record. Models invent methods, settings, citations and figures, and the inventions look internally consistent.
- Treat the model's explanation of its own work as part of the work, not as evidence for it.
- Before you accept, ask what the change touches beyond the request: security, existing data and anyone who depends on the current behaviour.
Problem definition
Find out what the requester needs before you start.
"Pull together a report on this" leaves a lot unanswered. Who will read it? Which period should it cover? What decision will it support? An assistant can fill in those gaps, but you may end up with a finished report based on assumptions nobody agreed to. Settle the important questions and write down the answers first.
- Write down the facts you have verified, the assumptions you are proceeding on and the decisions that have more than one defensible answer, before anything is produced.
- Write acceptance criteria that are observable, testable and finite.
- Decide how you'll check the work before asking the model to produce it. If you can't describe a check yet, spend more time on the request.
- Ask the requester about anything that could change the outcome. For smaller uncertainties, make a reasonable assumption, write it down and continue.
- Write down what must stay as it is. That gives you something to point to when an extra change appears.
Bounded delegation
Give the assistant only as much work as you can check.
Suppose you ask for a bug fix and get a refactor at the same time. You now have two changes to understand, and it may be hard to tell which one altered the behaviour. You could spend longer reviewing the result than you saved by generating it. Keep requests small enough to check each result independently, and make the decisions yourself.
- Split the work into investigation, decision and implementation. Never hand over all three in one request.
- Keep the decision. Ask for options and trade-offs if useful, then choose the approach yourself.
- Before delegating implementation, state a contract covering what may change, what must stay untouched and what counts as done.
- Save a checkpoint before each delegated step, whether that means committing the working state or keeping the last trusted version of the document. A clean revert has to stay possible.
- If the result includes work you didn't ask for, reject that part and make the next request more specific. Don't let it become extra work you have to approve.
Recovery
Change approach when the evidence shows yours is wrong.
You may need to undo your first attempt. Keep a note of what you learned before you do, so the next attempt starts from those findings. Be especially careful when you find yourself asking for the same fix again: unless you've found something new, another confident answer is unlikely to help.
- Watch for the loop: the last three attempts differ in wording rather than strategy, and no new evidence has appeared since the loop began.
- When nothing supports or rules out the current approach, stop generating and go and get evidence.
- Record the findings somewhere outside the conversation before you revert or start a fresh session. Get down the goal, the facts you established and the way you established them, what has been ruled out, and what the next attempt will do differently.
- Try a different approach next time. Rewording the same request won't address what you found.
- Restore the last version you trust before trying again. Use what you've learned to decide whether to try a stronger model, ask a colleague or finish the work yourself.
These four rules are practised live, on realistic work, in San Digital's two courses: AI-Assisted Engineering for software engineers and AI-Assisted Knowledge Work for people who work in documents and data.