Vector 02 · 7 min read · · AI in real businesses
The plan is not the outcome
What pilots learned about automation complacency, and why rubber-stamping AI pull requests is the same trap.
The forecast said we'd be in Phillip Island in about 2.8 hours.
I was flying a charter out of Tasmania, taking the team from a motorcycle shop across Bass Strait to the MotoGP. Like every flight, it started with a plan. The forecast had a headwind of about 40 knots on the first leg, easing to around 30 as we turned towards Phillip Island near the Victorian coast. I planned the fuel around that, added reserves, and we departed with just over four hours on board. That was a healthy margin.
About halfway across the first leg, the instruments stopped agreeing with the plan.
The headwind at our cruising altitude was stronger than forecast. That wasn't a problem yet, because we had fuel to spare. But the further north we went, the more the instruments told a story I couldn't ignore. The headwind had built to 60 knots, and our flight time was stretching out to 3.6 hours. I re-checked the weather and re-ran the numbers, and reality still didn't match the plan.
If I'd kept flying the plan, we'd have landed with most of our reserve fuel gone. By my numbers we'd have burned through 80 to 90 per cent of it, over the cold, rough water of Bass Strait. One go-around at Phillip Island and we'd have been out of fuel.
So we changed the plan. I slowed the aircraft to maximum range cruise, which trades speed for a lower fuel burn. I changed altitude to find a level where the wind swung around more to our side and lost some of its strength. We landed 30 to 40 minutes late, with exactly our minimum reserve fuel intact.
It was a perfect plan, drifted by reality. Getting home safely meant accepting a delay and having a rethink.
I've been thinking about that flight a lot lately, because it's exactly what working with AI feels like.
The plan is not the outcome
AI is very good at plans. You can give it a detailed brief and it'll write the code, the tests and the pull request description, and on paper it all looks right. The forecast was right on paper too.
The problem is that reality drifts, and someone has to be watching the instruments when it does.
In aviation, staying in the loop isn't left to good intentions. It's procedure. At regular, scheduled points in every flight you check your fuel burn, your position, your engine and your navigation against the plan. It's drilled into you in training until it's simply how you fly. The whole point is that no drift from the plan goes unnoticed.
Software has instruments too. We just don't always read them. When AI writes the code, we still need to understand what it's telling us:
- What does it actually do?
- How does it solve the problem, and is that the way we'd want it solved?
- What problems does it introduce later on?
- What shape is it? Does it fit the system we're building, or does it bolt something new onto the side?
Those are the readings that tell you where the project is really heading, whatever the plan says.
Two ways to build
I've built software both ways now, and the contrast taught me more than either did on its own.
In 2022 we rebuilt SquarePay's payments platform from an empty repository, with no AI. It took nearly a year. We built it line by line, with a clear picture of the architecture we wanted: something that could stretch to handle use cases we hadn't thought of yet. It was slow and steady, and some weeks it felt painfully slow. But we understood every step, and we always knew which direction we were heading.
Fast forward to more recently. AI had become my main tool, and orchestration tools that coordinate lots of AI agents at once had arrived. I wrote a very detailed plan for a feature I wanted to add to a project and handed it over. Over a couple of days it spun up 40 or 50 sub-agents, and at the end it delivered the feature.
I had no idea how it worked under the hood.
When I went digging, I found trouble. Code we'd normally have written once and shared had been written again by every individual agent. Agents that were supposed to reduce duplication had broken edge cases that the other agents had carefully handled. Changing anything broke something another agent had built.
None of it was obvious from the outside. The feature worked, and it passed the checks. I didn't see the problems because I hadn't been watching the direction each agent was taking. I'd handed over the plan and stopped reading the instruments.
Three reviews, and nobody in the loop
Here's a less dramatic version, and to be honest a more common one.
Recently, one of our developers raised a pull request that had been written with AI, which is completely normal for us now. The developer reviewed it. An AI reviewer reviewed it. A tech lead reviewed it. Everything looked perfectly fine.
We shipped it on a Friday afternoon, of course.
The deployment pipeline failed. Then production failed, even though the deploy hadn't finished.
The root cause was a database migration that added a foreign key constraint to a table with millions of rows. On a developer's machine, with maybe a hundred rows of test data, it ran in a blink. In production it locked things up. The deployment timed out waiting for it, requests started queuing up behind it, and then the whole thing fell over.
Three reviews, and nobody had really stopped to read that migration line by line and ask what it would do at scale. Everyone was approving the plan. Nobody was flying the aircraft.
That's what I mean by a review slave: someone who rubber-stamps output instead of reading the instruments. It isn't laziness. Usually it's good people under time pressure, trusting that someone else in the chain is doing the careful bit.
Fly AI like a second officer
None of this means you should use less AI. I use it every day, and I wouldn't go back. But I think about it the way I'd think about a second officer in the cockpit. They're capable and fast, and genuinely useful. You still watch them. You still monitor progress. And you still ask pointed questions when the outcome starts drifting from the plan.
Here's what that looks like in practice for us.
Question it relentlessly. Ask why it made each decision. Why did it build a new method when we already have tooling for that? The answers do two jobs. They show you what's actually happening, and they show you the limits of both your existing code and what's being built now. That questioning is your instrument scan. It's what gives you the project's real heading.
Make the review a procedure, not a vibe. Our process goes like this:
- An AI agent reviews the merge request first.
- The developer has to resolve every issue it raises.
- A senior developer re-reviews it and re-runs the AI review.
The goal isn't more reviews. It's making sure a human with context is reading the readings.
Review in pieces. We use a tool that breaks a merge request down by feature, flow or task, so a reviewer can look at each part on its own instead of drowning in one giant diff. Smaller pieces mean nothing slips past in the noise.
Flag the known dangers automatically. That same breakdown flags risky changes, such as migrations that touch large tables. After that Friday, it's the first thing I want a reviewer looking at.
Late is fine
We landed at Phillip Island half an hour late, and nobody minded. They'd have minded a lot more if we hadn't landed at all.
I think the same is true with AI. It's tempting to take the plan and let it fly itself, because it's fast and it's usually right. But the plan isn't the outcome. Somebody still has to read the instruments, notice the drift, and be willing to slow down and change course.
That somebody should be you.