On Friday 19 July 2024, passengers at Kuala Lumpur International Airport’s Terminal 2 were checked in by hand. Malaysia’s National Cyber Security Agency reported manual check-ins and long waits there, and the Digital Minister later said five government agencies had been affected. Microsoft estimated that 8.5 million Windows devices worldwide went down that day.
It was not an attack. A security vendor, CrowdStrike, shipped a faulty update. Its own review found that a bug in the tool that checks updates had let the bad file through, and its list of remedies was mostly a list of tests: developer testing, update and rollback testing, and releasing to a small group first.
That is the cost of a release that nobody checked properly. This article is about making sure yours are checked, at a time when more and more of the code is written by AI.
It is written for the people who approve the work, not the people who type the commands. You will not write a test. You can require that tests exist, hand over the instructions below, and ask to see the evidence.
Why this matters more when AI writes the code
Almost everyone building software now uses AI. Google’s DORA research surveyed nearly 5,000 technology professionals in 2025, and 90% reported using AI at work.
More code, delivered faster, is less stable unless something checks it. The same research found that AI adoption went with higher delivery speed and with lower delivery stability. Its explanation is worth quoting: “Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability.” Its summary: “AI doesn’t fix a team; it amplifies what’s already there.”
The people using the tools know the output needs checking. In Stack Overflow’s 2025 survey, 46% of developers said they distrust the accuracy of AI tools, against 33% who trust it. The most common frustration, named by 66%, was “AI solutions that are almost right, but not quite”.
Security is part of it. Veracode ran 80 coding tasks across more than 100 models in 2025 and reported that the AI-written code introduced a security vulnerability in 45% of cases.
“Almost right” is exactly what an automated test is for. A person reading two thousand lines of generated code will miss the one wrong rounding rule. A test that says “this invoice must total RM 1,060.00” will not.
And the economics have changed. Writing tests used to be the slow part that got cut when the deadline came. An AI coding agent will write a few hundred of them in an afternoon. If your vendor uses AI to build faster, the saving should buy you a tested system, not only a quicker invoice.
The tests worth asking for
You named three. There are a few more worth knowing, and you do not need all of them on day one.
| Test | The question it answers | When it runs |
|---|---|---|
| Unit test | Does one rule or calculation give the right answer? | On every change. Takes seconds |
| Integration test | Do the parts work together: the code, the database, the permissions? | On every change. Takes minutes |
| End-to-end test | Can a user complete a whole journey in a real browser? | Before every release |
| Smoke test | After deployment, is the live system actually up and usable? | After every deployment |
| Regression test | Has a bug we already fixed come back? | With the rest. Every fixed bug adds one |
| Static checks | Are there errors a machine can spot without running the code? | Before every commit |
| Security scans | Are there known vulnerable components, or passwords left in the code? | On every change |
The first three form a pyramid. Many unit tests, because they are fast and precise. Fewer integration tests. A small number of end-to-end tests, because they are slow and the most likely to fail for reasons unrelated to your code.
Add these when the system calls for them:
- Load tests, if many people use it at once or a deadline creates a rush.
- Accessibility checks, if the public uses it.
- Migration and restore tests, to prove a database change or a backup really works before the day you need it.
- Evaluations for AI features. If the product itself contains a language model, its answers need their own test set. We covered that in how we keep a person in the loop.
How AI gets tests wrong
An AI agent will happily produce a large test suite that proves nothing. Three patterns account for most of it, and the prompts below are written to block each one.
Tests that copy the code. Asked to “add tests”, an agent reads the code and writes tests confirming the code does what it does. If the code is wrong, the test agrees with it. Expected results must come from the requirement, not from the program.
Tests that test a stand-in. A mock is a fake version of a component, used so a test does not need the real thing. Overused, the test only checks the fake. A study accepted at MSR 2026 looked at 1.2 million commits and found that 36% of commits by coding agents added mocks to tests, against 26% for other commits.
Tests bent until they pass. An agent told to “make the tests pass” may change the test instead of the code. OpenAI published an example from its own research in March 2025: a model given failing tests reasoned, “So we can hack verify to always return true.”
None of this is a reason to avoid AI-written tests. It is a reason to be specific about how they are written and to have a second pair of eyes on them.
Nine prompts to hand over
These work with any current coding agent. Text in square brackets is for your team to fill in. The wording is deliberate: each line exists because of a failure described above.
1. Standing rules
This goes in the instruction file the agent reads at the start of every session, so it applies to all work, not one task.
Testing rules for this repository
- Every change in behaviour comes with a test. Write the test first, run it,
and show me that it fails for the right reason before you write the code.
- Tests describe what the business requires, not what the code currently does.
Take expected results from the requirement, the specification or a worked
example I give you. If you cannot tell what the correct result is, stop and ask.
- Never edit, skip, weaken or delete an existing test to make it pass.
If you believe a test is wrong, stop and explain why.
- Never add special handling for test inputs to production code.
- Mock only what is outside our system: payment gateways, email, third-party
APIs, the clock. Use a real test database. Never mock the code under test.
- Tests must be deterministic: no real network calls, no dependence on
today's date or on the order they run in, no fixed waits.
- When you report that work is done, paste the exact command you ran
and its output.
2. Plan before writing anything
Read this repository. Do not change any code yet. Produce a test plan:
1. How tests are run today, what exists, and what the last run shows.
2. The ten business rules where a mistake would cost us most (money,
compliance, customer data). For each, say whether a test covers it now.
3. The five user journeys the business cannot operate without.
4. The tools you recommend for unit, integration and end-to-end tests
in this stack. Prefer what the project already uses.
5. A proposed order of work, highest risk first.
List anything you could not determine, and the questions I must answer.
Item 2 is the one to read yourself. It is your business, described back to you as a list of rules. If something important is missing, say so now.
3. Unit tests
Write unit tests for [the invoice calculation module].
- First list the cases as a table: input, expected result, and the rule
it comes from. Wait for my approval before writing code.
- Cover the normal case, the boundaries (zero, negative, maximum, rounding,
empty, duplicate), invalid input and every error path.
- Use the worked examples in [document] as expected values. Do not obtain
expected values by running the code under test.
- One behaviour per test, named so that a failure reads as a sentence.
- When finished, break the code on purpose in three places, one at a time,
and show that a test fails each time. Then restore it.
The last line is the important one. A test that cannot fail is decoration.
4. Integration tests
Write integration tests for [the order API].
- Run against a real test database, started for the test run and reset
between tests. Build its schema by running our migrations from empty.
- For each endpoint: the success case, validation failures and the
shape of the response.
- Permissions: prove that a user without the role is refused, and that a
user from organisation A cannot read or change organisation B's records.
- Replace external services (payment, email, e-invoice submission) with
fakes at the network boundary. Include their failures: timeout, error
response, duplicate callback.
5. End-to-end tests
Write end-to-end tests for these journeys only: [sign in; create a
quotation; approve it; issue the invoice; export the monthly report].
- Drive the real application in a browser against a seeded test environment.
- Find elements the way a user would (labels, roles, visible text),
not by internal identifiers that change with the design.
- Each test creates its own data. Tests can run in any order and in parallel.
- No fixed waits. Wait for the visible result.
- On failure, save a screenshot and a recording as pipeline artefacts.
Keep this suite small. If a rule can be tested at a lower level, test it there.
6. Smoke test
Write a smoke test that runs against a deployed environment in under
two minutes and changes no business data.
Check that: the health endpoint responds and reports the expected version;
the sign-in page loads; a test account can sign in; one read-only page
loads data from the database; [storage, the job queue, the email provider]
can be reached.
Exit with a failure code on any problem, with a message a non-engineer
can read. This runs after every deployment. A failure stops the release.
7. Every bug becomes a test
We have a bug: [what happened, the steps, what should have happened].
First write a test that reproduces it, and show me that it fails.
Then fix the code. Then show that test passing and the full suite passing.
Do not change any other test.
Make this a rule of working with your team. A bug that is fixed without a test is a bug you are entitled to see again.
8. Wire up the gates
Set up two gates.
Pre-commit hook, finishing in under 30 seconds: formatting, linting, type
checking, a scan for secrets, and the unit tests related to changed files.
Install it automatically when a developer sets up the project.
CI pipeline, on every pull request and on the main branch: install from the
lockfile, lint, type check, unit tests, integration tests with a real
database, build, dependency vulnerability audit, end-to-end tests.
Any failure fails the pipeline. Publish test reports as artefacts.
Release: deploy only from the pipeline, and only from a commit whose
pipeline passed. Run the smoke test after deployment. On failure, roll back.
Then tell me which repository settings I must change by hand so that
nobody can merge or deploy while the pipeline is failing.
9. A sceptical review
Run this in a fresh session, or with a different model, so the reviewer is not the author.
You did not write this code. Review the tests in [folder] as a sceptic.
- Which tests would still pass if the feature were broken?
- Which only assert that a mock was called?
- Which expected values look copied from the code's own output?
- Which rules in [specification] have no test?
- Which tests are skipped, and why?
Give me a list, worst first. Do not fix anything.
Two gates: before the commit, and before the release
Tests that run only when someone remembers are not a control. They need to run automatically, in two places.
The pre-commit hook runs on the developer’s machine, or the AI agent’s, each time a change is saved into version control. It should take seconds and catch the cheap mistakes: a syntax error, a failed rule, a password pasted into a file. It gives the fastest feedback, which matters even more for an agent, because a failing check is something it can read and correct on its own. Anthropic’s guidance for its coding agent makes the same point: “Give Claude something it can run that comes back pass or fail.”
It is also optional by design. Git’s documentation says the hook “can be bypassed with the –no-verify option”. Treat it as a convenience, not as your control.
The CI pipeline is the control. CI stands for continuous integration: a server that takes every proposed change, builds the system from scratch and runs the full set of tests. Nobody’s laptop is involved, so “it works on my machine” stops being an answer. Configured properly, a change with a failing test cannot be merged, and a release can only be made from a version that passed.
| Stage | What runs | Time budget |
|---|---|---|
| Before each commit | Formatting, lint, types, secret scan, related unit tests | Seconds |
| On every proposed change | All unit and integration tests, build, security scans | A few minutes |
| Before every release | The above, plus end-to-end tests | Ten to twenty minutes |
| After every deployment | Smoke test, with rollback on failure | Under two minutes |
| Nightly, if needed | Load tests, full browser coverage, restore test | As long as it takes |
This is how we work. On one platform we built, every push runs linting, type checks, unit tests, the build and a code-quality scan before anything is deployed, and the backend has 589 automated tests. Another has more than 1,150.
What to ask your team or vendor
You do not need to read the tests. Ask for these.
- Show me the pipeline run for the last release. A real screen, with each stage and its result.
- How many tests are there, and how long do they take? You want a specific number and a time in minutes.
- What happens when one fails? The right answer is that the change cannot be merged. “We look into it” is the wrong one.
- Can anyone deploy without going through the pipeline? If yes, who, and when did it last happen?
- Break something on purpose. Ask them to change one business rule in a test environment and show the pipeline turning red. It takes five minutes and tells you more than any report.
- Which rules are tested? Ask for the list from prompt 2, and check that the rules that matter to you are on it.
- When did a test last catch a real problem? A team with working tests has a recent story.
Be careful with one number. Coverage, the percentage of code that tests execute, is easy to raise without testing anything. A high figure with weak tests is worse than a lower one with good tests, because it reassures.
What tests will not do
They do not tell you the requirement was right. A test confirms the system does what was specified. If the specification misunderstood your business, the tests pass and the system is still wrong. Your staff still need to use it before it goes live.
Someone still has to read them. AI-written tests need review like AI-written code. Prompt 9 helps. It does not replace an engineer who is accountable for the result.
They need upkeep. When the business changes, tests change. A suite that fails at random gets ignored, and an ignored suite is the same as none. Budget for maintenance.
The short version
AI has made writing software faster, and the research says speed without checks makes it less stable. AI has also made the checks cheap. Ask for both.
Put the rules in writing, require tests with every change, run them at commit and before every release, and ask to see a failing build once so you know the gate is real.
For the wider procurement picture, read how to choose a software development partner and seven questions to ask before you approve an AI project.
Sources
- Navigating the fallout: lessons from the CrowdStrike outage (KPMG Malaysia)
- Gobind: CrowdStrike event not cyberattack but failure of global cybersecurity firm (Malay Mail)
- Helping our customers through the CrowdStrike outage (Microsoft)
- Falcon content update: preliminary post incident review (CrowdStrike)
- Announcing the 2025 DORA Report (Google Cloud)
- 2025 Developer Survey: AI (Stack Overflow)
- AI-generated code poses major security risks in nearly half of all development tasks (Veracode)
- Are coding agents generating over-mocked tests? An empirical study (Hora and Robbes, MSR 2026)
- Detecting misbehavior in frontier reasoning models (OpenAI)
- Best practices for Claude Code (Anthropic)
- githooks: pre-commit (Git documentation)