From Prompt Engineering to Harness Engineering: Testing AI-Generated Code
Summary of the video “How to Move from Prompt Engineering to Harness Engineering in Testing” by Automation Testing with Joe Colantonio.
Matt Wyn, a former Cucumber Core team member, shares his journey from resisting AI to building harness systems that validate AI-generated code without humans reading it. The shift moves testing from prompt-based interactions to engineering the validation loops and constraints that guide AI agents, making testing more about system design than code inspection.
The Evolution: From BDD to AI-Driven Development
BDD's Lasting Impact on Test Structure
The Given-When-Then structure from BDD remains a universal language for organizing tests, predating even the Arrange-Act-Assert pattern. This natural unit of behavior description has spread across testing tools and now influences how LLMs structure their outputs.
The Real Bottleneck: Shared Understanding, Not Typing
While LLMs save typing effort, the deeper bottleneck has always been shared understanding between teams about the business problem itself. If the team truly understands the problem, typing becomes the bottleneck; if not, no amount of code generation helps.
Matt's Personal AI Journey: From Denial to Engagement
Matt spent months burying his head in the sand about AI, believing his TDD and refactoring expertise made the tools unnecessary. Around 18 months ago, he realized the tools weren't going away and leaned in, discovering he could encode his entire Gherkin coaching practice into 500 lines of markdown.
The Software Factory Concept: Hands-Off Development
What If Humans Never Write or Read Code?
The software factory concept, coined by Justin McCarthy at Strong DM, asks: what if humans never write or read the code generated by LLMs? This terrifies testers initially, but it reframes validation from code inspection to system-level guardrails and healing loops.
Validation Loops Replace Code Reading
Instead of humans reading generated code, the system uses LLM-powered validation loops to check for malleability, security best practices, and performance. When issues are found, the system heals the code automatically rather than requiring human intervention.
Steering the System, Not the Agent
The shift is from steering individual agent prompts to steering the system the agents work within. This means creating guardrails, constraints, and context that guide agents toward desired behavior, much like designing a slope so the right outcome inevitably emerges.
Understanding LLM Limitations and Capabilities
LLMs as Amplifiers and Semantic Transformers
LLMs are not intelligent entities but transformers of text and tokens. They amplify existing patterns in their training data—both good and bad. They excel at semantic zoom (compression and reorganization of existing information) but struggle when asked to generate novel information beyond their input.
The Three Phases of LLM Understanding
Users progress through three phases: initial magic (wow, this is amazing), disillusionment (it's not as good as I thought), and transcendence (understanding it's just an inference machine and learning to use it strategically).
Information Compression vs. Hallucination
LLMs perform best when compressing existing information (e.g., summarizing a transcript, extracting key points from a conversation). They hallucinate when asked to generate novel information from a small seed. The key is providing rich context on input so the model has all the information it needs.
Using Multiple LLMs for Better Validation
Running the same code or plan through three different LLM models (e.g., Claude, GPT, Gemini) provides diverse perspectives because they have different training data and weights. Each model catches different issues, creating a more robust validation through averaging.
Harness Engineering: The New Testing Discipline
What Is Harness Engineering?
Harness engineering is the infrastructure and constraints built around LLM calls to guide and validate their output. It includes skills (prompts that trigger automatically), hooks (deterministic processes), automated tests, adversarial sub-agents, and the entire system designed to narrow the LLM's work toward desired outcomes.
Moving from Prompt Engineering to Harness Engineering
The progression moves from copying-pasting into ChatGPT, to using cursor or Claude Code with manual prompts, to building automated systems where prompts trigger based on context. The goal is to stop repeating explanations and instead engineer the system so the LLM succeeds without constant human steering.
Coaching the Agent: Retro-Driven Improvement
When an LLM produces suboptimal code, instead of just fixing it, treat it like coaching a junior developer. Ask: what context could I have provided that would have made them more successful? Document architectural decisions, patterns, and preferences so the agent learns from the feedback loop.
Context Engineering: Making Implicit Knowledge Explicit
LLMs can only follow patterns they can see. Capture implicit architectural decisions, design heuristics, and team preferences in written form (ADRs, style guides, design documents) so agents can find and follow them, reducing churn in code reviews.
Practical Applications: Legacy Code and Brownfield Systems
Modernizing Legacy Mainframe Code with LLMs
At Mechanical Orchard, LLMs were used to modernize 30-40 year old COBOL mainframe code by generating characterization tests through coverage-guided fuzzing, then using those tests as a mold to cast equivalent Java versions. The LLM speeds up laborious work that would be tedious to do by hand.
Handling Archaeological Layers in Brownfield Code
Brownfield codebases have multiple generations of patterns and practices layered on top of each other. LLMs often follow old patterns or mix them together. The solution is to make best practices and architectural decisions legible in the codebase so agents generate code in the current style, not historical patterns.
Extracting Design Heuristics from Code Review Comments
By analyzing pull request review comments where humans say 'I prefer it like this' or 'we tend to do it this way,' an LLM can distill the team's design heuristics and house style. Documenting these in the codebase makes them available to code-generating agents, reducing review churn.
The Future of Testing and Development Roles
Testing as a Specialism in T-Shaped Teams
Rather than disappearing, testing becomes a specialism within T-shaped teams. People with brains wired to think about what could go wrong remain valuable, especially for testing the systems that produce code rather than the code itself.
New Role: Chief Code Reviewer and Validator
As code generation accelerates, the tester's role evolves from writing tests to designing validation systems, reviewing generated code at scale using LLMs, and ensuring the harness catches problems before production.
Convergence of Product, Development, and Testing
At Mechanical Orchard, product owners began implementing features directly, and traditional role boundaries blurred. The future likely involves more convergence, with people moving fluidly between roles as LLMs reduce the friction of context switching.
Job Reorganization Rather Than Replacement
Rather than job loss, expect reorganization. If a team of eight can do twice as much with LLM assistance, you might need four people to handle twice the workload rather than eight people doing the same work. The challenge is how to organize teams and share context as velocity increases.
Addressing Skepticism and Capitalism Concerns
LLMs Amplify Existing Societal Problems
LLMs are not neutral tools; they amplify existing biases, inequalities, and problems in society. They concentrate wealth toward a few companies (Anthropic, OpenAI, Google) and can perpetuate discrimination in their outputs. However, blaming the technology misses the point—the issue is the capitalist system organizing it.
Humans Make Mistakes Too
A common objection is that LLMs make mistakes. But humans also make mistakes. The question isn't whether LLMs are perfect, but whether they can be integrated into validation loops that catch errors reliably, just as human code review does.
Understanding the Technology Enables Better Testing
Reading papers like 'Attention Is All You Need' and understanding that LLMs are transformers, not magic genies, makes it easier to test and work with them effectively. Treating them as black boxes limits your ability to break them and find their weaknesses.
Notable quotes
What if the humans never write the code, never read the code? — Matt Wyn (describing the software factory concept)
I've now encapsulated that in like 500 lines of markdown and it's horrifying and amazing at the same time. — Matt Wyn (on encoding his Gherkin coaching practice)
We're engineering the system that produces the code rather than engineering the code itself. — Matt Wyn
Action items
- Get a modest LLM subscription (Claude, ChatGPT, or Gemini) and experiment in your spare time with low-risk projects like organizing shopping lists to build intuition for steering at the product level.
- Start documenting implicit architectural decisions and team design preferences in your codebase (ADRs, style guides) so code-generating agents can follow them.
- When an LLM produces suboptimal code, conduct a retro: ask what context you could have provided upfront to make it more successful, rather than just fixing the code.
- Implement a multi-LLM validation loop: run critical code or plans through three different models (Claude, GPT, Gemini) to catch issues each model might miss.
- Build a harness around your LLM interactions: define skills (context-triggered prompts), hooks (deterministic checks), and automated validation loops rather than relying on manual prompting.
- For brownfield codebases, extract design heuristics from pull request review comments and document them so agents generate code in your team's preferred style.
- Use LLMs to inspect large pull requests by asking for sequence diagrams, behavior impact analysis, or architectural summaries rather than reading line-by-line.
- Read foundational LLM papers (e.g., 'Attention Is All You Need') to understand how these systems work, making it easier to test and work with them effectively.