Build an AI Dark Factory: Autonomous Code Shipping System
Summary of the video “How to Build the Most Powerful System for AI Coding (Full Breakdown)” by Cole Medin.
An AI dark factory is a fully autonomous system that takes code specifications and ships production-ready, reviewed code without human intervention. Cole Medin breaks down how to build one using a skill-based framework, covering the workflow architecture, validation strategies, guidance layers, and deployment mechanisms that make autonomous AI coding reliable.
What is an AI Dark Factory
Definition and Core Concept
An AI dark factory is a repository that autonomously ships code based on specifications you provide. You submit a spec for what you want built, and the system outputs fully reviewed and validated code ready for production, eliminating the need for human code review or deployment steps.
Real-World Example: Chat.Dynamis.ai
Cole built a working agentic chat application that searches his entire YouTube channel and course content, citing sources. He built the entire application without reviewing a single line of code the system shipped, proving the dark factory concept works for real applications.
Not Vibe Coding
Dark factories are not reckless automation. Significant engineering effort goes into building the harness upfront so the agent can plan properly, including separate validation agents and strict guidance rules to ensure reliability.
AI Coding Autonomy Levels
Five Levels of AI Coding Autonomy
Coding autonomy ranges from Level 0 (manual with AI autocomplete) to Level 5 (full dark factory with no human steering). Most developers operate at Level 3 (hands-off code writing but involved in planning and validation). Dark factories represent Levels 4-5, where humans only provide high-level specs and the system handles everything else.
Why Level 3 is Most Reliable Today
Level 3 (hands-off code writing with human planning and validation) remains the most reliable because humans stay in the loop for oversight. The tradeoff is that humans become the bottleneck, slowing down shipping speed.
Dark Factory Removes the Bottleneck
By moving to Levels 4-5, humans are no longer the bottleneck. The agent receives only high-level direction (PRD or spec document), figures out implementation details, validates its own work, and deploys autonomously. This requires a well-engineered harness to be reliable.
Building Your Dark Factory: Setup and Installation
The Skill-Based Framework
Cole provides a reusable skill that guides you through building a dark factory. You input a PRD (Product Requirements Document), and the skill builds the entire system based on best practices learned from building his own dark factory.
Installation Options
If using Claude Code, install via marketplace plugin with two commands. For other agents (Pi, Codex), point them to the GitHub repository URL. You can also manually copy the build-dark-factory skill and customize it for your coding agent.
Starting Setup
Create an empty folder, copy in the dark factory skill, add any additional skills you want to use, and include your PRD. The skill then interviews you with detailed questions about autonomy level, validation strategy, and your preferred AI coding workflow.
Dark Factory Architecture and Workflow
GitHub Issue-Based Queue System
The dark factory uses GitHub issues as input specs. Each issue is triaged, labeled with status (accepted, in-progress, needs-review), and processed through the workflow. A cron job triggers every 30 minutes to handle the queue of pending work.
Issue Status Labels and Priorities
Issues move through states: accepted (ready to build), in-progress (currently being worked on), needs-review (PR open, awaiting validation). The 30-minute trigger prioritizes fixing existing PRs first, then reviewing PRs, then working on approved issues, then triaging new specs.
Headless Mode Execution
The coding agent runs in headless mode (background task) for each workflow step. This means the agent operates autonomously without interactive UI, which is essential for the dark factory to run without human steering.
Parallel Job Execution
You can configure multiple jobs to run in parallel during each 30-minute trigger, allowing several issues to be built and reviewed simultaneously. However, Cole recommends limiting parallelism to avoid hitting rate limits on your AI provider.
Deployment Strategy
Blue-Green Deployment Model
The dark factory uses blue-green deployment: two identical versions of the application run simultaneously. One is live for users; the other is on standby. When updates are ready, they're deployed to the standby version, then the versions are flipped. This eliminates downtime and allows rollback if issues occur.
Deployment is Non-Negotiable
If merged code doesn't automatically deploy to users, you've built a PR generator, not a dark factory. Autonomous deployment is essential to the system's value; otherwise, humans remain the final bottleneck.
Guidance Layer: Controlling Autonomous Behavior
Three Core Guidance Files
The dark factory uses three files to guide the agent: global rules (constraints and conventions for all work), factory rules (stricter boundaries specific to autonomous operation), and mission.md (goals and out-of-scope items derived from the PRD).
Global Rules vs. Factory Rules
Global rules apply whether the agent works autonomously or with you normally. Factory rules are stricter and only apply in the dark factory context, ensuring the agent handles bite-sized tasks one at a time and doesn't attempt overly complex work when unsupervised.
Mission.md and Spec Rejection
Mission.md encodes your application's goals and explicitly lists what's out of scope. This allows the agent to reject specs that violate scope, correcting you if you forget the original constraints. The agent can also split complex specs or request human collaboration if needed.
Validation Harness: Ensuring Reliability
Separate Builder and Validator Agents
Two distinct agents operate in the dark factory: the builder (implements code) and the validator (reviews PRs). This separation prevents bias, as the validator approaches the code with fresh eyes, unaware of implementation details or the original plan.
Holdout Scenarios: The Core Innovation
Holdout scenarios are test cases written before implementation and kept hidden from the builder agent. They define success at a high level without revealing implementation details. The builder cannot see them, preventing it from gaming the tests. The validator runs these scenarios separately to catch issues the builder's own tests might miss.
Dual Testing Approach
The builder agent has access to unit and integration tests during development to iterate on its work. The validator then runs a separate suite of tests, including holdout scenarios, to verify correctness without bias. Any failures are fed back to the builder for fixes.
Safety and Escalation
Human Escalation Failsafe
Although the dark factory aims for full autonomy, Cole built in a failsafe to escalate issues to humans when needed. You can configure during setup when escalation should trigger, preventing infinite loops where the agent repeatedly fails to fix a problem.
Current Experimental Status
Dark factories are still experimental. Level 3 (hands-off code writing with human oversight) remains the most reliable for production use. Dark factories work well for many applications but should be tested thoroughly before trusting them with critical systems.
Practical Considerations and Customization
Coding Agent Flexibility
The dark factory works with any coding agent that supports headless mode: Claude, Pi, Amp, Client, Codex, and others. You can use different agents or models for different workflow steps, mixing providers as needed.
Interview-Driven Customization
The skill's interview process asks about your autonomy preferences, validation strategy, existing AI coding workflow, and desired parallelism. This ensures the generated dark factory matches your specific needs and working style.
Game Development as a Testing Ground
Cole uses game development to test dark factories because complexity can be easily scaled by adding features. Other applications risk becoming feature-bloated, making them poor test cases for pushing the system's limits.
Notable quotes
An AI dark factory is a repository that ships its own code. — Cole Medin
If merging never puts code in front of a user, you built a PR generator, not a dark factory. — Cole Medin
Without holdout scenarios, there is no way I'd even run this as an experiment. — Cole Medin
Action items
- Install the dark factory skill from Cole's GitHub repository or via Claude Code marketplace plugin
- Prepare a detailed PRD (Product Requirements Document) for the application you want to build
- Run the skill's interview process and answer all questions about autonomy level, validation strategy, and coding workflow
- Set up a GitHub repository to use as your dark factory queue system
- Configure your preferred coding agent to run in headless mode
- Define global rules, factory rules, and mission.md for your specific application
- Write holdout scenarios before starting development to define success criteria
- Set up blue-green deployment infrastructure for your application
- Configure the 30-minute cron trigger and parallelism settings based on your token budget
- Test the dark factory with a non-critical application before using it for production systems
- Monitor the first few runs and adjust escalation thresholds as needed