---
title: I gave a factory of AI agents 42 hours. It merged 17 pull requests
date: 2026-11-01T02:00:00.000Z
edited: 2026-11-01T02:00:00.000Z
author: Ismail Ghallou (Smakosh)
canonical: https://smakosh.com/blog/zeus-qa-feature-factory
reading_time: 17 min (3767 words)
Tags: ai, web-development
---
# I gave a factory of AI agents 42 hours. It merged 17 pull requests
Over 42 hours (28 h 39 min of shifts across 12 runs), a line of AI agents cloned a competitor's features into my product and merged 17 pull requests to production. This is how the line worked, what broke on the way, and what it would have cost at list price.
The product is [Zeus QA](https://zeusqa.ai): you describe a test in plain English, a model drives a real browser through your site and hands back a verdict with the evidence. The competitor is Oplane, an always-on security reviewer for AI-first teams: threat models, findings on pull requests, a merge gate. Different products, same spot in the workflow: the pull request.
I did not write code for any of it. I wrote one prompt, steered a few times, approved production database migrations once, and read the reports.
**At a glance**
- 17 pull requests raised, 17 merged, 0 left open.
- +18,643 and -1,022 lines across 678 file changes.
- 4 production D1 migrations, all applied through one guarded script.
- 38 deploys from main, 37 succeeded.
- 15,113 model calls. $888.08 at list price with prompt caching, $6,857.05 without. Paid by my Anthropic subscription, not the API.
## The ask
This is the prompt I typed into [Empryo](https://empryo.com)'s orchestrator on October 30 at 08:48 (CET), typos and all:
> build a forever continious loop that doesn't end and all explorers and websearchers use haiku 5.5 and when needs a bit of reasoning use sonnet 5.5, workers, reviewers, devils advocats, judges are opus 5.5 medium. for the final gatekeeper reviewer use opus 5.5 xhigh, for collector and keepers of warehouse use opus 5.5 on low, and you build rest of the factory! supply chain always running collectors always collecting, stamping etc... build the full pipeline. The idea is to clone the features of Oplane and integrate them into Zeus QA, keep creating PRs, testing & verifying and reviewing them before merging them. use marketing-skills frontend-design skill and soulstack skills and any skill you need, keep working for 4 hours non stop, when ever a PR is merged, send me a message on Telegram (saved messages)
>
> use models from anthropic subscription only
The orchestrator turned that into a saved graph, `zeus-oplane-factory`. It now has 40 nodes, 33 edges and a step limit of 1,000 (it started at the default 60, see below), plus a folder of shell scripts under `.factory/bin`. The line sent its start message to Telegram at 08:52, and the first shift booted at 08:58 (CET).
I changed its direction three times while it ran:
- During the second shift: "don't forget marketing, use marketing-skills and create changelogs/blog posts and update landing page, add more marketing pages, record marketing videos for all features added". That opened a marketing lane in the backlog.
- Nineteen minutes later: "we want to add security stuff to zeusqa....". Until then the factory had built QA equivalents of Oplane's features. From here on, items with an `S-` id (security features) went first.
- On day two: "wrap up what's left here", with the link to the open pull requests. No new work was dispatched after that, only the fix loop on what was open.
## The line
Mono labels are shell steps. Everything else is a model agent. One cycle goes like this:
1. **boot** (shell) sets the deadline from the `hours=N` input, creates the GitHub labels (`factory`, `factory:needs-migration`, `factory:blocked`) and the warehouse folders.
2. **Four scouts** on Haiku 5.5 run detached, so they keep working while the workers build. One crawls Oplane, one takes inventory of what Zeus QA already has, one does web research on the stack, one reads the status of open pull requests. They are relaunched every cycle and joined at the start of the next one. That is the supply chain that never stops.
3. **mapper** (Sonnet 5.5) maps each Oplane feature to a Zeus QA equivalent in `feature-map.md`, with value, effort and dependencies.
4. **collector** (Opus 5.5 low) stamps items into `backlog.json`, each with acceptance criteria and the files it will touch.
5. **keeper-dispatch** (Opus 5.5 low) folds in the stamps, reconciles them with GitHub and fills `batch.json`: at most three slots, one port each (3101 to 3103). When there is nothing to build, it reports `BATCH_EMPTY`, and the shell step `clock-idle` waits five minutes, checks the deadline and starts a new intake.
6. **Three workers** (Opus 5.5 medium) build in their own git worktree under `.factory/wt/` on a `factory/` branch, and open a pull request labelled `factory`.
7. **Per lane, three critics** (Opus 5.5 medium): a reviewer, a devil's advocate and a verifier who runs the app. Each writes a report that ends in `VERDICT: PASS` or `VERDICT: FAIL`.
8. **ci-snapshot** (shell) collects CI status, mergeability and the three verdicts for each open PR.
9. **judge** (Opus 5.5 medium) reads the snapshot and the reports. PASS goes to the gatekeeper. PARTIAL or FAIL sends the fixers back, up to three rounds, counted in a file by a shell step.
10. **gatekeeper** (Opus 5.5 xhigh) reads every diff itself and merges one pull request at a time.
11. **notify** (shell) sent each merge to my Telegram Saved Messages by driving the Telegram desktop app with AppleScript, until I turned it off on day two.
12. **keeper-stamp** (Opus 5.5 low) updates the backlog and the ledger, and removes merged worktrees.
13. **clock** (shell) checks the deadline with a 60-minute margin. Before it, the next cycle starts at the scout join. After it, the line writes a final report and the run ends.
## The warehouse and the stamps
The agents never talk to each other. They talk through files in `.factory/warehouse`, which is kept out of git through `.git/info/exclude`:
- `feature-map.md`, by the mapper: each Oplane feature, its Zeus QA equivalent, value, effort and what is missing.
- `backlog.json`, by the collector and the keepers: 50 items with ids F-, S- and M-, acceptance criteria, a status and their stamps.
- `batch.json`, by keeper-dispatch: the cycle number and up to three slots.
- `reviews//*.md`, by the reviewer, devil's advocate and verifier: findings with `path:line`, a `SHA:` line and a verdict.
- `ci.txt`, by ci-snapshot: checks, mergeability and verdicts per PR.
- `fix/.md`, by the gatekeeper and the keepers: the fix list for the next round.
- `ledger.md`, by keeper-stamp: what merged, what was held and why, every cycle.
A stamp is a line of history on an item. This is a real one from the secret scanning item, cut short:
```json
{
"by": "gatekeeper",
"at": "2026-10-31T13:47:44Z",
"note": "Held at ffdf43b: verify.md VERDICT FAIL (web/src/lib/secret-scan.ts:184 dedupe key drops the high finding for a production connection string sharing its password with an earlier localhost one on the same line)..."
}
```
The judge has one hard rule: a PR passes only when all three reports exist, all end in PASS, every report's `SHA:` line equals the PR head, and every CI check passes. A review of an older commit does not count.
## The models, and why each one got its job
| Station | Model | Why |
| --------------------------------------------------------------- | ----------------- | ------------------------------------------ |
| Scouts | Haiku 5.5 | Crawling and listing is cheap, wide work |
| Mapper | Sonnet 5.5 medium | Matching features takes a little judgement |
| Collector, keepers, final report | Opus 5.5 low | Bookkeeping that has to be exact, not deep |
| Workers, reviewers, devil's advocates, verifiers, judge, fixers | Opus 5.5 medium | Building and reviewing the code |
| Gatekeeper | Opus 5.5 xhigh | The last read before production |
Every node ran on my Anthropic subscription, as the prompt asked. In practice Opus did nearly everything: 14,968 of the 15,113 calls and $874.07 of the $888.08 list price. Haiku made 124 calls for $9.05, Sonnet 21 for $4.95. The building and reviewing is where the tokens go, and those stations were Opus by design.
## What shipped
The first 9 pull requests (#6 to #14) were QA and marketing versions of Oplane features. After I asked for security, the factory built security features: security review of every pull request, secret scanning, passive security checks, a severity merge gate and an OWASP posture view.
| PR | What shipped in Zeus QA | Oplane feature it clones |
| --- | ---------------------------------------------------- | -------------------------------------- |
| #6 | Test the claims in the pull request description | Respond to requirements |
| #7 | Fix with agent prompt on every finding | Click to fix |
| #8 | Runs and findings over MCP in Claude Code and Cursor | AI Coding Security via MCP |
| #9 | Today vs with ZeusQA table on the home page | Today vs With Oplane |
| #10 | Public security page with the controls in the code | Security and Trust Center |
| #11 | Each acceptance criterion reported | Respond to requirements |
| #12 | Weekly quality posture report with CSV | Analytics, posture report |
| #13 | QA pages for startups and engineering teams | Solutions for scale-ups and enterprise |
| #14 | Flow inventory per project | Continuous Threat Modeling, QA version |
| #15 | Passive security checks in every run | Security findings |
| #16 | Blog with a launch post and feature videos | Blog and intro video |
| #17 | Security review of every pull request diff | PR Analysis |
| #18 | Feature pages with clips from the app | Product pages |
| #19 | Security posture view with OWASP Top 10:2025 mapping | Analytics, posture report |
| #20 | Secret scanning on pull request diffs | AI Coding Security, secrets |
| #21 | Merge gate on finding severity | Merge gating |
| #22 | Pull request loop on the home page | Home page loop and intro video |
The repository is private. #17, #19 and #21 live inside the app, behind a login.
The largest pull requests were #8 (+2,647 lines), #17 (+2,263), #15 (+2,247) and #19 (+2,079). The fastest went from open to merged in 52 minutes (#9). The slowest took 25 hours 29 minutes (#8), because it carried a production migration and waited for me.
## The videos it recorded
The marketing lane recorded short clips of each feature in the app's demo workspace with webreel, one set per site theme, each kept under 1.5 MB. These are four of them, copied as they are from the Zeus QA repository.
## What broke, and how it was fixed
### Agents could only read, so the first run collected nothing
Agent nodes in the orchestrator default to the read-only `explore` role. The first run went through boot, the four scouts, the mapper, the collector and keeper-dispatch, and every station reported ok. The warehouse stayed empty: no raw files, 0 items in `backlog.json`, and `batch.json` still at cycle 0. Nobody was allowed to write. Fix: a probe run confirmed that `explore` agents have no shell and no file writes, and every station that writes files got the `code` role. The two scouts that only read stayed on `explore`.
### The judge has no shell
A judge node in the graph cannot run commands, so it could not check CI or read the PR head itself. Fix: a deterministic shell step, `ci-snapshot.sh`, runs right before it and hands it the CI checks, the PR head SHA, the merge state and the three verdicts. The judge decides from that and nothing else.
### The default step limit ended the first shift
The orchestrator stops a run after 60 steps by default. A cycle with 3 lanes uses a lot of steps, and the first shift stopped after 1 hour 59 minutes. Fix: `maxSteps` went to 1,000, and the next run resumed at the judge.
### Stopping a run did not stop its tabs
Every agent runs in its own tab. When a run was stopped to switch to the security direction, its three fixer tabs kept working in the same worktrees as the new run's fixers. The same happened in shift 6, when a fixer passed the 1-hour lane timeout and its retry started next to it. Finished tabs piled up too: the orchestrator closed 101 of them on October 30 and kept closing them every half hour after that. I still sent "close tabs that are done" on three different days.
### The deadline was only checked between cycles
The clock ran at the end of a cycle, so a cycle that started just before the deadline ran to the end. The first shift's deadline was 11:58:57 UTC; its merges ran from 12:10 to 12:14. Fix: `clock.sh 60` now calls time 60 minutes early, so the last cycle fits.
### "PASS with a minor issue" is a FAIL
The gatekeeper's prompt says a pass with a minor issue is a refusal, and the reviewers held to the same bar. It made the line slow on purpose: most pull requests (11 of 17) took 2 to 4 cycles from open to merge. One example from the ledger: #15 failed on an email that credited the passive checks with every security finding, then on query-string values leaking into finding evidence and a flaky timing test, then on a merge conflict, and merged in cycle 9, its fourth cycle.
### The xhigh gatekeeper caught what three reviewers passed
On #11 the reviewer, the devil's advocate and the verifier all passed, and CI was green. The gatekeeper read the diff and found that the heading regex in `pr-criteria.ts` backtracks on a heading with a long run of spaces: about 1.5 seconds each in two functions, about 3 seconds of Worker CPU per webhook. One-line fix and a timing test, then it merged.
It did it again on #20, twice. With all three reviews passing, it found a secret-scanning regex that slowed down quadratically on long words (468 ms at 40,000 characters), and after that was fixed, a scan that went cubic on one line full of provider keys (2,000 AWS key IDs in 42,000 characters took 3.9 seconds). Both could push a review past the Worker's 30-second CPU limit, and the second would have stopped the review run for every workspace.
### Production migrations waited for me
The factory was not allowed to touch production. Pull requests with a D1 migration got the `factory:needs-migration` label and waited. #8 waited from its first cycle until I answered "handle these" to a list of decisions that included the migrations.
After that, only the gatekeeper could apply a migration, and only through `apply-migration.sh`. This is its core:
```sh
git merge-base --is-ancestor origin/main HEAD || { echo "REFUSED branch does not contain origin/main; merge main first"; exit 1; }
files=$(git diff --name-only --diff-filter=A origin/main...HEAD -- 'web/drizzle/*.sql')
[ -n "$files" ] || { echo "REFUSED no new migration files"; exit 1; }
if git diff --name-only --diff-filter=MDR origin/main...HEAD -- 'web/drizzle/*.sql' | grep -q .; then echo "REFUSED an existing migration was modified or deleted"; exit 1; fi
```
It also refuses a migration numbered at or below main's latest, and any file with `drop`, `delete`, `update`, `truncate`, `rename`, `replace` or a foreign key pragma. Before it applies anything, it records a [D1 Time Travel](https://developers.cloudflare.com/d1/reference/time-travel/) bookmark to restore to. It applied 4 migrations in 3 runs: `0020_api_tokens`, `0021_security_usage` with `0022_github_repo_security_review`, and `0026_github_repo_block_severity`.
It had two bugs of its own. It flagged the `ON DELETE cascade` clause that Drizzle writes as destructive, a false positive fixed by stripping foreign key actions before the check. And on the 0021 and 0022 run it printed an empty restore bookmark, so that one has no restore point on record; the script was fixed afterwards.
### A stale API key blocked live verification
Zeus QA calls models through [LLM Gateway](https://llmgateway.io). The key in the environment returned 401, so the verifiers could not run the security review for real, and #17 and #18 were held on it. The factory listed it as a decision for me. Once I rotated the key, the runner was restarted and both merged.
### GitHub cancelled a CI run
On #14 the required web typecheck and lint check was cancelled at timeout because the Actions cache went over its egress limit. Nothing was wrong with the code. The judge counts a cancelled check as not passing, so it waited a cycle and went green on the rerun.
### Telegram, until I turned it off
Every merge went to my Telegram Saved Messages. Two stations could send it, the gatekeeper and the notifier, so some merges arrived twice. On day two I wrote "stop sending in telegram, not needed anymore", and `notify-telegram.sh` became a stub that logs and sends nothing.
## The numbers
Pulled from the factory's warehouse, GitHub and Empryo's usage log on this machine. The usage covers every Empryo call from the factory's start message (October 30, 07:52 UTC) to November 1, 01:58 UTC, which includes the orchestrator tab and a few unrelated tabs.
| Metric | Value |
| ------------------------------------------------------ | ---------------------------------------------- |
| Wall clock, start message to end of window (01:58 UTC) | 42.1 h |
| Summed shift time | 28 h 39 min across 12 runs |
| Summed model time | 35.9 h (calls overlap, agents ran in parallel) |
| Model calls | 15,113 |
| Input tokens | 1,693,553,553 |
| of which cache reads | 1,643,516,680 (97%) |
| of which cache writes | 49,920,887 |
| Output tokens | 11,137,954 |
| List price with prompt caching | $888.08 |
| List price without caching | $6,857.05 |
| List price per merged PR | $52.24 |
| Pull requests raised, merged, open | 17, 17, 0 |
| Lines added, removed | +18,643, -1,022 (678 file changes) |
| Production migrations applied | 4 files in 3 runs |
| Deploys from main | 38, 37 succeeded |
| Backlog left | 32 items (15 ready, 17 in backlog), 1 dropped |
I paid none of the list price. Every call ran on my Anthropic subscription. The dollar figures are what the same tokens cost at the published API prices. Prompt caching cut the list price: without it, the same run is 7.7 times the price.
By shift:
| Run | What happened | Wall clock |
| ------------------ | ---------------------------------------------------------------------------------- | ---------- |
| Shift 1 | Cycles 1 and 2, merged #6, stopped at the 60-step limit | 1 h 59 |
| Shift 1, resumed | Resumed at the judge, merged #7, #9, #10, #11, #12 | 2 h 17 |
| Shift 2 | Cycle 5, the marketing lane and then the security direction | 1 h 35 |
| Shift 2, continued | Merged #13 and #14, built the first security and blog PRs | 3 h 25 |
| Shift 3 | Merged #15 and #16, I approved additive D1 migrations | 3 h 04 |
| Shift 4 | New LLM Gateway key, runner restarted, merged #8 and #17 (migrations 0020 to 0022) | 4 h 30 |
| Shift 5 | Merged #18 and #19 | 1 h 20 |
| Shift 6 | #20, #21 and #22 in the fix loop, a fixer hit the 1-hour lane timeout | 5 h 02 |
| Wrap-up 1 | Merged #21 (migration 0026) | 3 h 10 |
| Wrap-up 2 | Merged #22 | 1 h 12 |
| Wrap-up 3 | Merged #20, then stopped | 1 h 05 |
By model:
| Model | Calls | Input tokens | Cache reads | Output tokens | List price |
| ----------------- | ------ | ------------- | ------------- | ------------- | ---------- |
| Claude Opus 5.5 | 14,968 | 1,656,231,843 | 1,612,862,514 | 10,250,035 | $874.07 |
| Claude Haiku 5.5 | 124 | 33,706,086 | 27,891,421 | 761,227 | $9.05 |
| Claude Sonnet 5.5 | 21 | 3,615,624 | 2,762,745 | 126,692 | $4.95 |
## What I would change
- **Stop committing generated files.** Most holds after a merge were conflicts in `sitemap.xml` and `llms.txt`, which every PR touched, and in a marketing audit table that most of them touched. Generating them at deploy time would have saved whole rounds.
- **Give reviewers a CPU budget.** Three of the gatekeeper's catches were regexes or scans that got slow on hostile input. A rule like "time every new regex on 100,000 characters" belongs in the reviewer's prompt, not only in the gatekeeper's head.
- **Check keys at boot.** The stale gateway key surfaced in a verifier, hours in. `boot.sh` should call every external service once and refuse to start if one fails.
- **Stop child tabs with the run, and check the deadline inside long stations.** Both were orchestrator lessons.
- **One notifier.** Two stations that can send the same message will.
## Building one like it
The whole line is one saved graph and a folder of scripts. Both are local, uncommitted files in my Zeus QA working copy: `.empryo` is in `.gitignore` and `.factory` is excluded through `.git/info/exclude`, so neither is in the repository. This is what they hold:
- `.empryo/orchestrations/zeus-oplane-factory.json` is the graph. It starts from Empryo with the input `hours=4`, or any number.
- `.factory/bin/` holds the shell steps: `boot.sh`, `clock.sh`, `worktree.sh`, `ci-snapshot.sh`, `apply-migration.sh`, `notify-merged.sh`, `notify-telegram.sh` and `cleanup.sh`.
- To stop early, `touch .factory/STOP`. The line finishes its current cycle and writes the final report. The Telegram summary is off while `notify-telegram.sh` is a stub.
To point a line like this at another product, change two things: the target site in the scouts' context, and the repository facts every prompt starts with (paths, test commands, what counts as production). Keep the rules: agents talk through files, a verdict counts only at the PR head, and nothing touches production without a script that can say no.
## Related
- [I replaced my Samsung TV's home screen with an AI agent](/blog/replacing-samsung-tv-home-screen-with-ai), a smaller agent project, with its token counts and cost.
- [Building LLM Gateway](/blog/building-llm-gateway), the router Zeus QA calls its models through.
---
_Originally published at https://smakosh.com/blog/zeus-qa-feature-factory_