When CI can't keep up with your agents
One of our projects opened 73 pull requests in a single day, and burned through a month of CI minutes doing it. So we moved every check onto our own machines.
One day last week, a single project of ours opened 73 pull requests and shipped to its main branch 67 times. Not a big team working overtime: one engineer making the calls, and agents that never clock off.
That same day burned 3,186 GitHub Actions minutes. The month came to about 3,600, on a plan that includes 2,000, and ninety-two percent of it came from that one repository.
Nothing was broken. Our CI was simply built for a human pace, and we don’t work at one anymore.
What an agent-paced day looks like
Every run fanned out into eight jobs, and every job started with a full pnpm install. Then each merge ran the same suite again, on the push to main, against code that had passed it minutes earlier.
Most teams never open 73 PRs in a day, so a setup like that never shows its cost. Ours can, and on busy days they do. Agents don’t get tired: they open small, focused PRs all day and all night, which is exactly what we want from them. Hosted CI bills every one of those runs as if it were a rare event.
Do the math: some 140 runs, eight jobs each, a few minutes of setup per job. Most of that bill was downloading dependencies, not testing anything.
The machines were already there
Our agents work on dedicated boxes (how that runs, day to day). Each one holds every checkout, a warm package store and each repository’s pinned toolchain. Every check a hosted runner performs (format, lint, typecheck, test, build) a box can run faster, because nothing has to be fetched first.
So we stopped renting runners for checks and made the boxes the CI.
One moment per kind of check
The trick isn’t running everything everywhere. Each kind of check runs at exactly one moment: the cheapest one that still catches the problem in time.
| Moment | What runs | Where |
|---|---|---|
| commit | message format, staged-state guards (under 2 s) | box |
| push | format, lint, typecheck: affected projects only | box |
| merge | every unit test, integration test and build | box, one gate at a time |
| release | everything, end-to-end included, on clean runners | GitHub Actions |
| after release | deploy only, no re-run | GitHub Actions |
Push runs static checks and nothing else. Agents push at checkpoints rather than after every commit, so a test suite there would buy little early feedback for a lot of memory. Tests belong to the merge, where they run once per PR.
A gate that reports back to GitHub
The merge gate is one command. It checks that the working tree is exactly the PR’s head, takes a box-wide lock so gates queue instead of fighting over memory, runs the repository’s pre-merge task, re-checks the tree in case anything changed mid-run, and posts the result where a reviewer will see it: as a commit status on the PR (every finished run also counts toward our live stats).
gh api "repos/$OWNER/$REPO/statuses/$HEAD_SHA" \
-f context=devbox/pre-merge \
-f state=success \
-f description="tests + build passed on the box · 4m 12s"Statuses are free, and they show on the PR page just like a check. Our merge command refuses unless the PR’s current head carries a green one. A new push changes the SHA, so the gate runs again for exactly what merges.
Each repository opts in by naming its tasks after the moments:
[tasks.pre-push]
description = "Format, lint and typecheck what changed"
run = ["pnpm check:format", "pnpm check:lint", "pnpm typecheck"]
[tasks.pre-merge]
description = "Every test, then every build"
depends = ["test", "build"]No task, nothing runs at that moment. One command lists exactly what a repository checks, and there’s no YAML to keep in sync with it.
What still runs on GitHub
Some things really do need a clean machine. Two PRs can each pass alone and break together, and a release should be proven on a runner that has never seen the code before. So the release gate still runs on GitHub, every job in parallel, before the production branch moves. The deploy afterwards trusts that green SHA instead of testing it again. Scheduled jobs and anything that needs a secret the boxes don’t hold stay on GitHub too.
On top of that, one rule: any change that can spend Actions minutes needs a human’s approval first, with an estimate attached (minutes per run × runs per day). Cheap-looking CI changes are how we got here.
What we gave up
- Branch protection. The gate is a commit status, not a protection rule. Agents can only merge through the command that checks it, and a person can still merge from the web UI, on purpose.
- A pristine runner at merge time. A box is shared by every agent working on it, which is why the gate re-checks the tree after it runs and throws the result away if anything moved. The release gate’s clean runners catch the rest.
- Someone else’s uptime. A box can die like any machine, and keeping it alive is now our job. So none of them is special: each is provisioned from code (the server in Terraform, packages and services in Ansible, dotfiles in chezmoi, secrets encrypted in git), and a fresh, identical one takes a couple of minutes of anyone’s time.
The result
Speed was never the problem; checking at that speed was. Now the checks cost nothing extra and start instantly against a warm store, so moving faster never means a change gets checked less. GitHub’s minutes go to the handful of runs that genuinely need a clean machine. And when an agent opens its 73rd PR of the day, the only thing anyone notices is the progress.