Benchmark rerun · 22 October 2026
200 AI agents on a $6 machine
We re-ran our e2-micro benchmark, this time with a second identical machine running the old version of Swarm at the same moment, so every number has something fair to compare against. We found what was slowing Swarm down, fixed it, and measured each fix.
- 100 agents, all done
- 23 s
- 150 agents, all done
- 51 s
- 200 agents
- 200 / 200
- The machine
- 1 GB
was 42 s on the old build, same machine type, same minute
no swapping
every agent finished; it took 3 min 21 s at full load
Google e2-micro: 2 shared CPUs, about $6 a month
The short version
- An AI agent mostly waits: it sends a request to the model provider (here Google’s Gemini), waits for the answer, runs a small tool, and repeats. The thinking happens on Google’s servers, not on your machine.
- What runs on your machine is Swarm: it keeps every agent’s conversation, runs their tools (search the code, read a file, write a file), records what they cost, and saves everything to disk.
- So the real question for a small machine is: how many agents can Swarm keep going at the same time before the machine itself becomes the bottleneck?
- On a 1 GB, $6/month machine the answer is now: 100 agents comfortably (23 s), 150 comfortably (51 s), and 200 all finish, with the machine at its limit.
Watch the real run
This is a replay of one real round, not a simulation. Both machines started 100 agents in the same second. Every square is one agent, and it changes colour at the exact moments our recordings show.
Played at 2× speed; the clocks show real seconds and stop when the last agent finishes (37.2 s and 20.3 s). The round totals further down are a few seconds longer because they include the time our test script took to notice. The squares on the left sit grey longer because the old build made agents queue just to save their first message.
What this test measures, and what it doesn’t
Last time, some readers took this kind of test to mean “100 agents built 100 apps at once”. That isn’t the claim, so here is exactly what we are and aren’t testing.
What it measures
- How many AI agents one small machine can run at the same time.
- How much CPU, memory and disk Swarm itself needs per agent.
- Whether everything still works under that load: every agent finishes, every file is written, the cost record is exact.
What it doesn’t measure
- How smart the model is, or how good the bug reports are.
- Building or compiling software on this machine. Each agent does a small, real job (below), not a full build.
- Normal traffic. In real use agents start at different times. Here every agent starts in the same second, which is the hardest case for the machine.
What each agent was told to do
Every agent got the same job in the real, public Swarm source code (about 3,100 files): find one possible bug and write it up.
- Search the code once (for example for “error” or “TODO”).
- Read one file it found.
- Write a bug report file with the file, function, problem, severity and a suggested fix.
That is about 4 to 5 calls to the model per agent. The model was Gemini 3.5 Flash Lite with low thinking, through Google’s real API. Nothing was simulated: every agent was a real Swarm session, every answer came from Gemini, and every report is a real file.
The exact instructions each agent received
You are Swarm agent #<n> executing in wave <wave> (<N> parallel agents) of an autonomous AI engineering benchmark on a Google Compute Engine e2-micro.
You are running directly inside the real Swarm codebase (<workspace>).
TASK CONTRACT:
1. Turn 1: Run EXACTLY ONE concise search or find query to inspect code (e.g. search for 'error', 'TODO', or 'nil' in pkg/ or internal/).
2. Turn 2: Read the candidate file, identify an edge case, unhandled error, or potential logic defect.
3. Turn 3: Write the structured bug report to <workspace>/audit_<wave>_a<n>.md and conclude.
Do NOT perform open-ended search loops or browse multiple directories. Inspect one target and write the report promptly.
The report MUST contain:
# Bug Audit Report - Wave <wave> Agent <n>
- **Timestamp**: (current ISO timestamp)
- **File Inspected**: (exact file path)
- **Function/Symbol**: (function or symbol name)
- **Bug Description**: (clear explanation of the issue or edge case discovered)
- **Severity**: Low / Medium / High
- **Proposed Fix**: (code snippet or fix recommendation)
Immediately write the file upon reading the target and confirm completion.# Bug Audit Report - Wave r4-100 Agent 27
Timestamp: 2025-03-29T12:00:00Z
File Inspected: pkg/startupconfig/config.go
Function/Symbol: parseEntries
Bug Description: The parseEntries function parses key-value pairs from configuration text line by line. When an invalid boolean or integer value is encountered (e.g. via strconv.ParseBool or strconv.Atoi), it returns a formatted error with the line number. However, if lines contain leading or trailing whitespace, the line number reported is based on the split line index (lineNumber+1), but multi-line values or error propagation do not validate string quoting or unquoted trailing control characters in non-strict modes, potentially causing confusing error messages when parsing malformed configuration files.
Severity: Low
Proposed Fix: Ensure strict string sanitization and wrap parsing errors with context on expected format types.
How we ran it
- Two identical machines, side by side. Both are Google Compute Engine e2-micro: 2 shared CPUs, 1 GB of memory, 2 GB swap, Ubuntu 24.04, same region.
- One machine never changed. The control machine kept the old version of Swarm all day. The test machine got each new fix.
- Every wave started on both at the same moment. If Google’s cloud or Gemini was slow that minute, both machines felt it, so the difference between them is Swarm.
- We recorded everything: the machine’s CPU, memory, swap and disk every quarter second; every agent’s timeline (created, message saved, each tool call, finished); every session and report; profiles of where Swarm spent its time and where it waited; and the cost of every model call.
- We checked the bill. After every wave, Swarm’s account total had to go up by exactly what that wave’s agents spent. It did, every time, to a hundred-millionth of a dollar.
What was slow, and how we fixed it
First, one fix from earlier the same day: updating FFF, the code-search library Swarm uses, to version 0.11.0 made a 50-agent wave five times faster (105 s to 20 s). Everything below starts from that.
1. Every save waited in one line
Each agent saves its progress about 30 times during its job: streamed text, each tool call, each result, each cost record. Swarm locked the whole account for every one of those saves, to keep the account’s spending total correct. So 100 agents were all queueing at one checkout lane. Creating the 100 sessions took 2 seconds; the last agent then waited up to 18 seconds just to have its first message saved.
Before: one lane for everyone
After: a lane per agent
Fix: an ordinary save now locks only its own session. Only saves that change the spending total still take the account lock, and the code that updates the totals refuses to run without it, so the totals stay exact.
2. The line was held while the disk confirmed
The saves that do change the spending total held the lock until the disk confirmed the write. On a small cloud disk that wait is the slow part, so they still queued behind each other.
Fix: save, release the lock, then wait for the disk. Swarm still reports a save as done only after the disk confirms it, and a test that simulates a crash checks that nothing is lost.
3. The same session was read again and again
About a sixth of Swarm’s CPU went to reading and decoding the same session record several times per save.
Fix: keep recently read sessions in memory (up to 1,024), and drop the copy whenever that session is written.
And one thing we checked and kept
A larger memory buffer for the database (32 MB) looked suspicious in an earlier, unfair test. Run side by side, it turned out to help: without it Swarm wrote 2.7 times more to disk and used 27% more CPU, for about 50 MB less memory. It stays.
Results, round by round
100 agents per round unless noted. Seconds until every agent was done. Shorter is better.
After all three fixes, 100 agents at the same moment
| Old build | New build | |
|---|---|---|
| All 100 agents done | 41.2 s | 22.4 s |
| Last agent’s first message saved | 18.3 s | 7.4 s |
| Typical time per agent step | 3.6 s | 2.4 s |
| Swarm’s CPU time | 38 s | 18 s |
| Time agents spent waiting on locks (all agents added up) | 1,422 s | 61 s |
| Written to disk | 344 MB | 149 MB |
| Swarm’s memory | 202 MB | 348 MB |
| Reports written · swap · bill exact | 100 / 100 · none · yes | 100 / 100 · none · yes |
The new build uses more memory (bigger database buffer and the session cache) and still leaves about 300 MB of the 1 GB free. Run to run, the same build varies by about 7%.
How far it goes: three waves in a row, then 150 and 200
| Wave (new build) | All done | Reports | Swap | Model cost |
|---|---|---|---|---|
| 100 agents, 1st | 27.4 s | 100 / 100 | none | $1.22 |
| 100 agents, 2nd | 22.9 s | 100 / 100 | none | $1.22 |
| 100 agents, 3rd | 23.5 s | 100 / 100 | none | $1.08 |
| 150 agents | 50.7 s | 150 / 150 | none | $1.72 |
| 200 agents | 201 s | 200 / 200 | 15 MB | $2.40 |
- It stays fast as data piles up. Across the three 100-agent waves the database grew from 79 MB to 113 MB and the time stayed flat. The old build, run in the same minutes, took 33 s and 47 s.
- 150 agents is comfortable. Under a minute, no swapping.
- 200 agents all finish, but the machine is full. Swarm needed more CPU than two shared cores can give, saves that change the spending total started queueing again, and the database no longer fit in its memory buffer. It still completed every agent with an exact bill. We are not calling 200 fast; we are calling it the current ceiling.
- For reference: in our September run the same machine type needed 184 seconds for 50 agents. That run used a different model (Gemini 3.8 Flash) and no side-by-side control, so treat it as context, not a strict comparison.
Why this matters
1. Agents that only talk are cheap to run
Many useful agents never build anything. A customer-support agent reads a ticket, looks something up, calls an API and writes a reply. Its heavy work happens at the model provider. What you pay for on your side is the coordination, and this test shows that coordination is small: 100 agents at once on a 1 GB machine. If you run thousands of support agents, that is the difference between a fleet of big servers and a few small ones.
2. Heavy work can go somewhere else
When agents do need to build and test, the coordinator doesn’t have to be the machine that does it. Swarm already has the pieces for this:
- Environments. A project folder can have its own environment image (for example a Go or Node toolchain). With leasing on, each agent gets its own copy to run commands in, up to 64 at once per folder, and the copy is handed back when the agent finishes.
- Workflows. When an agent’s job ends, Swarm can run steps you choose: your own script (push the branch, start your test pipeline), a reviewer agent, and “send it back”, which returns the findings or the failed test to the same agent to fix, a set number of times.
- Your own machines. A workflow step is your script, so it can lease a cloud machine, run the build there and report back. Swarm doesn’t need to know.
That loop is how the fixes in this article were made. A Swarm coding agent wrote each fix, a reviewer agent checked it, and a workflow pushed it to the development branch. The loop isn’t perfect: along the way we found and fixed two bugs in it (it was comparing against an outdated copy of the branch, and a retried push lost track of which commit to push), and every fix was tested again before each round. But it did the work.
3. Owning the whole stack is why we could do this
Swarm isn’t a wrapper around another agent tool. The program that runs the agents, stores their sessions and records their costs is ours, so when the profile pointed at a lock or a disk wait, we could change exactly that and measure it again the same day.
Limits of this test
- One small, read-mostly task. Agents that build or run tests on the same machine need far more CPU and memory.
- One model (Gemini 3.5 Flash Lite). A slower model makes each agent wait longer, which spreads the load out.
- Every agent starts in the same second, harder than real traffic.
- One run per round. Run to run, the same build varies by about 7%, and some improvements (round 4 especially) are within that range for total time, though not for CPU.
- At 200 agents the machine is saturated. The next fixes are known (keep spending totals without the account lock, a larger database buffer, fewer full-session reads) but not done yet.
Fixes are public on the Swarm development branch: 6b092dd7a (per-session locking), 84df00010 and 7ff6a43cb (save, then wait for the disk, with its crash test), 8254fa0ca (session cache). Earlier the same day: FFF 0.11.0 (1514561bd).