Multi-Agent Bug Triage with a Human Gate

Muzamil Faisal, an engineer at Big Immersive, built a bug-triage system in which four agents hand work to one another, search a ticket tracker and a crash log, and pause for a person before anything high-severity is written. It runs on LlamaIndex's AgentWorkflow, on a hosted model or fully locally on a 7B one, and it scores 15 out of 15 on its own eval set. This is what we learned about where the model should decide and where the code should.

The problem

Feedback about a product arrives from everywhere at once: store reviews, a Discord server, a support inbox, an in-app form. Most of it is not a bug. Of what is, a good share duplicates something already in the tracker or something three other users reported that morning, and a good share is too vague to act on. Somebody has to read all of it, decide which is which, work out how bad the real bugs are and whose they are, and write the ticket. That somebody is usually an engineer, and the work is exactly the kind an agent should be able to do: read, search, compare, decide, write.

The part that makes people nervous is the last step. A system that files tickets on its own will, sooner or later, file a critical billing ticket that is wrong, or close a data-loss report as a duplicate of something unrelated. The design question is not whether a model can triage — it can — but where the boundaries are that a model must not be able to cross.

Four agents, one job each

The workflow is four FunctionAgents, each with one decision to make and a small set of tools to make it with. Control moves only when an agent calls the handoff tool, and each agent's can_handoff_to list restricts where it may send the work.

• Intake classifies the item — bug, feature request, praise, spam, support question — and extracts device, OS, app version, reproduction steps and language. Anything that is not a bug stops here.
• Investigator decides whether a bug is new, a duplicate, or too vague to act on. It searches reports already processed in this batch, searches the tracker, and queries the crash log.
• Triage assigns severity, component and owner to a confirmed new bug, from release data and a CODEOWNERS file.
• Writer produces the artifact: a new ticket, a +1 comment on an existing one, or a clarification reply to the user in their own language. It is the only agent with side effects.

Spam and praise never reach the investigator. A duplicate skips triage entirely and goes straight to the writer for a comment. The graph is what a human triager already does; the agents are just the parts written down.

Routing is the model's decision, nudged hard

Each agent chooses where to hand off — that is what makes it an agent rather than a pipeline. But a small model left to infer the next step from a system prompt will, often enough, stop early or hand off to the wrong place. So every record_* tool, the tool an agent calls to commit its decision, ends its return string with an explicit instruction: NOW call handoff(to_agent='investigator', reason='bug report needs duplicate check'). The routing is still the model's call; it is just told, at the exact moment it has finished its job, what the next job is. That one sentence is most of what makes a 7B model follow the graph reliably. Delete it and watch what happens — it is one of the experiments the repository suggests.

Structured output through tools, not prompting

Every decision an agent makes is recorded by calling a tool whose arguments are validated by a Pydantic model: record_intake, record_investigation, record_triage. If the model sends a severity that is not one of the four allowed values, or a confidence that arrives as a string, the tool does not fail. It returns the validation error as its result, and the model corrects itself and calls again. This is structured output without asking the model for JSON: the schema is enforced at the boundary where the model can see and fix the problem, and the shared workflow state only ever holds validated records.

The guards live in code

The interesting engineering is not in the prompts. It is in the tools' refusal to accept certain claims, whatever the model says.

• The duplicate-claim guard. When the investigator searches the tracker or the batch, every hit and its similarity score is written into the item's state. A duplicate verdict is accepted only for a candidate that appeared in this item's own search results with a score of at least 0.65. The model cannot promote a weak match, and it cannot name a ticket it never saw. If it tries, the tool says so and tells it to either cite a strong hit or change the verdict to new.
• The regression guard. A closed ticket whose fixed_in version is older than the version the user is reporting from cannot be a duplicate; the bug came back. The tool rejects the claim and tells the agent to file a new ticket that references the old one.
• The actionability guard. A verdict of new needs at least one of a device, an app version, reproduction steps, or a crash signature confirmed in telemetry. Without any of them engineering cannot act, so the tool sends the agent back to needs_info and the writer drafts a clarification reply instead.
• The ticket-id guard. If the writer tries to comment on a feedback id instead of a ticket id — a mistake small models make — the tool resolves it from the investigation or refuses with instructions.

None of this depends on the prompt being obeyed. That is the point. A prompt is a request; a tool that returns an error is a fact the model has to deal with.

The human gate is a tool, not a prompt

The boundary that matters most is the write. create_ticket is the only tool that files a ticket, and it holds the gate: if the recorded severity is high or critical, or the investigator's confidence is below 0.7, the tool itself pauses the workflow with ctx.wait_for_event(HumanResponseEvent, …) and writes nothing until a person answers. The person sees the proposed title, severity, component, owner and evidence, and replies yes, no, or a corrected severity. The ticket records who approved it.

Because the gate is inside the tool, the model cannot skip it, and the writer's prompt does not have to mention it. Anything below the floor is filed automatically, which is where autonomy is safe. On the sample set, a double-charge report pauses as high and a lost-progress report as critical; both wait for a human, and a reviewer can raise the first to critical before it is filed.

An interrupted run does not ask twice. If a person already decided on an earlier attempt, the decision is carried over and applied without a second prompt. In the dashboard, approvals survive a server restart for the same reason.

Memory across a batch

The first report of a crash is new. The second, from another user on another device an hour later, is a duplicate of the ticket the first one produced — but that ticket did not exist when the batch started, so a tracker search will not find it. The runner keeps a second vector index of reports already processed in this run and inserts a node after every item, so search_recent_reports lets the investigator match the second, third and fifteenth report to the first. A duplicate remembers the ticket it resolved to, so a chain of duplicates still lands on one ticket rather than on each other.

Evidence, not opinions

Severity is argued from numbers. The crash-log tool aggregates telemetry by signature and reports events, distinct users, versions, platforms and the top of the stack: the shop crash in the sample set shows 28 events from 28 users on version 2.4.1, all in ShopScreen.renderFeatured. Release data says 61 per cent of users are on that version and the next code freeze is days away. The CODEOWNERS file says the shop belongs to the monetisation team. The triage agent's rationale has to cite these, and the ticket carries them as evidence, which is what the engineer who picks it up actually wants to see.

An eval, so changes are measured rather than felt

The sample set is fifteen feedback items for a fictional mobile game, each with an expected category, outcome, duplicate target, component and minimum severity. triage eval runs the whole inbox with the gate auto-approved and scores every item on what actually happened, not on what the model said. A prompt change, a model swap or a new guard is a number before and a number after.

The local 7B model, qwen2.5:7b, scores 15 of 15 at best of three, in about 39 seconds per item, for nothing. GLM-5 over OpenRouter scored 14 of 15 on its first run; the miss became one of the guards above, and it now scores 15 of 15, in about 27 seconds per item, for roughly 20 cents a run. The larger model calibrates severity across all four levels where the small one rates everything high. Both follow the graph.

What a small model taught us

Writing for a 7B model is a forcing function. Prompts became numbered steps with one job per agent and an explicit ending. Tools became the teacher: the search tool marks each hit STRONG or weak so the model does not have to interpret a float, and the coercion layer accepts a list sent as a string, steps sent as "1. 2. 3." and numbers sent as words, because a small model will send all of those. Everything that made the system work on a weak model made it cheaper and more predictable on a strong one.

From an engine to a product

The engine is a Python package with a CLI. Around it we built a multi-tenant workspace — projects with their own agents, tasks, runs, approvals, memory, knowledge, files and integrations — on FastAPI and PostgreSQL with pgvector. Each project's tracker, batch memory and uploaded documents are separate vector collections filtered by project on every query, so an agent cannot see another project's tickets. Runs are a Postgres job queue claimed under a per-project advisory lock: runs within a project stay ordered, projects run in parallel, and a run parked at the human gate releases its worker. Live events reach the browser over LISTEN/NOTIFY and WebSockets. It deploys as three Railway services.

It is a reference implementation on sample data — a fictional game, a fictional tracker — not a client deployment, and we say so. What it demonstrates is the shape we now use for agent work: control flow and safety in code, judgment in the model, and an eval before anything is called done.

Run it yourself

Clone the repository, pull two models into Ollama, and run triage run FB-014 to watch a double-charge report reach the human gate; run triage eval to score the whole inbox. Replacing the sample tracker with Jira or GitHub touches one file.

If you are scoping an agent system — one that reads, searches, decides and writes into something that matters — this is the conversation we would want to have first: which decisions belong to the model, which boundaries belong to the code, and how you will know when a change has made it better.