Cyber Transcript Sweeps Need Search-Method Receipts
By wGrow Project Team ·
The Volume Problem in Incident Response
Picture a post-incident review that starts on a narrow slice of logs and keeps expanding — cyber-eval flags turning up in usage data until the search covers the entire production transcript corpus. That scenario tells you everything about how the review process has to change once you’re at that scale. Nobody reads hundreds of millions of transcripts. Not with a team of fifty. Not with five hundred. Say a careful manual review takes five minutes a transcript — enough to read it, place it in context, and decide if it’s real. One reviewer clears roughly 90 a day; five hundred reviewers, doing nothing else, clear about 45,000 a day. Point that workflow at two hundred million transcripts under a 30-day incident window and you’d need roughly 74,000 reviewers working full-time on nothing but this. Once you cross into nine-figure transcript counts, manual review stops being a staffing problem — it becomes a mathematical impossibility.
So you deploy agents. You point large language models at the corpus and ask them to flag anything that looks like malicious cyber-eval behaviour: reconnaissance patterns, exploit scaffolding, requests that map onto known attack chains. At that scale, agentic triage may be the only practical way to narrow the corpus down to something a human team can actually work. Whether it justifies the trade-off is a separate question, and it turns on numbers you have to go measure: the miss rate against a known sample, the sampling frame the sweep actually covered, and the cost of a missed incident. An LLM classifier has a false-negative rate. Unless you measure it, you don’t actually know what that trade-off cost you.
That’s where a second-order problem kicks in — one most teams underestimate. Finding the incident is step one. Proving you ran a comprehensive search is the actual engineering work. Hand someone a short list of flagged transcripts pulled from a corpus in the hundreds of millions, and that list isn’t the deliverable — it’s a claim. Claims need evidence. Without a record of how the search actually ran — what was included, what was excluded, how the agent performed against a known sample — there’s no way to tell whether you found the incidents, or just found the incidents the agent happened to be good at spotting that day. Those are two different statements. Only one of them survives an audit.
Query Provenance Is Not a New Concept
This isn’t a new problem wearing an AI costume. It’s a classical audit problem, decades older than the models causing the current panic about it.
We ran into it directly back in 2018, building an Elasticsearch-backed audit trail for a Singapore public-sector client’s access logging system — who touched what, when, from which role. The client’s auditors didn’t want a summary report of anomalous access events. They wanted the query that produced it. Every report we shipped carried the exact search parameters attached: time window, index pattern, role filters, the boolean logic that split “flagged” from “excluded.” If an auditor couldn’t rerun our query against the same index snapshot and land on the same result, the report bounced. Full stop.
None of that had anything to do with machine learning. It came from a basic compliance principle: a finding is only as trustworthy as the search that produced it. An agent reading a transcript today is doing the same job our Elasticsearch queries did in 2018 — a stochastic layer sitting on top of a structured search against a dataset. The abstraction changed. The compliance requirement didn’t move an inch. You still have to prove how you asked the question before anyone trusts the answer.
Architecting the Agentic Audit Trail
Translate the 2018 discipline into agentic terms and four concrete artefacts fall out — things you log as you go, not things you narrate afterward.
The sampling frame. Record exactly which transcripts entered the sweep and which got filtered out before an agent ever read a token. Excluded anything under a certain length, outside a date range, or from a specific product tier? That exclusion logic is itself a finding. A sampling frame that quietly drops a chunk of traffic because of a truncation bug isn’t a footnote — it’s the difference between “we searched everything” and “we searched most of it.”
The query set. Log the exact system prompt, temperature, and model version used to run the sweep. Sounds pedantic until you remember that a mid-sweep prompt edit — even a one-line clarification — changes the classifier underneath it. Two halves of the same sweep run under two different prompts aren’t one search. They’re two searches stapled together, and you need to disclose the staple.
Misses. Run the sweep against a labelled sample of transcripts you already know contain incidents, and write down what the agent failed to catch. A false-negative rate isn’t a nice-to-have metric — it’s the clearest measure of what your sweep didn’t find. “We searched hundreds of millions of transcripts,” without a stated miss rate attached, is a volume claim dressed up as a coverage claim.
Reruns and failures. Log every API timeout, every rate-limit backoff, every retry. A failed call is an unread transcript. If your pipeline silently treats a timeout as “no incident found” instead of “not evaluated,” your coverage number is fiction.
Tracing Agent-on-Agent Evals with Langfuse
We apply the same discipline internally, at a smaller scale, when we use agents to evaluate other agents. wGrow builds agent crews that run multi-step workflows — research, drafting, code review — and a second layer of evaluation agents grades those workflows for correctness and safety before anything ships.
We instrument this with Langfuse. We trace every observable part of the evaluation run before it resolves to a pass or fail: the rubric inputs, the intermediate judgments the agent actually emits, the tool calls, the model and prompt versions in play, and the specific span of the target transcript attached to the final verdict. This isn’t observability for chasing down latency bugs. It’s observability because we don’t trust a pass/fail verdict we can’t reconstruct.
Our internal rule is blunt: if we can’t pull the eval trace back out of the database and rerun the reasoning chain, the eval didn’t happen. A grade with no trace is just an opinion with a timestamp attached. We’ve killed eval runs before — thrown out the verdicts entirely — because the trace didn’t persist correctly and we had no way to show our own team, let alone a client, how the grade was reached. Agent-grading-agent only earns trust when the grading agent’s homework is checkable.
Defining Deterministic Human Handoffs
None of this replaces human judgment. It defines where human judgment starts.
Agents are a triage layer, not an adjudication layer. They cut hundreds of millions of transcripts down to a manageable shortlist; a security engineer makes the final call on each item in it. The audit trail has to capture the exact handoff point — which transcript ID, what timestamp, which model produced the flag a human is now sitting with.
When that engineer overturns a flag — calls it a false positive — that override is a database write, not a Slack message. It has to persist with the same rigour as the original flag. And it feeds back into the system: overridden flags become the few-shot examples that tune the next version of the search prompt. Skip that loop and you’re not improving the sweep. You’re just running the same mistakes at higher volume.
Version Control for Discovery Methods
Transcript volumes aren’t shrinking. Every product cycle that adds usage adds transcripts, and the next order of magnitude is a billion, not hundreds of millions. Search methodology is compliance methodology now — whether or not the compliance frameworks have caught up enough to say so out loud.
The operational mandate is simple: version control your search agents and their prompts with the same seriousness you apply to production application code. Tag the model version. Diff the prompt changes. Pin the sampling frame to a specific dataset snapshot — the same way we pinned Elasticsearch queries to index snapshots back in 2018.
None of this comes free. Logging sampling frames, prompt diffs, and miss rates on every sweep adds real overhead, and for a low-stakes internal search it probably isn’t worth the ceremony. But once the sweep feeds a security or compliance decision, the overhead isn’t a tax on the job — it is the job. Sweep a massive dataset and come back without the sampling frame, the query set, the miss rate, and the rerun log behind it, and what you’ve got isn’t an audit. It’s a guess that happened to run on a GPU.