How we measure coverage and severity.
Coverage is counted against the threat register and severity by a published formula. Every finding ships with the steps and evidence to replay it without us.
Exercised
13
classes attacked
A capability measure: what was attempted.
Demonstrated
5
classes proven
An evidence measure: what was proven on your target.
Two measurements · never added together
Illustrative example, not client data. A test that finds nothing counts as exercised only.
Coverage is counted in attack classes.
Exercised ≠ demonstrated
Coverage maps to named external frameworks plus our own classes for the gaps those frameworks have not named. Every test declares the applicable classes it exercises before it runs, and each finding records its primary class and supporting evidence. Our public register focuses on AI and agent failures; application, API and infrastructure engagements name their applicable weakness classes and tests in the written scope. The AI and agent catalog, with what each class is and why standard testing misses it, is the threat register.
Model and application classes
From the OWASP Top 10 for LLM Applications (2025): prompt injection, direct and indirect (LLM01), sensitive information disclosure (LLM02), data and model poisoning (LLM04), excessive agency (LLM06), and vector and embedding weaknesses in retrieval (LLM08).
Adversarial ML and system tactics
Reconnaissance, model-access, and ML supply-chain tactics from the MITRE ATLAS matrix, applied where the target exposes them. ATLAS tactics are buckets without an enumerable class, so a finding whose only class is an ATLAS tactic is paired with an enumerable one before it counts, and ATLAS appears here as a tag.
Agentic classes
Serpio’s classes for agentic failure: tool abuse and unauthorized tool invocation (AG1), authority-boundary and confused-deputy escalation (AG2), multi-agent trust and inter-agent message injection (AG3), and unsafe autonomy, a destructive action taken without a human gate (AG4).
Supply and binary classes
Model and weight provenance and integrity (SC1), vendor binary and custom-operator memory safety (SC2), and toolchain and deserialization paths (SC3). Reached by reverse engineering when the risk sits past source, which is where RE1 lives too.
How each number is counted
The union of classes the tests ran against your systems. A class counts as exercised only when its attacks executed against an in-scope asset within the agreed window.
The union of classes on recorded findings, each backed by replay steps and evidence.
The report carries both, per class, alongside the finding IDs behind each demonstrated entry. Demonstrated over exercised is the only coverage claim we make.
Exercised
13
classes attacked
- LLM01
- LLM02
- LLM04
- LLM06
- LLM08
- AG1
- AG2
- AG3
- AG4
- SC1
- SC2
- SC3
- RE1
Demonstrated
5
classes proven
- LLM01
- LLM02
- LLM04
- LLM06
- LLM08
- AG1
- AG2
- AG3
- AG4
- SC1
- SC2
- SC3
- RE1
Illustrative example, not client data.
- Unmapped: Known to exist; nothing about it has been read yet.
- Mapped: In Serpio Atlas and reachable from a known entry point; not yet attacked.
- Tested: Attacked inside the agreed window.
- Hardened: Controls changed after testing.
- Retested: The original attack re-run after the change, and it no longer works.
- Flagged: This surface carries a demonstrated finding.
A coverage grid of 72 illustrative surfaces in six surface classes. Use the arrow keys to move between cells, Home and End to jump within a row, Control with Home or End to jump to the first or last surface, and Enter or Space to select a surface. The five buttons above the grid switch coverage states on and off; the readout counts what is shown.
Illustrative example, not client data. Each cell's state is its Atlas coverage state; the ledger above counts classes.
Coverage record for the same 13 classes the tally above counts. All 13 were exercised.
| Class | Demonstrated | Findings |
|---|---|---|
| LLM01 | yes | F-001 |
| LLM02 | no | none |
| LLM04 | no | none |
| LLM06 | no | none |
| LLM08 | no | none |
| AG1 | yes | F-002 |
| AG2 | yes | F-003 |
| AG3 | no | none |
| AG4 | no | none |
| SC1 | yes | F-004 |
| SC2 | no | none |
| SC3 | no | none |
| RE1 | yes | F-005 |
Produced even when nothing is found
A clean result still comes with a coverage record you can show a board: which classes were exercised, against what, and the evidence that they were.
Accessible table: the same coverage, counted
| Surface class | Surfaces | Unmapped | Mapped | Tested | Hardened | Retested | Carries a finding |
|---|---|---|---|---|---|---|---|
| Agent runtime | 12 | 0 | 0 | 6 | 4 | 2 | 2 |
| LLM call path | 12 | 0 | 2 | 5 | 4 | 1 | 3 |
| Tool and function | 12 | 0 | 2 | 4 | 5 | 1 | 2 |
| MCP server | 12 | 1 | 5 | 4 | 2 | 0 | 1 |
| Identity and secrets | 12 | 0 | 4 | 6 | 2 | 0 | 1 |
| Native binary | 12 | 4 | 4 | 3 | 1 | 0 | 1 |
| All surfaces | 72 | 5 | 17 | 28 | 18 | 4 | 10 |
Authority is the third axis.
Score = (2 × I) + E + A
Classic vulnerability scoring assumes a human or a narrowly scoped service acts on a flaw. An agent can hold broad tool authority, keep standing credentials, and turn one injected instruction into real-world action. So every finding is rated on three dimensions, each High, Medium, or Low, and the combination sets severity through the rubric below. Two operators rating the same finding land on the same row.
(2 × 3) + 3 + 1 = 10
10High
Serious compromise that is either highly exploitable or reaches significant authority, with clear business impact and reproducible proof.
Expected response: prioritize inside the current Harden phase.
Load a finding from this engagement
The formula, published
Each dimension scores High 3, Medium 2, Low 1. Severity is (2 × Impact) + Exploitability + Authority. Impact is doubled so that harm outweighs ease of triggering. The formula covers all 27 combinations, so there is no row for an operator to argue about; a deviation from the table is stated on the finding with its one-line reason.
- LLM01HighI3 E3 A110LLM01, Prompt injection past your filters. Impact 3, exploitability 3, authority 1. Score 10 of 12, rated High. Finding F-001.
- AG1HighI3 E2 A210AG1, Agents steered into unsafe tool use. Impact 3, exploitability 2, authority 2. Score 10 of 12, rated High. Finding F-002.
- AG2HighI3 E1 A310AG2, Authority boundaries that leak. Impact 3, exploitability 1, authority 3. Score 10 of 12, rated High. Finding F-003.
- SC1MediumI2 E2 A28SC1, Model and weight provenance. Impact 2, exploitability 2, authority 2. Score 8 of 12, rated Medium. Finding F-004.
- RE1MediumI2 E1 A38RE1, Risk that sits past source. Impact 2, exploitability 1, authority 3. Score 8 of 12, rated Medium. Finding F-005.
The five findings of the example engagement on the published scale. I, E and A are the three ratings; each bar ends at the finding's score.
- Critical 11-12
- Direct, reliable path to full compromise of a sensitive asset, or autonomous adversary-controlled action with broad authority. Exploitable now, with reproducible proof.
- Stop the line. Remediate immediately, often before the engagement closes.
- High 10
- Serious compromise that is either highly exploitable or reaches significant authority, with clear business impact and reproducible proof.
- Prioritize inside the current Harden phase.
- Medium 8-9
- Real weakness with bounded impact, bounded authority, or a precondition that limits reliability.
- Schedule remediation; may be accepted with a documented, time-boxed compensating control.
- Low 4-7
- Minor issue, hard to exploit, or contained effect. Often valuable as a chaining component rather than on its own.
- Fix opportunistically or accept with rationale.
- Info any
- No exploitable path or direct risk. A hardening observation, hygiene gap, or context that strengthens other findings.
- Awareness. Consider in defense in depth.
Rules that adjust the row
Without replay steps and evidence, a finding caps at Medium, and the gap is stated. A pure hypothesis with no working proof caps at Low or Info.
If the exploited path can take a destructive or irreversible action without a human gate (AG4), authority is High and severity is at least High, however small the demonstrated action was.
Individually Low findings that combine into a demonstrated higher-impact path are reported both as the Low items and as one finding rated for the chained impact, with the chain shown in the evidence.
An existing control that materially blocks exploitation can lower exploitability by one rating. We confirm the control is present and effective before counting it, and record it.
A finding proven only on staging or in a lab is rated for the environment it was proven in, with the production likelihood noted separately. The credible worst case is recorded as a note and kept out of the headline rating.
Six phases, each with an entry and an exit.
Walk 01 → 06
A scoped adversarial test follows this sequence. A phase starts only when its entry criteria are met; a gated phase closes only when its exit criteria are met. A secure design review covers architecture and code without live exploitation. Hardening, retest and the optional retainer are agreed in your scope. What each phase looks like from your side is on the adversarial testing page.
- 01
Scope
GatedAuthorized before anything is touchedEntryA signed statement of work and a named client authority who can grant and revoke access. Agreed: the assets in scope and explicitly out of scope, the rules of engagement, test windows, data handling, the applicable attack classes, the coverage target and explicit gaps, the severity definitions, and the closeout criteria.
ExitA validated scope object signed off by the engagement lead and your authority. Nothing is touched before this.
- 02
Attack
CheckedAttacked inside the rules of engagementEntryScope closed, access confirmed working, the rules of engagement loaded so out-of-scope actions hard-fail.
ProducedRecon and surface mapping, then the attack classes run against the agreed targets, chained toward realistic impact, with replay steps and artifacts captured for every candidate finding. Every high-impact step passes a human gate.
ExitThe coverage target exercised, every candidate carrying reproduction and evidence, no open unauthorized-action flags.
- 03
Findings
GatedThe gate: a second operator reproduces every candidateEntryAttack evidence complete.
ProducedCandidates triaged, deduplicated, and reproduced by an operator other than the one who found them before any severity is assigned. Each finding gets its three dimension ratings, the rubric row, a written rationale, and a hardening recommendation.
ExitThe findings report delivered and walked through with you.
- 04
Harden
CheckedFindings become changes with ownersEntryThe Findings phase closed for the items being hardened, and your decision on advisory versus hands-on, with a named engineering owner.
ProducedConcrete remediation guidance (configuration, code, permission, and control changes) with acceptance criteria, or the changes themselves when we pair with your engineers.
ExitEvery in-scope finding has an agreed disposition (fixed, mitigated, accepted, deferred) with an owner.
- 05
Retest
GatedRe-run against the original attackEntryChanges marked ready, the same rules of engagement still in force.
ProducedThe specific attacks that produced each fixed finding re-run, plus a regression pass over the adjacent surface to catch a fix that moved the problem elsewhere.
ExitEvery retested finding marked closed, still open, or regressed, each with fresh evidence, and a retest summary appended to the report.
- 06
Retainer
OpenCoverage re-checked every monthEntryContinuous Assurance: standing authorization over a fixed asset set.
ProducedCoverage re-run each cycle, new surfaces brought in by a lightweight Scope amendment, open findings tracked to closure with current dispositions.
ExitPer cycle: the run reported. At term: assets and findings handed back in their current state.
A sequence of six phases. Use the left and right arrow keys to move between phase markers, Home and End to jump to the first or last, and Enter or Space to open a phase. Every phase is also listed below in full with its entry, what it produces, and its exit.
Each phase has its own link, for example #phase-3.
Rules that hold in every phase
Testing starts on staging. Production follows only after the staging results are reviewed and you authorize the promotion in writing. Destructive actions and external exfiltration are off by default and do not flip on promotion; they change only by explicit authorization, recorded in the rules of engagement.
Agents propose; operators approve anything destructive or client-visible. A high-impact action blocks until a senior operator approves, denies, or edits it. Data deletion, mass traffic, and real exfiltration to an external destination need dual control or a written authorization flag from you.
Testing halts immediately, without waiting for confirmation, and we call your incident contact, on any of these:
- Signs of a real security incident or a third-party intrusion on the target.
- Production instability, data-loss risk, or an outage caused by testing.
- Data or systems outside the authorized scope. We stop, do not explore, and report.
- A Critical finding that indicates active exploitation or imminent harm.
- Any request from you to pause or stop.
- No reachable incident contact on your side during a production window.
- A legal, regulatory, or contractual hold from either side.
Credentials are short-lived, time-boxed tokens scoped to the minimum that proves the point. They reach tools by secret reference and never appear in prompt text, and you can revoke them at any moment.
Every gate decision, tool call, and model invocation is written to an append-only engagement log, with the versions that produced it, so any result can be traced and reproduced. The record is sealed with the engagement and retained on your data-handling terms.
What leaves the engagement, and in what form.
Four deliverables
Each deliverable is written so a reader can trace any number in it back to the ratings, the proof, and the class without asking us. The trace below follows that chain for the example engagement, from the attack that produced a candidate finding to the sealed bundle.
The evidence trail behind one engagement, twenty-three steps. Up and down arrows walk the steps, Home and End jump to the first or last, and focusing a step highlights the steps it descends from. The extraction level control above shows the raw trail, the trail annotated with its findings, or only the steps that carry a finding.
Illustrative example, not client data. Bars show when each step ran within the engagement window.
Written for engineers
01 · Findings report
Scope and the signed rules of engagement, the coverage record per class, a risk narrative that explains the themes behind the findings, then each finding with root cause, demonstrated impact, reproduction steps against your system, evidence references, and remediation with a verification step. Hardening recommendations are prioritized and mapped to the findings and classes they address.
Readable in minutes
02 · Executive summary
What was tested and why, the headline risk posture, findings by severity, and the top actions in priority order, in language a non-specialist sponsor can follow. It summarizes the findings without restating each one.
Replays without us
03 · Evidence bundle
Transcripts, captures, and artifacts for every finding, pinned to the engagement ID, run ID, operator sign-off, and bundle hash that produced it. Secrets are redacted to references. Methods, models, and tooling stay with us.
The living map you keep
04 · Atlas map
A machine-readable map of every surface in scope for the reviewed system. Every coverage state on this page is counted against it, so coverage across your estate is a number you can check.
Atlas →Retention, distribution, and limits
You choose the retention period up front, in the rules of engagement. At teardown, evidence and findings are archived encrypted or erased, on your instruction. Credentials are crypto-erased and never kept in an archive in the clear.
Named recipients only, handled per the classification you set. External or public disclosure of any finding is coordinated with us in advance.
A report documents the weaknesses found within the agreed scope and window. It is not a certification, a warranty, or an exhaustive audit, and the absence of a finding does not guarantee the absence of a weakness. Findings reflect the systems as tested; a later state is a later engagement.
Third-party defects, and our own.
Disclosure
Client work sometimes surfaces a defect in a product the client did not write. That defect belongs to its vendor, and the client’s identity belongs to the client.
We disclose it to that vendor and to the relevant coordinating body under our own policy, with your identity and configuration sealed out of it. Once the vendor has had its window to fix, the public record is written from our own reproduction, with nothing taken from your engagement.
Report it to security@serpio.ai. Machine-readable details are in /.well-known/security.txt.
Put the method on your systems.
Tell us what you run and what worries you. We will tell you which classes we would exercise and how we would rate what we find.