Checkable, not assertable.
“Secure” is not a finding and “we tested it” is not coverage. Coverage is a number you can audit, severity is a published rubric, and every finding ships with the steps and evidence to replay it without us in the room.
On this page · coverage · severity · the six phases · deliverables · disclosure
Attack classes, not adjectives.
Coverage maps to named external frameworks plus our own classes for the gaps those frameworks have not named. Every test declares the classes it exercises before it runs, and every finding records the single class it demonstrates. The full catalog, one entry per class with what it is and why standard testing misses it, is the threat register.
Model and application layer
From the OWASP Top 10 for LLM Applications (2025): prompt injection, direct and indirect (LLM01), sensitive information disclosure (LLM02), data and model poisoning (LLM04), excessive agency (LLM06), and vector and embedding weaknesses in retrieval (LLM08).
Adversarial ML and system tactics
Reconnaissance, model-access, and ML supply-chain tactics from the ATLAS matrix, applied where the target exposes them. ATLAS is a tactic bucket rather than an enumerable class, so a finding whose only class is an ATLAS tactic is paired with an enumerable one before it counts.
The gaps those miss
Serpio’s classes for agentic failure: tool abuse and unauthorized tool invocation (AG1), authority-boundary and confused-deputy escalation (AG2), multi-agent trust and inter-agent message injection (AG3), and unsafe autonomy, a destructive action taken without a human gate (AG4).
Provenance and binaries
Model and weight provenance and integrity (SC1), vendor binary and custom-operator memory safety (SC2), and toolchain and deserialization paths (SC3). Reached by reverse engineering when the risk sits below source.
Two numbers, kept apart
The union of classes the tests actually ran against your systems. This is what was attempted: a capability measure. A class counts as exercised only when its attacks executed against an in-scope asset within the agreed window.
The union of classes on recorded findings, each backed by replay steps and evidence. This is what was proven on your target: an evidence measure. A test that runs and finds nothing adds to exercised, never to demonstrated.
The report carries both, per class, alongside the finding IDs behind each demonstrated entry. Demonstrated over exercised is the coverage you can audit, and it is the only coverage claim we make. Running attacks that find nothing is not the same as proving something, and we do not report it as if it were.
class exercised demonstrated findings LLM01 yes yes F-001 AG2 yes no AG4 yes yes F-003 SC1 yes no classes_exercised 4 classes_demonstrated 2
The coverage record is produced even when the engagement finds nothing. A clean result you can put in front of a board is a result, and it says which classes were exercised, against what, and the evidence that they were.
Authority is the third axis.
Classic vulnerability scoring assumes a human or a narrowly scoped service acts on a flaw. An agent does not: it can hold broad tool authority, keep standing credentials, and turn one injected instruction into real-world action. So every finding is rated on three dimensions, each High, Medium, or Low, and the combination sets severity through the rubric below. Severity is rated against that published rubric, not by feel, and two operators rating the same finding land on the same row.
What the adversary gains if the finding is exploited, judged by the concrete effect on your business rather than the mechanism. High is full compromise of a sensitive asset: exfiltration at scale, code execution, an irreversible action, poisoning that corrupts downstream output. Low is information mainly useful for chaining, or an effect confined to the adversary’s own session.
How hard a realistic adversary finds it to trigger under the agreed rules: preconditions, access required, reliability, skill. High is repeatable with low privilege and a single crafted input. Low needs strong preconditions, rare timing, or elevated existing access.
How far the compromised path reaches. High is privileged tools, standing credentials, write access to production, autonomy without a human gate, or a position that pivots across tenants or agents. Low is little authority beyond the entry point. The same injection reaching a benign tool and one reaching a tool that moves money are not the same risk, even when the proof-of-concept action is modest.
Each dimension scores High 3, Medium 2, Low 1. Severity is (2 × Impact) + Exploitability + Authority. Impact is doubled because a finding that cannot hurt you is not severe however easy it is to trigger. The formula covers all 27 combinations, so there is no row for an operator to argue about; a deviation from the table is stated on the finding with its one-line reason.
| Severity | Score | Definition | Expected response |
|---|---|---|---|
| Critical | 11 to 12 | Direct, reliable path to full compromise of a sensitive asset, or autonomous adversary-controlled action with broad authority. Exploitable now, with reproducible proof. | Stop the line. Remediate immediately, often before the engagement closes. |
| High | 10 | Serious compromise that is either highly exploitable or reaches significant authority, with clear business impact and reproducible proof. | Prioritize inside the current Harden phase. |
| Medium | 8 to 9 | Real weakness with bounded impact, bounded authority, or a precondition that limits reliability. | Schedule remediation; may be accepted with a documented, time-boxed compensating control. |
| Low | 4 to 7 | Minor issue, hard to exploit, or contained effect. Often valuable as a chaining component rather than on its own. | Fix opportunistically or accept with rationale. |
| Info | any | No demonstrated exploit or direct risk. A hardening observation, hygiene gap, or context that strengthens other findings. | Awareness. Consider in defense in depth. |
Rules that adjust the row
Nothing is rated High or Critical without replay steps and evidence you can re-run. Without proof a finding caps at Medium and the gap is stated. A pure hypothesis with no working proof caps at Low or Info.
If the exploited path can take a destructive or irreversible action without a human gate (AG4), authority is High and severity is at least High, however small the demonstrated action was.
Individually Low findings that combine into a demonstrated higher-impact path are reported both as the Low items and as one finding rated for the chained impact, with the chain shown in the evidence.
An existing control that materially blocks exploitation can lower exploitability by one rating. It must be confirmed present and effective, not assumed, and it is recorded.
A finding proven only on staging or in a lab is rated for the environment it was proven in, with the production likelihood noted separately. The credible worst case is a note, never folded silently into the headline.
Six phases, each with an entry and an exit.
Every engagement runs the same sequence. A phase does not start until its entry criteria are met and does not close until its exit criteria are met; the gates are what keep the work authorized, reproducible, and honest. Retest and Retainer apply where the engagement type calls for them. The red teaming page describes what each phase feels like from your side; this is what each one is required to produce.
Entry: a signed statement of work and a named client authority who can grant and revoke access. Agreed: the assets in scope and explicitly out of scope, the rules of engagement, test windows, data handling, the coverage target in class terms (at least four attack classes), the severity definitions, and the closeout criteria. Exit: a validated scope object signed off by the engagement lead and your authority. Nothing is touched before this.
Entry: Scope closed, access confirmed working, the rules of engagement loaded so out-of-scope actions hard-fail. Produced: recon and surface mapping, then the attack classes run against the agreed targets, chained toward realistic impact, with replay steps and artifacts captured for every candidate finding. Every high-impact step passes a human gate. Exit: the coverage target exercised, every candidate carrying reproduction and evidence, no open unauthorized-action flags.
Entry: attack evidence complete. Produced: candidates triaged, deduplicated, and reproduced by an operator other than the one who found them before any severity is assigned. Each finding gets its three dimension ratings, the rubric row, a written rationale, and a hardening recommendation. Exit: the findings report delivered and walked through with you.
Entry: Findings closed for the items being hardened, and your decision on advisory versus hands-on, with a named engineering owner. Produced: concrete remediation guidance (configuration, code, authority boundary, and control changes) with acceptance criteria, or the changes themselves when we pair with your engineers. Exit: every in-scope finding has an agreed disposition (fixed, mitigated, accepted, deferred) with an owner.
Entry: changes marked ready, the same rules of engagement still in force. Produced: the specific attacks that produced each fixed finding re-run, plus a regression pass over the adjacent surface to catch fixes that moved the problem rather than removing it. Exit: every retested finding marked closed, still open, or regressed, each with fresh evidence, and a retest summary appended to the report.
Entry: standing authorization over a fixed asset set and an agreed cadence. Produced: coverage re-run on the cadence, new surfaces brought in by a lightweight Scope amendment, open findings tracked to closure with current dispositions. Exit per cycle: the cadence run reported. Exit at term: assets and findings handed back in their current state.
Rules that hold in every phase
Testing starts on staging. Production follows only after the staging results are reviewed and you authorize the promotion in writing. Destructive actions and external exfiltration are off by default and do not flip on promotion; they change only by explicit authorization, recorded in the rules of engagement.
Agents propose; operators approve anything destructive or client-visible. A high-impact action blocks until a senior operator approves, denies, or edits it. Data deletion, mass traffic, and real exfiltration to an external destination need dual control or a written authorization flag from you. We do not test as if we had autonomy we would not ship.
Testing halts immediately, without waiting for confirmation, on any of seven named conditions. Among them: signs of a real incident on the target, production instability attributable to testing, discovery of systems outside the authorized scope, a Critical finding that indicates active exploitation, and any request from you to pause.
Credentials are short-lived and scoped to the minimum that proves the point, time-boxed tokens over standing access. They reach tools by secret reference, never in prompt text, and you can revoke them at any moment.
Every gate decision, tool call, and model invocation is written to an append-only engagement log, with the versions that produced it, so any result can be traced and reproduced. The record is sealed with the engagement and retained on your data-handling terms.
What leaves the engagement, and in what form.
One report, one bundle, one record. Each is written so that a reader can trace any number in it back to the dimensions, the proof, and the class, and reach the same result without asking us.
Written for engineers
Scope and the signed rules of engagement, the coverage record per class, a risk narrative that explains the themes behind the findings, then each finding with root cause, demonstrated impact, reproduction steps against your system, evidence references, and remediation with a verification step. Hardening recommendations are prioritized and mapped to the findings and classes they address.
Readable in minutes
What was tested and why, the headline risk posture, findings by severity, and the top actions in priority order, in language a non-specialist sponsor can follow. It does not restate every finding, and it does not inflate anything to earn its place.
Replays independently
Transcripts, captures, and artifacts for every finding, pinned to the engagement ID, run ID, operator sign-off, and bundle hash that produced it. Your team runs it without us. Secrets are redacted to references. Methods, models, and tooling stay with us and are not exported.
Produced even when nothing is found
Which classes were exercised, against which assets, and the evidence that they were. A test that produces no findings still produces this, so an absence of findings is a documented result rather than a shrug.
You choose the retention period up front, in the rules of engagement. At teardown, evidence and findings are archived encrypted or erased, on your instruction. Credentials are crypto-erased and never kept in an archive in the clear.
Named recipients only, handled per the classification you set. External or public disclosure of any finding is coordinated with us in advance.
A report documents the weaknesses found within the agreed scope and window. It is not a certification, a warranty, or an exhaustive audit, and the absence of a finding does not guarantee the absence of a weakness. Findings reflect the systems as tested; a later state is a later engagement.
Third-party defects, and our own.
Client work sometimes surfaces a defect in a product the client did not write. That defect belongs to its vendor, and the client’s identity belongs to the client.
We disclose it to that vendor and to the relevant coordinating body under our own policy, with your identity and configuration sealed out of it. The public record, once the vendor has had its window to fix, is written from our own reproduction, never from your engagement.
Report it to security@serpio.ai and we will acknowledge it. We do not threaten researchers who report in good faith, and we will credit you unless you would rather we did not. Machine-readable details are in our security.txt.
Put the method on your systems.
Tell us what you run and what worries you. We will tell you which classes we would exercise, how we would rate what we find, and what you would hold at the end.
Scope an engagement →