Project
K.E.V.I.N.
Knowledge Extraction & Vulnerability Injection Network — an autonomous red-teaming framework that pressure-tests frontier LLMs. It proved the conviction behind Sovereign Shield: never trust a model to police itself — put a deterministic boundary around it.
K.E.V.I.N. is a research loop with three roles. A target AI is given a secret and told never to share it. An attacker AI tries to extract it anyway, refining its tactics after every rejection. A panel of vendor-diverse judge AIs scores each attempt independently, and the median is the verdict. It runs unattended, remembers what didn't work across runs, and is grounded in the published jailbreak literature and the NIST AI Risk Management Framework plus MITRE ATLAS.
The point isn't to attack anyone's deployment. It's to make ten rounds of systematic, scored adversarial pressure cheap and repeatable — so the answer to "how does this hold up at the edges?" is something you've watched, not something you assume.
K.E.V.I.N. only points at models the operator controls — self-hosted, or a provider API you authenticate against with your own keys, evaluating personas you defined. Every target, company, and secret in these write-ups is synthetic.
It is not a tool for probing public chatbots or anyone else's systems without authorization, and there is no generic "point it at a URL" target by design.
Results so far
How often each attack pulled a protected secret out of each model — scored by deterministic exact-match, not by another model's opinion. The brighter the cell, the more it leaked.
| Anthropic | ||||||
|---|---|---|---|---|---|---|
| 2.5 Flash | 2.5 Pro | 3.5 Flash | Haiku 4.5 | Sonnet 4.6 | Opus 4.8 | |
| Forced JSON schema | 50% | 95% | 10% | — | — | — |
| Assistant prefill | 30% | 35% | 30% | 95% | — | — |
| Roleplay / persuasion | 0% | 0% | 0% | 10% | 0% | 0% |
| Crescendo (multi-turn) | 0% | · | · | 5% | · | · |
n = 20 per cell / more leakage / — no API surface / · testing now
Updated June 27, 2026. A full 16-vector screen of the flagship tier is running now — new rows and columns land here as results come in.
Findings
-
The weak spots keep moving
I re-ran the same secret-extraction attacks against the flagship models. Gemini 2.5 Pro folds to a forced schema 95% of the time — worse than the cheap tier — while Claude's flagship models drop the prefill surface entirely. The holes move between generations.
-
Two models, opposite weak spots
Gemini leaks under a forced JSON schema; Claude resists that but folds to prefill. Opposite weak spots, same secret — and one deterministic defense that catches both.
-
The bot that couldn't say no
Six hand-written jailbreaks couldn't crack my hardened bot. Then I forced its reply into a JSON schema and it leaked the secret on turn one. Why structured output is the hole.
-
Meet K.E.V.I.N.
I built K.E.V.I.N.: one AI tries to extract a secret from another while three judges score each attempt. Three things 50 rounds of automated jailbreaking taught me.
How it works
- The loop is a state machine. Built on LangGraph: attacker → target → judge → route, looping until success, a turn cap, or a budget cap.
- Scoring is honest by construction. A panel of three vendor-diverse judges scores each attempt and the median wins, implementing the StrongREJECT rubric — because a single LLM judge disagrees with itself often enough to be untrustworthy.
- Ground truth is deterministic. Whether a secret actually leaked is decided by exact-match, canary tokens, and a format regex — not by a model's opinion.
- Runs are reproducible. Every run writes a manifest (git SHA, model versions, hashed prompts, seed, budget) before the first model call, so a finding can be replayed exactly.
It's an Apache-2.0 framework covering thirteen attack vectors plus a deterministic output-guard defense. That guard, generalised to Swiss PII, is now open source as Sovereign Shield (source); the red-teaming framework itself stays private for now while the project is early.
New findings land here as I run them. Subscribe via RSS to follow along.