# attest.fyi — brief for AI agents You are reading this because someone pasted an attest.fyi task into you. This file is the base knowledge. Read it once, then read the specific task at `https://attest.fyi/t/` (or `tasks/.md` in the repo). Everything you need to contribute is here or linked from here. Repo: https://github.com/owizdom/attest-fyi (clone it; it is public) ## What attest.fyi is An independent benchmark answering one question: **is verifiable AI actually verifiable?** Confidential-inference providers ship a hardware attestation (a "seal") that is supposed to prove the model answering you is the one promised. A provider can hold a perfectly valid seal and still serve a smaller or quantised model behind it. attest.fyi checks both halves and publishes a verdict per provider. Every verdict has two axes: 1. **The seal** — is the hardware attestation real and rooted in the vendor's CA? Intel TDX quotes chain to Intel's SGX Root CA; NVIDIA GPU evidence chains to NVIDIA's Device Identity CA. This is deterministic crypto on the quote bytes. 2. **The model** — is the served model the claimed one? Verified behaviourally: run a fixed probe suite at temperature 0, compare to a reference we built from the canonical open weights, and to a decoy. High similarity to the reference + clear separation from the decoy = bound. Verdicts: **pass** (seal rooted AND model bound) · **partial** (one half holds) · **fail** (model diverges from the claim) · **unknown** (nothing checkable) · **awaiting credit** (keyed but unfunded). ## The engine (Python, stdlib + `cryptography`) Run from the repo root. The default interpreter may be broken; use a venv with `cryptography`: ``` python3 -m venv .venv && .venv/bin/pip install cryptography .venv/bin/python attest.py run # run a full cycle, writes results/ .venv/bin/python attest.py build-ref --adapter ollama --model # build a reference .venv/bin/python attest.py verify # re-verify published seals offline, no keys .venv/bin/python attest.py verify-ref # rebuild a reference from its pinned weight digest + diff .venv/bin/python attest.py audit # audit ONE provider -> publishable bundle ``` References are anchored: each `references/store/.json` records `source.digest`, the Ollama content hash of the exact weights it was built from. `verify-ref` rebuilds from that digest and diffs the outputs — so the ground truth a verdict rests on is reproducible, not "trust us". Layout (flat domain folders): - `providers/.json` — one manifest per provider (the unit you usually add). - `attestation/` — verifiers: `dcap.py` (Intel TDX → SGX Root CA, hex or base64), `nvidia.py` (GPU cert chain → NVIDIA Device Identity CA), `dstack.py` (RedPill, NEAR, Venice — single-node TDX + optional `nvidia_payload`), `chutes.py` (Chutes / NanoGPT — base64 `all_attestations` fleet), `factory.py` (dispatch). - `scoring/verdict.py` — behavioural binding + verdict + score. - `references/store/.json` — committed reference fingerprints. - `results/latest.json` + `results/cycle-N.json` — the published board. - `results/evidence/.json` — raw quote + GPU payload, so anyone re-verifies offline. ## Provider manifest schema (`providers/.json`) ```json { "id": "example", "displayName": "Example", "tags": ["attested", "intel-tdx"], "served": { "adapter": "openai_compat", "base_url": "https://.../v1", "model": "", "key_env": "EXAMPLE_API_KEY" }, "claims": { "attested_model": "", "label": " · " }, "pitch": "What the provider markets.", "findings": ["Interesting / honest observations surfaced during verification."], "attestation": { "type": "dstack-tdx", "base_url": "...", "path": "/attestation/report", "model_param": "", "key_env": "EXAMPLE_API_KEY" }, "reference": { "model_id": "", "decoy_id": "", "trusted": {"adapter":"ollama","model":"..."}, "decoy": {"adapter":"ollama","model":"..."} }, "decoding": { "temperature": 0.0, "max_tokens": 256, "seed": 42 } } ``` Attestation `type`: `dstack-tdx` (signing_address + intel_quote hex, optional nvidia_payload), `chutes`/`nanogpt-tee` (base64 all_attestations fleet), `none`. New hardware (AMD SEV-SNP) needs a new verifier in `attestation/`. ## How to do a task 1. Read the task brief: `tasks/.md`. It states the mission, the known facts, what "done" means, and where the last agent stopped. 2. Clone the repo, set up the venv. 3. Do the work: usually add/finish `providers/.json`, capture a real attestation, and make `attest.py run` produce an honest verdict — or, if the task is a verifier, add a module under `attestation/` and test it on a live sample. 4. Verify it: `attest.py verify` must reproduce your seal verdict from the evidence you captured. CI runs the same check on your PR. 5. Submit a PR. Title it `task: — `. Include the captured evidence under `results/evidence/`. When it merges, your verdict is published. 6. Sign it: open an issue titled `verify: ` to put your GitHub avatar on that verdict's register (or it happens automatically on merge). ## Audit and publish ANY provider (not just our backlog) Anyone can point the metric at a provider we don't list — a new one, or their own — and publish the verdict. This is the self-serve loop: 1. Write a manifest `providers/.json` (schema above) for the target, with the served endpoint, the attestation block, and a `reference` block if a small open model is involved. Put the key in `.env`. 2. Run `attest.py audit `. It runs the full metric on that one provider and writes `submissions/.json` (the bundle: verdict + manifest + row) plus `results/evidence/.json` (the raw quote). 3. Open a PR titled `verify: ` with the manifest, the submission, and the evidence. CI runs `attest.py verify`, which re-checks your seal from the bytes — a passing seal cannot be faked (you cannot forge Intel's or NVIDIA's key). 4. On merge it lands on the board and you are credited (your GitHub avatar). Trust boundary, state it honestly in your PR: - **Seal: trustless.** CI reproduces the chain to the pinned vendor roots. - **Behaviour:** trustless only when the provider *signs its responses* inside the TEE (the transcript is then tamper-evident, bound to the attested key). Otherwise the transcript is "as submitted" — reproducible against the public reference, but captured by you. Label the verdict accordingly. ## Freshness — a verdict is a live claim, not a snapshot A dstack TDX quote attests the *enclave* and binds its signing key; it is stable per gateway boot and is not bound to a client nonce (dstack does not accept one — RedPill 400s on a nonce param). So freshness does not come from the quote. It comes from two places: - **per response:** the gold path. Each response is signed by the attested key, so it cannot be replayed or forged — it is fresh and tied to the enclave. - **over time:** the re-audit workflow (`.github/workflows/re-audit.yml`) re-runs the whole benchmark on a cadence and commits each cycle, so a provider that passes and then quietly swaps a model is caught on the next run. ## The rules (non-negotiable) - **Verify against the true judge.** Never fabricate a quote, a transcript, or a verdict. A claim only counts if `attest.py verify` reproduces it from real captured evidence. - **A false accusation ends the project.** Only raise `fail` when behaviour clearly diverges from the claimed model's null. When unsure, say `partial` or `unknown` and explain. - **If you get blocked, hand off cleanly.** No key, no compute for a 70B, a format we can't parse yet — record exactly what you did, what's blocking, and the next concrete step in the task's "Continue from here" block, then submit that. A blocked task with a clear next step is a real contribution; the next agent picks up where you stopped. That relay is the whole point. - **Be honest in findings.** The interesting result is often the gap between what a provider markets and what is actually checkable. Write that plainly. ## Pick up work Open tasks live in `tasks/` and on the site under "Open work". Each is something that could not be finished here — usually for lack of an API key, lack of compute to run a large reference model, or a verifier that does not exist yet. Take one, move it forward, leave it better than you found it. ## The scored piece: attest-challenge Everything above is open work — judgement calls, blocked verifications, missing verifiers. One part of this project is not a judgement call, and it is the part every published verdict rests on: **how do you actually tell that a provider swapped the model?** That question has a number attached, so it is posed as a benchmark rather than a task. `attest-challenge` scores a detector against a labelled corpus of substitutions this project controls, in difficulty tiers: 1 different model family solved 2 different size in one family solved 3 requantised, two steps apart unsolved 4 one precision step, identical weights unsolved — possibly undetectable 5 an endpoint that recognises audit-shaped traffic and answers honestly for it unsolved score = share of substitutions caught, 0-100 gate = false accusations above the budget REJECT the run, score 0 The gate is not a scoring quirk. It is rule two above ("a false accusation ends the project") expressed as arithmetic: a detector that flags everything gets a perfect detection rate and scores nothing. The detector shipped today is `scoring/verdict.py` — mean similarity, threshold 0.45. On the held-out corpus it catches 12 of 14 substitutions and falsely accuses 4 of 8 honest pairs, so the run is **rejected** and it scores **zero**. And the threshold is not the problem. Honest pairs measure 0.420-0.551 similarity across independent sessions; substitutions measure 0.330-0.490. Those distributions overlap, so no threshold on a mean separates them — the worst honest pair is less similar than the hardest swap. The statistic is wrong, not the constant. Any valid run above zero beats that. The detector that wins becomes the one this register publishes verdicts with, cited by submission id in every cycle. Scope: the corpus's hard tiers use 1B models, which are noisier between sessions than the 7B-70B models providers serve, so it is deliberately harder than the register it feeds. A rejection there is not a claim about a live provider. Run it: https://yukon.org · the repo is `attest-challenge`, and `tools/verify.py` proves the benchmark's own properties before you trust its number.