How we build the registry
A dataset is only as trustworthy as its weakest row. Here's exactly how we source, verify, and publish every incident — and what we refuse to do.
The VERBATIM contract
Every incident in this registry obeys five rules:
- A public source URL is required. No source, no row. Period. We link to vendor forums, named reporters, affected-user posts, and news outlets — primary sources where possible, secondary where validated.
- Amounts are as reported. "$72,000" means the source reported $72,000. We
never round up, never annotate with "estimated," and never backfill a missing dollar figure.
Undisclosed amounts carry
amount_usd: nullandverified: false. - Dates are as reported. We record the incident date (or report date, if the incident date is unclear) as the source states it. If only a year is known, we record the year and mark the record as partially verified.
- No surveys. No incidence rates. We do not publish claims like "X% of teams experience runaway spend." Vague rates are how dataset trust dies. If we cannot source each row individually, we do not ship an aggregate.
- Verified flag means everything is confirmed. A record is
verified: trueonly when the amount, date, cause, and provider are all confirmed by the source. If any field is approximate,verified: false— so downstream users can filter confidently.
The sipi.bot codebase has a file called fix_fabricated_benchmarks.py. It exists
because we once shipped a survey-based incidence claim ("312 teams surveyed, 67% had incidents")
that was not sourceable. We scrubbed it. This methodology page is the permanent commitment that
it won't happen again — every number carries a source, or it doesn't ship.
How an incident enters the registry
- We find a credible public report. Vendor community forums (OpenAI, GCP), Reddit megathreads (r/LangChain, r/googlecloud), named tech journalism (The Register, AlphaSignal), and affected-user posts (LinkedIn, dev.to) are all acceptable. Anonymous forum posts without supporting detail are not.
- We normalize the fields. Amount →
amount_usd(float or null), date → ISO month, cause → one of the five normalized categories (retry loop, prompt injection, unattended session, leaked credential, failed cap), provider → normalized vendor name. - We map the cause to a rule. Each incident is tagged with the sipi.bot rule type that would have stopped it — per-transaction cap, daily total, velocity limit, merchant allowlist, category rule, or time-of-day. This is the conversion mechanic: it turns fear into a checklist.
- We regenerate. Adding an incident to
data/incidents.pytriggers a rebuild of every artifact: all format files (JSON/CSV/JSONL/min), all hub pages, all OG cards, all sitemaps, and IndexNow pings. One source of truth.
Can I contribute?
Yes. Open a PR against the dataset repo with:
- A public source URL (required)
- The amount as reported (or "undisclosed")
- The date as reported
- A neutral, factual summary — no marketing language
Do not submit unsourced incidents, rounded-up amounts, or invented statistics. They will be
rejected. The verified flag lets you submit a well-sourced report even if some
details remain approximate.
Use the dataset
Download as JSON, CSV, or JSONL. Cite it in your own research. Embed the incident widget on your site with one script tag. CC BY 4.0 — free forever.