Honest numbers.
Including what failed.
This ledger keeps the measurements we can currently document — including wins, caveats, and failed experiments. It is a project-authored record, not an independent certification. Read the scope and reproduction status before using a number to make a buying decision.
Evidence-state notice · reviewed September 3, 2026: this page preserves earlier in-project measurements. Some historical reproduction paths no longer exist on current main. These results are not independent replication, certification, or proof of generalization. Use the current measured scope and dated operations snapshot before quoting them.
Looking for Kaggle, product, npm, GitHub, or Clay capability evidence? Open the Operations Evidence explorer. It keeps public scores, rules and forum review freshness, controlled experiments, deterministic checks, model records, and missing proof in separate lanes.
1. Three-mechanism combined defense Archived in-project result
An earlier project experiment used six synthetic attack types with 50 trials each (300 total) against scbe_14layer_reference.py and recorded a combined mean AUC of 0.9942. Treat this as development evidence until the exact runner, corpus, environment, and result bundle are restored or pinned publicly.
| Attack type | Phase AUC | Tonic AUC | Drift AUC | Combined |
|---|---|---|---|---|
| A. Wrong tongue | 0.642 | 1.000 | 0.468 | 0.9992 |
| B. Replay attack | 0.525 | 0.997 | 0.569 | 0.9924 |
| C. Synthetic bypass | 0.567 | 0.993 | 1.000 | 0.9998 |
| D. Wrong frequency | 0.465 | 1.000 | 0.508 | 0.9992 |
| E. Scale anomaly | 0.992 | 0.926 | 1.000 | 0.9902 |
| F. Adaptive / rounded | 0.497 | 0.502 | 1.000 | 0.9842 |
| Mean | 0.6146 | 0.9029 | 0.7575 | 0.9942 |
Each detector alone was incomplete in this archived experiment — phase caught E but failed A/B/D/F, tonic caught A/B/D but failed F, and drift caught C/E/F but failed A/D. The combined score was the strongest tested configuration in that experiment; it is not a current cross-domain guarantee.
experiments/three_mechanism_results.json2. Red team regression corpus In-project result
The 91 adversarial prompts span 10 project-authored categories. The pipeline was developed against this corpus, so 91/91 is useful regression coverage but does not measure generalization. The current README separately reports a held-out paraphrase lane.
| System | Blocked | False Positives | Rate |
|---|---|---|---|
| SCBE-AETHERMOORE | 91/91 | 0 | 100% |
| ProtectAI DeBERTa v2 | 62/91 | — | 68% |
| Keyword filter | 27/91 | — | 30% |
| Meta Prompt Guard | 15/91 | — | 16% |
| No protection | 0/91 | — | 0% |
This is an archived project table. Comparator lanes in the default runner may be simulated unless external models are explicitly enabled; do not read it as an independent head-to-head model benchmark.
huggingface.co/datasets/issdandavis/scbe-red-team-benchmarks3. Framework-mapped project checks Not certification
150 project-authored automated checks were mapped to 13 framework references, 14 pipeline layers, and 12 security axioms. The linked table includes zero-test areas, including FedRAMP and layers L2–L6; passing these checks is not independent compliance validation or certification.
Framework references in the project mapping (150/150 authored checks passing)
HIPAA/HITECH · NIST 800-53 · FIPS 140-3 · PCI-DSS v4.0 · SOX Section 302/404 · GDPR · ISO 27001:2022 · IEC 62443 · SOC 2 Type II · FedRAMP · CMMC 2.0 · FDA 21 CFR Part 11 · NERC CIP
Axioms validated
A1 Boundedness · A2 Continuity · A3 Encryption (AES-256-GCM) · A4 Nonce Uniqueness · A5 Pseudonymization · A6 Least Privilege · A7 Fail-to-Noise · A8 Key Lifecycle · A9 Context Binding · A10 Audit Completeness · A11 Monotonic Recovery · A12 Bounded Failure
tests/scbe_compliance_report.md compliance_report.json (raw)4. Latency Caveat
Per-agent timing for the full 14-layer pipeline: 0.328 ms/agent (σ 0.222), measured over 100 runs, 30 agents, 50 steps. That's sub-millisecond — much faster than our old site claimed.
What this does NOT measure: end-to-end request-response wall clock through a real API gateway. That benchmark is pending. Don't take the 0.33 ms/agent number as "your chatbot will add 0.33 ms per message" — network, serialization, and policy load-time add overhead.
Any end-to-end latency target must be measured in the buyer's actual stack. No production latency guarantee is inferred from this microbenchmark.
experiments/pipeline_vs_baseline_results.json5. Pipeline-depth ablation Caveat — important
We measured AUC across 4 pipeline depths (2-layer, 3-layer, 6-layer, 14-layer) on 3 attack classes (random, subtle, adaptive). The results are surprising and worth knowing:
| Attack | 2-layer | 6-layer | 14-layer |
|---|---|---|---|
| Random | 0.9987 | 0.7945 | 0.9780 |
| Subtle | 0.9837 | 0.9247 | 0.0536 |
| Adaptive | 0.8864 | 0.8220 | 0.6905 |
The raw 14-layer pipeline scored 0.054 AUC on subtle attacks — worse than random guessing. In a separate archived experiment, the phase + tonic + drift combination reached 0.9942 mean AUC on its six in-project attack types. Those results motivate the combined design, but they do not establish performance on a new client system.
Publishing the failed ablation matters because architecture depth alone was not the answer. A buyer-facing engagement still needs a fresh baseline, relevant controls, held-out cases, and acceptance criteria on the target stack.
experiments/pipeline_vs_baseline_results.json6. What we disproved Disproven
Not every hypothesis survived contact with data. These are ideas we tried, measured, and rejected:
X1. Hyperbolic distance is superior to Euclidean for anomaly detection
100 runs, 20 agents, 5 malicious. Euclidean AUC 0.9995, hyperbolic 0.9553. Hyperbolic lost. We still use Poincaré embeddings for policy-region geometry, but the raw distance metric is Euclidean because that's what measures best.
X2. GeoSeal swarm coordination
Measured 0.543 AUC on swarm detection. Below useful threshold. Retired.
X3. Constant-time cryptographic operations in pure Python/numpy
Not established at the Python/numpy layer. Constant-time properties, where documented, belong to the underlying cryptographic implementation and still require platform-specific verification. We stopped claiming that property for our Python layer.
X4. Tripoint centroid hyperbolic advantage
Tried using three-point hyperbolic centroids as a trust anchor. Measured no advantage over the Euclidean baseline. Removed from the pipeline.
docs/CLAIMS_EVIDENCE_LEDGER.md — full ledger7. What's still untested Honest status
The evidence ledger in the repo lists patent claims 5, 6, 7, 8, 12, 13, 17, 18, 20, and 21 as CODE_EXISTS_UNTESTED. The code is written, the math is described, but I haven't run controlled experiments yet. We don't advertise these as proven.
The full ledger: docs/CLAIMS_AUDIT_V4.md
Verify the current surface
The historical experiment commands previously printed here no longer resolve on current main, so they have been removed rather than presented as runnable. Start with the repository's current measured detection section, which states the corpus, limitations, dependencies, and current command.
Current runnable check:
git clone https://github.com/issdandavis/SCBE-AETHERMOORE.git
cd SCBE-AETHERMOORE && pip install -e ".[test]"
pip install ".[ml-onnx]"
SCBE_INJECTION_MODEL=1 pytest tests/test_intent_model_benchmark.py -q (use the platform-appropriate environment-variable syntax)
Questions I get asked
Why publish the disproven claims?
Because the honest ledger is itself the credibility signal. Any vendor willing to say "this idea didn't work" in public is more trustworthy than one who only publishes wins. If you're buying audit evidence from me, you need to trust that the evidence I deliver is accurate — which means you need to see that I kill my own darlings when the data says so.
Why is the raw 14-layer pipeline so bad on subtle attacks?
Because layer depth is a blunt instrument. More layers means more parameters to fit the training distribution, which makes subtle out-of-distribution attacks easier to miss. The three-mechanism combined defense (phase + tonic + drift) corrects for this by using complementary detection strategies — each covers attack types the others miss.
Can I trust any of the claims on the solution pages?
Treat solution pages as product descriptions, then verify each metric against a dated artifact, its stated corpus, and its limitations. This archived ledger is useful context, but it is not a blanket validation of every solution-page claim.
Where are the 29,000 tests?
The earlier claim of "29,000+ tests passing" conflated TypeScript assertions across the ecosystem with the Python count. This archived snapshot recorded 638 Python tests, 150 project compliance checks, and 91 project-authored red-team prompts. Each count still needs its own version, date, runner, and scope before reuse as a current headline.