ISNAD

The Problem#

When a multi-agent AI system answers you, that claim has passed through many hands — a source, a scraper, an ingestion model, a synthesis model. Each hand can drop, distort, or invent.

A single factual claim may pass through document retrieval, extraction, summarization, knowledge integration, and answer synthesis before reaching a user. Provenance systems can reconstruct that chain after the fact. They provide limited guidance on the question users actually care about: this specific claim, having passed through these specific transformers — should I trust it?

Provenance tools record what happened. Nothing grades who did it — or tells you how much to trust the result. A confident model at the end of a chain cannot repair a corrupted extraction at the start of it.

The Idea#

Classical Islamic hadith science spent twelve centuries on a structurally identical problem: whether to trust knowledge transmitted through chains of human narrators. ISNAD transfers that methodology to AI pipelines.

Concept Mapping: From Ḥadīth Science to AI Pipelines#

The intellectual heart of the framework — the paper's Table 1, mapping each classical term to its ISNAD usage.

Refining the Weakest-Link Rule

In hadith transmission a weak link is unconditionally corrupting. AI transformations are not all alike. A destructive transformation — extraction, chunking, lossy summarization — can only lose or corrupt information, and there the strict minimum applies: nothing downstream recovers what the extractor dropped. A generative transformation can either repair upstream noise or introduce fresh corruption by favouring its parametric prior over retrieved evidence. For generative steps the rule is therefore not a blind minimum but a bounded one: a generative narrator can raise the floor only up to its own grade, and only when corroboration supports the repair — and it can always lower it.

This mirrors a distinction the muḥaddithūn drew themselves, between riwāya bi-l-lafẓ (transmission of exact wording) and riwāya bi-l-maʿnā (transmission by meaning). The weakest-link principle survives; it is refined by transformation type.

The Decision Matrix#

The framework's most actionable artifact — chain grade × content verdict → action. The third column is the binding constraint on real-world coverage.

Contradictions go to humans by default. Knowledge-conflict research consistently finds LLMs unreliable at representing and reconciling competing evidence, and during prototype development LLM synthesis handled contradiction cases worse than an extractive baseline until contradiction markers were made explicitly visible in context — after which auto-resolution was still left disabled in favour of gated review.

Worked Example: End to End#

Claim: "the momentum of a photon is p = h/λ", ingested from an openly licensed physics text.

Chain: [OpenStax University Physics Vol. 3 — source; publisher-trusted, ʿadālah high] → [pdf-scraper v1.2 — ḍabṭ graded: high extraction fidelity on text-layer PDFs] → [ingest-analysis model M@v — ungraded, recent version bump] → [ingest-renderer M@v — ungraded]

Chain grade: complete (ittiṣāl holds), but two narrators ungraded → ḥasan-tier by the weakest-link rule.

Matn criticism: the corpus already contains a page asserting momentum as p = mv without regime qualification. Contradiction detected.

Matrix cell: ḥasan × contradiction → review queue; do not serve.

Adjudication: a human reviewer recognises both claims as valid in different regimes — classical and quantum — and updates both pages with regime qualifiers, resolving the contradiction once, at ingest time, rather than leaving every future query to renegotiate it. The resolution feeds back: the ingestion model's contradiction-detection behaviour on this case becomes taʿdīl evidence toward its grade.

This example is implemented as a runnable demo and an integration test in the reference implementation (make demo, examples/worked_example.py, tests/test_worked_example.py).

What ISNAD Adds Over Existing Provenance#

Each Mechanism Against Its Strongest Existing Ancestor

Capability Coverage

Chain capture means claim-level transmission chains are recorded per claim; graded transmitters means each transformer carries a maintained reliability grade; content criticism means content is evaluated independently of chain quality; decision routing means the system emits a concrete serve/review/quarantine action. A ✅ indicates the approach addresses that capability as a first-class concern.

Deep lineage: Condorcet's jury theorem (1785), Hume on testimony (1748), the hearsay-within-hearsay rule (US FRE 805), Dempster–Shafer evidence combination, Dawid & Skene (1979), Jøsang's subjective logic (2001), TruthFinder (2008), Knowledge Vault (2014), Knowledge-Based Trust (2015). The paper's claim is not novelty for its own sake but the transfer of the most operationally complete solution to a domain that lacks it.

Quickstart#

Install

pip install isnad

npm i isnad

Also on npm: the JS/TS audit-record verifier — verifies SHA-256 / Merkle / detached-signature integrity of ISNAD audit records. Grading stays in the Python core.

60-Second Example

from isnad import Registry, Chain, ChainLinkSpec, grade_chain, decide
from isnad.types import NarratorGrade, ContentVerdict
from isnad.critics import EmbeddingCritic

# Build a chain: source → scraper → model
chain = Chain([
    ChainLinkSpec("openstax-textbook", 0, domain="physics"),
    ChainLinkSpec("pdf-scraper-v2", 1),
    ChainLinkSpec("ingest-model-v3", 2),
])

# Seed-grade known narrators
reg = Registry()
reg.register("openstax-textbook", "physics", grade=NarratorGrade.RELIABLE)
reg.register("pdf-scraper-v2",    "physics", grade=NarratorGrade.RELIABLE)
reg.register("ingest-model-v3",   "physics", grade=NarratorGrade.ACCEPTABLE)

# Grade the chain
grades     = [reg.get_grade(l.narrator_id, l.domain) for l in chain.links]
transforms = [l.transform_type for l in chain.links]
chain_grade = grade_chain(grades, transforms, is_complete=True)

# Content criticism — decoupled from chain grading
critic  = EmbeddingCritic()
verdict = critic.evaluate("p = h/λ", "p = h/lambda", ["p = mv"])
action  = decide(chain_grade, verdict)

print(f"Chain: {chain_grade.value.upper()} | Content: {verdict.value} | Action: {action.value}")

LangChain Integration (5 Lines)

pip install isnad[langchain]

from isnad.integrations.langchain import IsnadTracer, seed_registry
from isnad.critics import EmbeddingCritic

reg = seed_registry({"source:docs": "reliable", "model:gpt-4o": "acceptable"})
tracer = IsnadTracer(registry=reg, critic=EmbeddingCritic())
chain.invoke("What is F=ma?", config={"callbacks": [tracer]})
print(tracer.report())

MCP Integration — Grade MCP Servers as Narrators

from isnad.integrations.mcp import MCPToolObserver, build_mcp_tools, handle_grade_claim

obs = MCPToolObserver(registry, domain="finance")
obs.record_call(tool_name="query", server_name="data-warehouse")
result = obs.grade("revenue grew 12% YoY")
print(result["chain_grade"])   # weakest-link grade over the tool narrators

The Model Context Protocol treats every tool as a transmitter of claims. ISNAD grades MCP servers as narrators — tool outputs are destructive links, so tools stay majhūl (UNGRADED → capped at ḍaʿīf) until the operator seeds or logs evidence. A grade_claim tool exposes the operator's registry to agents at runtime — grades only, never manufactured.

Run the Experiments

git clone https://github.com/alizahidraja/isnad && cd isnad
make install   # uv sync
make test      # zero config, SQLite fallback
make demo      # the paper's worked example (§4.5)
make check     # lint + type-check + test

No database required for pure-logic tests; PostgreSQL is optional (docker compose up, set ISNAD_DATABASE_URL).

Architecture & Module Map#

Pluggable Strategies — Updated Defaults

Reference Schema

CREATE TABLE rijal_claims (
    claim_id         TEXT PRIMARY KEY,
    page_slug        TEXT NOT NULL,
    claim_text       TEXT NOT NULL,
    narrator_chain   JSONB NOT NULL,
    chain_confidence NUMERIC(4,3),
    valid_from       TIMESTAMPTZ,
    valid_until      TIMESTAMPTZ,
    superseded_by    TEXT,
    chain_status     TEXT DEFAULT 'active'
);

CREATE TABLE narrator_registry (
    narrator_id      TEXT NOT NULL,
    domain_tag       TEXT NOT NULL,
    narrator_type    TEXT NOT NULL,
    grade            TEXT NOT NULL,
    known_error_rate NUMERIC(4,3),
    model_version    TEXT,
    is_active        BOOLEAN DEFAULT TRUE,
    PRIMARY KEY (narrator_id, domain_tag)
);

Because the pipeline's transformation points already emit trace identifiers, populating narrator_chain is an instrumentation task, not an architectural change.

Evaluation: What's Validated, What Failed, What's Measured#

ISNAD-Bench — Measured Against 1,200 Years of Ground Truth

The framework's newest and strongest validation. ISNAD-Bench grades 577,024 real hadith chains that classical scholars already graded by hand, then asks one falsifiable question: when ISNAD is given the scholars' own narrator grades, does its weakest-link rule reach the same chain verdicts they did?

The answer is reported as Cohen's κ — never raw accuracy, which is meaningless on a corpus dominated by ṣaḥīḥ.

Three things the benchmark taught

1. Mutābaʿa was already load-bearing. The largest "disagreement" was chains scholars called weak-alone-but-ḥasan-if-corroborated. ISNAD said weak; they said acceptable. Checking the corpus, 88–92% of those chains have an independent corroborating route — the scholars were applying mutābaʿa, and it validated ISNAD's corroboration engine from the outside.
2. Reliability has a timeline. 161 narrators declined over their lifetime (ikhtilāṭ), touching 35% of all chains. The scholars didn't discard them — they dated the decline. ISNAD's get_grade_as_of() period-slicing, built on principle, turns out to be load-bearing across a third of the corpus.
3. The default was wrong. When a narrator had no grade (majhūl), ISNAD treated them as passable — generous, reasonable, wrong. κ = 0.761. The classical position caps unknown at ḍaʿīf. Flipping the default: κ = 0.871. Eleven points of κ from a 1,200-year-old method being right where modern intuition was wrong.

Full results, preregistered mapping, negative controls and the complete disagreement buckets: bench/docs/RESULTS.md · preregistered mapping.

RAGTruth — Hallucination Detection on Real LLM Output

A second, independent benchmark. On RAGTruth — 17,790 responses across 6 models — ISNAD's grounding critic catches 97.6% of hallucinations at κ = 0.575 (79.3% accuracy against a 71.4% baseline). Reported on ISNAD Cloud with the same preregistered discipline as ISNAD-Bench.

Setup

• 20,000 atomic claims extracted from real PDF text: OpenStax University Physics Vols 1–3 (CC BY 4.0) and Crowell's Light and Matter (CC BY-SA), split evenly, 10,000 per source.
• Domain tags: mechanics (7,600), general (6,890), electromagnetism (3,054), modern-quantum (1,728), optics-waves (728).
• Extraction via deepseek-chat over chunked PDF text.
• Split 30% calibration / 70% evaluation → 14,001 evaluation claims, across 10 random seeds.
• Four narrators with designed fault rates: pdf-scraper@1.2 (1%), ingest@good (2%), ingest@weak (15%), pdf-scraper@0.9-legacy (18%).
• Faults: OCR noise, digit swap, sign flip, negation drop, unit corruption, formula mangling, entity swap, fabricated numerics, regime confusion. ~5% of chains marked incomplete.
• Review budgets: B ∈ {2%, 5%, 10%, 20%}.

Status at a Glance

Validated: Weakest-Link Quarantine

Every claim whose chain contained a narrator graded rejected was quarantined — 4,057 claims, 29% of the evaluation split — and every quarantine was traceable to the specific narrator grade that caused it.

Partial Failure: The jarḥ–taʿdīl Loop

The loop recovered three of four narrator grades from audit evidence alone:

• pdf-scraper@1.2 (1% fault) → reliable in all 50 (narrator, domain) cells
• ingest@good (2%) → acceptable in all 50
• ingest@weak (15%) — deliberately left ungraded — independently discovered and driven to rejected in 49/50 cells, weak in one

Grade recovery is bounded by calibration coverage per (narrator, domain) cell, not only by the transition policy. Domain-conditioned grading multiplies the number of cells that must be filled, and a narrator rare within a domain is invisible to jarḥ–taʿdīl regardless of how unreliable it is. A deployment should monitor per-cell evidence counts as a first-class operational metric and treat a persistently under-evidenced cell as a risk, not a neutral absence.

Validated: Corroboration Across Three Corpora

The eight negative controls (v2) correctly produced no upgrade: no matching text; shared model family (the madār case); corroborators below the grade gate; a rejected base chain; the ṣaḥīḥ cap; shared upstream source; insufficient independent chains; empty corpus.

Matched-Coverage: The Measured Finding

The paper reported the matched-coverage comparison as "inconclusive" — ISNAD-gated coverage could not be driven above 4.8% with the reference critic. That question is now resolved as a measured, three-part finding (#126), not a shrug.

1. Chain grading + jarḥ–taʿdīl is the primary gate, and it works. With the critic treated as a no-op (acceptance riding entirely on the chain and narrator grades), ISNAD reaches 47.9% coverage at 3.0% served-error — the corrupt narrator is caught by the evidence loop, not by a content critic.

2. The matn critic is a rejection gate, not an acceptance gate. On a sample of the residual "leaks" (corrupted claims with a sound chain), the LLM critic caught 0 of 30 — subtle single-claim corruption is exactly what a content critic cannot reliably detect. Its false-consistent rate on meaning-changing corruption is 39.1% (#126), so it is unsafe as an acceptance gate.

3. The missing discriminator — corroboration — is structurally absent from this corpus. The classical defense against "sound chain, subtly wrong claim" is independent corroboration (mutābaʿāt). The §8 corpus has zero cross-source overlap, so corroboration fired 0 times by construction. Matched coverage on this corpus is structurally unattainable, not merely unfinished.

Full write-up: experiments/s8_gated_vs_ungated/results/MATCHED_COVERAGE_FINDING.md.

End-to-End Utility: The Fidelity→Utility Crossing

Every prior validation used synthetic fault injection or extracted sentences. Here the claims are generated by a live LLM (DeepSeek), labelled by a deterministic oracle (not the critic), then routed through the real pipeline.

The gate served only the correct in-corpus restatements (with caveat) and routed all 12 novel-topic claims to review.

Full write-up: experiments/e2e_utility/RESULTS.md.

The Confidence Baseline Is Uninformative by Construction

A confidence-gated condition was included to represent current practice. It performs at chance — but for a reason that sharply limits what can be concluded from it. The corpus's model_confidence field is computed before fault injection and never updated. Confidence is statistically independent of corruption by construction. The point-biserial correlation is r = 0.04; mean confidence on corrupted claims (0.853) and clean claims (0.848) differ by less than one hundredth.

Honest Negatives: Transition Policy & Content Critic

Sweeping the downgrade threshold shows a direct tradeoff with no sweet spot:

There is no setting in this range that delivers both coverage and grade recovery. More audit evidence did not help — increasing the per-cell audit budget reduced coverage from 10.0% to 5.9% at 40 and above. The default policy over-penalises reliable narrators as evidence accumulates on small samples.

Only the LLM tier is near-perfect on genuine contradictions (100% recall, zero false-consistents); the offline tiers are safe but conservative. But on subtle meaning-changing corruption, the critic's false-consistent rate is 39.1% (#126) — it is a rejection gate, not an acceptance gate.

The earlier "~90% LLM / ~40% NLI / ~30% LocalNLI" figures were estimated on a template-injected word-swap set and have been retracted (issue #96); the measured hierarchy above replaces them.

The honest negative stands: with most real claims returning "unverifiable," the critic — not the grading mechanism — caps end-to-end coverage. The hierarchy is measured; the open piece is the hard case (a confident wrong claim within a covered domain).

Case Study: Contradiction Detection on Real Texts

A prototype compiled-wiki system ingested OpenStax University Physics Vols 1–3 and Crowell's Light and Matter via chunked PDF extraction, with DeepSeek-chat as the ingestion model, single run.

Result: 19 contradictions surfaced across the corpus, each manually reviewed and confirmed genuine — among them classical momentum (p = mv) versus photon momentum (p = h/λ); wave versus particle treatments of light; Newtonian versus relativistic kinematics.

All 19 were Type B — regime distinctions where both claims are true in their domains and resolution means qualification — not Type A errors. This validates the detection substrate and routing path; it does not demonstrate error-correction capability.

Limitations#

Scope exclusions: Byzantine collusion among agents, cryptographic attestation of traces (complementary, not competing), and open-ended or creative generation. ISNAD grades claims that can in principle be true or false and checked against a corpus. It is a framework for knowledge accumulation, not for tasks where there is no fact of the matter to transmit.

What's Next#

In the Wild#

Listed in LangChain's Middleware Directory

ISNAD ships as a community middleware integration in LangChain's official docs. In the "All middleware" directory — sorted by monthly downloads — it ranks sixth, right after OpenAI, Anthropic, AWS, Microsoft Foundry, and CopilotKit.

Indexing, Archiving & Aggregation

Also discoverable via NASA ADS, Google Scholar, Semantic Scholar, Connected Papers, alphaXiv, CatalyzeX, Litmaps, scite, OpenAIRE, and Hugging Face Daily Papers. Reddit launch threads — URLs pending; will be linked when recovered.

Prior & Adjacent Work

• PROV-AGENT (Souza et al., IEEE e-Science 2025) — extends W3C PROV to capture agent interactions.
• Agent-Sentry (Sequeira et al., 2026) — bounding LLM agents via execution provenance.
• HadithRank (Hawramani) — models isnād topology using information theory, treating independent chains as redundant channels. That a mathematician grading hadith and this paper grading agents converge on the same redundancy rule is evidence the mapping is structural, not decorative.
• The Provenance Paradox (Prakash, 2026) — when delegates can inflate self-reported quality, quality-based routing can perform worse than random. Consistent with ISNAD's design intuition but not a replication.
• RAPS (2026) — reputation-aware publish–subscribe with Bayesian overlays.
• TrustTrade (2026) — replaces "uniform trust" with cross-agent consistency weighting.
• AgentPoison (Chen et al., NeurIPS 2024) — red-teaming LLM agents by poisoning memory/knowledge bases; the demonstrated threat the mawḍūʿ tier defends against.
• Karpathy's "LLM Wiki" sketch (2026) — the compiled-knowledge-base system class that motivates the whole framework.

Two public repositories independently reference isnad-style provenance chains: cognalith/isnad (agent skill security with provenance verification) and VAMaksimov/threat-intel-format (threat intelligence schema with isnad-style provenance chains). Neither is affiliated.

Links#

Cite#

Paper

@article{raja2026grading,
  author        = {Ali Zahid Raja},
  title         = {Grading the Narrators: An Isnad--Rijal Framework for
                   Claim-Level Provenance in Multi-Agent Knowledge Systems},
  year          = 2026,
  doi           = {10.48550/arXiv.2607.24117},
  eprint        = {2607.24117},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI}
}

Software

@software{raja2026isnad,
  author = {Ali Zahid Raja},
  title  = {Isnad--Rijal Framework: Reference Implementation},
  year   = 2026,
  doi    = {10.5281/zenodo.21216872},
  orcid  = {0009-0003-7875-4590}
}

LLM usage disclosure: drafting and editing were assisted by a large language model; all technical claims, design decisions, experimental results, and final text were reviewed by and are the responsibility of the author.

Frequently Asked Questions#

What is ISNAD in one sentence?

ISNAD is an open-source framework that grades every agent, scraper, and model in a claim's transmission chain — adapted from classical Islamic hadith science — to tell you how much to trust a claim that has passed through multiple AI hands.

How is this different from W3C PROV or PROV-AGENT?

PROV records what happened — it captures chains, agents, and activities. ISNAD grades who did it and whether to trust the result. PROV gives you a transmission log; ISNAD tells you which link in that log is the weakest, and what action to take. See the positioning tables for a detailed comparison.

Does ISNAD outperform confidence-based gating?

The matched-coverage question is now a measured finding, not an open one. Chain grading alone reaches 47.9% coverage at 3.0% served-error (no content critic); the matn critic adds nothing on subtle single-claim corruption (0/30 caught); and corroboration — the one mechanism that could have discriminated — is structurally absent from the §8 corpus. See the measured finding.

What does "the chain is graded by its weakest link" mean in practice?

A ṣaḥīḥ model at the end of a chain cannot rescue a ḍaʿīf scraper at the start. The chain's grade is the minimum grade among its narrators — adjusted by transformation type (destructive vs. generative) — because information lost early in the chain cannot be recovered downstream. The chain animation demonstrates this visually.

What is mutābaʿāt / corroboration, and when does it not fire?

Mutābaʿāt is the mechanism by which independent chains asserting the same claim corroborate each other, upgrading the claim's grade (capped at ṣaḥīḥ). It does not fire when chains share a narrator, model family, or upstream source — the classical madār case. It is validated at 603/603 on Wikipedia and 104/104 on physics textbooks, with 8/8 negative controls passing. No negative controls have been run on the physics corpus yet.

Why grade narrators per domain instead of globally?

A model precise on historical dates may be unreliable on quantum-mechanical equations. The registry key is (narrator, domain), not narrator alone. This mirrors classical practice — rijāl scholars graded transmitters domain-specifically as a matter of course.

What happens when a model version changes?

A model-version bump resets the narrator to ungraded until re-evaluated. Version drift is treated as a new narrator — track record does not carry forward. This is intentional: a model at v2 may behave differently from the same model at v1, and inherited reputation would be misleading. Known blind spot: hosted models whose weights change without a version-string change will silently retain a grade they no longer merit.

What is the "unverifiable" verdict and why does it matter?

"Unverifiable" is the third column of the decision matrix — the classical position of tawaqquf (suspension of judgment). A content critic that cannot evaluate a claim must be able to say so rather than defaulting to "consistent." Routing unverifiable claims conservatively is what makes the framework safe under a weak critic. It is also the binding constraint on practical coverage — most real claims receive this verdict under the reference critic.

What did the evaluation fail to show?

Three things, reported in the same typographic weight as the successes: (1) The jarḥ–taʿdīl loop recovered 3 of 4 narrator grades but missed the highest-fault narrator through insufficient calibration evidence. (2) The confidence baseline is synthetic and proves nothing about real LLM self-confidence. (3) The matn critic's false-consistent rate on meaning-changing corruption is 39.1% (#126) — it is a rejection gate, not an acceptance gate. See the evaluation section.

Is this a religious claim or ruling?

No. ISNAD borrows a methodology — the epistemological architecture of isnād criticism — from classical Islamic hadith science. It makes no religious claims, issues no rulings, and does not position any computational system as an authority within the religious sciences. See the positionality statement.

Can I use this in production today?

The offline pipeline has been exercised end to end on 20,000 claims, and the end-to-end utility experiment (#128) runs live LLM-generated claims through the real pipeline — gated serving at 0.0% vs 50.0% ungated served-error. The LangChain integration is shipped and tested. A live production deployment where chains are written automatically at ingest and the registry gates a serving path under real query load is still open work.

How do I cite it?

Two BibTeX entries are available in the Cite section — one for the arXiv paper and one for the software. Paper: arXiv:2607.24117 (cs.AI). Software DOI: 10.5281/zenodo.21216872.

What licence is it under?

Code: Apache-2.0 — with a public commitment that src/isnad/ stays Apache-2.0 in perpetuity (see LICENSING.md). Paper and documentation: CC BY 4.0.

How can I contribute?

The CONTRIBUTING.md lists good first issues — alternative critics, seed-grade bootstrappers, correlation detectors, and pipeline adapters. The highest-impact open problem is chain independence (#54). Pull requests welcome.