🏛️ EntLORE

A Graph-Grounded Benchmark for Latent Organizational Reasoning
in Enterprise Question Answering

Akrin Zheng^*   Alexander Wu^*   Alaia Liu^*†  —  ScitiX.ai
_{^* Equal contribution  ·  ^† Corresponding author: alaia@scitix.ai}

📄 Paper  ·  🤗 Hugging Face  ·  📊 Data & code

TL;DR — Enterprise answers often depend on organizational relations that no single document states. EntLORE reconstructs an audited enterprise truth graph, releases an anonymized document world whose gold answers and proofs are computed from that graph, and withholds the target derived relations from evaluated systems. Across 8 models × 7 knowledge-access conditions, how the released world is organized matters more than how much retrieval machinery is applied to it — and even perfect evidence leaves latent-relation questions unanswered. Full construction methodology is in the paper.

---

🚀 Quick start

Needs Python 3.10+ and API access — no local GPUs. All generation and embedding calls go through remote endpoints; the dataset is ~15 MB.

pip install -r requirements.txt
cp .env.example .env          # fill in your API endpoint(s) + model names

Model calls go through src/llm.py: point API_BASE at any OpenAI-compatible endpoint and ANTHROPIC_BASE_URL at Anthropic (or a compatible relay). Routing is by model-name prefix — claude* uses the Anthropic native protocol, everything else uses the OpenAI-compatible endpoint.

Smoke test first (a few cents). Verify credentials and the whole build→run→score wiring on the 5-question subset (dataset/smoke.json) before spending budget on a full run. It uses only closed_book + bm25, so no embedding endpoint is needed.

scripts/smoke.sh <your-model>   # ~10 chat + ~10 judge calls

A clean pass writes runs/smoke/score_summary.json with "complete": true. A nonzero bm25 mean means retrieval + scoring work; closed_book is expected near 0.

Full evaluation.

# 1) build retrieval indexes (bm25/rag are fast; okf/graphrag are heavier, many LLM calls)
python scripts/build_indexes.py --baseline bm25,rag        # add okf,graphrag for the structured conditions

# 2) run a model over the access conditions
python scripts/run_eval.py --models <your-model> \
    --pipes closed_book,bm25,rag,agentic_rag,okf,graphrag,oracle_rag \
    --questions dataset/questions.json --out runs/main --workers 32

# 3) score (deterministic gates + LLM judge; set JUDGE_MODEL in .env)
python scripts/score.py --root runs/main --models <your-model> \
    --pipes closed_book,bm25,rag,agentic_rag,okf,graphrag,oracle_rag \
    --bank dataset/golden_packets.jsonl --questions dataset/questions.json

All paths and parameters are overridable via EKWB_* environment variables (single table in src/baselines/config.py).

⚙️ Configuration notes (build model, concurrency, thinking models, disk/RAM)

• okf / graphrag build model. bm25 uses no LLM and rag only calls the embedding endpoint (EMBED_MODEL); the okf/graphrag compilers run an LLM per document using CHAT_MODEL from .env (recorded as build_model in their manifests). build_indexes.py --model / EKWB_SUT_MODEL set the answering model and do not change the compiled index. An OpenAI-family CHAT_MODEL builds fine — routing is by name prefix.
• Concurrency. run_eval.py --workers N (default 32) sets question-level fan-out. LLMWIKI_MAX_CONCURRENCY (env) governs index build, embedding, and the judge. A comma-separated API_KEY=k1,k2,k3 round-robins across keys to raise the effective rate limit. Single-key users: keep --workers ≈ 4–8 and LLMWIKI_MAX_CONCURRENCY low to avoid rate-limit errors.
• Thinking models. Set EKWB_NO_THINK=1 to inject enable_thinking=false for open "thinking" models (Qwen/GLM) that otherwise burn their token budget on hidden reasoning and return empty completions. It is a no-op for gpt-5* and claude*.
• Disk / memory. Indexes are not shipped and are rebuilt locally: bm25/rag are a few hundred MB, the OKF/GraphRAG substrates reach ~1–4 GB. bm25/rag/closed-book run in a few GB of RAM; plan for ~16 GB to evaluate with graphrag. Developed on Linux (x86-64); macOS works. No CUDA required.

Bring your own system. Your system only needs to read dataset/corpus/ and the question text in dataset/questions.json, then emit one answer per question. Score it with scripts/score.py against dataset/golden_packets.jsonl. To add it as a baseline, subclass Baseline and register with @register("name") (see src/baselines/).

📊 Main results

Answer accuracy (%) across all 8 answer models and 7 knowledge-access conditions, split into four views: Overall (tier-weighted over L1 469 / L2 204 / L3 234) and per level. Read column-wise: the highlighted cell is the best model under that access condition. The Mean row is an aggregate and is never marked; the all-zero closed-book column on L2 carries no signal and is left unmarked. Columns: CB closed-book · Ag agentic retrieval · Wiki LLM Wiki · GRAG GraphRAG · Ω Oracle ceiling (n=216 on L3).

Open-weight models take 12 of the 20 deployable-condition columns — including GraphRAG at every level (GLM-5.2 on Overall/L1/L3, DS-V4-Flash on L2). On L3 the proprietary models lead every column except GraphRAG: where documents stop stating the relation, the winner changes with the substrate, not with the model tier.

_{Overall (tier-weighted) accuracy of a representative open-weight model (GLM-5.2) vs a proprietary flagship (GPT-5.4) across the six access conditions. The open model leads on the structured substrates (GraphRAG, OKF) and ties at the Oracle ceiling; the proprietary model edges ahead only on flat retrieval (BM25 / RAG / agentic).}

🔑 Key findings

• Access organization beats retrieval machinery. All five deployable conditions read the same corpus and differ only in how they index and present it. They split into two far-apart tiers: BM25 (0.529), GraphRAG (0.522), LLM Wiki (0.509) on top, Agentic Retrieval (0.365) and flat dense RAG (0.360) ~15 points below (tier-weighted overall).
• Even perfect evidence leaves latent relations unanswered. Supplying the gold documents (Oracle) still leaves 30.4% of latent (L3) questions unanswered, versus 12.6% for explicit (L1) and 6.2% for compositional (L2) — the residual is reasoning, not retrieval.
• Flat dense retrieval falls below a plain lexical index. Released documents identify their organizational region through sparse anchors (project aliases, module names, ticket terms) that BM25 ranks first and a single dense space washes out.
• Where documents stop helping, the ranking inverts. 70% of L3 items resist flat retrieval (the target relation is stated in no released document). On that subset GraphRAG leads (0.310) and BM25 falls to 0.251 — GraphRAG answers with a relation it materialized offline, not by retrieving better. This is the sense in which EntLORE measures relation recovery, not recall.
• A mismatched harness burns the budget reasoning would have used. On non-retrievable L3 items, 30-step agentic retrieval scores 0.088 — below the no-corpus closed-book floor (0.077).
• The gap concentrates in hierarchy attribution. Department attribution: lexical retrieval 0.02 vs. the induced graph 0.53, on items the oracle answers 84% of the time.
• Open-weight models match or surpass the proprietary frontier where it counts. On corpus-induced GraphRAG — the strongest deployable condition — the top four scores on the hardest, latent-reasoning tier (L3) are all open-weight: GLM-5.2 (0.431), DeepSeek-V4-Flash (0.398), Qwen3.5-397B (0.396), DeepSeek-V4-Pro (0.386) — ahead of the best proprietary model (Claude-Sonnet-4.6, 0.383) and well ahead of GPT-5.4 (0.341). Kimi-K2.6 is the only model that leads with BM25 at all three levels. The proprietary edge appears only at the Oracle ceiling (Claude, 0.763 on L3) — raw synthesis given perfect evidence, not the retrieval-grounded settings a deployed system actually runs. Organizing the knowledge well closes the open-vs-closed gap more than scaling the answer model does.

📦 What's in this repository

Path	Contents
dataset/corpus/	2,341 markdown documents (1,194 reports · 794 knowledge-base · 353 tickets)
dataset/questions.json	907 questions ({id, question})
dataset/golden_packets.jsonl	907 gold packets — required facts, evidence pointers, proof/scoring mode
dataset/SCHEMA.md	field-level schema for the dataset
src/baselines/	the 7 access conditions + oracle upper bounds (self-contained)
src/evaluator.py	deterministic gates + LLM-judge scorer
scripts/	build_indexes.py, run_eval.py, score.py, smoke.sh
third_party/	vendored GraphRAG (MIT) and OKF (Apache-2.0)

Question tiers. L1 (explicit fact, 469) · L2 (compose across facts, 204) · L3 (derive a latent relation, 234). 62 operators (question types); 18 verified-unanswerable questions (abstention). The truth graph behind construction holds 1,153 entities and 3,784 typed relations.

Access conditions (paper name ↔ baseline id):

Condition	Baseline id	Measures
Closed-book	closed_book	parametric-memory floor (no retrieval)
BM25	bm25	naive sparse lexical retrieval
RAG	rag	flat dense top-k single-turn retrieval
Agentic Retrieval	agentic_rag	multi-turn autonomous tool loop over the dense index
LLM Wiki	okf	navigation loop over an LLM-compiled offline knowledge base
GraphRAG	graphrag	dual-tool loop over an induced entity graph + Leiden community reports
Oracle Ω	oracle_rag / oracle_agentic_rag / oracle_okf	perfect-evidence ceilings (gold packet fed directly)

🔖 Citation

If you use EntLORE, please cite:

@article{entlore2026,
  title         = {{EntLORE}: A Graph-Grounded Benchmark for Latent Organizational
                   Reasoning in Enterprise Question Answering},
  author        = {Zheng, Akrin and Wu, Alexander and Liu, Alaia},
  journal       = {arXiv preprint arXiv:2608.10679},
  year          = {2026},
  eprint        = {2608.10679},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR}
}

📜 License & data statement

Code is released under the Apache-2.0 License (see LICENSE); the dataset (dataset/) is released under CC BY-NC-SA 4.0. The corpus is fully synthetic and anonymized (English): an audited real enterprise world is reconstructed into a fictional organization, with persons, projects, aliases, and dates mapped through a shared identity map and private metadata never verbalized into the documents. See NOTICE for the full data statement and the licenses of vendored components (third_party/).

_{Made with 🧠 & 📈 by ScitiX.ai}