Hebrew Coreference Resolution System

A comprehensive toolkit for Hebrew coreference resolution that integrates multiple components into a unified pipeline. This system provides mention detection, web-based annotation, neural coreference models, and LLM evaluation capabilities.

🚀 Features

Core Components

• Mention Detection: Advanced NP chunking with multiple parser backends (Stanza, Trankit, Gold)
• Web Annotation: Interactive web interface for coreference and NP-relation annotation
• Neural Models: State-of-the-art neural coreference resolution (LingMess-Coref, WL-Coref)
• LLM Evaluation: Comprehensive evaluation of Large Language Models for coreference tasks

Key Capabilities

• ✅ Multi-Parser Support: Stanza, Trankit, and Gold standard parsing
• ✅ Interactive Annotation: Web-based interface for manual annotation
• ✅ Neural Training: End-to-end neural model training and evaluation
• ✅ LLM Integration: Zero-shot and few-shot evaluation of LLMs
• ✅ Comprehensive Evaluation: MUC, B³, CEAF metrics
• ✅ Production Ready: Clean CLI interface and modular architecture
• ✅ Extensible: Easy to add new models and components

📋 Table of Contents

• Installation
• Quick Start
• System Architecture
• Usage
• Configuration
• Examples
• Contributing
• License

🛠️ Installation

Prerequisites

• Python 3.8+
• Git

Setup

# Clone the repository
git clone https://github.com/shaked571/hebrew_coreference.git
cd hebrew_coreference

# Clone the data submodule
git submodule update --init --recursive

# Install dependencies
pip install -r requirements.txt

🚀 Quick Start

1. Mention Detection

# Run NP chunking with Stanza parser
python src/mention_detection/stanza_parser/stanza_chunker.py

# Run NP chunking with Trankit parser
python src/mention_detection/trankit_parser/trankit2spacy.py

2. Neural Coreference

# Train LingMess-Coref model
cd src/neural_models/neural_coref/src/lingmess-coref
python training.py

# Evaluate model
python eval.py

3. LLM Evaluation

# Compare LLM outputs
cd error_analysis/llm_comparison
python compare_multi_system_outputs.py --approaches raw gold_tokenized sota_tokenized

4. Web Annotation

# Start annotation server
cd src/annotation/tne_ui
python annotationServer.py

🏗️ System Architecture

hebrew_coreference/
├── src/
│   ├── mention_detection/          # NP chunking and parsing
│   ├── neural_models/             # Neural coreference models
│   ├── llm_evaluation/            # LLM evaluation framework
│   └── annotation/                # Web-based annotation tools
├── error_analysis/                # Error analysis and comparison tools
├── statistics/                    # Statistical analysis tools
├── tests/                         # Test suite
├── data/                          # Data submodule (separate repo)
└── scripts/                       # Utility scripts

📖 Usage

Mention Detection

The system supports multiple parsing backends for robust mention detection:

• Stanza: Stanford's NLP toolkit with Hebrew support
• Trankit: Unified multilingual NLP toolkit
• Gold: Manual annotation for high-quality reference

Neural Models

Two state-of-the-art neural architectures:

• LingMess-Coref: End-to-end neural coreference resolution
• WL-Coref: Span-based coreference with BERT encoding

LLM Evaluation

Comprehensive evaluation framework for Large Language Models:

• Raw Tokenization: Direct LLM output evaluation
• Gold Tokenization: Aligned with reference tokenization
• SOTA Tokenization: State-of-the-art tokenization alignment

Web Annotation

Interactive annotation interface for:

• Coreference annotation
• NP-relation annotation
• Mention boundary correction
• Quality control and validation

⚙️ Configuration

Environment Variables

export PYTHONPATH="${PYTHONPATH}:$(pwd)/src"
export CUDA_VISIBLE_DEVICES=0  # For GPU training

Model Configuration

Models can be configured through YAML files in their respective directories:

• src/neural_models/neural_coref/src/lingmess-coref/config.yaml
• src/neural_models/neural_coref/src/wl-coref/config.toml

📚 Examples

Basic NP Chunking

from src.mention_detection.np_chunker.chunker import NPChunker

chunker = NPChunker()
chunks = chunker.chunk_text("הטקסט בעברית עם שמות עצם")
print(chunks)

Coreference Evaluation

from src.neural_models.neural_coref.src.lingmess_coref.metrics import CorefEvaluator

evaluator = CorefEvaluator()
scores = evaluator.evaluate(predictions, gold)
print(f"MUC: {scores['muc']}, B³: {scores['b3']}, CEAF: {scores['ceaf']}")

LLM Comparison

from error_analysis.llm_comparison.compare_multi_system_outputs import MultiSystemComparisonRunner

runner = MultiSystemComparisonRunner()
results = runner.run_comprehensive_analysis()

🧪 Testing

Run the comprehensive test suite:

# Run all tests
python -m pytest tests/

# Run specific test categories
python -m pytest tests/test_mention_detection.py
python -m pytest tests/test_neural_models.py
python -m pytest tests/test_llm_evaluation.py

📊 Error Analysis

The system includes comprehensive error analysis tools:

• Cluster-level Analysis: Detailed breakdown of correct/missed/extra clusters
• Multi-system Comparison: Compare different approaches side-by-side
• Visualization: Generate charts and graphs for analysis
• Statistical Testing: Significance testing for performance differences

🤝 Contributing

We welcome contributions! Please see our contributing guidelines:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Hebrew NLP community for language resources
• Stanford NLP Group for Stanza
• Microsoft Research for Trankit
• Contributors to LingMess-Coref and WL-Coref

📞 Support

For questions and support:

• Open an issue on GitHub
• Check the documentation in each component directory
• Review the error analysis outputs for troubleshooting

📝 Citation

If you use this repository, please cite the following paper:

@inproceedings{greenfeld-tsarfaty-2026-beyond,
    title = "Beyond Word Boundaries: A {H}ebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text",
    author = "Greenfeld, Refael Shaked  and
      Tsarfaty, Reut",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.488/",
    doi = "10.18653/v1/2026.acl-long.488",
    pages = "10669--10683",
    ISBN = "979-8-89176-390-6",
    abstract = "Coreference Resolution (CR) is a fundamental NLP task critical for long-form tasks as information extraction, summarization, and many business applications. However, CR methods originally designed for English struggle with Morphologically Rich Languages (MRLs), where mention boundaries do not necessarily align with word boundaries, and a single token may consist of multiple anaphors. CR modeling and evaluation protocols standardly assume that, as in English, words and mentions mostly align. However, this assumption breaks down in MRLs, particularly in the context of LLMs' raw-text processing and end-to-end tasks. To assess and address this challenge, we introduce KibutzR, the first comprehensive CR dataset for Modern Hebrew, an MRL rich with complex words and pronominal clitics. We deliver an annotated dataset that identifies mentions at word, sub-word and multi-word levels, and propose an evaluation protocol that directly addresses word/morpheme boundary discrepancies. Our experiments show that contemporary LLMs perform significantly worse on Hebrew than on English, and that performance degrades on raw unsegmented text. Crucially, we show an inverse performance-trend in Hebrew relative to English, where smaller encoders perform far better than contemporary decoder models, leaving ample space for investigation and improvement. We deliver a new benchmark for Hebrew coreference resolution and a segmentation-aware evaluation protocol to inform future work on other MRLs."
}

---

Note: This system is designed specifically for Hebrew text and includes Hebrew-specific optimizations and linguistic features.