ScarfBench is a comprehensive benchmark suite of Java applications across Jakarta EE, Quarkus, and Spring frameworks, designed to systematically evaluate AI agents' ability to migrate enterprise Java applications while preserving functionality, idiomatic patterns, and architectural integrity.
| Applications | Layers | Frameworks | Tests |
|---|---|---|---|
| 102 | 6 | 3 | 1,331 |
All applications have been manually converted and verified by experienced developers. Each implementation has undergone rigorous testing to ensure functional correctness, adherence to framework-specific idioms, and preservation of architectural integrity.
| Repository | Description |
|---|---|
| benchmark | The core benchmark suite — all application examples with Dockerfiles, Makefiles, and smoke tests |
| CLI | General purpose CLI to undertake all benchmark related tasks like pull/update, evaluate, etc. |
| Version | Date | Description |
|---|---|---|
| v0.1.2 | 2026-03-25 | Standardized all Dockerfiles to use framework-native run commands (spring-boot:run, quarkus:run, liberty:run) |
| v0.1.1 | 2026-03-24 | Consolidated multi-module coffee-shop (Jakarta, Spring) into single Maven modules for strict architectural parity across frameworks; Dockerfile updates for PetClinic |
| v0.1.0 | 2026-03-19 | Initial release — 87 focused examples across 5 layers and 3 frameworks, plus 15 whole application variants |
ScarfBench includes two categories of examples spanning 7 architectural layers and 5 production-grade whole applications, each implemented across all three target frameworks.
Layer-specific demonstrations, each isolating a particular Java EE technology:
| Layer | Examples | Key Technologies |
|---|---|---|
| Business Domain | cart, converter, counter, helloservice, standalone | EJBs (stateful, stateless, singleton), session beans |
| Dependency Injection | billpayment, decorators, encoder, guessnumber, producerfields, producermethods, simplegreeting | CDI, qualifiers, interceptors, decorators, producers, observers |
| Infrastructure | concurrency-jobs, concurrency-taskcreator, ejb-async, ejb-interceptor, ejb-timersession | Managed executors, async EJBs, interceptors, timer services |
| Persistence | address-book, order, roster | JPA, CRUD, entity relationships, composite keys, JPQL |
| Presentation | dukeetf, dukeetf2, fileupload, hello-servlet, jaxrs-customer, jaxrs-hello, jaxrs-rsvp, mood, websocketbot | Servlets, JAX-RS, WebSockets, SSE, filters, listeners |
Complete, production-grade systems demonstrating multi-layer coordination:
| Application | Description |
|---|---|
| CargoTracker (code) | DDD cargo shipping tracker — Jakarta Faces, CDI, EJBs, JPA, REST, Batch, JMS. Implements Eric Evans' DDD patterns. |
| Coffee Shop (code) | Event-driven microservices (Orders, Barista, Kitchen) via Kafka. MicroProfile, reactive messaging, eventual consistency. |
| DayTrader (code) | High-performance stock trading benchmark — stateless session beans, JPA optimistic locking, transaction management. |
| PetClinic (code) | Veterinary clinic management — Jakarta Faces (PrimeFaces), complex JPA relationships, CDI, Bean Validation. |
| RealWorld (code) | Medium.com clone (Conduit) — MicroProfile JWT, JAX-RS, JPA, Testcontainers integration tests. |
Links:
- Documentation & Guide: scarfbench.github.io/site
- Quickstart: scarfbench.github.io/site/quickstart
- Leaderboard: scarfbench.github.io/site/leaderboard
- Submit Your Agent Results: scarfbench.github.io/site/submit
- Contributing Guide: CONTRIBUTING.md
- Citation: scarfbench.github.io/site/citation