Skip to content

VALIDATION_RESULTS.md

Latest commit

 

History

History
113 lines (84 loc) · 4.49 KB

VALIDATION_RESULTS.md

File metadata and controls

113 lines (84 loc) · 4.49 KB
Raw

MCP Java SDK Conformance Test Validation Results

Last validated: 2026-08-17 against conformance suite @modelcontextprotocol/conformance@0.2.0-alpha.11 (SDK at main, 2.0.1-SNAPSHOT), targetting version 2025-11-25 (--spec-version 2025-11-25).

Summary

Server Tests (active suite): 73/73 checks passed (31 scenarios, 100%) Server Tests (SEP-1613 json-schema-2020-12): 5/5 checks passed (SEP-2106 checks skipped — post-2025-11-25 spec additions) Client Tests: 3/4 scenarios passed; sse-retry fails (tracked in conformance-baseline.yml) Auth Tests: 14/14 scenarios passing (193 checks, 0 failed, 0 warnings)

Baseline check passed on every run: all failures are expected per conformance-baseline.yml.

Server Test Results

Active Suite — Passing (31/31 scenarios, 73/73 checks)

  • Lifecycle & Utilities: initialize, ping, logging-set-level, completion-complete
  • Tools (13/13): all scenarios including progress notifications, sampling, elicitation
  • Elicitation: SEP-1034 defaults (6 checks), SEP-1330 enums (6 checks)
  • Resources: list, read-text, read-binary, templates-read, subscribe, unsubscribe
  • Prompts: list, simple, with-args, embedded-resource, with-image
  • SSE Transport: multiple streams
  • Security: DNS rebinding protection

SEP-1613 — JSON Schema 2020-12 (5/5 checks)

  • json_schema_2020_12_tool found; $schema, $defs, and additionalProperties fields preserved; every JSON-RPC message valid per the spec JSON schema for the negotiated spec version (wire-schema-valid)
  • SEP-2106 checks (composition/conditional/anchor keywords) reported SKIPPED: they postdate the 2025-11-25 spec release and are excluded from scoring

Client Test Results

Passing (3/4 scenarios)

  • initialize (1/1): protocol negotiation, clientInfo, capabilities
  • tools_call (2/2): tool discovery and invocation
  • elicitation-sep1034-client-defaults (5/5): default values for string, integer, number, enum, boolean

Failing — in baseline (1/4 scenarios)

  • sse-retry: client does not parse/respect the retry: SSE field timing and does not send the Last-Event-ID header (SHOULD requirement). Expected failure, listed in conformance-baseline.yml.

Auth Test Results (Spring HTTP Client)

Status: 193 checks passed, 0 failed, 0 warnings across 14 scenarios

Uses the client-spring-http-client module with Spring Security OAuth2 and the mcp-client-security library.

Fully passing: metadata-default, metadata-var1/2/3, basic-cimd, scope-from-www-authenticate, scope-from-scopes-supported, scope-omitted-when-undefined, scope-step-up, scope-retry-limit, token-endpoint-auth-basic/post/none, pre-registration.

Note: auth/resource-mismatch (present in earlier suite versions) is no longer part of the 0.2.0-alpha auth suite.

Known Limitations

  1. Client SSE Retry: client doesn't parse or respect the retry: field, reconnects immediately, and doesn't send the Last-Event-ID header

Running Tests

Server (active suite)

# Build and start server
./mvnw clean install -DskipTests
mvn exec:java -pl conformance-tests/server-servlet \
  -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet"

# Run tests (in another terminal, from the repo root)
npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
  --url http://localhost:8080/mcp --suite active \
  --expected-failures ./conformance-tests/conformance-baseline.yml

Server (SEP-1613 scenario)

npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
  --url http://localhost:8080/mcp --scenario json-schema-2020-12

Client

for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do
  npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \
    --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-*.jar" \
    --scenario $scenario \
    --expected-failures ./conformance-tests/conformance-baseline.yml
done

Auth (Spring HTTP Client)

npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \
  --spec-version 2025-11-25 \
  --command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-*.jar" \
  --suite auth \
  --expected-failures ./conformance-tests/conformance-baseline.yml

Recommendations

High Priority

  1. Fix client SSE retry field handling in HttpClientStreamableHttpTransport