Add Chrysalis E3SM v3 archive ingestion

[tomvothecoder]

Description

Adds a targeted E3SM v3 archive ingestor that runs on Chrysalis and uploads cases to SimBoard through the remote HPC endpoint. This is required because v3 performance data is stored on Chrysalis rather than a filesystem accessible from NERSC.

• Defines documented v3 simulation names and scans archive snapshots from 2024-01
• Reuses remote discovery, validation, deduplication, packaging, and upload logic
• Skips whole-snapshot checkpoints for safe filtered reconciliation
• Adds reconciliation summaries, regression tests, and operator documentation
• Closes [DevOps]: Add v3 ingestion script for Chrysalis #280

Steps

• 1. Reference the data docs under "Simulation" column for the case names (define statically in the script)
• 2. Start discovery in the 2024-01 directory
• 3. Iterate over performance_archive<timestamp> sub-directories to discover matching case names.
• 4. Attempt ingestion on Chrysalis with dry-run
• 5. Validate against the case names in the data docs list to ensure all cases were ingested.
• 6. Ensure docs are updated (related to [DevOps]: Add guidelines for specific material to ingest from performance archive directory #259)
• 7. Attempt full ingestion on Chrysalis
• 8. Merge changes

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings
• Tests added or updated
• All tests pass locally
• Documentation/comments updated
• No breaking change

Deployment Notes (if any)

Run dry-run reconciliation on Chrysalis with an externally reachable SIMBOARD_API_BASE_URL and service-account token before enabling uploads.

[Copilot]

Pull request overview

Adds a Chrysalis-specific, targeted E3SM v3 archive ingestor that filters to a
static list of documented v3 simulations, scans archive snapshots from 2024-01,
packages each selected case, and uploads via the remote
/api/v1/ingestions/from-hpc-upload route (without using whole-snapshot
checkpoints).

Changes:

• Introduces chrysalis_v3_archive_ingestor.py and a v3 reconciliation summary
  that fails the run when expected simulations are missing/transient/incomplete.
• Extends the shared archive ingestion plumbing to support optional case-path
  filtering, optional archive checkpointing, and a run-report object for
  specialized runners.
• Adds operator documentation and dedicated regression tests for the targeted
  v3 backfill behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/deploy/hpc-api-token-authentication.md	Documents Chrysalis v3 backfill workflow and upload endpoint usage.
docs/architecture/metadata-ingestion.md	Explains how the v3 backfill specializes remote automated upload mode and why checkpointing is disabled.
backend/tests/features/ingestion/test_chrysalis_v3_archive_ingestor.py	Adds regression coverage for v3 filtering, forced config, checkpoint bypass, and reconciliation outcomes.
backend/app/scripts/README.md	Documents the new Chrysalis v3 backfill script entry point and operational expectations.
backend/app/scripts/ingestion/nersc_archive_ingestor.py	Adds IngestorRunReport, configurable env overrides, and optional case filtering + report capture during archive scans/ingest.
backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py	Adds hooks for case filtering, disabling archive checkpointing, and emitting run-report details.
backend/app/scripts/ingestion/chrysalis_v3_archive_ingestor.py	Implements the targeted v3 archive scan/upload runner with reconciliation logging and non-checkpoint behavior.

[tomvothecoder]

Need clarification on what directories to process v3 data. The v3 simulations were ran on Chrysalis, which means the performance data is there too.

On the E3SM Data Docs, there are HPSS links to the output as well. We can either

1. Parse the original performance data on Chrysalis
2. See if the required metadata is available on NERSC

[tomvothecoder]

Need clarification on what directories to process v3 data. The v3 simulations were ran on Chrysalis, which means the performance data is there too.

On the E3SM Data Docs, there are HPSS links to the output as well. We can either

1. Parse the original performance data on Chrysalis

2. See if the required metadata is available on NERSC

Hey @chengzhuzhang, can you clarify where we should be ingesting v3 data?

[tomvothecoder]

LCRC E3SM V3 dry-run results

Summary: The V3 archive dry run completed successfully: all 30 expected simulations were found, 423 cases are ready for ingestion, and no missing, transient, or invalid executions were detected.

A dry run completed successfully against the Chrysalis archive.

Metric	Result
Exit code	0
Duration	248.046s
Expected simulations	30
Matched simulations	30
Missing simulations	0
Matching case directories	542
Accepted execution directories	632
Selected submission cases	423
Transient execution IDs	0
Invalid execution IDs	0
Incomplete execution IDs skipped	243
Ingestion failures	0

The runner found every expected V3 simulation, completed traversal, and found no transient or invalid execution metadata. The incomplete executions were excluded from submission as intended.

Reproduce on Chrysalis

Run from the repository's backend/ directory at the tested revision:

Run from the repository's backend/ directory at the tested revision:

git checkout devops/280-v3-ingestion
git rev-parse HEAD
uv sync

Create a local credentials file outside the repository:

mkdir -p ~/.config/simboard
cp app/scripts/ingestion/v3_data/lcrc-v3.env.example \
  ~/.config/simboard/lcrc-v3.env
chmod 600 ~/.config/simboard/lcrc-v3.env
Edit ~/.config/simboard/lcrc-v3.env and set:
SIMBOARD_API_BASE_URL=https://<simboard-api-host>
SIMBOARD_API_TOKEN=<service-account-token>
DRY_RUN=true

Optionally set OLD_PERF_ARCHIVE_ROOT when the archive is mounted somewhere other than the documented Chrysalis default.

Run the dry run:
LCRC_V3_ENV_FILE="$HOME/.config/simboard/lcrc-v3.env" \ ./app/scripts/ingestion/v3_data/lcrc_v3.sh

Confirm the final v3_ingestion_summary reports:

• missing_simulations=[]
• transient_execution_ids=0
• rejected_invalid_execution_ids=0
• exit_code=0

Live run
After reviewing the dry-run output, change DRY_RUN=false in the external env file and rerun the same wrapper:

LCRC_V3_ENV_FILE="$HOME/.config/simboard/lcrc-v3.env" \ ./app/scripts/ingestion/v3_data/lcrc_v3.sh

Then restore DRY_RUN=true and rerun the dry run. Successful uploads should leave selected_submission_cases=0 and increase the existing/processed execution counts.

[tomvothecoder]

LCRC E3SM V3 single-case ingestion validation

Summary: With MAX_CASES_PER_RUN=1, SimBoard successfully ingested one V3 case, v3.LR.piControl, containing three executions. A subsequent dry run confirmed those three executions are now skipped.

Live run

Metric	Result
Mode	ingest
Exit code	0
Duration	68.65s
Expected simulations	30
Matched simulations	30
Missing simulations	0
Accepted execution directories	632
Qualified submission cases	423
Selected submission cases	1
Deferred execution IDs	629
Ingestion successes	1
Ingestion failures	0
Transient execution IDs	0
Invalid execution IDs	0
Incomplete execution IDs skipped	243

The selected case, v3.LR.piControl, was accepted by the API with 201 Created. Its three executions were ingested successfully:

• 481004.240305-220806
• 481005.240305-221648
• 481051.240305-222449

SimBoard page: https://simboard-dev.e3sm.org/cases/6f05ef88-79fc-45b8-be41-7ca28b26d69d

Post-ingestion dry run

Metric	Result
Mode	dry-run
Expected simulations	30
Matched simulations	30
Missing simulations	0
Accepted execution directories	629
Existing execution IDs skipped	3
Qualified submission cases remaining	422
Selected submission cases	1
Deferred execution IDs	627
Transient execution IDs	0
Invalid execution IDs	0
Incomplete execution IDs skipped	243
Ingestion failures	0

The follow-up dry run confirms that the uploaded executions are recognized as existing and will not be submitted again.

[tomvothecoder]

LCRC E3SM V3 full ingestion run

Summary: SimBoard successfully ingested the remaining V3 workload: 422 submission cases containing 629 executions. Together with the three executions from the prior single-case validation, all 632 qualified executions are now registered. A final dry run confirmed the full backfill is idempotent.

Live run

Metric	Result
Mode	ingest
Exit code	0
Duration	3601.53s
Expected simulations	30
Matched simulations	30
Missing simulations	0
Accepted execution directories	629
Existing execution IDs skipped	3
Qualified submission cases	422
Selected submission cases	422
Deferred execution IDs	0
Ingestion successes	422
Ingestion failures	0
Transient execution IDs	0
Invalid execution IDs	0
Incomplete execution IDs skipped	243

All remaining qualified V3 cases were ingested successfully.

Post-ingestion dry run

Metric	Result
Mode	dry-run
Exit code	0
Duration	219.77s
Expected simulations	30
Matched simulations	30
Missing simulations	0
Accepted execution directories	0
Existing execution IDs skipped	632
Qualified submission cases remaining	0
Selected submission cases	0
Deferred execution IDs	0
Transient execution IDs	0
Invalid execution IDs	0
Incomplete execution IDs skipped	243
Ingestion failures	0

The final dry run confirms every qualified V3 execution is recognized as existing and will not be submitted again.

[tomvothecoder]

@TonyB9000 @chengzhuzhang @rljacob @wlin7 @golaz The remote V3 ingestion path from LCRC to NERSC worked end-to-end, including archive creation and upload. This successfully processed 422 cases and 629 executions, giving us confidence that the same workflow should work for testing against both the staging and archive directories.

More details above.

You can view the results on the SimBoard UI here: https://simboard-dev.e3sm.org/cases (filter on Chrysalis, optionally Case Group set to v3.LR)

[TonyB9000]

@tomvothecoder I have had the chrysalis collection running continuously for the past 20 hours, launching every 15 minutes.

In the listing below, the first two runs were "MAX=1" and can be ignored. The next launch (MAX=unlimited) took just over two hours (log size 3996606) and you can see that every 15 minutes cron would try to start up collection and abort with SKIP launch simboard collection, lock already held, pid 3630681

-rw-rw-r--+ 1 ac.bartoletti1 E3SM2 2725257 Aug 26 12:26 SBCS-20260826_171201.log
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2 2362408 Aug 26 13:14 SBCS-20260826_181201.log
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 14:05 SBCS-20260826_190504.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 14:20 SBCS-20260826_192002.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 14:35 SBCS-20260826_193502.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 14:50 SBCS-20260826_195003.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 15:05 SBCS-20260826_200503.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 15:20 SBCS-20260826_202004.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 15:35 SBCS-20260826_203502.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 15:50 SBCS-20260826_205001.log    (skipping)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2 3996606 Aug 26 16:01 SBCS-20260826_185001.log  (finally completed, et 7883.398)
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2     650 Aug 26 16:20 SBCS-20260826_212001.log
-rw-rw-r--+ 1 ac.bartoletti1 E3SM2 2389245 Aug 26 16:24 SBCS-20260826_210501.log

Here are the last lines of the 2 hr run. I am unsure how to interpret these numbers:

2026-08-26 16:01:32,774 [INFO]: archive_ingestor_core.py(_log_event:893) >> event=run_completed accepted_execution_ids=1921 deferred_execution_ids=0 execution_dirs_accepted=1921 execution_dirs_scanned=1 failure_count=11 rejected_existing_execution_ids=2 rejected_incomplete_execution_ids=2144 rejected_invalid_execution_ids=0 scanned_cases=942 selected_submission_cases=942 skipped_incomplete=2144 skipped_invalid=0 skipped_transient=0 submission_qualified_cases=942 success_count=931 transient_execution_ids=0
2026-08-26 16:01:32,774 [INFO]: archive_ingestor_core.py(_log_event:893) >> event=run_summary_counts mode=ingest scanned_cases=942 submission_qualified_cases=942 selected_submission_cases=942 execution_dirs_scanned=1 execution_dirs_accepted=1921 skipped_incomplete=2144 skipped_invalid=0 skipped_transient=0
2026-08-26 16:01:32,774 [INFO]: archive_ingestor_core.py(_log_event:893) >> event=run_summary_outcomes success_count=931 failure_count=11 accepted_execution_ids=1921 rejected_existing_execution_ids=2 rejected_incomplete_execution_ids=2144 rejected_invalid_execution_ids=0 transient_execution_ids=0 deferred_execution_ids=0
2026-08-26 16:01:32,776 [INFO]: archive_ingestor_core.py(_log_event:893) >> event=run_finished mode=ingest scan_mode=staging exit_code=1 duration_seconds=7883.398

I am thinking of issuing n "archive" run - but wanted feedback on this first, in case something is amiss.

[tomvothecoder]

@TonyB9000 I think you meant to post your latest comment above in #169. I'll reply there.