Feature/datajoint json config by esutlie · Pull Request #266 · MMathisLab/FreelyMovingVR4Mice

esutlie · 2026-02-05T20:16:35Z

Note: This branch builds on the Dj2.0 migration PR #249

Summary

Replaces manual dj.config setup with DataJoint 2.0's declarative config files:

datajoint.json - non-sensitive settings (host, port, prefix)
.secrets/datajoint.json - credentials (gitignored, template provided)

Changes

Add dj_pipeline/datajoint.json and .secrets/ directory
Simplify connect_to_database() - all params now optional
Deprecate LoginUser class with warning, fix multiple init bug
Remove DB credentials from env.py

Backward Compatible

Existing code still works - parameters override the config file when provided.

- Create tests/ directory with pytest setup - Add conftest.py with shared fixtures for test data: - Path fixtures for all test data files - Data loading fixtures (pickle, JSON, HDF5, NPY) - Expected values fixtures for assertions - Dataset/video/DLC key fixtures - Add pytest.ini with test configuration - Add test_fixtures.py with 24 smoke tests to verify fixtures - Update .gitignore to exclude venv/ All 24 fixture tests pass. Test data: test_data/Celia_Set_14012026/ - Nightingale_2024-08-16_1.pickle (53 keys) - Nightingale_2024-08-16_1.json (33 keys) - Imagingsource_*_DLC.hdf5 (281748, 83) - Imagingsource_*_TS.npy (455965,) - Imagingsource_*_PROC (11 arrays) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Tests for DLC data serialization and processing functions: TestDfToDj (11 tests): - Returns dict with data/headers keys - Handles 2-level MultiIndex (no scorer) - Handles 3-level MultiIndex (with scorer) - Preserves data shape, values, and header tuples TestDjToDf (8 tests): - Round-trip preserves shape, values, columns - Reconstructs MultiIndex correctly - Preserves column names for 2-level and 3-level TestDlcInterpolate (6 tests): - Interpolates low confidence points - Preserves high confidence points - Documents NaN behavior when all points low confidence TestDlcSavgolFilter (5 tests): - Smooths noisy data - Preserves shape - Documents NaN handling (raises ValueError) TestFindClosestIndices (7 tests): - Binary search for closest timestamps - Handles edge cases (before first, after last, empty) TestConvertAngles (5 tests): - Angle conversion with shift - Output range [-180, 180] TestFilterDlc (4 tests): - Full filtering pipeline - Preserves shape and columns TestComputeHeadAngles (3 tests): - Returns expected columns - Preserves row count TestH5ToDj (4 tests): - Loads real HDF5 file - Correct data shape Also updated conftest.py to add analysis/actions paths to sys.path. Total: 53 new tests (77 total) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add testcontainers-based MySQL integration tests (10 tests) - Add unit tests for helpers_dj.py and populate_rig.py - Add data roundtrip tests for transformation functions - Fix lazy import in helpers_dj.py to avoid DB connection on import - Update conftest.py with additional fixtures Test summary: - 157 unit tests (dlc_helpers, helpers_dj, populate_rig, data_roundtrips) - 10 integration tests (schema creation, table population) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add TestGetPath class with tests for: - File path construction from raw_data - File existence checks (returns False for nonexistent files) - File moving logic when move=True - No-move behavior when move=False - exp_session_filepath key behavior (uses .npy extension) - Windows path separator handling - Double slash path handling - Add TestGetRemotePath class with tests for: - Full remote path construction - Returns None for empty/None src - Transformer key mapping - Complex nested path handling - Add TestAdditionalUtilityFunctions for: - no_value_opto, no_joystick, no_force_field - Add TestGetStateEdgeCases for short state arrays Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add TestCheckKeysEdgeCases for edge case coverage: - Transformer key found after being flagged missing (lines 135-137) - Multiple transformers with mixed found/missing keys - Add TestPopulate class with tests for: - Data insertion from raw_data into tables - local_def function usage for value generation - Transformer mapping for attribute names - skip_duplicates=True behavior - Numpy array handling - Parameter passing to local_def functions - Add TestPopulateRig class with tests for: - Pickle file discovery and processing - Skipping existing datasets in database - Empty directory handling - GUI mode requiring .npy files - Processing .npy files when no pickle exists - File path generation without GUI - Graceful exception handling All tests use mocks for database operations to ensure fast, isolated unit testing. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add TestH5ToDj class with tests for: - HDF5 file reading and conversion to DJ format - Data and headers extraction - 2-level and 3-level MultiIndex handling - Round-trip verification with dj_to_df - Add TestSyncDlcWithGame class with tests for: - DLC/game data synchronization - Step and step_time column addition - Row count matching game data - pose_time adjustment by start_time - Add TestGetOfflineDlcVariables class with tests for: - Kinematic variable computation - heading_dir and head_angle columns - Time column preservation - Angle conversion (90 degree shift) - Add TestComputeHeadAnglesEdgeCases for: - Zero-length vector handling (ValueError in acos) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create capture_golden_master.py script to capture test outputs and data structures before migration for comparison after DJ 2.0 upgrade - Capture baseline: test results, data structures, sample values, metadata - Fix tuple key serialization for MultiIndex DataFrame columns in JSON output - Move test_data_roundtrips.py from unit/ to integration/ (requires test data) - Enhance tests with Golden Master assertions for spot-checking values - Add golden_master/ directory with captured baseline (DJ 0.14.6) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This commit implements the complete migration from DataJoint 1.x to 2.0. ## Config Changes (3 files) Removed deprecated `dj.config["enable_python_native_blobs"] = True` from: - dj_pipeline/vr4mice/utils/schema_config.py - dj_pipeline/base/base_actions/base_actions/utils/schema_config.py - tests/integration/conftest.py ## Blob Type Changes (6 schema files, ~180 fields) Changed all `longblob`, `mediumblob`, `blob` to `<blob>` syntax in: - dj_pipeline/vr4mice/schema/vr4mice.py (83 fields) - dj_pipeline/vr4mice/schema/base_analysis.py (56 fields) - dj_pipeline/vr4mice/schema/dlc.py (22 fields) - dj_pipeline/vr4mice/schema/session_metrics.py (7 fields) - dj_pipeline/vr4mice/schema/latency_tests.py (12 fields) - dj_pipeline/vr4mice/schema/interpolated_trajectories.py (41 fields) ## Fetch Pattern Changes (13 files, ~71 occurrences) Updated fetch patterns to DJ 2.0 API: - `.fetch(as_dict=True)` → `.to_dicts()` - `.fetch(*cols, as_dict=True)` → `.proj(*cols).to_dicts()` - `pd.DataFrame(table.fetch())` → `table.to_pandas()` - `.fetch("col")[0]` → `.fetch1("col")` Files modified: - dj_pipeline/vr4mice/actions/fetch_data.py (12 occurrences) - dj_pipeline/vr4mice/schema/vr4mice.py (8 occurrences) - dj_pipeline/vr4mice/schema/base_analysis.py (8 occurrences) - dj_pipeline/vr4mice/schema/dlc.py (6 occurrences) - dj_pipeline/vr4mice/schema/interpolated_trajectories.py (3 occurrences) - dj_pipeline/vr4mice/schema/latency_tests.py (3 occurrences) - dj_pipeline/vr4mice/schema/base.py (2 occurrences) - dj_pipeline/vr4mice/actions/populate_rig.py (1 occurrence) - dj_pipeline/vr4mice/analysis/analysis.py (1 occurrence) - dj_pipeline/vr4mice/analysis/dlc_helpers.py (1 occurrence) - dj_pipeline/vr4mice/analysis/utils.py (1 occurrence) - dj_pipeline/vr4mice/analysis/summary_dj.py (1 occurrence) ## Test Updates - Added tests/compare_golden_master.py for migration verification - Updated test mocks in tests/unit/test_populate_rig.py to use .to_dicts() ## Verification - All 175 unit tests pass - Golden Master comparison shows no differences from pre-migration baseline Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Rename tests/golden_master/ to tests/golden_baseline/ - Rename capture and compare scripts accordingly - Update all internal references to use golden_baseline - Revert .gitignore changes to keep PR focused on DJ 2.0 migration Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create scripts/migrate_to_dj2.py: standalone migration script that uses datajoint.migrate.analyze_columns() and migrate_columns() to add type labels to column comments. Supports --dry-run and --analyze-only modes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create scripts/validate_migration.py: standalone script that validates the DJ 1.x -> 2.x migration by fetching blob data from the database and comparing against original Nightingale golden dataset files. Validates tables: - Dataset (sanity check) - MouseState (10 blob columns) - State (9 blob columns) - Metadata (3 blob columns) Reports pass/fail for each table and exits with non-zero code on failure. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Replace dj.config custom keys with module-level variables for schema config - Change dj.schema() to dj.Schema() (9 occurrences) - Replace .fetch() with .to_arrays(), .to_dicts(), .keys() (29 occurrences) - Remove validate_migration.py from tracking (kept locally) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

DataJoint 2.0 Migration

- Replace module-level globals with DJ_SCHEMA_PREFIX and DJ_CREATE_TABLES environment variables for schema configuration - Simplify dj.Schema() calls in exp.py and mice.py by removing redundant locals() and create_tables=True arguments (both are defaults) - Add limit=1 to to_dicts() and to_arrays() calls in base_analysis.py for improved query performance Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use dj.config['database.schema_prefix'] and dj.config['database.create_tables'] instead of DJ_SCHEMA_PREFIX and DJ_CREATE_TABLES environment variables. This aligns with the official DataJoint 2.0 config API added in PR #1346. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Update config key from database.schema_prefix to database.database_prefix to align with DataJoint 2.0.0a25 API changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Refactor schema config to use dj.config attributes

Changes: - Revert .to_dicts(), .to_arrays(), .to_pandas() calls back to .fetch() - Fix table instantiation in test_db_populate.py (Table.fetch -> Table().fetch) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add test_run.py: Tests for CLI utilities and argument parser - Add test_run_modes.py: Full pipeline tests for each run.py mode - Update conftest.py: Rename SCENE_ROOT to PROJECT_ROOT, look for test_data inside project directory - Add test_data/ to .gitignore (download separately) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The original DataJoint version is 0.14.1, not 1.x. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Reverting removal of these arguments to match scene-pre-migration pattern. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add migration documentation: - docs/migration/minimum_migration_guide.md: Step-by-step DJ 0.x to 2.0 guide - docs/migration/recommended_improvements.md: Post-migration improvements - Consolidate integration tests: - Delete redundant test_db_populate.py (overlapped with test_run_modes.py) - Fix DLC test failures caused by conflicting test data - Reorganize golden baseline directory structure: - Move golden_baseline/*.json -> golden_baseline/unit/ - Move golden_master/mode_outputs/ -> golden_baseline/integration/ - Standardize on "golden baseline" terminology throughout - Add .env.test.local.example for test environment configuration Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- golden_baseline/unit/ -> golden_baseline/migration/ (captures data before/after DB datatype migration) - golden_baseline/integration/ -> golden_baseline/pipeline/ (captures pipeline outputs for end-to-end validation) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Custom Numpy Codec should be Built-in Numpy Codec Co-authored-by: Dimitri Yatsenko <dimitri@datajoint.com> Signed-off-by: Elissa Sutlief <elissasutlief@gmail.com>

Co-authored-by: Dimitri Yatsenko <dimitri@datajoint.com> Signed-off-by: Elissa Sutlief <elissasutlief@gmail.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Integration tests added for run.py Summary Add comprehensive integration tests for the dj_pipeline/run.py Add migration documentation with a minimal migration guide and recommended improvements Reorganize test infrastructure with golden baseline pattern to accommodate new integration tests Fix test conflicts and standardize terminology Changes Integration Tests (+1,180 lines) test_run_modes.py: Full pipeline tests covering all run.py modes (populate, offkin, dlc, all) test_run.py: CLI utilities and argument parser tests Tests run against a real MySQL database using testcontainers Golden baseline pattern captures expected row counts and sample values Migration Documentation minimum_migration_guide.md: Step-by-step guide for DJ 0.x → 2.0 migration recommended_improvements.md: Post-migration improvements organized by priority Test Infrastructure Reorganized golden baseline directory structure: golden_baseline/migration/ - Captures data before/after DB datatype migration golden_baseline/pipeline/ - Captures pipeline outputs for end-to-end validation Deleted redundant test_db_populate.py (consolidated into test_run_modes.py) Fixed DLC test failures caused by conflicting test data Added .env.test.local.example for test environment configuration Bug Fixes Restored locals() and create_tables=True to dj.Schema() calls Fixed version references (DJ 1.x → DJ 0.x)