Dj2.0 migration by esutlie · Pull Request #249 · MMathisLab/FreelyMovingVR4Mice

esutlie · 2026-01-23T15:37:20Z

This PR migrates the codebase from DataJoint 0.14 to DataJoint 2.0, including comprehensive test infrastructure to validate the migration.

Summary

Upgrade all schema definitions and queries to DJ 2.0 API
Add test infrastructure (unit + integration tests with Docker)
Add migration tools
Validate blob data integrity with golden baseline comparisons

Changes

API Updates

Replace dj.schema() with dj.Schema()
Replace dj.config custom keys with module-level variables (DJ 2.0 uses Pydantic settings that don't allow arbitrary keys)
Replace .fetch() with DJ 2.0 equivalents:
- .fetch(as_dict=True) → .to_dicts()
- .fetch("column") → .to_arrays("column")
- .fetch(*primary_key, as_dict=True) → .keys()
- .fetch("col1", "col2", as_dict=True) → .proj("col1", "col2").to_dicts()
Update blob type annotations to syntax

Test Infrastructure

Unit tests: Mock-based tests for helpers_dj, dlc_helpers, populate_rig
Integration tests: Real MySQL via Docker (testcontainers) for schema creation and data population
Golden baseline: Capture and compare data structures to verify migration correctness
Data roundtrip tests: Verify pickle/JSON → DB → fetch produces identical results

Migration Tools

scripts/migrate_to_dj2.py - Applies DJ 2.0 type labels to column comments

maryapp

@esutlie, great work on this PR — especially the testing and validation!

One structural concern: removing fetch() is a major breaking change for long-lived pipelines (we have 100+ usages per pipeline, deeply coupled with downstream logic). This creates real long-term maintenance risk and version lock-in, especially for production systems that must remain reproducible across OS/driver/Python changes.

Would it be possible to keep fetch() as a compatibility layer (even internally mapped to the new API)? This would allow gradual transition, protect existing pipelines, and improve long-term sustainability without blocking DJ 2.0 adoption.

dimitri-yatsenko · 2026-01-28T14:34:05Z

-    dj.config["database.misc.create_tables"] = create_tables
-    dj.config["enable_python_native_blobs"] = True
+    dj.config['database.database_prefix'] = prefix
+    dj.config['database.create_tables'] = create_tables


Why are these set here instead of datajoint.json?

A more natural way to configure settings is in the prjoect's datajoint.json file. Here is the how-to: https://docs.datajoint.com/how-to/configure-database/

@dimitri-yatsenko
As explained earlier, the base_schemas (and base_actions) package is shared across all our pipelines and dates back to the very early days of DataJoint.

This is precisely why backward and API compatibility with older versions is critical for us when introducing new DataJoint releases. These base packages are imported across many existing pipelines, and even small syntax changes would require widespread refactoring.

This also explains why certain elements are intentionally not configured differently, and why changes such as the CamelCase updates (reported below) are particularly important for us. Our goal is to avoid refactoring legacy pipelines while still being able to support and adopt the newer DataJoint core.

We believe this is a reasonable and pragmatic requirement, given the shared and long-lived nature of these base schemas

dimitri-yatsenko · 2026-01-28T14:35:38Z

            if mice.Mouse() & mouse_key:
                pk = mice.Mouse().primary_key
-                mouse = (mice.Mouse() & mouse_key).fetch(*pk, as_dict=True)[0]
+                mouse = (mice.Mouse() & mouse_key).proj(*pk).to_dicts()[0]


Suggested change

mouse = (mice.Mouse() & mouse_key).proj(*pk).to_dicts()[0]

mouse = (mice.Mouse() & mouse_key).keys(limit=1)[0]

dimitri-yatsenko · 2026-01-28T14:37:32Z

+                session = (exp.Session() & mouse_key & session_key).proj(
+                    *pk
+                ).to_dicts()[0]


Suggested change

session = (exp.Session() & mouse_key & session_key).proj(

*pk

).to_dicts()[0]

session = (exp.Session() & mouse_key & session_key).keys(limit=1)[0]

dimitri-yatsenko · 2026-01-28T15:06:21Z

Good news! We've just opened a PR to restore backward-compatible fetch() in DataJoint 2.0:

datajoint/datajoint-python#1355

This adds:

fetch() with a deprecation warning that maps to the new 2.0 methods

So the migration can be much simpler — you don't need to replace all fetch() calls immediately. They'll work with a deprecation warning, allowing gradual migration.

Once merged, you could simplify this PR to focus on the essential changes:

dj.schema() → dj.Schema()
Config key updates
Type annotation updates

The 100+ fetch() calls can stay as-is initially.

lecriste · 2026-01-29T17:37:31Z

Thanks @dimitri-yatsenko, really appreciated!

I’m the Lead Software Engineer in the MMathis Lab and I’m coordinating this migration on our side. I see that datajoint/datajoint-python#1355 has been merged. Could the DataJoint team please update this PR accordingly?

To further reduce backward incompatibility, would it be possible to provide a compatibility path for the other two breaking changes as well?

dj.schema() → dj.Schema()
Config key updates

Our goal is to complete the DataJoint 2.0 migration by next week, so an initial minimal-change approach would allow us to validate the whole pipeline accurately at the infrastructure level. We’ll migrate the APIs incrementally afterward, once we’ve had time to run more comprehensive tests.

esutlie · 2026-01-29T21:50:58Z

Hi @lecriste,

Thanks for coordinating this. Happy to help make the migration as smooth as possible.

The backward-compatible fetch() with deprecation warnings should let you identify any remaining calls without breaking your pipeline. That said, we'd still recommend updating to the new data-fetching methods in the core codebase as part of the initial migration.

For dj.schema() → dj.Schema() and the config key updates, we won't be adding a compatibility layer, so these will need to be part of the minimum-viable migration. Fortunately, these two changes are straightforward and won't require logic changes.

Your incremental approach after the initial migration sounds great. We'd be happy to put together a prioritized list of recommendations for the project if that would be useful.

Would a call next week work for you? We could walk through the migration process together and address any questions in real time.

lecriste · 2026-01-30T09:20:49Z

Hi @esutlie, thanks for the input.

Happy to schedule a call next week (Wednesday?).
In the meantime, could you please update this PR in light of datajoint/datajoint-python#1355 by reverting the .fetch()-related changes? That will help us review the remaining required updates and get a clearer view of what we still need to address.

esutlie · 2026-02-01T23:59:08Z

Hi @lecriste,

All of the .fetch() calls have been reverted. The current changes should be quite minimal, other than the addition of the test suite. Let me know if there are any other changes you'd like to see, and we can discuss further on Wednesday.

lecriste

Thanks for the update and for providing the test suite, @esutlie.
I've added my first round of comments.

Have you run tests/compare_golden_baseline.py? If so, what does the report look like?

lecriste · 2026-02-02T11:28:45Z

+  "capture_timestamp": "2026-01-20T15:25:11.035586",
+  "capture_timestamp_utc": "2026-01-20T20:25:11.036924Z",
+  "python_version": "3.12.3 (main, Jan  8 2026, 11:30:50) [GCC 13.3.0]",
+  "datajoint_version": "0.14.6",


With datajoint_version 0.14.6, running:
from vr4mice.schema import base
raises:
DataJointError: ClassName must be alphanumeric in CamelCase, begin with a capital letter

Could you please update the tests to include this import (using a previous version, e.g. 0.14.0), and more generally add test coverage for dj_pipeline/run.py as well?

Hi @lecriste,

The ClassName error is caused by four table classes in base_schemas/schemas/mice.py that use underscores in their names:

MouseScoreSheet_BodyCondition

MouseScoreSheet_GeneralAssay

MouseScoreSheet_HousingAssesment

MouseScoreSheet_WaterRestriction

The latest version of datajoint that supports that syntax is 0.14.1

Tomorrow I can implement more testing support and on Wednesday we can discuss how to fix this issue in the minimum viable migration.

@esutlie, thanks for the detailed analysis. Yes, this is exactly the issue we’re facing: the current ClassName restriction prevents us from upgrading base_schemas to newer DataJoint versions without breaking existing pipelines.

These tables are part of a shared, long-lived base package used across multiple pipelines, GUIs, and notebooks. A simple “rename and mysqldump” approach isn’t practical for us, as it would require coordinated changes across many components and affect long-term maintainability and reproducibility.

This is why we’ve remained on DataJoint 0.14.1 so far, and why backward compatibility at the API and naming levels is important for us. We’re open to discussing a minimal and safe migration path, as long as it doesn’t require refactoring legacy schemas.

@lecriste @maryapp

Two things to address here:

In the most recent commits I've added test coverage for dj_pipeline/run.py. The tests use the dataset Celia sent me as a golden dataset and expect the files to be in test_data\golden_dataset next to the tests folder. Let me know if you have any trouble using these.

The ClassName error occurs because of the way datajoint converts between class names and table names, where the class names are in camel case and the table names are in snake case. To make this compatible for your use case, we've updated DataJoint 2.0 to allow for underscores in the class names with a warning message. The only issue this could present down the line is if you were to reverse engineer the class names from the tables names, they would be missing the underscore. Hopefully this won't be a problem. You will need to migrate directly from 0.14.1 to 2.0 to skip over the versions where camel case is enforced.

lecriste · 2026-02-04T18:50:02Z

Hi @esutlie, can you please update the pipeline Dockerfile? Specifically this section:

RUN pip install datajoint==0.14 \
    # Note: this version of seaborn, as otherwise it installs 0.12.2 and updates numpy to 1.26,
    # and then Numba stops to work: err "Numba needs NumPy 1.24 or less" 
    seaborn==0.13.0 \
    pandas==1.4.3 \
    numpy==1.22.4 \
    jupyter \
    umap-learn \ 
    black==22.6
    # TODO: make pip via req
    # with newer 2.0.3, tom's code fails

This Dockerfile defines the environment we'll use to run the tests from this PR, and eventually the updated pipeline.

esutlie · 2026-02-04T21:40:27Z

I've rebased this PR to include all the recent changes in the main branch, and updated the dockerfile to install datajoint 2.0.1

It should be ready to go, so please let me know how testing goes!

lecriste · 2026-02-04T23:56:07Z

    && apt-get clean

-RUN pip install datajoint==0.14 \
+RUN pip install datajoint==2.0.1 \


@esutlie this is not enough. The current base image ships an old Python version that is not compatible with DJ 2 (have you tried to build the image?).
You can switch the base image to deeplabcut/deeplabcut:latest-jupyter.

The tests added to this PR should then be executed inside the container built from this Dockerfile.

maryapp · 2026-02-05T14:13:01Z

Good news! We've just opened a PR to restore backward-compatible fetch() in DataJoint 2.0:

@dimitri-yatsenko, thanks for bringing back the fetch() function. However, it looks like there are some differences in behavior that have been noticed: datajoint/datajoint-python#1381, which means this may block the pipelines.
@esutlie, did you happen to notice any issues like that while testing?

maryapp

As discussed yesterday, the goal is to merge this “migration PR,” ASAP, but in order to approve the changes, the tests and environment need to be confirmed, as well as the back-end DataJoint internal changes.

Review the new PR (# ?) with the DataJoint config file implementation proposal
Have a stable test environment (compatible with the Python version and libraries required in the current environment and requirements (to run analysis)
Have a small test database showing that the entire pipeline is still functional and populates from A to Z (regardless of fetch() implementation changes, etc.)
Bring it up to date with main and cover new schemas

cc @lecriste, @esutlie

esutlie · 2026-02-05T20:22:24Z

Hi @lecriste @maryapp,

Here are the latest updates:

The most recent commits provide a method for running the tests with the updated dockerfile. The old testcontainers method will still work as well.
I've also submitted a new PR built on top of this one with the datajoint.json config changes we would recommend: Feature/datajoint json config #266
As of yesterday's commits, everything should be up to date with the latest version of main.

Let me know if you have any issues running the tests.

lecriste · 2026-02-06T13:12:19Z

Hi @esutlie, we provided a GitHub Action to test the Dockerfile build.
Can you please rebase this PR onto the latest main? so we can see the test outcome.

- Create tests/ directory with pytest setup - Add conftest.py with shared fixtures for test data: - Path fixtures for all test data files - Data loading fixtures (pickle, JSON, HDF5, NPY) - Expected values fixtures for assertions - Dataset/video/DLC key fixtures - Add pytest.ini with test configuration - Add test_fixtures.py with 24 smoke tests to verify fixtures - Update .gitignore to exclude venv/ All 24 fixture tests pass. Test data: test_data/Celia_Set_14012026/ - Nightingale_2024-08-16_1.pickle (53 keys) - Nightingale_2024-08-16_1.json (33 keys) - Imagingsource_*_DLC.hdf5 (281748, 83) - Imagingsource_*_TS.npy (455965,) - Imagingsource_*_PROC (11 arrays) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Tests for DLC data serialization and processing functions: TestDfToDj (11 tests): - Returns dict with data/headers keys - Handles 2-level MultiIndex (no scorer) - Handles 3-level MultiIndex (with scorer) - Preserves data shape, values, and header tuples TestDjToDf (8 tests): - Round-trip preserves shape, values, columns - Reconstructs MultiIndex correctly - Preserves column names for 2-level and 3-level TestDlcInterpolate (6 tests): - Interpolates low confidence points - Preserves high confidence points - Documents NaN behavior when all points low confidence TestDlcSavgolFilter (5 tests): - Smooths noisy data - Preserves shape - Documents NaN handling (raises ValueError) TestFindClosestIndices (7 tests): - Binary search for closest timestamps - Handles edge cases (before first, after last, empty) TestConvertAngles (5 tests): - Angle conversion with shift - Output range [-180, 180] TestFilterDlc (4 tests): - Full filtering pipeline - Preserves shape and columns TestComputeHeadAngles (3 tests): - Returns expected columns - Preserves row count TestH5ToDj (4 tests): - Loads real HDF5 file - Correct data shape Also updated conftest.py to add analysis/actions paths to sys.path. Total: 53 new tests (77 total) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create capture_golden_master.py script to capture test outputs and data structures before migration for comparison after DJ 2.0 upgrade - Capture baseline: test results, data structures, sample values, metadata - Fix tuple key serialization for MultiIndex DataFrame columns in JSON output - Move test_data_roundtrips.py from unit/ to integration/ (requires test data) - Enhance tests with Golden Master assertions for spot-checking values - Add golden_master/ directory with captured baseline (DJ 0.14.6) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This commit implements the complete migration from DataJoint 1.x to 2.0. Removed deprecated `dj.config["enable_python_native_blobs"] = True` from: - dj_pipeline/vr4mice/utils/schema_config.py - dj_pipeline/base/base_actions/base_actions/utils/schema_config.py - tests/integration/conftest.py Changed all `longblob`, `mediumblob`, `blob` to `<blob>` syntax in: - dj_pipeline/vr4mice/schema/vr4mice.py (83 fields) - dj_pipeline/vr4mice/schema/base_analysis.py (56 fields) - dj_pipeline/vr4mice/schema/dlc.py (22 fields) - dj_pipeline/vr4mice/schema/session_metrics.py (7 fields) - dj_pipeline/vr4mice/schema/latency_tests.py (12 fields) - dj_pipeline/vr4mice/schema/interpolated_trajectories.py (41 fields) Updated fetch patterns to DJ 2.0 API: - `.fetch(as_dict=True)` → `.to_dicts()` - `.fetch(*cols, as_dict=True)` → `.proj(*cols).to_dicts()` - `pd.DataFrame(table.fetch())` → `table.to_pandas()` - `.fetch("col")[0]` → `.fetch1("col")` Files modified: - dj_pipeline/vr4mice/actions/fetch_data.py (12 occurrences) - dj_pipeline/vr4mice/schema/vr4mice.py (8 occurrences) - dj_pipeline/vr4mice/schema/base_analysis.py (8 occurrences) - dj_pipeline/vr4mice/schema/dlc.py (6 occurrences) - dj_pipeline/vr4mice/schema/interpolated_trajectories.py (3 occurrences) - dj_pipeline/vr4mice/schema/latency_tests.py (3 occurrences) - dj_pipeline/vr4mice/schema/base.py (2 occurrences) - dj_pipeline/vr4mice/actions/populate_rig.py (1 occurrence) - dj_pipeline/vr4mice/analysis/analysis.py (1 occurrence) - dj_pipeline/vr4mice/analysis/dlc_helpers.py (1 occurrence) - dj_pipeline/vr4mice/analysis/utils.py (1 occurrence) - dj_pipeline/vr4mice/analysis/summary_dj.py (1 occurrence) - Added tests/compare_golden_master.py for migration verification - Updated test mocks in tests/unit/test_populate_rig.py to use .to_dicts() - All 175 unit tests pass - Golden Master comparison shows no differences from pre-migration baseline Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Rename tests/golden_master/ to tests/golden_baseline/ - Rename capture and compare scripts accordingly - Update all internal references to use golden_baseline - Revert .gitignore changes to keep PR focused on DJ 2.0 migration Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create scripts/migrate_to_dj2.py: standalone migration script that uses datajoint.migrate.analyze_columns() and migrate_columns() to add type labels to column comments. Supports --dry-run and --analyze-only modes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create scripts/validate_migration.py: standalone script that validates the DJ 1.x -> 2.x migration by fetching blob data from the database and comparing against original Nightingale golden dataset files. Validates tables: - Dataset (sanity check) - MouseState (10 blob columns) - State (9 blob columns) - Metadata (3 blob columns) Reports pass/fail for each table and exits with non-zero code on failure. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Replace dj.config custom keys with module-level variables for schema config - Change dj.schema() to dj.Schema() (9 occurrences) - Replace .fetch() with .to_arrays(), .to_dicts(), .keys() (29 occurrences) - Remove validate_migration.py from tracking (kept locally) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Replace module-level globals with DJ_SCHEMA_PREFIX and DJ_CREATE_TABLES environment variables for schema configuration - Simplify dj.Schema() calls in exp.py and mice.py by removing redundant locals() and create_tables=True arguments (both are defaults) - Add limit=1 to to_dicts() and to_arrays() calls in base_analysis.py for improved query performance Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use dj.config['database.schema_prefix'] and dj.config['database.create_tables'] instead of DJ_SCHEMA_PREFIX and DJ_CREATE_TABLES environment variables. This aligns with the official DataJoint 2.0 config API added in PR #1346. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Update config key from database.schema_prefix to database.database_prefix to align with DataJoint 2.0.0a25 API changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Changes: - Revert .to_dicts(), .to_arrays(), .to_pandas() calls back to .fetch() - Fix table instantiation in test_db_populate.py (Table.fetch -> Table().fetch) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add test_run.py: Tests for CLI utilities and argument parser - Add test_run_modes.py: Full pipeline tests for each run.py mode - Update conftest.py: Rename SCENE_ROOT to PROJECT_ROOT, look for test_data inside project directory - Add test_data/ to .gitignore (download separately) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The original DataJoint version is 0.14.1, not 1.x. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Reverting removal of these arguments to match scene-pre-migration pattern. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add migration documentation: - docs/migration/minimum_migration_guide.md: Step-by-step DJ 0.x to 2.0 guide - docs/migration/recommended_improvements.md: Post-migration improvements - Consolidate integration tests: - Delete redundant test_db_populate.py (overlapped with test_run_modes.py) - Fix DLC test failures caused by conflicting test data - Reorganize golden baseline directory structure: - Move golden_baseline/*.json -> golden_baseline/unit/ - Move golden_master/mode_outputs/ -> golden_baseline/integration/ - Standardize on "golden baseline" terminology throughout - Add .env.test.local.example for test environment configuration Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- golden_baseline/unit/ -> golden_baseline/migration/ (captures data before/after DB datatype migration) - golden_baseline/integration/ -> golden_baseline/pipeline/ (captures pipeline outputs for end-to-end validation) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Custom Numpy Codec should be Built-in Numpy Codec Co-authored-by: Dimitri Yatsenko <dimitri@datajoint.com> Signed-off-by: Elissa Sutlief <elissasutlief@gmail.com>

Co-authored-by: Dimitri Yatsenko <dimitri@datajoint.com> Signed-off-by: Elissa Sutlief <elissasutlief@gmail.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Update Dockerfile base image to deeplabcut:latest-jupyter for DJ 2.0.1 compatibility - Add test dependencies (pytest, scipy, tables) to Dockerfile - Create docker-compose.test.yml to orchestrate MySQL + test runner - Add docs/docker_testing.md with usage instructions - Disable TLS in conftest.py for MySQL 5.7 compatibility Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Some scipy versions raise ValueError on NaN input to savgol_filter, while others propagate NaNs. Updated test to accept either behavior. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

From #268 (Fix primary key of DecisionPoints): - Fix DecisionPoints primary key: PredictionModel -> PredictionModel.SessionPrediction - Add logger.info for population tracking in PredictionModel and DecisionPoints From #265 (Fix leaking in trial history feature): - Fix trial_history calculation to prevent data leaking between trials - Add length check for label_info to catch multiple entries From upstream vr4mice.py: - Add missing FailedSession.should_skip() method Test fix: - Fix test assertion: populate_rig returns None on early return Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

esutlie · 2026-02-06T21:05:13Z

Hey @lecriste, I've rebased the PR onto main (as of commit #268). The branch now includes the recent upstream commits (#265, #266, #268) with a clean linear history.

I noticed #267 (AWS cron execution) was pushed after I rebased. Since it only touches infrastructure files (Makefile, cron scripts) and shouldn't conflict with the migration changes, I've left the PR as-is rather than rebasing again. Let me know if you'd prefer I pull it in.

Let me know if you have any questions or if the GitHub Action reveals any issues.

lecriste · 2026-02-07T11:10:05Z

@esutlie, your rebase dropped the changes from "main".
Please keep https://github.com/MMathisLab/FreelyMovingVR4Mice/pull/249/changes#diff-8c59f87a2ad20469e89ead472b2a4714dcce03631d983387812292af6b244d10L18 and https://github.com/MMathisLab/FreelyMovingVR4Mice/pull/249/changes#diff-8c59f87a2ad20469e89ead472b2a4714dcce03631d983387812292af6b244d10L33-L35, otherwise the GH action will fail.

Add back changes from upstream that were lost during rebase conflict resolution: - Add pip/setuptools/wheel/packaging upgrade before main pip install - Add default values for ARG (user_name=user, uid=1000, gid=1000) for CI workflow Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

esutlie · 2026-02-07T16:29:48Z

Hey @lecriste, I've pushed a fix that restores those changes.

Quick question: since this PR is from my fork, you can't push to it directly. If you anticipate needing more changes, I can move this to a branch in the main repo so you can commit directly. Just let me know.

lecriste · 2026-02-07T16:40:10Z

Hi @esutlie, yes please. Working on a branch in this repo will be easier.

esutlie · 2026-02-07T17:02:27Z

Sounds good! Moved to #269.

maryapp self-requested a review January 28, 2026 10:56

maryapp requested changes Jan 28, 2026

View reviewed changes

maryapp added the proposal label Jan 28, 2026

dimitri-yatsenko reviewed Jan 28, 2026

View reviewed changes

dimitri-yatsenko mentioned this pull request Jan 28, 2026

feat: Add backward-compatible fetch() with deprecation warning datajoint/datajoint-python#1355

Merged

maryapp requested a review from lecriste January 29, 2026 16:57

lecriste linked an issue Jan 30, 2026 that may be closed by this pull request

Migration from DJ 1.0 --> 2.0 #242

Closed

lecriste reviewed Feb 2, 2026

View reviewed changes

Comment thread dj_pipeline/base/base_min_schemas/base_schemas/schemas/exp.py Outdated

lecriste reviewed Feb 2, 2026

View reviewed changes

Comment thread dj_pipeline/base/base_min_schemas/base_schemas/schemas/mice.py Outdated

maryapp self-requested a review February 3, 2026 15:02

esutlie marked this pull request as ready for review February 4, 2026 21:33

lecriste reviewed Feb 4, 2026

View reviewed changes

maryapp requested changes Feb 5, 2026

View reviewed changes

lecriste mentioned this pull request Feb 5, 2026

Update Dockerfile and docs #261

Merged

esutlie mentioned this pull request Feb 5, 2026

Feature/datajoint json config #266

Closed

esutlie and others added 2 commits February 6, 2026 09:25

esutlie and others added 22 commits February 6, 2026 12:24

Add DJ_SCHEMA_PREFIX and DJ_CREATE_TABLES to env.py

fe299e4

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Rename schema_prefix to database_prefix per PR #1351

0ca2bd7

Update config key from database.schema_prefix to database.database_prefix to align with DataJoint 2.0.0a25 API changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Revert fetch() calls to use backward-compatible DJ 2.0 API

02b0ede

Changes: - Revert .to_dicts(), .to_arrays(), .to_pandas() calls back to .fetch() - Fix table instantiation in test_db_populate.py (Table.fetch -> Table().fetch) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix version references: DJ 1.x -> DJ 0.x

d9b1119

The original DataJoint version is 0.14.1, not 1.x. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Restore locals() and create_tables=True to dj.Schema() calls

d057f2b

Reverting removal of these arguments to match scene-pre-migration pattern. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Update docs/migration/recommended_improvements.md

c9e5055

Custom Numpy Codec should be Built-in Numpy Codec Co-authored-by: Dimitri Yatsenko <dimitri@datajoint.com> Signed-off-by: Elissa Sutlief <elissasutlief@gmail.com>

Update docs/migration/recommended_improvements.md

6677715

Co-authored-by: Dimitri Yatsenko <dimitri@datajoint.com> Signed-off-by: Elissa Sutlief <elissasutlief@gmail.com>

Add zarr@ to schema-addressed storage options

f05473a

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix test to handle scipy version differences in NaN handling

1cc8de2

Some scipy versions raise ValueError on NaN input to savgol_filter, while others propagate NaNs. Updated test to accept either behavior. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

esutlie force-pushed the dj2.0-migration branch from 159982a to e6ea6da Compare February 6, 2026 20:54

lecriste reviewed Feb 7, 2026

View reviewed changes

esutlie closed this Feb 7, 2026

	mouse = (mice.Mouse() & mouse_key).proj(*pk).to_dicts()[0]
	mouse = (mice.Mouse() & mouse_key).keys(limit=1)[0]

Conversation

esutlie commented Jan 23, 2026

Uh oh!

maryapp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maryapp Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lecriste commented Jan 29, 2026

Uh oh!

esutlie commented Jan 29, 2026

Uh oh!

lecriste commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

esutlie commented Feb 1, 2026

Uh oh!

lecriste left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lecriste Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

esutlie Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

maryapp Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

esutlie Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lecriste commented Feb 4, 2026

Uh oh!

esutlie commented Feb 4, 2026

Uh oh!

lecriste Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maryapp commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maryapp left a comment • edited by esutlie Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esutlie commented Feb 5, 2026

Uh oh!

lecriste commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

esutlie commented Feb 6, 2026

Uh oh!

lecriste Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

maryapp left a comment •

edited

Loading

dimitri-yatsenko Jan 28, 2026 •

edited

Loading

maryapp Feb 3, 2026 •

edited

Loading

dimitri-yatsenko Jan 28, 2026 •

edited

Loading

dimitri-yatsenko commented Jan 28, 2026 •

edited

Loading

lecriste commented Jan 30, 2026 •

edited

Loading

lecriste left a comment •

edited

Loading

lecriste Feb 4, 2026 •

edited

Loading

maryapp commented Feb 5, 2026 •

edited

Loading

maryapp left a comment •

edited by esutlie

Loading

lecriste commented Feb 6, 2026 •

edited

Loading

lecriste Feb 7, 2026 •

edited

Loading