Bug: DataJoint silently discards exception info for every uncaught exception

[arturoptophys]

Importing datajoint installs a process-wide sys.excepthook that routes every uncaught
exception in the process — not just DataJoint-related ones — through the package's own logger.
That logger's formatter (LevelAwareFormatter) never renders exc_info, so the exception type,
message, and traceback are discarded before anything is printed. The operator sees only:

[2026-07-21 16:00:13][ERROR]: Uncaught exception

with no indication of what actually failed. This is strictly worse than doing nothing: Python's
default sys.excepthook would have printed the full traceback for free, and DataJoint's hook
suppresses that.

LLM-analysis of the issue

Environment

• datajoint==2.3.1
• Python 3.12.11 (Anaconda build, Linux x86_64)
• Reproduced both inside a project container and with a standalone script with no DB connection
  required — the bug does not depend on being connected to a database.

Root cause

datajoint/logging.py does two things at import time (module-level code, so it runs the instant
anything does import datajoint):

1. Configures a logger named "datajoint" with a StreamHandler using a custom formatter:

   class LevelAwareFormatter(logging.Formatter):
       """Format INFO messages cleanly, show level for warnings/errors and JOBS."""
   
       def format(self, record):
           timestamp = self.formatTime(record, "%Y-%m-%d %H:%M:%S")
           if record.levelno >= logging.WARNING:
               return f"[{timestamp}][{record.levelname}]: {record.getMessage()}"
           elif record.levelno == JOBS:
               return f"[{timestamp}][JOBS]: {record.getMessage()}"
           else:
               return f"[{timestamp}] {record.getMessage()}"

   This completely overrides logging.Formatter.format() instead of extending it. The base
   implementation, after formatting the message, checks record.exc_info and appends
   self.formatException(record.exc_info) (plus record.stack_info if present). This override
   does neither — it has no branch that ever looks at record.exc_info or
   record.stack_info, so any exception object attached to the record is never rendered,
   regardless of log level.

2. Replaces sys.excepthook:

   def excepthook(exc_type, exc_value, exc_traceback):
       if issubclass(exc_type, KeyboardInterrupt):
           sys.__excepthook__(exc_type, exc_value, exc_traceback)
           return
   
       logger.error("Uncaught exception", exc_info=(exc_type, exc_value, exc_traceback))
   
   sys.excepthook = excepthook

   sys.excepthook is global interpreter state. Setting it as a side effect of import datajoint
   changes how any unhandled exception in the process is reported, including exceptions with no
   relationship to DataJoint at all (argument-parsing bugs, typos, KeyErrors in unrelated
   application code, etc.). The exc_info tuple passed into logger.error(...) here is correct and
   complete at this point — the data exists — it is only lost one step later, inside
   LevelAwareFormatter.format().

The combination means: the moment a process imports datajoint, every uncaught exception
anywhere in that process is silently reduced to a single, contentless log line.

This also isn't limited to the excepthook path. Any code — including inside datajoint itself —
that logs through logging.getLogger("datajoint") with exc_info set (e.g. logger.exception(...)
during a populate() call) loses the traceback the same way, since the defect is in the shared
formatter, not the hook.

Minimal reproduction

No database connection or project code required:

import datajoint  # import alone installs sys.excepthook

def inner():
    raise ValueError("something specific and diagnosable went wrong")

def outer():
    inner()

outer()

Actual output:

[2026-07-21 16:38:16][ERROR]: Uncaught exception

(Process exits 1. No exception type, message, or traceback anywhere in stdout/stderr.)

For contrast, the same script without import datajoint:

Traceback (most recent call last):
  File "<string>", line 8, in <module>
  File "<string>", line 6, in outer
  File "<string>", line 3, in inner
ValueError: something specific and diagnosable went wrong

Impact

We hit this in production debugging: a CLI entry point (an ingestion script using DataJoint
tables) failed with a schema mismatch (KeyError from Table.insert), but the only visible
output was [ERROR]: Uncaught exception. Diagnosing the real cause required bypassing DataJoint's
sys.excepthook entirely — wrapping the call in our own try/except and calling
traceback.print_exc() manually, which formats and prints without going through DataJoint's
logger/formatter at all.

Because sys.excepthook is global, this silently degrades error visibility for every script and
CLI entry point in any project that imports datajoint, for any exception, not just DataJoint's
own. It is easy to miss because everything looks fine until the first uncaught exception occurs
in production, at which point the actual cause is unrecoverable from the process's own output —
the traceback is simply gone, not just hard to read.

Suggested fix

LevelAwareFormatter.format() should render exc_info/stack_info the same way the base
logging.Formatter does, e.g.:

class LevelAwareFormatter(logging.Formatter):
    def format(self, record):
        timestamp = self.formatTime(record, "%Y-%m-%d %H:%M:%S")
        if record.levelno >= logging.WARNING:
            message = f"[{timestamp}][{record.levelname}]: {record.getMessage()}"
        elif record.levelno == JOBS:
            message = f"[{timestamp}][JOBS]: {record.getMessage()}"
        else:
            message = f"[{timestamp}] {record.getMessage()}"

        if record.exc_info:
            if not record.exc_text:
                record.exc_text = self.formatException(record.exc_info)
        if record.exc_text:
            message = f"{message}\n{record.exc_text}"
        if record.stack_info:
            message = f"{message}\n{self.formatStack(record.stack_info)}"
        return message

Separately/optionally: consider whether datajoint should install a global sys.excepthook at
all as an import side effect, given it affects exceptions unrelated to DataJoint in the importing
process. At minimum, whatever hook is installed must not be lossy relative to Python's default
behavior — today it is strictly worse than doing nothing.

[dimitri-yatsenko]

Thanks @arturoptophys — excellent write-up, and confirmed on master. For the record on how it got here: the hook was originally non-lossy (it inlined the exception message, and rendered a full traceback under DJ_LOG_LEVEL=DEBUG via the stdlib formatter), but two later changes compounded into the silent-discard behavior — the excepthook was simplified to rely solely on exc_info (a9dee8d), and the custom LevelAwareFormatter that replaced the stdlib one never renders exc_info (41dad5d). Rather than just make the hook non-lossy, we're removing the process-wide sys.excepthook entirely (a library shouldn't reformat unrelated exceptions in the host process) and hardening the formatter as defense-in-depth; fix incoming.