fix(integ): update debugger/profiler for opt-in defaults - #6189
Open
nargokul wants to merge 4 commits into
Open
Conversation
PR aws#6188 makes DebuggerHookConfig and active ProfilerConfig opt-in rather than attached by default. Update the integ tests that assumed the old default-on behavior, and fix one source gap: - estimator.py: auto-attach a ProfilerConfig when a ProfilerRule is passed, mirroring the existing debugger-rule handling. Without this, a profiler rule was emitted alongside DisableProfiler=True (an invalid request shape). - test_debugger.py: pass debugger_hook_config=True in the PyTorch/TensorFlow/ XGBoost checkpointing blocks so the force-disable logic has a hook to act on. - test_profiler.py: request profiling explicitly via profiler_config in the two tests that relied on the default-on profiler. - test_training_steps.py: request profiling explicitly in the debugger+profiler pipeline test.
CreateTrainingJob rejects the request with:
ValidationException: "ProfilerRuleConfigurations" cannot be provided
without "ProfilerConfig".
Two unit tests asserted a populated profiler_rule_configs alongside
profiler_config={"DisableProfiler": True}, a shape the service will not
accept. Verified against the real SageMaker API in us-west-2. Update both
to expect the ProfilerConfig that is now attached when a ProfilerRule is
passed.
nargokul
force-pushed
the
fix-integ-tests-debugger-profiler-opt-in
branch
from
August 18, 2026 21:14
d0c24ca to
6cea507
Compare
nargokul
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on / continues #6188, which makes
DebuggerHookConfigand an activeProfilerConfigopt-in instead of attached by default. The integ suite for #6188 failed because several tests (and one source path) still assumed the old default-on behavior.This PR fixes those 5 integ-test failures. Because #6188 is not yet merged into
master-v2, this branch includes #6188's two commits plus the fixes here, so the changes are coherent on their own.Failing tests fixed
test_debugger.py::test_debug_hook_disabled_with_checkpointingtest_profiler.py::test_mxnet_with_default_profiler_config_and_profiler_ruletest_profiler.py::test_mxnet_profiling_with_disable_debugger_hooktest_profiler.py::test_mxnet_with_built_in_profiler_rule_with_custom_parameterssagemaker/workflow/test_training_steps.py::test_training_job_with_debugger_and_profilerChanges
Source (
src/sagemaker/estimator.py)_prepare_profiler_for_training, auto-attach aProfilerConfigwhen aProfilerRuleis passed, mirroring the existing debugger-rule handling. Without this, passing a profiler rule emitted aProfilerRuleConfigurationalongsideProfilerConfig: {"DisableProfiler": True}— an invalid request shape (a profiler rule with profiling disabled). Passing aProfilerRuleis itself an opt-in, so it now gets a config to attach to.Tests
test_debugger.py: addeddebugger_hook_config=Trueto the PyTorch, TensorFlow, and XGBoost checkpointing blocks. Debugger is now opt-in, so a hook must exist for the "checkpointing + distributed disables the hook" logic to act on. The block asserting the hook survives (XGBoost) and the blocks asserting it is force-disabled toFalse(PyTorch/TF) both need an explicit opt-in now.test_profiler.py(2 tests): request profiling explicitly viaprofiler_config=ProfilerConfig(system_monitor_interval_millis=500). Thedisable_debugger_hooktest keepsProfilingStatus == "Enabled"so it still proves that turning off the debug hook does not turn off profiling.test_training_steps.py: request profiling explicitly in the debugger+profiler pipeline test (plus theProfilerConfigimport).Testing
All 5 tests were run for real against AWS in
us-west-2(Python 3.10, matching CI'spy310) — the profiler and workflow tests launched actual training jobs / a pipeline execution:test_debug_hook_disabled_with_checkpointingtest_mxnet_with_default_profiler_config_and_profiler_ruletest_mxnet_profiling_with_disable_debugger_hooktest_mxnet_with_built_in_profiler_rule_with_custom_parameterstest_training_job_with_debugger_and_profiler