Skip to content

Fix test_basic_deployment in CI#2767

Open
Dreamsorcerer wants to merge 3 commits into
mainfrom
sam/fix-test-basic-deployment
Open

Fix test_basic_deployment in CI#2767
Dreamsorcerer wants to merge 3 commits into
mainfrom
sam/fix-test-basic-deployment

Conversation

@Dreamsorcerer

Copy link
Copy Markdown
Collaborator

No description provided.

@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adjusts test_basic_deployment so it can run more reliably in the intended CI lane. The main changes are:

  • Moves the test onto the self_hosted pytest marker.
  • Replaces the fixed sleep with a bounded polling loop.
  • Waits for movement, odometry, and lidar message counts before asserting.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
dimos/core/test_core.py Updates test_basic_deployment marker selection and replaces a fixed wait with count-based polling.

Reviews (3): Last reviewed commit: "Update test_core.py" | Re-trigger Greptile

Comment thread dimos/core/test_core.py
Comment thread dimos/core/test_core.py
Comment thread dimos/core/test_core.py
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2689 2 2687 70
View the full list of 2 ❄️ flaky test(s)
dimos.e2e_tests.test_dimsim_walk_forward::test_walk_forward

Flake rate in main: 23.85% (Passed 83 times, Failed 26 times)

Stack Traces | 206s run time
lcm_spy = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7a7a3da720f0>
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7a7a3a5ef060>
human_input = <function human_input.<locals>.send_human_input at 0x7a7a3a5ee980>
dim_sim = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7a7a3c559040>

    @pytest.mark.self_hosted_large
    def test_walk_forward(lcm_spy, start_blueprint, human_input, dim_sim) -> None:
        start_blueprint(
            "run",
            "--disable",
            "spatial-memory",
            "--disable",
            "security-module",
            "unitree-go2-agentic",
            simulator="dimsim",
        )
        lcm_spy.save_topic(".../McpClient/on_system_modules/res")
        lcm_spy.wait_for_saved_topic(".../McpClient/on_system_modules/res", timeout=1200.0)
    
        origin_x, origin_y = 1, 2
        dim_sim.set_agent_position(origin_x, origin_y)
    
        human_input("move forward 3 meter")
    
>       lcm_spy.wait_until_odom_position(origin_x + 3, origin_y, threshold=0.4, timeout=120)

dim_sim    = <dimos.e2e_tests.dim_sim_client.DimSimClient object at 0x7a7a3c559040>
human_input = <function human_input.<locals>.send_human_input at 0x7a7a3a5ee980>
lcm_spy    = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7a7a3da720f0>
origin_x   = 1
origin_y   = 2
start_blueprint = <function start_blueprint.<locals>.set_name_and_start at 0x7a7a3a5ef060>

dimos/e2e_tests/test_dimsim_walk_forward.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dimos/e2e_tests/lcm_spy.py:167: in wait_until_odom_position
    self.wait_for_message_result(
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7a7a3a5ecfe0>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7a7a3da720f0>
        threshold  = 0.4
        timeout    = 120
        x          = 4
        y          = 2
dimos/e2e_tests/lcm_spy.py:153: in wait_for_message_result
    wait_until(
        event      = <threading.Event at 0x7a7a3c5593a0: unset>
        fail_message = 'Failed to get to position x=4, y=2'
        listener   = <function LcmSpy.wait_for_message_result.<locals>.listener at 0x7a7a3a5ef6a0>
        predicate  = <function LcmSpy.wait_until_odom_position.<locals>.predicate at 0x7a7a3a5ecfe0>
        self       = <dimos.e2e_tests.lcm_spy.LcmSpy object at 0x7a7a3da720f0>
        timeout    = 120
        topic      = '/odom#geometry_msgs.PoseStamped'
        type       = <class 'dimos.msgs.geometry_msgs.PoseStamped.PoseStamped'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

predicate = <bound method Event.is_set of <threading.Event at 0x7a7a3c5593a0: unset>>

    def wait_until(
        predicate: Callable[[], bool],
        *,
        timeout: float,
        interval: float = 0.1,
        message: str | None = None,
    ) -> None:
        """Poll ``predicate`` until it returns truthy or ``timeout`` elapses."""
        deadline = time.monotonic() + timeout
        while time.monotonic() < deadline:
            if predicate():
                return
            time.sleep(interval)
>       raise TimeoutError(message or f"Timed out after {timeout}s waiting for condition")
E       TimeoutError: Failed to get to position x=4, y=2

deadline   = 3535953.085535148
interval   = 0.1
message    = 'Failed to get to position x=4, y=2'
predicate  = <bound method Event.is_set of <threading.Event at 0x7a7a3c5593a0: unset>>
timeout    = 120

.../utils/testing/waiting.py:35: TimeoutError
dimos.perception.test_image_embedding.TestImageEmbedding::test_clip_embedding_initialization

Flake rate in main: 30.00% (Passed 7 times, Failed 3 times)

Stack Traces | 27.9s run time
request = <SubRequest 'monitor_threads' for <Function test_clip_embedding_initialization>>

    @pytest.fixture(autouse=True)
    def monitor_threads(request):
        # Capture threads before test runs
        test_name = request.node.nodeid
        with _seen_threads_lock:
            _before_test_threads[test_name] = {
                t.ident for t in threading.enumerate() if t.ident is not None
            }
    
        yield
    
        with _seen_threads_lock:
            before = _before_test_threads.get(test_name, set())
    
        # Threads intentionally left running for the whole process and cleaned up on
        # exit, so they don't count as per-test leaks.
        expected_persistent_thread_prefixes = [
            "Dask-Offload",
            # HuggingFace safetensors conversion thread - no user cleanup API
            # https://github..../transformers/issues/29513
            "Thread-auto_conversion",
        ]
    
        def live_new_threads():
            # Threads created during this test that are still running. A thread that
            # has already stopped is not a leak -- it's done, just not yet reaped
            # from threading's registry -- so we key on is_alive(), not presence.
            result = []
            for t in threading.enumerate():
                if t.ident is None or t.ident in before or t.name == "MainThread":
                    continue
                if any(t.name.startswith(prefix) for prefix in expected_persistent_thread_prefixes):
                    continue
                if t.is_alive():
                    result.append(t)
            return result
    
        # Some C extensions tear their callback threads down asynchronously, so a
        # thread can stay alive briefly after the test cleaned up its owner (notably
        # zenoh's pyo3-closure threads, freed shortly after the session is closed).
        # A single snapshot races that teardown and flags a thread that is about to
        # exit. Give new threads a grace period to drain; only ones that stay alive
        # are real leaks (a genuinely leaked thread never exits).
        deadline = time.monotonic() + 5.0
        leaked = live_new_threads()
        while leaked and time.monotonic() < deadline:
            time.sleep(0.02)
            leaked = live_new_threads()
    
        if not leaked:
            return
    
        with _seen_threads_lock:
            # Report each leaked thread only once across the session.
            truly_new = [t for t in leaked if t.ident not in _seen_threads]
            for t in leaked:
                _seen_threads.add(t.ident)
    
        if not truly_new:
            return
    
        thread_names = [t.name for t in truly_new]
    
>       pytest.fail(
            f"Non-closed threads created during this test. Thread names: {thread_names}. "
            "Please look at the first test that fails and fix that."
        )
E       Failed: Non-closed threads created during this test. Thread names: ['Thread-252']. Please look at the first test that fails and fix that.

before     = {124717805999808, 124717814392512, 124717831177920, 124717839570624, 124717856356032, 124717864748736, ...}
deadline   = 4527038.613525735
expected_persistent_thread_prefixes = ['Dask-Offload', 'Thread-auto_conversion']
leaked     = [<TMonitor(Thread-252, started daemon 124717789214400)>]
live_new_threads = <function monitor_threads.<locals>.live_new_threads at 0x716ee7303d80>
request    = <SubRequest 'monitor_threads' for <Function test_clip_embedding_initialization>>
t          = <TMonitor(Thread-252, started daemon 124717789214400)>
test_name  = 'dimos/perception/test_image_embedding.py::TestImageEmbedding::test_clip_embedding_initialization'
thread_names = ['Thread-252']
truly_new  = [<TMonitor(Thread-252, started daemon 124717789214400)>]

dimos/conftest.py:273: Failed

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant