Skip to content

Add active vision - #1072

Open
Flova wants to merge 10 commits into
mainfrom
feature/active_vision
Open

Add active vision#1072
Flova wants to merge 10 commits into
mainfrom
feature/active_vision

Conversation

@Flova

@Flova Flova commented Sep 7, 2026

Copy link
Copy Markdown
Member

Summary

Adds a sampling based active vision algorithm

Proposed changes

Samples head targets and scores them based on some cost function including ball, field, robot coverage.

Related issues

Superceded #856

Checklist

  • Run pixi run build
  • Write documentation
  • Test on your machine
  • Test on the robot
  • Create issues for future work
  • Triage this PR and label it

We tested this on the IFA 2026

Flova and others added 8 commits August 16, 2026 23:03
Adds an ACTIVE_VISION head mode that samples candidate head trajectories
every cycle and scores them against the ball, team ball, robot detections
and a decaying field coverage map, instead of replaying a fixed pattern.

Extracts the existing head mover into a tested library first; behaviour of
the other head modes is unchanged. Missing inputs are reported and the
update is dropped rather than substituted for.

Also fixes a division by zero producing NaN keyframes for a single scan
line, and an off centre coverage grid when the field extent is not a
multiple of the cell size.
No invented defaults for missing input, throttled logging that names what
is missing, drop the update rather than the process, and keep retrying
inputs that legitimately arrive late.
Removing build/, install/ or log/ by hand breaks every package that is not
rebuilt with it, including ones outside the selection being worked on, and
recovering needs a full workspace rebuild.
Signed-off-by: Florian Vahl <florian@flova.de>
Signed-off-by: Florian Vahl <florian@flova.de>
Signed-off-by: Florian Vahl <florian@flova.de>
@github-project-automation github-project-automation Bot moved this to 🆕 New in Software Sep 7, 2026
@Flova Flova moved this from 🆕 New to 🏗 In progress in Software Sep 7, 2026
@Flova Flova added enhancement New feature or request motion labels Sep 7, 2026
@ayin21 ayin21 moved this from 🏗 In progress to 👀 In review in Software Sep 7, 2026
root_link:
type: string
default_value: "base_link"
description: "Link the sampled camera poses are resolved against. The map to this link transform places the robot on the field."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The map to this link transform places the robot on the field. This sentence is grammatically wrong or just really hard to parse.

control_period:
type: double
default_value: 0.05
description: "Time step (seconds) the controller advances the setpoint by each cycle. Should match the control loop period."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should make it match

debug:
enabled:
type: bool
default_value: true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably get turned off

@jaagut jaagut left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Written by GPT-6 on my behalf.

The sampling approach and separation of the planner from ROS integration make sense. Before merging, I’d like to address the following integration issues.

  1. Make field-dimension retrieval asynchronous.
    The retry timer calls get_parameters_from_other_node(), which internally uses spin_until_future_complete() on the head-mover node. After startup, that node already belongs to the EventsExecutor, so the retry throws when it attempts to attach it to another executor. Starting the parameter blackboard late therefore leaves active vision disabled. The separate timer also runs on the same executor thread, so its synchronous wait still blocks head updates.
    Please use an asynchronous request with a response callback and bounded retries. Add a regression test where the parameter blackboard becomes available after the initial attempt. Relevant code

  2. Synchronize parameter access across the two executor threads.
    behave() replaces params_ while the explicitly created detection thread reads params_.active_vision.map_frame. The listener’s internal mutex does not protect this separate copy. This can race during runtime parameter updates, even when only a scoring weight changes.
    Please give detection callbacks a synchronized configuration snapshot and release the lock before blocking TF lookups. The default EventsExecutor serializes its own callbacks, but it does not serialize them with the added detection executor. Relevant code

  3. Give the timer and controller one source for their period.
    The timer uses a hardcoded 50ms, while controller.control_period is independently configurable. Changing that parameter changes the setpoint step without changing the update frequency.
    Please derive the controller period from the duration used to create the timer, removing the separate parameter unless changing both together is supported. Relevant code

  4. Make the exposed parameters match what is actually applied.
    sampling.random_seed is never passed to the sampler. Coverage margin, cell_size, and half_life are loaded during field-dimension retrieval but ignored by subsequent parameter refreshes.
    Please wire the seed through and either implement updates for the coverage settings or declare them read-only. Frame settings should likewise be startup-only unless changing them also rebuilds or invalidates the dependent state. Add focused coverage for the supported update behavior. Relevant code

  5. Retry search initialization after incomplete joint states.
    prev_head_mode_ is updated before trajectory construction succeeds. If the current joint-state message lacks a head joint, construction fails and later valid messages never trigger another attempt for that mode.
    Please retry initialization once usable joint data arrives. Test an incomplete message followed by a complete message without another mode change. Relevant code

There are also a few optional simplifications:

  • Remove active_vision_hold_position_; it is written and reset but never read.
  • Record coverage before constructing a fully configured scorer, so preparation happens once per cycle.
  • Have debug markers use kinematics directly instead of constructing a scorer and accessing the world model.
  • Move the unrelated AGENTS.md changes out of this PR and update comments that still describe trajectory sampling.

The existing unit tests are useful, but they do not exercise the node-level startup and concurrency paths above.

@github-project-automation github-project-automation Bot moved this from 👀 In review to 🏗 In progress in Software Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request motion

Projects

Status: 🏗 In progress

Development

Successfully merging this pull request may close these issues.

4 participants