Skip to content

Web API Proposal

Jump to bottom
Paul Nechifor edited this page Jul 22, 2026 · 4 revisions

API Proposal

(caveat: this is what I'm working towards, there could be some differences when I get to implementing everything)

Terms:

  • cockpit: the UI part of the frontend
  • relay: the deno app which runs hosted (or locally) and serves the cockpit and relays data between robot and UI
  • relay bridge: the DimOS module which encodes data from DimOS to be sent to/from the relay

How to start

Normally, nothing happens:

dimos run unitree-go2

You can start a local relay:

dimos run --local-relay unitree-go2

Or connect to a remote one:

dimos run --relay http://re.lay unitree-go2

Cockpit panels/pages

The cockpit contains:

  • panels
  • layout elements (columns and rows)
  • pages (full page panels)

The cockpit is a fully built webapp which can only be configured from the backend. We can't ship local code to a remote site, so the cockpit can only be configured through some config (in the blueprints). (Locally, we may want to allow for custom panels/pages, but in hosted mode we can't allow users to build their own frontend code that we run on our site. We can only allow customizing the interface from pre-built elements.)

The cockpit and relay are built into the dimos package, so running from the package should work as well

Cockpit layout configuration

Each robot knows what it requires, so the cockpit layout is configured also where the robot is defined: in the blueprints.

That is, if a robot has two cameras, we need to define a cockpit layout which supports two video feeds.

Here's an example of a very simple layout.

from dimos.web.cockpit import Map2D, Row, Col, Stats, Teleop, Video, cockpit

unitree_go2_cockpit = autoconnect(
    unitree_go2,
    cockpit(
        layout=Row(
            Video("color_image", max_hz=15),
            Col(
                Map2D(costmap="global_costmap", pose="odom"),
                Teleop(max_linear=0.8),
                shares=[3, 1],
            ),
            shares=[2, 1],
        ),
        pages=[Stats()],
    ),
)

  • Row/Col are just layouts.

  • Video/Map2D/Teleop are panels.

  • Stats is a page (a full page panel accessible through a tab)

  • Panels contain implicit channel declarations. Video says it needs to listen to the /color_image topic and Map2D says it needs the /global_costmap channel and the /odom topic.

  • Video even contains an implicit transformation: it throttles the images to 15 frames per second.

The above is equivalent to this:

from dimos.web.cockpit import Map2D, Row, Col, Stats, Teleop, Video, cockpit

unitree_go2_cockpit = autoconnect(
    unitree_go2,
    cockpit(
        layout=Row(
            Video("color_image"),
            Col(
                Map2D(costmap="global_costmap", pose="odom"),
                Teleop(max_linear=0.8),
                shares=[3, 1],
            ),
            shares=[2, 1],
        ),
        pages=[Stats()],
        channels=[
            Channel("color_image", encoding="jpeg.v1", max_hz=15),
            Channel("global_costmap", encoding="costmap.zlib.v1"),
            Channel("odom", encoding="pose.v1"),
        ],
    ),
)

Or, to put it another way, simply declaring the need for a topic in any panel adds it to cockpit.channels with all-default values. Some panels have shortcuts to control the stream (like max_hz example). But obviously, if two panels have conflicts this will error out, and you'll have to declare the channel transformation explicitly in cockpit.channels.

To reduce the number of voxels in a 3D voxel map you could do something like this:

cockpit(
    layout=[Voxels3d('global_map', voxel_size=0.2, max_points=50000, max_hz=1)]
)

Again, this is the same as:

cockpit(
    layout=[Voxels3d('global_map')],
    channels=[Channel('global_map', encoding="voxels.v1", voxel_size=0.2, max_points=50000, max_hz=1)]
)

encoding=... represents the encoding method for the messages. These are strings, with versions, because they need to reference the precise decoding used in the frontend. Since the frontend code is fixed, you need to reference pre-registered encodings. That is, you can't add your own encoding, on the fly because there's no way to decode on the frontend.

Each encoding could take additional params. For example "jpeg.v1" can also take quality=75 (the JPEG quality).

We can have alternative encoders for a format. So jpeg.pillow.v1 could use Pillow instead of TurboJPEG (just an illustrative example).

We can have a generic json.v1 encoder which just does msg.__dict__. It could also maybe take omit=['field1', 'field2'] to drop some fields.

All the encodings are in a separate place, not methods on the classes themselves. This should be separate.

Over the wire

An example of how this looks over the wire. In the protocol we send a 'hello' message to establish a connection. It would look something like this:

{
  "version": 1,
  "robot": {"id": "go2-lab", "name": "go2-lab", "model": "unitree_go2"},
  "channels": [
    {"id": "color_image", "dir": "rx", "encoding": "jpeg.v1", "delivery": "latest", "maxHz": 14, "params": {"quality": 75}},
    {"id": "odom", "dir": "rx", "encoding": "pose.v1", "delivery": "reliable", "maxHz": 20},
    {"id": "global_costmap", "dir": "rx", "encoding": "costmap.zlib.v1", "delivery": "latest", "maxHz": 5},
    {"id": "cmd_vel", "dir": "tx", "encoding": "twist.v1", "params": {"maxLinear": 0.8, "watchdogMs": 300}}
  ],
  "panels": [
    {"id": "p0", "type": "video", "channels": {"image": "color_image"}},
    {"id": "p1", "type": "map2d", "channels": {"costmap": "global_costmap", "pose": "odom"}},
    {"id": "p2", "type": "teleop", "channels": {"cmd": "cmd_vel"}}
  ],
  "layout": {"row": ["p0", {"col": ["p1", "p2"], "shares": [3, 1]}], "shares": [2, 1]},
  "pages": []
}

The frontend

The cockpit is data-driven. It renders whatever the manifest describes, using two registries that are fixed at build time:

  • decoders: encoding id -> function turning payload bytes into a value
  • panels: panel type string -> React component

Adding frontend capability means adding entries to these registries. Everything else (which panels appear, which channels exist, the layout) comes from the manifest at runtime.

Data path:

relay -> one uni stream per message -> decoder (by encoding) -> channel store -> panels

(If you're worried about one stream per message: don't. QUIC streams have no round trip costs and it allows you to reset an in-flight message if a newer one is done.)

Session flow

What happens when you open the cockpit:

  1. Fetch /api/info, connect WebTransport, send hello with role: "viewer".
  2. The relay answers welcome and pushes the robots list.
  3. The cockpit picks the robot and sends watch.
  4. The relay sends back that robot's manifest. The layout renders from it.
  5. The visible panels need channels, so the cockpit subscribes to them. Frames start flowing.

On disconnect the transport reconnects with backoff and re-subscribes. If the manifest changed in the meantime (the robot was restarted with an edited blueprint), the layout remounts.

Subscribing to streams

Panels never talk to the network. We have an intermediary (the channel store) which does this. This is necessary because multiple pannels could be accessing the same channel.

  • When the first consumer of a channel appears, the store sends {"t": "sub", "ch": "odom"} (t is type). When the last one goes away it sends unsub.
  • The relay refcounts subscriptions across all viewers and notifies the relay bridge. So if there are 0 subscribers, the DimOS sends nothing at all.
  • Visibility drives consumption. A panel on a hidden page drops its subscriptions, so a backgrounded cockpit costs the robot nothing. (We may want to customise this, for example maybe odom should always render, but old images are definitly useless)

Two panels reading the same channel share one subscription and one stored value.

The store keeps one latest-wins slot per channel: {value, seq, ts}. The decoder runs once per frame, before the store. There are two ways to read a slot, and the difference is the core frontend rule:

// Low-rate values, normal React. Re-renders per message.
const pose = useChannel<Pose>(channels.pose);

// High-rate values (video, maps). Draw directly. React is not involved.
useEffect(() => store.subscribe(channels.image, draw), [channels.image]);

That is, draw is a function which alters the DOM without React involvement.

High-rate data must never cause React renders. Video and map panels draw to a canvas from store.subscribe. Readouts and the status bar use useChannel. Per-channel Hz and staleness are computed from the frame headers ({ch, seq, ts, meta}) and shown by the panel chrome for free.

Panel components

A panel type is a React component registered under its type string. It receives its channel bindings and params from the manifest, never hardcoded stream names:

type PanelProps = {
  channels: Record<string, string>; // role -> channel id, e.g. {pose: "odom"}
  params: Record<string, unknown>;
};

function PosePanel({ channels }: PanelProps) {
  const pose = useChannel<Pose>(channels.pose);
  if (!pose) return <WaitingForData />;
  return <div>{pose.x.toFixed(2)}, {pose.y.toFixed(2)}, yaw {pose.yaw.toFixed(2)}</div>;
}

registerPanel("pose", PosePanel);

Every panel gets the same chrome (PanelFrame): title bar, Hz/staleness dot, maximize. A panel whose channel has not produced data yet shows "waiting for data".

Some of the panel times which will be added: video, map2d, teleop, stats, chat (humancli), voxels3d.

Sending to the robot (tx)

tx channels flow the other way. The cockpit encodes a small JSON message, the relay forwards it, and the relay bridge decodes it and publishes on the DimOS topic. Panels send through the session by channel id, and the channel's delivery class picks the path. Loss-tolerant tx (teleop twist at publish_hz) rides datagrams. Reliable tx (chat) rides the control stream. The safety net for teleop is robot-side either way: the relay bridge watchdog zeroes the twist on silence, stop, or disconnect.

Custom panels

A custom panel is four small pieces, two per side. Example: draw the planned path.

Robot side, an encoder and a panel declaration:

@encoder("path.v1", input=Path)
def encode_path(msg, params):
    points = [[round(p.x, 3), round(p.y, 3)] for p in msg.poses]
    return json.dumps(points).encode(), {"frame": msg.frame_id}
cockpit(
    layout=Row(
        Video("color_image"),
        Panel("path2d", channels={"path": "planned_path"}),  # generic form, no dedicated class needed
    ),
)

(Note, I haven't decided on how to declare encoders. Either @encoder like above, or plain ENCODERS = {"path.v1": encode_path, ...})

Cockpit side, a decoder and a component:

registerDecoder("path.v1", (payload, meta) => JSON.parse(new TextDecoder().decode(payload)));
registerPanel("path2d", PathPanel); // a component like PosePanel above

But like I've said before, custom panels only work locally. You can't run custom panels in the hosted relay version because you can't deliver the code to the frontend.

Clone this wiki locally