Object3d device fixes - #487
Conversation
|
Should we merge 417 first? |
I am not entirely sure what the standard practice should be here since this is a follow-up PR. But maybe it is good to merge 417 first? This PR inherits all the commits previously made by @salmanmkc, so it should still merge correctly after 417 is merged. |
|
Thank you Xincheng! If this is merged, #417 can be closed I guess? |
|
i recommend making this a stacked pr, it came out 6 days ago:
it let's you see them both at the same time, #417 will be under and this on top and then they can both merge separately but in the same stack 
or merge 417 first + rebase of main, it has another few commits that this pr doesn't have yet (probably conflicts) |
|
@Peter-Pater Can you rebase onto main? |
PR 417 was using the virtual camera's camera parameter rather than the physical ones. Also improved the real-world boundingbox performance by snapping it to the room frame rather than cardinal axises
e63078b to
35f82d6
Compare
Done :) |
|
Thanks for this @Peter-Pater! Was this on Galaxy XR that you tested this on? |
dli7319
left a comment
dli7319
left a comment
There was a problem hiding this comment.
Tested it on Galaxy XR.
It's technically working but the whole app intermittently freezes on device during the detection.
And the WebGPU errors are still there.
| /** A mesh of only horizontal (floor) triangles. */ | ||
| function floorMesh(): THREE.Mesh { | ||
| const positions: number[] = []; | ||
| for (let i = 0; i < 20; ++i) { |
There was a problem hiding this comment.
nit: comment on magic number
/*room_dim_m=*/20
or const ROOM_DIM_N = 20
| }); | ||
|
|
||
| it('recovers a rotated room', () => { | ||
| for (const d of [17, 30, 41]) { |
There was a problem hiding this comment.
nit: magic number
const d of /room_dimension=/[17, 30, 41]
or const
| const DEFAULTS = { | ||
| maxTriangles: 20000, | ||
| maxEdge: 0.2, | ||
| maxAbsNy: 0.25, |
|
|
||
| const DEFAULTS = { | ||
| maxTriangles: 20000, | ||
| maxEdge: 0.2, |
There was a problem hiding this comment.
what's the range, or a threshold?
4 participants
Object3d device fixes (on top of PR 417)
Description
Continues #417, which extracted the objects_3d pipeline into the objects3d addon. The extraction was right, but the pipeline didn't work on a headset: boxes were the wrong size, in the wrong place, all facing the same direction.
Root cause — the wrong camera. Rays were cast through a clone of the XR render camera (the ~90° union frustum between the eyes) while the pixels came from the physical passthrough camera (~48° FOV, near the right eye). Different intrinsics, different pose, so every ray left at the wrong angle.
Orientation is selectable via {mode: 'roomFrame' | 'free' | 'cardinal'}, defaulting to roomFrame, which degrades to today's output if estimation fails.
Type of Change
Checklist
Disclaimer:
Real-world performers suffers from small objects, sensor accuracy, and object cluttering, so it is not as perfect as in the simulator.