fix(vom): vom support observe iframe - #105
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR extends the VOM (View Observation Model) pipeline to correctly discover, capture, merge, and render iframe / OOPIF content by introducing a CDP-level frame graph, per-frame DOM/AX capture, cross-frame coordinate projection, and frame-aware paint-order comparisons.
Changes:
- Add frame identity fields (frameId/contextScopeId/backendNodeId) to VOM nodes/results and prevent cross-frame handle-context leakage.
- Capture DOMSnapshot + AX trees per frame/target (including OOPIF sessions) and merge them into a single scene with correct parent/owner attachment and top-level coordinate projection.
- Make occlusion/blocking-layer logic frame-aware (paint order is treated as document-scoped; cross-frame comparisons are resolved via iframe owners).
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/vom/src/types.ts | Add per-node frame/backend identity and a richer VomRef; track rootFrameId for paint-order boundaries. |
| packages/vom/src/render.ts | Enforce contextScope boundaries for handle context; add frame-aware paint-order comparators; emit refs with frameId/backendNodeId. |
| packages/vom/src/layers.ts | Restrict blocking-layer detection/membership to the root document’s frame paint-order space. |
| packages/vom/src/index.ts | Re-export VomRef type. |
| packages/vom/src/tests/render.test.ts | Add tests for context scoping and frame-scoped paint-order blocking behavior. |
| apps/extension/wxt.config.ts | Raise minimum Chrome version to support flattened debugger sessions for OOPIF. |
| apps/extension/src/tools/vom/frame-document.ts | Partition/deduplicate AX nodes by frame ownership and cut cross-frame parent edges. |
| apps/extension/src/tools/vom/frame-capture.ts | Capture missing OOPIF DOM snapshots and per-frame AX trees; build per-frame documents. |
| apps/extension/src/tools/vom/capture.ts | Track snapshot frameId/ancestry, populate frameNodes maps, and add form value extraction fields. |
| apps/extension/src/tools/vom/tests/frame-document.test.ts | Validate ownership partitioning, dedupe, and collision handling for frame documents. |
| apps/extension/src/tools/vom/tests/frame-capture.test.ts | Validate OOPIF DOM capture + top-level projection for multiple child targets. |
| apps/extension/src/tools/vom/tests/capture.test.ts | Validate frameId propagation and captured frame ancestry outputs. |
| apps/extension/src/tools/snapshot-ref.ts | Make ref resolution tab-bound and reject iframe-target refs as unsupported (for now). |
| apps/extension/src/tools/shared.ts | Add target-aware CDP send helpers and extend CdpRunner with optional frame-graph APIs. |
| apps/extension/src/tools/observation.ts | Route observation through per-frame capture/scene merge; store refs with optional child session identity. |
| apps/extension/src/tools/frame-geometry.ts | Implement cross-target iframe geometry projection and polygon clipping utilities. |
| apps/extension/src/tools/tests/snapshot-ref.test.ts | Test rejection of iframe-target refs in resolveSnapshotRef/lookupSnapshotRef. |
| apps/extension/src/tools/tests/observation.test.ts | Add coverage for multi-frame scene merging and attachment rules. |
| apps/extension/src/tools/tests/human-loop.test.ts | Update tests to use real RefStore instances with the new ref identity shape. |
| apps/extension/src/tools/tests/frame-geometry.test.ts | Test projective mapping, polygon clipping, and OOPIF boundary transform behavior. |
| apps/extension/src/session-manager/ref-store.ts | Store CDP session identity alongside backendNodeId/tabId for refs. |
| apps/extension/src/session-manager/tests/ref-store.test.ts | Add coverage for persisting child CDP session identity in refs. |
| apps/extension/src/browser-driver/frame-graph.ts | Introduce frame graph builder that merges multiple frame-tree sources by target boundary. |
| apps/extension/src/browser-driver/chromium-cdp.ts | Add iframe target auto-attach discovery, recursive session tracking, and frame graph assembly. |
| apps/extension/src/browser-driver/tests/frame-graph.test.ts | Test frame-graph merging across sibling/nested OOPIFs and deep trees/cycles. |
| apps/extension/src/browser-driver/tests/chromium-cdp.test.ts | Test multi-session discovery, recursive attachment behavior, and frame graph correctness. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Ljy-0827
marked this pull request as ready for review
August 18, 2026 06:06
Ljy-0827
force-pushed
the
fix/vom-iframe
branch
2 times, most recently
from
August 18, 2026 07:24
b01a400 to
78d763b
Compare
iuyo5678
force-pushed
the
fix/vom-iframe
branch
from
August 18, 2026 11:18
78d763b to
8d7c0aa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
现有问题
原有 VOM 主要基于顶层文档的 AX Tree 和 DOMSnapshot 构建观察结果,没有完整处理 DOMSnapshot 中的 iframe 子文档及其 AX 节点归属。因此 iframe 内的文本、控件和交互节点无法稳定出现在 VOM 中;当页面存在多个或嵌套 iframe 时,还可能出现节点归属错误、backend node ID 冲突和结构挂载不正确的问题。
解决方案
扩展 VOM 采集和渲染链路,解析 DOMSnapshot 中的 iframe 子文档,并按照明确的 frameId、父 Frame 和 iframe owner 关系组织各 Frame 的 DOM 与 AX 数据。VOM 渲染时保留 Frame 边界,为 iframe 内可交互节点生成有效 ref,同时支持多个及嵌套 iframe。
后续接入
本 PR 是 iframe 观察与录制缺失 issue 的问题拆分,仅解决 iframe 内容进入 VOM 的采集、归属和渲染问题,不引入 OOPIF session 管理和完整的跨 Frame Geometry 机制。
后续 PR 将在此基础上增加 OOPIF target/session 发现、跨 target 数据采集和统一 Geometry,使 iframe 内的 ref 可以进一步用于点击、截图和 help 高亮。后续 PR3 可继续构建统一 VOM 语义图,改善节点命名、结构层级和跨 Frame 语义一致性。