Skip to content

fix(vom): vom support observe iframe - #105

Merged
iuyo5678 merged 1 commit into
mainfrom
fix/vom-iframe
Aug 18, 2026
Merged

fix(vom): vom support observe iframe#105
iuyo5678 merged 1 commit into
mainfrom
fix/vom-iframe

Conversation

@Ljy-0827

@Ljy-0827 Ljy-0827 commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

现有问题

原有 VOM 主要基于顶层文档的 AX Tree 和 DOMSnapshot 构建观察结果,没有完整处理 DOMSnapshot 中的 iframe 子文档及其 AX 节点归属。因此 iframe 内的文本、控件和交互节点无法稳定出现在 VOM 中;当页面存在多个或嵌套 iframe 时,还可能出现节点归属错误、backend node ID 冲突和结构挂载不正确的问题。

解决方案

扩展 VOM 采集和渲染链路,解析 DOMSnapshot 中的 iframe 子文档,并按照明确的 frameId、父 Frame 和 iframe owner 关系组织各 Frame 的 DOM 与 AX 数据。VOM 渲染时保留 Frame 边界,为 iframe 内可交互节点生成有效 ref,同时支持多个及嵌套 iframe。

后续接入

本 PR 是 iframe 观察与录制缺失 issue 的问题拆分,仅解决 iframe 内容进入 VOM 的采集、归属和渲染问题,不引入 OOPIF session 管理和完整的跨 Frame Geometry 机制。

后续 PR 将在此基础上增加 OOPIF target/session 发现、跨 target 数据采集和统一 Geometry,使 iframe 内的 ref 可以进一步用于点击、截图和 help 高亮。后续 PR3 可继续构建统一 VOM 语义图,改善节点命名、结构层级和跨 Frame 语义一致性。

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the VOM (View Observation Model) pipeline to correctly discover, capture, merge, and render iframe / OOPIF content by introducing a CDP-level frame graph, per-frame DOM/AX capture, cross-frame coordinate projection, and frame-aware paint-order comparisons.

Changes:

  • Add frame identity fields (frameId/contextScopeId/backendNodeId) to VOM nodes/results and prevent cross-frame handle-context leakage.
  • Capture DOMSnapshot + AX trees per frame/target (including OOPIF sessions) and merge them into a single scene with correct parent/owner attachment and top-level coordinate projection.
  • Make occlusion/blocking-layer logic frame-aware (paint order is treated as document-scoped; cross-frame comparisons are resolved via iframe owners).

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/vom/src/types.ts Add per-node frame/backend identity and a richer VomRef; track rootFrameId for paint-order boundaries.
packages/vom/src/render.ts Enforce contextScope boundaries for handle context; add frame-aware paint-order comparators; emit refs with frameId/backendNodeId.
packages/vom/src/layers.ts Restrict blocking-layer detection/membership to the root document’s frame paint-order space.
packages/vom/src/index.ts Re-export VomRef type.
packages/vom/src/tests/render.test.ts Add tests for context scoping and frame-scoped paint-order blocking behavior.
apps/extension/wxt.config.ts Raise minimum Chrome version to support flattened debugger sessions for OOPIF.
apps/extension/src/tools/vom/frame-document.ts Partition/deduplicate AX nodes by frame ownership and cut cross-frame parent edges.
apps/extension/src/tools/vom/frame-capture.ts Capture missing OOPIF DOM snapshots and per-frame AX trees; build per-frame documents.
apps/extension/src/tools/vom/capture.ts Track snapshot frameId/ancestry, populate frameNodes maps, and add form value extraction fields.
apps/extension/src/tools/vom/tests/frame-document.test.ts Validate ownership partitioning, dedupe, and collision handling for frame documents.
apps/extension/src/tools/vom/tests/frame-capture.test.ts Validate OOPIF DOM capture + top-level projection for multiple child targets.
apps/extension/src/tools/vom/tests/capture.test.ts Validate frameId propagation and captured frame ancestry outputs.
apps/extension/src/tools/snapshot-ref.ts Make ref resolution tab-bound and reject iframe-target refs as unsupported (for now).
apps/extension/src/tools/shared.ts Add target-aware CDP send helpers and extend CdpRunner with optional frame-graph APIs.
apps/extension/src/tools/observation.ts Route observation through per-frame capture/scene merge; store refs with optional child session identity.
apps/extension/src/tools/frame-geometry.ts Implement cross-target iframe geometry projection and polygon clipping utilities.
apps/extension/src/tools/tests/snapshot-ref.test.ts Test rejection of iframe-target refs in resolveSnapshotRef/lookupSnapshotRef.
apps/extension/src/tools/tests/observation.test.ts Add coverage for multi-frame scene merging and attachment rules.
apps/extension/src/tools/tests/human-loop.test.ts Update tests to use real RefStore instances with the new ref identity shape.
apps/extension/src/tools/tests/frame-geometry.test.ts Test projective mapping, polygon clipping, and OOPIF boundary transform behavior.
apps/extension/src/session-manager/ref-store.ts Store CDP session identity alongside backendNodeId/tabId for refs.
apps/extension/src/session-manager/tests/ref-store.test.ts Add coverage for persisting child CDP session identity in refs.
apps/extension/src/browser-driver/frame-graph.ts Introduce frame graph builder that merges multiple frame-tree sources by target boundary.
apps/extension/src/browser-driver/chromium-cdp.ts Add iframe target auto-attach discovery, recursive session tracking, and frame graph assembly.
apps/extension/src/browser-driver/tests/frame-graph.test.ts Test frame-graph merging across sibling/nested OOPIFs and deep trees/cycles.
apps/extension/src/browser-driver/tests/chromium-cdp.test.ts Test multi-session discovery, recursive attachment behavior, and frame graph correctness.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/extension/src/tools/vom/capture.ts
@Ljy-0827
Ljy-0827 requested review from iuyo5678 and shnpd August 18, 2026 06:06
@Ljy-0827 Ljy-0827 self-assigned this Aug 18, 2026
@Ljy-0827
Ljy-0827 marked this pull request as ready for review August 18, 2026 06:06
@Ljy-0827
Ljy-0827 force-pushed the fix/vom-iframe branch 2 times, most recently from b01a400 to 78d763b Compare August 18, 2026 07:24
@iuyo5678
iuyo5678 merged commit 22019cd into main Aug 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants