diff --git a/.changeset/sync-node-modules.md b/.changeset/sync-node-modules.md new file mode 100644 index 00000000..31904923 --- /dev/null +++ b/.changeset/sync-node-modules.md @@ -0,0 +1,5 @@ +--- +"@cloudflare/computer": patch +--- + +Sync `node_modules` by default so package manager installs persist across workspace runtimes. diff --git a/docs/02_sync_protocol.md b/docs/02_sync_protocol.md index 83f58177..610bd4f6 100644 --- a/docs/02_sync_protocol.md +++ b/docs/02_sync_protocol.md @@ -405,56 +405,33 @@ first-class conflict primitives. ## Ignore lists - -The `ignore` option hides path segments from the pull. Excluded -paths are still written and read inside the container — the bytes just -never cross the wire back to the DO. This is essential for any large -directory of derived files: `node_modules`, `.next`, `target`, -`__pycache__`, `dist`. Without an ignore, a single `npm install` would -push tens of thousands of small files through the sync wire on the -next pull. - -The default is `["node_modules"]`, applied server-side when `ignore` is -omitted. A caller-supplied list **replaces** the default — it does not -extend it. Pass `[]` to disable ignoring entirely, or pass your full -list (including `"node_modules"` if you still want it) to customise. - -### Ignored entries - -Ignored paths are **invisible to the `Workspace.fs` API**. They do not -appear in `readdir`, `stat` returns `ENOENT`, and `readFile` returns -`ENOENT`. The bytes still live inside the container, so anything that -*uses* the ignored files — `exec("node ...")`, build tools, anything -running container-side — keeps working. The exclusion only affects what -crosses the wire **and** what the DO-side API surfaces. - -This is a deliberately narrow surface for the initial release. Whether -ignored entries should be representable to the DO at all (as stubs, as -a separate shell-only namespace, or not at all) is left to a future -iteration — see [Future considerations](#future-considerations). +The optional `ignore` list hides matching path segments from a sync +stream. Filtering is opt-in and applies only while coalescing changes +for the wire; the filesystem retains and exposes the paths normally. +An ignored path is therefore not deleted or hidden from `Workspace.fs`, +but it is omitted from that particular pull. This is useful for large +directories of derived files such as `.next`, `target`, `__pycache__`, +or `dist`. + +The default is `[]`. A caller-supplied list replaces the configured +list for that fetch. Pass an explicit list when a sync peer should omit +paths, or pass `[]` to include every path. To omit `node_modules`, for +example, configure `ignore: ["node_modules"]` explicitly. + +### Filtered entries + +Filtering affects only the change stream. Files remain readable and +writable through `Workspace.fs`, and container-side commands such as +`exec("node ...")` can use them normally. Filtered entries still advance +the receiving peer's fetch cursor, so an ignore configuration should be +kept stable for a sync relationship; removing a pattern does not replay +changes that were already filtered. ## Future considerations Items deferred from the initial design. File an issue if a real use case depends on a particular resolution. -### Representing ignored entries to the DO - -Today ignored paths are entirely invisible to `Workspace.fs`. That is -the simplest contract but it loses one piece of information: tools that -want to enumerate "everything the agent's exec can see" can't get it -from the DO. Two options worth weighing later: - -- **Stub entries with an `ignored` flag** on `stat()`, surfaced via - `readdir`. Easy to retrofit; surprising for tools that walk the tree - and don't check the flag. -- **An explicit shell-only namespace** — e.g. `workspace.runtime.readdir` - returns container-only entries, `workspace.fs.readdir` stays clean. - Cleaner separation, larger API surface. - -Either way, the bytes never cross the wire; the question is purely how -much the DO admits exists. - ### Bloom/cuckoo filter over `vfs_blobs.hash` Every pull does a `hasObjects` probe round-trip. With tens of thousands diff --git a/docs/03_filesystem_schema.md b/docs/03_filesystem_schema.md index 9df4dfaa..5de4ec54 100644 --- a/docs/03_filesystem_schema.md +++ b/docs/03_filesystem_schema.md @@ -74,9 +74,10 @@ The `vfs_nodes_by_rev` index supports `coalesceChanges`'s cursor scan over live inodes, which the sync protocol calls once per pull to enumerate everything modified after the last fetch cursor. -There is no `ignored` column: ignored paths are entirely invisible to -the DO-side filesystem API (see -[02. Sync Protocol → Ignored entries](./02_sync_protocol.md#ignored-entries)). +There is no `ignored` column. Ignore patterns are an ephemeral sync +option, applied while building a change stream; they do not change the +filesystem namespace or the paths visible through `Workspace.fs`. See +[02. Sync Protocol → Ignore lists](./02_sync_protocol.md#ignore-lists). ### `vfs_dirents` — name → inode mapping diff --git a/docs/06_mount_interface.md b/docs/06_mount_interface.md index 3c911fb9..6cb830fc 100644 --- a/docs/06_mount_interface.md +++ b/docs/06_mount_interface.md @@ -210,7 +210,7 @@ provider-specific config: | Option | Default | Meaning | | --- | --- | --- | | `mode` | `"read-only"` | `"read-only"` or `"read-write"`. | -| `ignore` | `[]` | Path segments hidden from the pull *and* from `Workspace.fs`. Composed with the top-level `ignore` by union. See [02. Sync Protocol → Ignored entries](./02_sync_protocol.md#ignored-entries). | +| `ignore` | `[]` | Path segments omitted from the mount's sync pull. The paths remain visible through the underlying `Workspace.fs`; the option is composed with the top-level `ignore` by union. See [02. Sync Protocol → Ignore lists](./02_sync_protocol.md#ignore-lists). | | `writeBack` | `"debounce"` | `"debounce"` (default) or `"manual"`. See “Write-back gating” above. | | `writeBackMs` | `500` | Debounce window in milliseconds. Ignored when `writeBack: "manual"`. | | `maxBytes` | unbounded | Hard cap on total bytes indexed from this mount. Exceeding throws at index time before any data lands in `vfs_nodes`. | @@ -218,7 +218,7 @@ provider-specific config: The workspace-level `ignore` option (the renamed `pullIgnore`) applies to every mount and to top-level paths. Mount-level `ignore` extends it -for that mount only. +for that mount only. No paths are ignored by default. ## Mount conflicts diff --git a/docs/08_capnweb_interface.md b/docs/08_capnweb_interface.md index d708b0d4..72672373 100644 --- a/docs/08_capnweb_interface.md +++ b/docs/08_capnweb_interface.md @@ -312,7 +312,7 @@ type WireError = { | Code | Meaning | | --- | --- | -| `ENOENT` | Path does not exist on the receiver (covers ignored paths, which are invisible to `Workspace.fs`), or `getExec` / `disposeExec` referenced an unknown id. | +| `ENOENT` | Path does not exist on the receiver, or `getExec` / `disposeExec` referenced an unknown id. Sync ignore patterns do not alter filesystem path lookups. | | `EUNKNOWN_HASH` | `fetchObjects` referenced a hash the receiver has no record of; raised via `createWorkspaceError`. | | `EEXEC_BUSY` | `exec` was called with an `id` that's already in use by a live run. | | `ELOG_TRUNCATED` | `getExec` resume point is older than the retained log. | diff --git a/packages/dofs/README.md b/packages/dofs/README.md index 2027226a..94bd285f 100644 --- a/packages/dofs/README.md +++ b/packages/dofs/README.md @@ -17,7 +17,7 @@ This package exposes a JavaScript module, not a CLI. It bundles three layers tha - A `Database` wrapper around Durable Object SQL storage plus `initializeSchema` for the `vfs_*` tables. - Filesystem primitives under `src/fs/*` (`mkdir`, `writeFile`, `readFile`, `rm`, `readdir`, `stat`, `lstat`, `chmod`, `find`, `ls`, `grep`, `symlink`, `readlink`, `gc`, `watch`) operating on a `Database`. - `SQLiteWorkspaceProvider`, a `@platformatic/vfs` adapter that composes those primitives into a node-shaped filesystem (fd table, positional `readSync`/`writeSync`, `watchSync`, symlinks). This is what `computerd` mounts via FUSE. -- Sync protocol building blocks operating on the same `Database`: `applyChanges`, `stageBlob`, `materialiseChange`, `coalesceChanges`, `fetchChanges`, `fetchObjects`, `hasObjects`, `pushObjects`, `buildManifest`, `currentRev`, `compareChangeCursors`, `readWatermark`/`writeWatermark`, `assertAppliedPushCursor`, and `DEFAULT_IGNORE`/`isIgnored`. The wire wiring lives in `@cloudflare/computer-rpc`. +- Sync protocol building blocks operating on the same `Database`: `applyChanges`, `stageBlob`, `materialiseChange`, `coalesceChanges`, `fetchChanges`, `fetchObjects`, `hasObjects`, `pushObjects`, `buildManifest`, `currentRev`, `compareChangeCursors`, `readWatermark`/`writeWatermark`, `assertAppliedPushCursor`, and the opt-in ignore matcher `isIgnored` (the default ignore list is empty). The wire wiring lives in `@cloudflare/computer-rpc`. Minimal DO-side usage — initialize the schema; the `Database` becomes the handle every other helper takes: diff --git a/packages/dofs/src/sync/ignore.test.ts b/packages/dofs/src/sync/ignore.test.ts index 37975760..7082b434 100644 --- a/packages/dofs/src/sync/ignore.test.ts +++ b/packages/dofs/src/sync/ignore.test.ts @@ -1,10 +1,15 @@ import { describe, expect, it } from "vitest"; -import { isIgnored } from "./ignore.js"; +import { DEFAULT_IGNORE, isIgnored } from "./ignore.js"; describe("isIgnored", () => { const list = ["node_modules", ".next", "target"]; + it("has no default exclusions", () => { + expect(DEFAULT_IGNORE).toEqual([]); + expect(isIgnored("/node_modules/package/index.js", DEFAULT_IGNORE)).toBe(false); + }); + it("returns false for paths that don't intersect the list", () => { expect(isIgnored("/src/index.ts", list)).toBe(false); expect(isIgnored("/README.md", list)).toBe(false); diff --git a/packages/dofs/src/sync/ignore.ts b/packages/dofs/src/sync/ignore.ts index 91f7b841..fca1e56e 100644 --- a/packages/dofs/src/sync/ignore.ts +++ b/packages/dofs/src/sync/ignore.ts @@ -1,14 +1,14 @@ -// Path segment matcher for the container-side ignore list. The -// container uses this to drop paths from coalesceChanges before -// they hit the wire; the DO's Workspace.fs surface uses the same -// helper to make ignored paths invisible to API consumers. +// Path-segment matcher for an explicitly configured sync ignore list. +// Matching is whole-segment: a pattern matches that segment anywhere in +// the path, but does not match a longer segment. Patterns are plain +// strings, not globs; we can extend to globs later if a real case +// demands it. // -// Matching is whole-segment: "node_modules" matches the segment -// node_modules anywhere in the path but does not match -// node_modules_old or my_node_modules. Patterns are plain strings, -// not globs; we can extend to globs later if a real case demands it. +// The default is intentionally empty. Ignoring paths is opt-in because +// the filesystem and sync surfaces should preserve the same namespace. -export const DEFAULT_IGNORE = ["node_modules"]; +// Kept as a named export for callers that need the package default. +export const DEFAULT_IGNORE: string[] = []; export function isIgnored(path: string, patterns: string[]): boolean { if (patterns.length === 0) return false; diff --git a/packages/rpc/src/server.ts b/packages/rpc/src/server.ts index ebe7c467..c9c46dcb 100644 --- a/packages/rpc/src/server.ts +++ b/packages/rpc/src/server.ts @@ -12,7 +12,6 @@ import { compareChangeCursors, currentRev, type Database, - DEFAULT_IGNORE, fetchObjects, hasObjects, materialiseChange, @@ -178,8 +177,7 @@ class SyncRPCServer extends RpcTarget implements SyncRPC { } } const after = input.after ?? { rev: 0, path: null }; - const ignore = - input.ignore ?? (this.options.ignore.length > 0 ? this.options.ignore : DEFAULT_IGNORE); + const ignore = input.ignore ?? this.options.ignore; const snapshotRev = currentRev(this.db); const currentCursor = { rev: snapshotRev, path: null }; return { diff --git a/packages/rpc/src/sync-driver.test.ts b/packages/rpc/src/sync-driver.test.ts index 46c6e294..5fb6b8a7 100644 --- a/packages/rpc/src/sync-driver.test.ts +++ b/packages/rpc/src/sync-driver.test.ts @@ -1082,7 +1082,7 @@ describe("sync driver — blob-stage recovery", () => { source.mkdirSync("/repo/node_modules/tiny", { recursive: true }); source.writeFileSync("/repo/package.json", '{"name":"fixture"}\n'); source.writeFileSync("/repo/dist/result.txt", "installed"); - source.writeFileSync("/repo/node_modules/tiny/index.js", "ignored"); + source.writeFileSync("/repo/node_modules/tiny/index.js", "module.exports = 'installed';"); let injected = false; const failingDb = new Database({ @@ -1109,7 +1109,9 @@ describe("sync driver — blob-stage recovery", () => { const destination = new SQLiteWorkspaceProvider(receiver.db, { now: () => 1 }); expect(resumed.applied).toBeGreaterThan(0); expect(destination.readFileSync("/repo/dist/result.txt", "utf8")).toBe("installed"); - expect(destination.existsSync("/repo/node_modules")).toBe(false); + expect(destination.readFileSync("/repo/node_modules/tiny/index.js", "utf8")).toBe( + "module.exports = 'installed';", + ); expect(readFetchCursor(receiver.db)).toEqual({ rev: currentRev(upstream.db), path: null,