Skip to content

perf: scan the workspace once per create instead of once per port - #11

Merged
laysakura merged 2 commits into
mainfrom
perf/single-workspace-port-scan
Sep 10, 2026
Merged

laysakura merged 2 commits into
mainfrom
perf/single-workspace-port-scan

Conversation

@okazu-dm

Copy link
Copy Markdown
Contributor

Summary

  • sprout create walked the entire workspace once per {{ auto_port() }} placeholder. It now walks it once per create, and the walk itself skips dependency and cache directories
  • On the workspace that motivated this (~490k files under .sprout/, 8 auto-assigned ports), the .env generation step went from 97s to 2s; a single walk went from 10.0s to 0.66s
  • Port results are unchanged: on a real workspace with 661 .env files, the old and new walks return identical file sets and identical port sets

Problem

Two independent causes, both measured with per-phase timestamps:

  1. find_available_port() called get_used_ports() on every call, so .sprout/ was walked once per placeholder — 9 walks for a template with 8 ports, at ~11s each.
  2. Every walk descended into node_modules, .venv, .git and friends, which held 77% of the files under .sprout/.

Changes

  • find_available_port(used_ports=None) accepts a pre-computed port set. parse_env_template() builds that set once per file and reuses it, so get_used_ports() runs once per create instead of once per port.
  • iter_env_files() replaces Path.rglob("*.env") with os.walk(), pruning SCAN_EXCLUDED_DIRS in place.
  • The scan is deferred until the first {{ auto_port() }} is reached, so a template without one never walks the worktrees — and never shells out to git rev-parse.

Correctness notes

Three ways this can go wrong, and how each is handled:

  • A worktree can be named after an excluded directory. sprout create venv (or fix/venv) puts a real worktree behind an excluded name. Pruning by name alone made its .env invisible and handed its ports out again. is_port_available() is not a safety net here: it only sees a conflict while the other stack is running, so the collision surfaces later as a failure to bind. An excluded name is still walked when the directory holds a .git entry, which every worktree root does.
  • used_ports replaces the scan rather than adding to it. An empty set means "nothing is taken"; pass None to have the function scan. Both docstrings spell this out, because a partial set silently gives up collision detection against the other worktrees.
  • Non-regular files. Dropping the is_file() guard let a fifo named .env block read_text() indefinitely. The guard is back, applied only to names that already end in .env, so it costs one stat per candidate.

Test Plan

  • make all-checks green: ruff, mypy, 103 tests (up from 70)
  • Old vs new walk compared on a real 490k-file workspace: identical file sets (661 files) and identical port sets (48 ports)
  • Parity checked for dotfiles, local.env-style names, directories named *.env, dangps, symlinked worktrees and permission errors
  • Mutation tested: 11 mutations — prune removal, dropped worktree-root escape, unconditling in both functions, per-file rescan, shrunken exclude list, exact-name match, missingfile_ports.add(), dropped is_file() — each kills at least one test
  • End-to-end sprout create on the original workspace: .env generation 97s -> 2s

okazu-dm and others added 2 commits September 9, 2026 19:00
`sprout create` spent 97 of its 125 seconds generating .env files on a
workspace holding ~490k files under .sprout/. Two causes:

- find_available_port() called get_used_ports() on every call, so the
  entire workspace was walked once per {{ auto_port() }} placeholder:
  9 walks at ~11s each for a template with 8 auto-assigned ports.
- get_used_ports() descended into node_modules, .venv, .git and friends,
  which hold most of those files and never a sprout-generated .env.

find_available_port() now accepts a pre-computed port set, and
parse_env_template() builds that set once per file, so the workspace is
scanned once per create instead of once per port. The walk prunes
dependency, cache and VCS directories, which also stops ports from being
reserved out of .env files that sprout did not write.

Measured on the same workspace: the .env generation step went from 97s
to 2s, and a single scan from 10.0s to 0.66s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZFDWL7uPZTBDWvLDsTdJD
Review of the previous commit found three defects in it:

- The prune matched by directory name at any depth, so a worktree whose
  own name is on the exclude list (a branch called "venv", or "fix/venv")
  had its .env skipped and its ports handed out to the next worktree.
  is_port_available() does not catch that: it only sees a conflict while
  the other stack is running. An excluded name is now still walked when
  the directory holds a .git entry, which every worktree root does.
- parse_env_template() scanned the workspace even for templates with no
  {{ auto_port() }}, dragging git rev-parse and a full walk into every
  call. That also broke test isolation: 15 unit tests started depending
  on being run inside a git repository. The scan now happens on the first
  placeholder that needs a port, and not at all otherwise.
- Dropping the is_file() guard let a fifo named .env reach read_text()
  and block indefinitely. The guard is back, applied only to names that
  already end in .env, so it costs one stat per candidate.

The tests grew from 70 to 103. The new ones pin what mutation testing
showed was unprotected: the set() vs None distinction, the prune itself,
walking a worktree root that carries an excluded name, suffixed names
such as local.env, in-file port uniqueness, and one workspace scan per
create. CHANGELOG now credits the speedup to both changes rather than to
the rescan removal alone, and the usage doc that describes the same
scanning behaviour as the README was brought in line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZFDWL7uPZTBDWvLDsTdJD
@okazu-dm
okazu-dm requested a review from laysakura September 10, 2026 03:36

@laysakura laysakura left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx. I'll merge it and upgrade to 0.9.0.

@laysakura
laysakura merged commit 2f4585f into main Sep 10, 2026
7 checks passed
@laysakura
laysakura deleted the perf/single-workspace-port-scan branch September 10, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants