refactor(test): restructure e2e test tiers (L3/L4 split)#84
Merged
Conversation
Move 14 mock-server tests from L4 (VM) to L3 (binary-only): - auth_e2e_test.go: login/logout OAuth flow (6 tests) — no system state needed - snapshot_api_e2e_test.go: publish/import (5 tests) — mock HTTP, no brew - cli_binary_e2e_test.go: new file for version + invalid-preset (2 tests) Delete 5 invalid/redundant L4 tests: - TestE2E_InstallWithInvalidPackage — tested brew behavior, not openboot - TestE2E_DryRunDoesNotInstall — tmpConfig never wired up; covered by TestSmoke_DryRunNoSideEffects - TestE2E_SnapshotCapture — covered by TestSmoke_InstallAndVerifySnapshot - misc_e2e_test.go (TestE2E_FullPreset_DryRun) — only asserted NoError Fix L4 correctness bugs: - Add `install` subcommand prefix to all vmRunDevBinaryWithGit calls - vmInstallViaBrew: new helper using brew tap (no curl|bash, no TTY) - vm_interactive_test.go: use vmInstallViaBrew + install expect idempotently - vm_user_journey_test.go: cleanup openboot before FirstTimeUser, rename subtest - installOhMyZsh: make idempotent (skip if ~/.oh-my-zsh exists) Parallel VM execution: - Makefile: add test-vm-parallel target (2 VMs, VM_A/VM_B split by test name) - scripts/vm/run.sh: support OPENBOOT_VM_TEST env var for test filtering - Remove --exclude='/openboot' from rsync to reuse pre-built binary in VM Agent feedback loop: - post-tool-use.sh: handle "matched no packages" from vet on e2e&&vm files L4 now: 21 tests, all genuinely destructive (real brew install, macOS defaults write, dotfiles symlink, Oh-My-Zsh). L3 now: 16 mock-server tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
auth_e2e_test.go(login/logout, 6 tests),snapshot_api_e2e_test.go(publish/import, 5 tests), newcli_binary_e2e_test.go(version + invalid-preset, 2 tests, plusTestE2E_Install_FetchesCloudConfigalready in L3). None of these tests require real system state — they were in L4 only by historical accident.TestE2E_InstallWithInvalidPackage(tested brew, not openboot),TestE2E_DryRunDoesNotInstall(broken setup — tmpConfig never wired up; covered byTestSmoke_DryRunNoSideEffects),TestE2E_SnapshotCapture(covered by smoke tests),misc_e2e_test.go(TestE2E_FullPreset_DryRunonly assertedNoError).installsubcommand prefix on allvmRunDevBinaryWithGitcalls;installOhMyZshmade idempotent;TestVM_Journey_FirstTimeUsercleans up leftover openboot + renames subtest.vmInstallViaBrewhelper (brew tap + brew install, nocurl | bash, no TTY) — fixes SSH-without--tfailures inTestVM_Interactive_InstallScriptandTestVM_Journey_FirstTimeUser.make test-vm-parallelruns VM A (4 long-running journey tests) and VM B (remaining 17) concurrently;scripts/vm/run.shgainsOPENBOOT_VM_TESTfilter; rsync no longer excludes the pre-built binary (saves ~3 min per VM).post-tool-use.shnow silences false-positivego veterrors one2e && vmfiles when built without tags.Result: L4 = 21 tests (all genuinely destructive), L3 = 16 mock-server tests runnable on any Mac without a VM.
Test plan
go build -tags=e2e ./test/e2e/...— compiles cleango build -tags="e2e,vm" ./test/e2e/...— compiles cleango test -timeout 5m ./...(L1) — all passgo test -tags=e2e -list '.' ./test/e2e/...— 16 L3 tests listed, no VM tests includedgo test -tags="e2e,vm" -list '.' ./test/e2e/...— 21 L4 tests listed, no auth/snapshot_api tests included