Repository navigation
Only parse files that can declare a backend - #444
Open
benlangfeld wants to merge 1 commit into
Open
benlangfeld wants to merge 1 commit into
benlangfeld wants to merge 1 commit into
Conversation
terraform-backend loads the whole root module in order to read one backend block. The cost scales with the size of the module rather than with the thing being looked up, and on a large root module that is most of a minute before terraform has been asked to do anything. A block cannot appear in a file that does not mention its name, so the files worth parsing can be picked out by a substring check first. Files that mention a word only in a comment or a string are still parsed, so the result is unchanged; the scan only decides what to skip, never what the answer is. A file that cannot be read is treated as a candidate so that the usual parsing and error handling still applies to it. On a generated 261 file module with one backend block, reading the backend type goes from 34.07s to 0.20s for the same answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 23, 2026
benlangfeld
marked this pull request as ready for review
September 23, 2026 14:37
benlangfeld
marked this pull request as draft
September 23, 2026 14:41
benlangfeld
marked this pull request as ready for review
September 23, 2026 14:44
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One of the two approaches offered in #443. The other is #445, which adds a
backend_typeinput; they are independent and this one needs nothing from the caller.The change
terraform-backendcallsload_module(), which parses every.tffile in the root module, so thatget_backend_type()can read a singlebackendblock. The cost scales with the size of the module rather than with the thing being looked up.load_module()gains an optionaldeclaringargument. When given, a file is only parsed if its text mentions one of those words:A block cannot appear in a file that does not mention its name, so this cannot change the answer — a file mentioning
backendonly in a comment or a string is still parsed in full, and the HCL parser still decides what it means. The scan only chooses what to skip. A file that cannot be read is treated as a candidate, so the existing parse-and-ignore-failures behaviour still applies to it.Nothing else calls
load_module()with the new argument, so every other caller is unaffected.Effect
Measured on the real root module this came from — 245
.tffiles, of which 12 mentionbackendorcloud:load_module(path)s3load_module(path, declaring=['backend', 'cloud'])s319x, for the same answer. That is consistent with the 46s measured in #443 on the same module inside the action.
How selective the scan is naturally depends on the module. Here only
main.tfmentionsbackendat all; the other 11 matches arecloudflareandpowerapp.cloudappearing in unrelated files. A module that says "backend" in many descriptions or resource names would save less — but never less than today, since the scan only ever skips work.I originally quoted 167x from a synthetic module. That was too flattering: the generated files were smaller and simpler than real ones. The 19x above is the number to go on.
Tests
Six added to
tests/test_module.py, covering: words found anywhere including comments, unreadable files treated as candidates, the backend still found with the filter, skipped files genuinely not parsed,localreported when nothing declares a backend, andcloudblocks found.pytest tests --ignore=tests/github_pr_commentgives 153 passed against 147 onmain, with the same 2 failures and 7 errors before and after — those need S3 and some environment variables I do not have.tests/github_pr_commentfails to collect identically before and after, on a missingTERRAFORM_ACTIONS_*variable.A larger related saving
The same root module is parsed twice per step.
terraform-versioncallsload_module()on it in its own process, immediately beforeterraform-backenddoes.That the first one is a parse rather than the archive download is now measured rather than inferred: the reporter cached the archive with
actions/cacheand the time did not move at all, with gpg verification landing at the identical offset on a cold and a warm run. See #443 (comment).Together the two parses are around 80 seconds of a ~5m30s plan.
#446 removes the first one for workflows that pin an exact version, by taking the existing argument fast path in
main()whenTERRAFORM_VERSIONnames a version rather than a constraint — which also skips the release index fetch. With this PR and #446 together, a large root module with a pinned version parses nothing at all before terraform runs.🤖 Generated with Claude Code