Skip to content

Only parse files that can declare a backend - #444

Open
benlangfeld wants to merge 1 commit into
dflook:mainfrom
benlangfeld:only-parse-files-that-can-declare-a-backend
Open

benlangfeld wants to merge 1 commit into
dflook:mainfrom
benlangfeld:only-parse-files-that-can-declare-a-backend

Conversation

@benlangfeld

@benlangfeld benlangfeld commented Sep 23, 2026 •

Copy link
Copy Markdown

One of the two approaches offered in #443. The other is #445, which adds a backend_type input; they are independent and this one needs nothing from the caller.

The change

terraform-backend calls load_module(), which parses every .tf file in the root module, so that get_backend_type() can read a single backend block. The cost scales with the size of the module rather than with the thing being looked up.

load_module() gains an optional declaring argument. When given, a file is only parsed if its text mentions one of those words:

module = load_module(Path(os.environ.get('INPUT_PATH', '.')), declaring=['backend', 'cloud'])

A block cannot appear in a file that does not mention its name, so this cannot change the answer — a file mentioning backend only in a comment or a string is still parsed in full, and the HCL parser still decides what it means. The scan only chooses what to skip. A file that cannot be read is treated as a candidate, so the existing parse-and-ignore-failures behaviour still applies to it.

Nothing else calls load_module() with the new argument, so every other caller is unaffected.

Effect

Measured on the real root module this came from — 245 .tf files, of which 12 mention backend or cloud:

Time Backend
load_module(path) 32.07s s3
load_module(path, declaring=['backend', 'cloud']) 1.67s s3

19x, for the same answer. That is consistent with the 46s measured in #443 on the same module inside the action.

How selective the scan is naturally depends on the module. Here only main.tf mentions backend at all; the other 11 matches are cloudflare and powerapp.cloud appearing in unrelated files. A module that says "backend" in many descriptions or resource names would save less — but never less than today, since the scan only ever skips work.

I originally quoted 167x from a synthetic module. That was too flattering: the generated files were smaller and simpler than real ones. The 19x above is the number to go on.

Tests

Six added to tests/test_module.py, covering: words found anywhere including comments, unreadable files treated as candidates, the backend still found with the filter, skipped files genuinely not parsed, local reported when nothing declares a backend, and cloud blocks found.

pytest tests --ignore=tests/github_pr_comment gives 153 passed against 147 on main, with the same 2 failures and 7 errors before and after — those need S3 and some environment variables I do not have. tests/github_pr_comment fails to collect identically before and after, on a missing TERRAFORM_ACTIONS_* variable.

A larger related saving

The same root module is parsed twice per step. terraform-version calls load_module() on it in its own process, immediately before terraform-backend does.

That the first one is a parse rather than the archive download is now measured rather than inferred: the reporter cached the archive with actions/cache and the time did not move at all, with gpg verification landing at the identical offset on a cold and a warm run. See #443 (comment).

Together the two parses are around 80 seconds of a ~5m30s plan.

#446 removes the first one for workflows that pin an exact version, by taking the existing argument fast path in main() when TERRAFORM_VERSION names a version rather than a constraint — which also skips the release index fetch. With this PR and #446 together, a large root module with a pinned version parses nothing at all before terraform runs.

🤖 Generated with Claude Code

terraform-backend loads the whole root module in order to read one backend
block. The cost scales with the size of the module rather than with the thing
being looked up, and on a large root module that is most of a minute before
terraform has been asked to do anything.

A block cannot appear in a file that does not mention its name, so the files
worth parsing can be picked out by a substring check first. Files that mention
a word only in a comment or a string are still parsed, so the result is
unchanged; the scan only decides what to skip, never what the answer is. A file
that cannot be read is treated as a candidate so that the usual parsing and
error handling still applies to it.

On a generated 261 file module with one backend block, reading the backend type
goes from 34.07s to 0.20s for the same answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant