From 548566f4b8b2f87ebb8e69d42edb9356228fe62a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 06:06:22 +0000 Subject: [PATCH 01/11] test(projectgraph): measure the Python receiver forms and method targets The Python plan starts with a number. The link step reads no type hint of Python, and no Python project is in the method corpus, so nobody could say how often a Python method target ends uncertain. receivers_test.go has 18 Python rows against a pypi unit under testdata/receivers/python. Two rows give their answer today. The other 16 name the type-hint step of the plan and their answer before it. A call on a variable that holds the result of an external call gets the false key dep//new_thing//do and no record, so the method of the real type is no-such-target. The method corpus has 10 Python projects with 14 pinned dependencies. OSV lists no symbols for a PyPI advisory, so scripts/pypi_advisories.py holds the symbols of 72 advisories by hand, as the advisory text names them, and writes them with the ranges from OSV. fetch_method_corpus.sh unpacks a dependency from its wheel, so the unit has the layout of site-packages, and a project from its sdist. methods_test.go reads the pypi ecosystem and takes PROJECTGRAPH_METHOD_LANGUAGES to index one language. report.md has the baseline: 5 of 107 targets found, none not-found and none no-such-target. Every search meets an uncertainty that concerns every target. The causes are the bundled JavaScript that the index truncates, the decorator-wrap flag on every framework decorator, a truncated test file, and the unknown receiver of 27 of 63 methods. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- scripts/fetch_method_corpus.sh | 69 +- scripts/pypi_advisories.py | 194 ++ test/projectgraph/README.md | 19 +- test/projectgraph/methods_test.go | 50 +- .../public/methods/advisories.json | 1950 ++++++++++++++++- test/projectgraph/public/methods/corpus.txt | 10 + test/projectgraph/receivers_test.go | 180 +- test/projectgraph/report.md | 113 + .../receivers/python/dep/dep/__init__.py | 45 + .../receivers/python/dep/pyproject.toml | 3 + 10 files changed, 2613 insertions(+), 20 deletions(-) create mode 100755 scripts/pypi_advisories.py create mode 100644 test/projectgraph/testdata/receivers/python/dep/dep/__init__.py create mode 100644 test/projectgraph/testdata/receivers/python/dep/pyproject.toml diff --git a/scripts/fetch_method_corpus.sh b/scripts/fetch_method_corpus.sh index 7597a726..ff722436 100755 --- a/scripts/fetch_method_corpus.sh +++ b/scripts/fetch_method_corpus.sh @@ -1,17 +1,80 @@ #!/bin/bash # Fetch the projects and the dependencies of the method measure into the # directory of the first argument. Go modules come from the module proxy -# into /gomod, and crates from crates.io into /crates. The list -# is test/projectgraph/public/methods/corpus.txt. Point +# into /gomod, crates from crates.io into /crates, and PyPI +# packages from pypi.org into /pypi/-. A dependency +# comes as a wheel, unpacked, so its directory has the layout of +# site-packages with the .dist-info of the package. A project comes as its +# sdist, so its directory has the manifests. The list is +# test/projectgraph/public/methods/corpus.txt. Point # PROJECTGRAPH_METHOD_CORPUS at the directory to measure them. set -euo pipefail dir="${1:?usage: fetch_method_corpus.sh }" list="$(dirname "$0")/../test/projectgraph/public/methods/corpus.txt" -mkdir -p "$dir/gomod" "$dir/crates" +mkdir -p "$dir/gomod" "$dir/crates" "$dir/pypi" dir="$(cd "$dir" && pwd)" +# pypi_url prints the download URL of a package version: the sdist for +# the project, and for a dependency a pure wheel, else a manylinux x86_64 +# wheel, else any wheel, else the sdist. +pypi_url() { + local name="$1" version="$2" kind="$3" + curl -sSfL "https://pypi.org/pypi/$name/$version/json" | jq -r --arg kind "$kind" ' + .urls as $urls + | ($urls | map(select(.packagetype == "sdist")) | .[0].url) as $sdist + | ($urls | map(select(.packagetype == "bdist_wheel"))) as $wheels + | if $kind == "project" then $sdist + else ( + ($wheels | map(select(.filename | test("none-any[.]whl$"))) | .[0].url) + // ($wheels | map(select(.filename | test("manylinux.*x86_64"))) | sort_by(.filename) | reverse | .[0].url) + // ($wheels | .[0].url) + // $sdist + ) end' +} + +# pypi_fetch unpacks a package version into /pypi/-. +pypi_fetch() { + local item="$1" kind="$2" + local name="${item%@*}" + local version="${item#*@}" + local out="$dir/pypi/$name-$version" + if [ -d "$out" ]; then + return + fi + local url + url="$(pypi_url "$name" "$version" "$kind")" + if [ -z "$url" ] || [ "$url" = "null" ]; then + echo "no file of $item on pypi.org" >&2 + exit 1 + fi + local tmp + tmp="$(mktemp -d)" + curl -sSfL "$url" -o "$tmp/package" + mkdir -p "$out" + case "$url" in + *.whl) + unzip -q "$tmp/package" -d "$out" + ;; + *.zip) + unzip -q "$tmp/package" -d "$tmp/src" + mv "$tmp"/src/*/* "$out" + ;; + *) + tar -xzf "$tmp/package" -C "$out" --strip-components=1 + ;; + esac + rm -rf "$tmp" +} + grep -v '^#' "$list" | while read -r language _ project deps; do + if [ "$language" = python ]; then + pypi_fetch "$project" project + for item in $deps; do + pypi_fetch "$item" dependency + done + continue + fi for item in $project $deps; do case "$language" in go) diff --git a/scripts/pypi_advisories.py b/scripts/pypi_advisories.py new file mode 100755 index 00000000..6c0c88a3 --- /dev/null +++ b/scripts/pypi_advisories.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +"""Write the PyPI advisories of the method measure into advisories.json. + +OSV has no symbol list for a PyPI advisory, unlike the Go vulndb. So this +script holds the symbols by hand, as the text of each advisory names +them, and reads the id and the version ranges from OSV. Run it to refresh +the ranges. Add a row to SYMBOLS when the corpus gains an advisory. + + python3 scripts/pypi_advisories.py + +A version in a range is padded to three parts, so the Go test compares it +as semver. A pre-release, as 2.0.0a1, becomes 2.0.0-a1. +""" + +import json +import re +import sys +import urllib.request +from pathlib import Path + +OSV = "https://api.osv.dev/v1/vulns/" +OUT = Path(__file__).resolve().parent.parent / "test/projectgraph/public/methods/advisories.json" + +# SYMBOLS maps an OSV id to the module and the symbols that the advisory +# names, as the Go vulndb lists them: a module path and a symbol, with a +# dot between a class and its method. The source of each row is +# safedep:hand, from the advisory text. +SYMBOLS = { + # PyYAML: code execution through full_load and FullLoader before 5.4. + "PYSEC-2021-142": {"yaml": ["load", "load_all", "full_load", "full_load_all"]}, + # Jinja2: attribute injection through the xmlattr filter, and the + # sandbox breakouts through format. + "GHSA-h5c8-rqwp-cp95": {"jinja2.filters": ["do_xmlattr"]}, + "GHSA-h75v-3vvj-5mfj": {"jinja2.filters": ["do_xmlattr"]}, + "GHSA-q2x7-8rv6-6q7h": {"jinja2.sandbox": ["SandboxedEnvironment.call", "SandboxedEnvironment.format_string"]}, + "GHSA-cpwx-vrp4-4pq7": {"jinja2.sandbox": ["SandboxedEnvironment.call", "SandboxedEnvironment.is_safe_attribute"], "jinja2.filters": ["do_attr"]}, + # Tornado: the open redirect of StaticFileHandler, and the cookie and + # multipart parsers. + "PYSEC-2023-75": {"tornado.web": ["StaticFileHandler.get", "StaticFileHandler.validate_absolute_path"]}, + "GHSA-8w49-h785-mj3c": {"tornado.httputil": ["parse_cookie"]}, + "GHSA-7cx3-6m66-7c5m": {"tornado.httputil": ["parse_multipart_form_data", "parse_body_arguments"]}, + "GHSA-qjxf-f2mg-c6mc": {"tornado.httputil": ["parse_multipart_form_data"]}, + "GHSA-pr2v-jx2c-wg9f": {"tornado.web": ["RequestHandler.set_status"]}, + "GHSA-fqwm-6jpj-5wxc": {"tornado.web": ["RequestHandler.set_cookie"]}, + "GHSA-c2m8-h5v5-343r": {"tornado.web": ["StaticFileHandler.get", "StaticFileHandler.validate_absolute_path"]}, + # requests: the Proxy-Authorization leak on a redirect, the lost + # verify=False, and the .netrc credentials. + "PYSEC-2023-74": {"requests.sessions": ["Session.rebuild_proxies", "Session.send"], "requests.adapters": ["HTTPAdapter.send"]}, + "GHSA-9wx4-h78v-vm56": {"requests.adapters": ["HTTPAdapter.send", "HTTPAdapter.get_connection"], "requests.sessions": ["Session.request"]}, + "GHSA-9hjg-9r4m-mvj7": {"requests.utils": ["get_netrc_auth"], "requests.sessions": ["Session.prepare_request"]}, + # urllib3: the headers and the body that a redirect keeps. + "PYSEC-2023-192": {"urllib3.poolmanager": ["PoolManager.urlopen"], "urllib3.connectionpool": ["HTTPConnectionPool.urlopen"]}, + "PYSEC-2023-212": {"urllib3.poolmanager": ["PoolManager.urlopen"], "urllib3.connectionpool": ["HTTPConnectionPool.urlopen"]}, + "GHSA-34jh-p97f-mpxf": {"urllib3.poolmanager": ["PoolManager.urlopen"], "urllib3.connectionpool": ["HTTPConnectionPool.urlopen"]}, + # idna: quadratic time in encode. + "PYSEC-2024-60": {"idna": ["encode"], "idna.core": ["encode"]}, + # Flask: the session cookie in a cached response. + "PYSEC-2023-62": {"flask.sessions": ["SecureCookieSessionInterface.save_session"]}, + # Werkzeug: the nameless cookie, the multipart parser, the debugger + # and safe_join. + "PYSEC-2023-57": {"werkzeug.http": ["parse_cookie"], "werkzeug._internal": ["_cookie_parse_impl"]}, + "PYSEC-2023-58": {"werkzeug.formparser": ["MultiPartParser.parse", "FormDataParser.parse"]}, + "PYSEC-2023-221": {"werkzeug.formparser": ["MultiPartParser.parse"], "werkzeug.sansio.multipart": ["MultipartDecoder.next_event"]}, + "GHSA-2g68-c3qc-8985": {"werkzeug.debug": ["DebuggedApplication.__call__"]}, + "GHSA-f9vj-2wh5-fj8j": {"werkzeug.security": ["safe_join"]}, + "GHSA-q34m-jh98-gwm2": {"werkzeug.formparser": ["MultiPartParser.parse"], "werkzeug.sansio.multipart": ["MultipartDecoder.next_event"]}, + "PYSEC-2026-4312": {"werkzeug.security": ["safe_join"]}, + # paramiko: the race in write_private_key_file. + "PYSEC-2022-166": {"paramiko.pkey": ["PKey.write_private_key_file", "PKey._write_private_key_file"]}, + # Pillow: the open of an image and the pure Python callers of the + # decoders that the advisories name. + "PYSEC-2022-42980": {"PIL.Image": ["open"], "PIL.TiffImagePlugin": ["TiffImageFile._setup"]}, + "PYSEC-2023-227": {"PIL.ImageFont": ["FreeTypeFont.getmask2"], "PIL.ImageDraw": ["ImageDraw.text"]}, + "PYSEC-2026-457": {"PIL.ImageMath": ["eval"]}, + "GHSA-74r2-7ff9-mjj2": {"PIL.Image": ["open"], "PIL.GifImagePlugin": ["GifImageFile._open"]}, + "GHSA-xgh8-jp6q-p5x8": {"PIL.Image": ["open"], "PIL.XbmImagePlugin": ["XbmImageFile._open"]}, + "PYSEC-2026-2257": {"PIL.ImageShow": ["WindowsViewer.get_command"]}, + "PYSEC-2026-3453": {"PIL.ImageCms": ["ImageCmsTransform.apply"]}, + # cryptography: the PKCS7 and PKCS12 loaders and the SSH certificates. + "PYSEC-2023-254": {"cryptography.hazmat.primitives.serialization.pkcs7": ["load_pem_pkcs7_certificates", "load_der_pkcs7_certificates"]}, + "PYSEC-2024-225": {"cryptography.hazmat.primitives.serialization.pkcs12": ["serialize_key_and_certificates"]}, + "PYSEC-2023-112": {"cryptography.hazmat.primitives.serialization.ssh": ["load_ssh_public_identity", "SSHCertificateBuilder.sign"]}, + # lxml: iterwalk and canonicalize, and iterparse. The code is compiled, + # so the link step has no source for it. + "PYSEC-2022-230": {"lxml.etree": ["iterwalk", "canonicalize"]}, + "GHSA-vfmq-68hx-4jfw": {"lxml.etree": ["iterparse", "ETCompatXMLParser"]}, + # Django: the validators, the text utilities, the template filters and + # the query set methods that the advisories name. + "PYSEC-2023-100": {"django.core.validators": ["EmailValidator.__call__", "URLValidator.__call__"]}, + "PYSEC-2023-222": {"django.contrib.auth.forms": ["UsernameField.to_python"]}, + "PYSEC-2023-225": {"django.utils.encoding": ["uri_to_iri"]}, + "PYSEC-2023-226": {"django.utils.text": ["Truncator.chars", "Truncator.words"]}, + "PYSEC-2023-61": {"django.forms.fields": ["FileField.clean"], "django.forms.widgets": ["ClearableFileInput.value_from_datadict"]}, + "PYSEC-2024-28": {"django.contrib.humanize.templatetags.humanize": ["intcomma"]}, + "PYSEC-2024-47": {"django.utils.text": ["Truncator.words"]}, + "PYSEC-2024-56": {"django.utils.html": ["urlize"], "django.template.defaultfilters": ["urlize", "urlizetrunc"]}, + "PYSEC-2024-57": {"django.contrib.auth.backends": ["ModelBackend.authenticate"]}, + "PYSEC-2024-58": {"django.core.files.storage.base": ["Storage.save"], "django.db.models.fields.files": ["FieldFile.save"]}, + "PYSEC-2024-59": {"django.utils.translation": ["get_supported_language_variant"]}, + "PYSEC-2024-67": {"django.template.defaultfilters": ["floatformat"]}, + "PYSEC-2024-68": {"django.utils.html": ["urlize", "Urlizer.__call__"]}, + "PYSEC-2024-69": {"django.utils.html": ["urlize", "Urlizer.__call__"]}, + "PYSEC-2024-70": {"django.db.models.query": ["QuerySet.values", "QuerySet.values_list"]}, + "PYSEC-2024-102": {"django.utils.html": ["urlize", "Urlizer.__call__"]}, + "PYSEC-2024-156": {"django.utils.html": ["strip_tags"]}, + "PYSEC-2024-157": {"django.db.models.fields.json": ["HasKey.as_oracle"]}, + "PYSEC-2025-1": {"django.core.validators": ["validate_ipv6_address"], "django.utils.ipv6": ["is_valid_ipv6_address"]}, + "PYSEC-2025-13": {"django.utils.text": ["wrap"], "django.template.defaultfilters": ["wordwrap"]}, + "PYSEC-2025-37": {"django.utils.html": ["strip_tags"]}, + "PYSEC-2025-47": {"django.utils.log": ["log_response"]}, + "PYSEC-2025-106": {"django.db.models.query": ["QuerySet.annotate", "QuerySet.alias"]}, + "PYSEC-2025-108": {"django.db.models.query": ["QuerySet.filter", "QuerySet.exclude", "QuerySet.get"]}, + "PYSEC-2026-45": {"django.utils.text": ["Truncator.chars", "Truncator.words"]}, + "PYSEC-2026-47": {"django.db.models.query": ["QuerySet.order_by"]}, + "PYSEC-2026-48": {"django.http.multipartparser": ["MultiPartParser.parse"]}, + # sqlparse: the lexer and the grouping of nested lists. + "PYSEC-2023-87": {"sqlparse.lexer": ["Lexer.get_tokens"], "sqlparse": ["parse", "format"]}, + "GHSA-2m57-hf25-phgg": {"sqlparse": ["parse"], "sqlparse.engine.grouping": ["group"]}, + "GHSA-27jp-wm6q-gp25": {"sqlparse": ["format"]}, + "GHSA-cfqr-cjx5-5jcm": {"sqlparse": ["format"]}, + "GHSA-f2ff-p2ww-7p4p": {"sqlparse": ["parse"], "sqlparse.engine.grouping": ["group_comments"]}, +} + +# PACKAGES are the PyPI names of the dependencies of corpus.txt, in the +# case that OSV uses as the key of advisories.json. +PACKAGES = [ + "pyyaml", "jinja2", "tornado", "requests", "urllib3", "idna", "flask", "werkzeug", "paramiko", "pillow", + "cryptography", "lxml", "django", "sqlparse", +] + +PRERELEASE = re.compile(r"^(\d+(?:\.\d+)*)(?:\.?(a|b|rc|post|dev)(\d*))?$") + + +def semver(version): + """Pad a PyPI version to three parts for the semver compare of the Go test.""" + m = PRERELEASE.match(version) + if not m: + return version + parts = m.group(1).split(".") + while len(parts) < 3: + parts.append("0") + out = ".".join(parts[:3]) + if m.group(2): + out += "-" + m.group(2) + (m.group(3) or "0") + return out + + +def fetch(vuln_id): + with urllib.request.urlopen(OSV + vuln_id, timeout=60) as r: + return json.load(r) + + +def ranges_of(vuln, package): + out = [] + for a in vuln.get("affected", []): + pkg = a.get("package", {}) + if pkg.get("ecosystem") != "PyPI" or pkg.get("name", "").lower() != package: + continue + for r in a.get("ranges", []): + if r.get("type") != "ECOSYSTEM": + continue + for e in r.get("events", []): + if "introduced" in e: + out.append({"introduced": semver(e["introduced"])}) + if "fixed" in e: + out.append({"fixed": semver(e["fixed"])}) + return out + + +def main(): + data = json.loads(OUT.read_text()) + pypi = {} + for vuln_id, modules in sorted(SYMBOLS.items()): + vuln = fetch(vuln_id) + names = {a["package"]["name"].lower() for a in vuln.get("affected", []) if a.get("package", {}).get("ecosystem") == "PyPI"} + matched = [p for p in PACKAGES if p in names] + if not matched: + sys.exit(f"{vuln_id} affects none of PACKAGES: {sorted(names)}") + for package in matched: + pypi.setdefault(package, []).append({ + "id": vuln_id, + "source": "safedep:hand", + "ranges": ranges_of(vuln, package), + "imports": [{"path": path, "symbols": symbols} for path, symbols in sorted(modules.items())], + }) + for rows in pypi.values(): + rows.sort(key=lambda a: a["id"]) + data["pypi"] = dict(sorted(pypi.items())) + OUT.write_text(json.dumps(data, indent=1, sort_keys=True) + "\n") + print(f"{sum(len(v) for v in pypi.values())} advisories of {len(pypi)} packages") + + +if __name__ == "__main__": + main() diff --git a/test/projectgraph/README.md b/test/projectgraph/README.md index bc78a905..18cbbcc9 100644 --- a/test/projectgraph/README.md +++ b/test/projectgraph/README.md @@ -122,8 +122,11 @@ Set `PROJECTGRAPH_PUBLIC_DUMP` to a case name, as `classes/self_call`, or to method of a dependency ends uncertain on real projects. `public/methods/corpus.txt` lists the projects and the dependencies, and `public/methods/advisories.json` holds their advisories from the Go vulndb -and from RustSec through OSV. The code is not in the repository, so fetch -it first: +and from RustSec through OSV. OSV lists no symbols for a PyPI advisory, so +`scripts/pypi_advisories.py` holds the symbols of each PyPI advisory by +hand, as the advisory text names them, and writes them with the ranges +from OSV under `pypi`. Run it to refresh the ranges or after you add a row +to its table. The code is not in the repository, so fetch it first: ```bash export PROJECTGRAPH_METHOD_CORPUS=$(bash scripts/fetch_method_corpus.sh "$(mktemp -d)") @@ -132,11 +135,19 @@ go test -tags projectgraph -run TestMethodTargets -count=1 -v ./test/projectgrap The test logs the count of each answer per language and per symbol kind, and the same per project. Set `PROJECTGRAPH_METHOD_REPORT` to a file path -to write the block to that file. `report.md` says how the corpus was +to write the block to that file. Set `PROJECTGRAPH_METHOD_LANGUAGES` to a +list of languages, as `python`, to index those alone. A Python project +bundles the JavaScript of its web pages, and a run of `python` alone +leaves that out. `report.md` says how the corpus was chosen and what each cause means. +A Go project of the corpus comes from the module proxy, a crate from +crates.io, and a PyPI package from pypi.org: a dependency as its wheel, +unpacked, so its directory has the layout of site-packages, and a project +as its sdist, so its directory has the manifests. + `receivers_test.go` holds one row per form of the receiver of a method -call, for Go and for Rust, against the small dependency units under +call, for Go, for Rust and for Python, against the small dependency units under `testdata/receivers`. It runs in `go test ./...`. A row that the link step does not answer yet names the issue that fixes it and its answer before that fix, so the test stays green until the fix changes the answer. diff --git a/test/projectgraph/methods_test.go b/test/projectgraph/methods_test.go index 5b2b5ea7..23727a7a 100644 --- a/test/projectgraph/methods_test.go +++ b/test/projectgraph/methods_test.go @@ -17,9 +17,11 @@ import ( "unicode" "unicode/utf8" + "github.com/safedep/code/core" "github.com/safedep/code/fileindex" codefs "github.com/safedep/code/fs" "github.com/safedep/code/lang" + "github.com/safedep/code/langcode" "github.com/safedep/code/plugin/indexer" "github.com/safedep/code/projectgraph" "github.com/safedep/code/projectgraph/reach" @@ -32,8 +34,15 @@ import ( // fills with the projects and the dependencies of the method measure. const methodCorpusEnv = "PROJECTGRAPH_METHOD_CORPUS" +// methodLanguagesEnv names the languages that the index step reads, as +// python,javascript. Empty reads every language. A Python project bundles +// the JavaScript of its web pages, and a minified file of it is truncated, +// so a run of python alone shows the Python causes. +const methodLanguagesEnv = "PROJECTGRAPH_METHOD_LANGUAGES" + // methodsDir holds the list of the projects and the advisories of their -// dependencies, from the Go vulndb and from RustSec through OSV. +// dependencies, from the Go vulndb and from RustSec through OSV, and the +// PyPI advisories with the symbols that scripts/pypi_advisories.py holds. var methodsDir = filepath.Join("public", "methods") // methodProject is one line of corpus.txt. @@ -43,9 +52,13 @@ type methodProject struct { } // advisory is one advisory of a package: the events of its affected -// ranges, and the symbols of Go or the functions of Rust that it names. +// ranges, and the symbols of Go or the functions of Rust that it names. A +// PyPI advisory has the Go shape, with a module path and its symbols, and +// Source says that a person wrote the symbols from the advisory text, +// because OSV lists none for PyPI. scripts/pypi_advisories.py holds them. type advisory struct { ID string `json:"id"` + Source string `json:"source,omitempty"` Ranges []struct { Introduced string `json:"introduced"` Fixed string `json:"fixed"` @@ -123,8 +136,8 @@ func methodTargets(language, dep string, advisories map[string]map[string][]advi out = append(out, methodTarget{key: key, kind: kind}) } } - if language == "go" { - for _, a := range advisories["go"][name] { + if eco, lang := ecosystemOf(language); eco != "cargo" { + for _, a := range advisories[eco][name] { if !a.affects(version) { continue } @@ -134,7 +147,7 @@ func methodTargets(language, dep string, advisories map[string]map[string][]advi if strings.Contains(sym, ".") { kind = kindMethod } - add(fileindex.Join(fileindex.Namespace("go", imp.Path), strings.Split(sym, ".")...), kind) + add(fileindex.Join(fileindex.Namespace(lang, imp.Path), strings.Split(sym, ".")...), kind) } } } @@ -167,14 +180,32 @@ func startsUpper(s string) bool { return unicode.IsUpper(r) } +// ecosystemOf returns the ecosystem of the packages of a language of the +// corpus, as the keys of advisories.json and the units write it, and the +// language code of the import namespaces of its keys. +func ecosystemOf(language string) (string, langcode.Code) { + switch language { + case "go": + return "go", langcode.Go + case "python": + return "pypi", langcode.Python + } + return "cargo", langcode.Rust +} + // methodDir returns the directory of a project or a dependency in the -// corpus, and its unit. +// corpus, and its unit. A PyPI dependency is an unpacked wheel, so its +// directory has the layout of site-packages, and the unit is rooted there. func methodDir(t *testing.T, corpus, language, item string) (string, string) { name, version, _ := strings.Cut(item, "@") - if language == "go" { + eco, _ := ecosystemOf(language) + switch eco { + case "go": escaped, err := module.EscapePath(name) require.NoError(t, err) return filepath.Join(corpus, "gomod", filepath.FromSlash(escaped)+"@"+version), "go:" + item + case "pypi": + return filepath.Join(corpus, "pypi", name+"-"+version), "pypi:" + item } return filepath.Join(corpus, "crates", name+"-"+version), "cargo:" + item } @@ -239,6 +270,11 @@ func TestMethodTargets(t *testing.T) { } languages, err := lang.AllLanguages() require.NoError(t, err) + if names := os.Getenv(methodLanguagesEnv); names != "" { + languages = slices.DeleteFunc(languages, func(l core.Language) bool { + return !slices.Contains(strings.Split(names, ","), string(l.Meta().Code)) + }) + } projects, advisories := readMethodCorpus(t) counts := methodCounts{} diff --git a/test/projectgraph/public/methods/advisories.json b/test/projectgraph/public/methods/advisories.json index b8e158a2..be336e10 100644 --- a/test/projectgraph/public/methods/advisories.json +++ b/test/projectgraph/public/methods/advisories.json @@ -4292,5 +4292,1953 @@ ] } ] + }, + "pypi": { + "cryptography": [ + { + "id": "PYSEC-2023-112", + "imports": [ + { + "path": "cryptography.hazmat.primitives.serialization.ssh", + "symbols": [ + "load_ssh_public_identity", + "SSHCertificateBuilder.sign" + ] + } + ], + "ranges": [ + { + "introduced": "40.0.0" + }, + { + "fixed": "41.0.2" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-254", + "imports": [ + { + "path": "cryptography.hazmat.primitives.serialization.pkcs7", + "symbols": [ + "load_pem_pkcs7_certificates", + "load_der_pkcs7_certificates" + ] + } + ], + "ranges": [ + { + "introduced": "3.1.0" + }, + { + "fixed": "41.0.6" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-225", + "imports": [ + { + "path": "cryptography.hazmat.primitives.serialization.pkcs12", + "symbols": [ + "serialize_key_and_certificates" + ] + } + ], + "ranges": [ + { + "introduced": "38.0.0" + }, + { + "fixed": "42.0.4" + } + ], + "source": "safedep:hand" + } + ], + "django": [ + { + "id": "PYSEC-2023-100", + "imports": [ + { + "path": "django.core.validators", + "symbols": [ + "EmailValidator.__call__", + "URLValidator.__call__" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.3" + }, + { + "introduced": "4.0.0" + }, + { + "fixed": "4.1.10" + }, + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.20" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-222", + "imports": [ + { + "path": "django.contrib.auth.forms", + "symbols": [ + "UsernameField.to_python" + ] + } + ], + "ranges": [ + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.23" + }, + { + "introduced": "4.1.0" + }, + { + "fixed": "4.1.13" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.7" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-225", + "imports": [ + { + "path": "django.utils.encoding", + "symbols": [ + "uri_to_iri" + ] + } + ], + "ranges": [ + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.21" + }, + { + "introduced": "4.1.0" + }, + { + "fixed": "4.1.11" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.5" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-226", + "imports": [ + { + "path": "django.utils.text", + "symbols": [ + "Truncator.chars", + "Truncator.words" + ] + } + ], + "ranges": [ + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.22" + }, + { + "introduced": "4.1.0" + }, + { + "fixed": "4.1.12" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.6" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-61", + "imports": [ + { + "path": "django.forms.fields", + "symbols": [ + "FileField.clean" + ] + }, + { + "path": "django.forms.widgets", + "symbols": [ + "ClearableFileInput.value_from_datadict" + ] + } + ], + "ranges": [ + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.19" + }, + { + "introduced": "4.0.0" + }, + { + "fixed": "4.1.9" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.1" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-102", + "imports": [ + { + "path": "django.utils.html", + "symbols": [ + "urlize", + "Urlizer.__call__" + ] + } + ], + "ranges": [ + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.1" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.9" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.16" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-156", + "imports": [ + { + "path": "django.utils.html", + "symbols": [ + "strip_tags" + ] + } + ], + "ranges": [ + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.4" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.10" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.17" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-157", + "imports": [ + { + "path": "django.db.models.fields.json", + "symbols": [ + "HasKey.as_oracle" + ] + } + ], + "ranges": [ + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.4" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.10" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.17" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-28", + "imports": [ + { + "path": "django.contrib.humanize.templatetags.humanize", + "symbols": [ + "intcomma" + ] + } + ], + "ranges": [ + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.24" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.10" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.2" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-47", + "imports": [ + { + "path": "django.utils.text", + "symbols": [ + "Truncator.words" + ] + } + ], + "ranges": [ + { + "introduced": "3.2.0" + }, + { + "fixed": "3.2.25" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.11" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.3" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-56", + "imports": [ + { + "path": "django.template.defaultfilters", + "symbols": [ + "urlize", + "urlizetrunc" + ] + }, + { + "path": "django.utils.html", + "symbols": [ + "urlize" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.14" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.7" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-57", + "imports": [ + { + "path": "django.contrib.auth.backends", + "symbols": [ + "ModelBackend.authenticate" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.7" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.14" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-58", + "imports": [ + { + "path": "django.core.files.storage.base", + "symbols": [ + "Storage.save" + ] + }, + { + "path": "django.db.models.fields.files", + "symbols": [ + "FieldFile.save" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.7" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.14" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-59", + "imports": [ + { + "path": "django.utils.translation", + "symbols": [ + "get_supported_language_variant" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.7" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.14" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-67", + "imports": [ + { + "path": "django.template.defaultfilters", + "symbols": [ + "floatformat" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.8" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.15" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-68", + "imports": [ + { + "path": "django.utils.html", + "symbols": [ + "urlize", + "Urlizer.__call__" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.8" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.15" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-69", + "imports": [ + { + "path": "django.utils.html", + "symbols": [ + "urlize", + "Urlizer.__call__" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.8" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.15" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2024-70", + "imports": [ + { + "path": "django.db.models.query", + "symbols": [ + "QuerySet.values", + "QuerySet.values_list" + ] + } + ], + "ranges": [ + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.8" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.15" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2025-1", + "imports": [ + { + "path": "django.core.validators", + "symbols": [ + "validate_ipv6_address" + ] + }, + { + "path": "django.utils.ipv6", + "symbols": [ + "is_valid_ipv6_address" + ] + } + ], + "ranges": [ + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.5" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.11" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.18" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2025-106", + "imports": [ + { + "path": "django.db.models.query", + "symbols": [ + "QuerySet.annotate", + "QuerySet.alias" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.25" + }, + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.13" + }, + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.7" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2025-108", + "imports": [ + { + "path": "django.db.models.query", + "symbols": [ + "QuerySet.filter", + "QuerySet.exclude", + "QuerySet.get" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.26" + }, + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.14" + }, + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.8" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2025-13", + "imports": [ + { + "path": "django.template.defaultfilters", + "symbols": [ + "wordwrap" + ] + }, + { + "path": "django.utils.text", + "symbols": [ + "wrap" + ] + } + ], + "ranges": [ + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.7" + }, + { + "introduced": "5.0.0" + }, + { + "fixed": "5.0.13" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.20" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2025-37", + "imports": [ + { + "path": "django.utils.html", + "symbols": [ + "strip_tags" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.21" + }, + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.9" + }, + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.1" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2025-47", + "imports": [ + { + "path": "django.utils.log", + "symbols": [ + "log_response" + ] + } + ], + "ranges": [ + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.2" + }, + { + "introduced": "5.1.0" + }, + { + "fixed": "5.1.10" + }, + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.22" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-45", + "imports": [ + { + "path": "django.utils.text", + "symbols": [ + "Truncator.chars", + "Truncator.words" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.28" + }, + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.11" + }, + { + "introduced": "6.0.0" + }, + { + "fixed": "6.0.2" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-47", + "imports": [ + { + "path": "django.db.models.query", + "symbols": [ + "QuerySet.order_by" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.28" + }, + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.11" + }, + { + "introduced": "6.0.0" + }, + { + "fixed": "6.0.2" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-48", + "imports": [ + { + "path": "django.http.multipartparser", + "symbols": [ + "MultiPartParser.parse" + ] + } + ], + "ranges": [ + { + "introduced": "4.2.0" + }, + { + "fixed": "4.2.30" + }, + { + "introduced": "5.2.0" + }, + { + "fixed": "5.2.13" + }, + { + "introduced": "6.0.0" + }, + { + "fixed": "6.0.4" + } + ], + "source": "safedep:hand" + } + ], + "flask": [ + { + "id": "PYSEC-2023-62", + "imports": [ + { + "path": "flask.sessions", + "symbols": [ + "SecureCookieSessionInterface.save_session" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.2.5" + }, + { + "introduced": "2.3.0" + }, + { + "fixed": "2.3.2" + } + ], + "source": "safedep:hand" + } + ], + "idna": [ + { + "id": "PYSEC-2024-60", + "imports": [ + { + "path": "idna", + "symbols": [ + "encode" + ] + }, + { + "path": "idna.core", + "symbols": [ + "encode" + ] + } + ], + "ranges": [ + { + "introduced": "0.1.0" + }, + { + "fixed": "3.7.0" + } + ], + "source": "safedep:hand" + } + ], + "jinja2": [ + { + "id": "GHSA-cpwx-vrp4-4pq7", + "imports": [ + { + "path": "jinja2.filters", + "symbols": [ + "do_attr" + ] + }, + { + "path": "jinja2.sandbox", + "symbols": [ + "SandboxedEnvironment.call", + "SandboxedEnvironment.is_safe_attribute" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.1.6" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-h5c8-rqwp-cp95", + "imports": [ + { + "path": "jinja2.filters", + "symbols": [ + "do_xmlattr" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.1.3" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-h75v-3vvj-5mfj", + "imports": [ + { + "path": "jinja2.filters", + "symbols": [ + "do_xmlattr" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.1.4" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-q2x7-8rv6-6q7h", + "imports": [ + { + "path": "jinja2.sandbox", + "symbols": [ + "SandboxedEnvironment.call", + "SandboxedEnvironment.format_string" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.1.5" + } + ], + "source": "safedep:hand" + } + ], + "lxml": [ + { + "id": "GHSA-vfmq-68hx-4jfw", + "imports": [ + { + "path": "lxml.etree", + "symbols": [ + "iterparse", + "ETCompatXMLParser" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.1.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2022-230", + "imports": [ + { + "path": "lxml.etree", + "symbols": [ + "iterwalk", + "canonicalize" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "4.9.1" + } + ], + "source": "safedep:hand" + } + ], + "paramiko": [ + { + "id": "PYSEC-2022-166", + "imports": [ + { + "path": "paramiko.pkey", + "symbols": [ + "PKey.write_private_key_file", + "PKey._write_private_key_file" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.9.3" + }, + { + "introduced": "2.10.0" + }, + { + "fixed": "2.10.1" + } + ], + "source": "safedep:hand" + } + ], + "pillow": [ + { + "id": "GHSA-74r2-7ff9-mjj2", + "imports": [ + { + "path": "PIL.GifImagePlugin", + "symbols": [ + "GifImageFile._open" + ] + }, + { + "path": "PIL.Image", + "symbols": [ + "open" + ] + } + ], + "ranges": [ + { + "introduced": "5.4.0" + }, + { + "fixed": "13.0.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-xgh8-jp6q-p5x8", + "imports": [ + { + "path": "PIL.Image", + "symbols": [ + "open" + ] + }, + { + "path": "PIL.XbmImagePlugin", + "symbols": [ + "XbmImageFile._open" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "13.0.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2022-42980", + "imports": [ + { + "path": "PIL.Image", + "symbols": [ + "open" + ] + }, + { + "path": "PIL.TiffImagePlugin", + "symbols": [ + "TiffImageFile._setup" + ] + } + ], + "ranges": [ + { + "introduced": "9.2.0" + }, + { + "fixed": "9.3.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-227", + "imports": [ + { + "path": "PIL.ImageDraw", + "symbols": [ + "ImageDraw.text" + ] + }, + { + "path": "PIL.ImageFont", + "symbols": [ + "FreeTypeFont.getmask2" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "10.0.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-2257", + "imports": [ + { + "path": "PIL.ImageShow", + "symbols": [ + "WindowsViewer.get_command" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "12.3.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-3453", + "imports": [ + { + "path": "PIL.ImageCms", + "symbols": [ + "ImageCmsTransform.apply" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "12.3.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-457", + "imports": [ + { + "path": "PIL.ImageMath", + "symbols": [ + "eval" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "10.2.0" + } + ], + "source": "safedep:hand" + } + ], + "pyyaml": [ + { + "id": "PYSEC-2021-142", + "imports": [ + { + "path": "yaml", + "symbols": [ + "load", + "load_all", + "full_load", + "full_load_all" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "5.4.0" + } + ], + "source": "safedep:hand" + } + ], + "requests": [ + { + "id": "GHSA-9hjg-9r4m-mvj7", + "imports": [ + { + "path": "requests.sessions", + "symbols": [ + "Session.prepare_request" + ] + }, + { + "path": "requests.utils", + "symbols": [ + "get_netrc_auth" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.32.4" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-9wx4-h78v-vm56", + "imports": [ + { + "path": "requests.adapters", + "symbols": [ + "HTTPAdapter.send", + "HTTPAdapter.get_connection" + ] + }, + { + "path": "requests.sessions", + "symbols": [ + "Session.request" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.32.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-74", + "imports": [ + { + "path": "requests.adapters", + "symbols": [ + "HTTPAdapter.send" + ] + }, + { + "path": "requests.sessions", + "symbols": [ + "Session.rebuild_proxies", + "Session.send" + ] + } + ], + "ranges": [ + { + "introduced": "2.3.0" + }, + { + "fixed": "2.31.0" + } + ], + "source": "safedep:hand" + } + ], + "sqlparse": [ + { + "id": "GHSA-27jp-wm6q-gp25", + "imports": [ + { + "path": "sqlparse", + "symbols": [ + "format" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "0.5.4" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-2m57-hf25-phgg", + "imports": [ + { + "path": "sqlparse", + "symbols": [ + "parse" + ] + }, + { + "path": "sqlparse.engine.grouping", + "symbols": [ + "group" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "0.5.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-cfqr-cjx5-5jcm", + "imports": [ + { + "path": "sqlparse", + "symbols": [ + "format" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "0.6.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-f2ff-p2ww-7p4p", + "imports": [ + { + "path": "sqlparse", + "symbols": [ + "parse" + ] + }, + { + "path": "sqlparse.engine.grouping", + "symbols": [ + "group_comments" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "0.6.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-87", + "imports": [ + { + "path": "sqlparse", + "symbols": [ + "parse", + "format" + ] + }, + { + "path": "sqlparse.lexer", + "symbols": [ + "Lexer.get_tokens" + ] + } + ], + "ranges": [ + { + "introduced": "0.1.15" + }, + { + "fixed": "0.4.4" + } + ], + "source": "safedep:hand" + } + ], + "tornado": [ + { + "id": "GHSA-7cx3-6m66-7c5m", + "imports": [ + { + "path": "tornado.httputil", + "symbols": [ + "parse_multipart_form_data", + "parse_body_arguments" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.5.0" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-8w49-h785-mj3c", + "imports": [ + { + "path": "tornado.httputil", + "symbols": [ + "parse_cookie" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.4.2" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-c2m8-h5v5-343r", + "imports": [ + { + "path": "tornado.web", + "symbols": [ + "StaticFileHandler.get", + "StaticFileHandler.validate_absolute_path" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.5.9" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-fqwm-6jpj-5wxc", + "imports": [ + { + "path": "tornado.web", + "symbols": [ + "RequestHandler.set_cookie" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.5.5" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-pr2v-jx2c-wg9f", + "imports": [ + { + "path": "tornado.web", + "symbols": [ + "RequestHandler.set_status" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.5.3" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-qjxf-f2mg-c6mc", + "imports": [ + { + "path": "tornado.httputil", + "symbols": [ + "parse_multipart_form_data" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.5.5" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-75", + "imports": [ + { + "path": "tornado.web", + "symbols": [ + "StaticFileHandler.get", + "StaticFileHandler.validate_absolute_path" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "6.3.2" + } + ], + "source": "safedep:hand" + } + ], + "urllib3": [ + { + "id": "GHSA-34jh-p97f-mpxf", + "imports": [ + { + "path": "urllib3.connectionpool", + "symbols": [ + "HTTPConnectionPool.urlopen" + ] + }, + { + "path": "urllib3.poolmanager", + "symbols": [ + "PoolManager.urlopen" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "1.26.19" + }, + { + "introduced": "2.0.0" + }, + { + "fixed": "2.2.2" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-192", + "imports": [ + { + "path": "urllib3.connectionpool", + "symbols": [ + "HTTPConnectionPool.urlopen" + ] + }, + { + "path": "urllib3.poolmanager", + "symbols": [ + "PoolManager.urlopen" + ] + } + ], + "ranges": [ + { + "introduced": "2.0.0" + }, + { + "fixed": "2.0.6" + }, + { + "introduced": "0.0.0" + }, + { + "fixed": "1.26.17" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-212", + "imports": [ + { + "path": "urllib3.connectionpool", + "symbols": [ + "HTTPConnectionPool.urlopen" + ] + }, + { + "path": "urllib3.poolmanager", + "symbols": [ + "PoolManager.urlopen" + ] + } + ], + "ranges": [ + { + "introduced": "2.0.0" + }, + { + "fixed": "2.0.7" + }, + { + "introduced": "0.0.0" + }, + { + "fixed": "1.26.18" + } + ], + "source": "safedep:hand" + } + ], + "werkzeug": [ + { + "id": "GHSA-2g68-c3qc-8985", + "imports": [ + { + "path": "werkzeug.debug", + "symbols": [ + "DebuggedApplication.__call__" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.0.3" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-f9vj-2wh5-fj8j", + "imports": [ + { + "path": "werkzeug.security", + "symbols": [ + "safe_join" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.0.6" + } + ], + "source": "safedep:hand" + }, + { + "id": "GHSA-q34m-jh98-gwm2", + "imports": [ + { + "path": "werkzeug.formparser", + "symbols": [ + "MultiPartParser.parse" + ] + }, + { + "path": "werkzeug.sansio.multipart", + "symbols": [ + "MultipartDecoder.next_event" + ] + } + ], + "ranges": [ + { + "introduced": "2.0.0-rc1" + }, + { + "fixed": "3.0.6" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-221", + "imports": [ + { + "path": "werkzeug.formparser", + "symbols": [ + "MultiPartParser.parse" + ] + }, + { + "path": "werkzeug.sansio.multipart", + "symbols": [ + "MultipartDecoder.next_event" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.3.8" + }, + { + "introduced": "3.0.0" + }, + { + "fixed": "3.0.1" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-57", + "imports": [ + { + "path": "werkzeug._internal", + "symbols": [ + "_cookie_parse_impl" + ] + }, + { + "path": "werkzeug.http", + "symbols": [ + "parse_cookie" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.2.3" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2023-58", + "imports": [ + { + "path": "werkzeug.formparser", + "symbols": [ + "MultiPartParser.parse", + "FormDataParser.parse" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "2.2.3" + } + ], + "source": "safedep:hand" + }, + { + "id": "PYSEC-2026-4312", + "imports": [ + { + "path": "werkzeug.security", + "symbols": [ + "safe_join" + ] + } + ], + "ranges": [ + { + "introduced": "0.0.0" + }, + { + "fixed": "3.1.9" + } + ], + "source": "safedep:hand" + } + ] } -} \ No newline at end of file +} diff --git a/test/projectgraph/public/methods/corpus.txt b/test/projectgraph/public/methods/corpus.txt index 06069a2e..14db9e55 100644 --- a/test/projectgraph/public/methods/corpus.txt +++ b/test/projectgraph/public/methods/corpus.txt @@ -21,3 +21,13 @@ rust selectors selectors@0.22.0 smallvec@1.6.0 rust tokio-postgres tokio-postgres@0.7.7 postgres-protocol@0.6.11 tokio@1.13.0 rust lettre_email lettre_email@0.9.4 lettre@0.9.5 time@0.1.45 rust rusqlite_migration rusqlite_migration@0.5.0 rusqlite@0.22.0 +python mkdocs mkdocs@1.1.2 pyyaml@5.3.1 jinja2@2.11.3 tornado@6.3.1 +python httpie httpie@3.2.1 requests@2.28.2 urllib3@1.26.14 idna@3.6 +python requests-toolbelt requests-toolbelt@0.9.1 requests@2.30.0 urllib3@1.26.16 +python flask-cors flask-cors@3.0.10 flask@2.2.4 werkzeug@2.2.2 +python fabric fabric@2.7.1 paramiko@2.9.2 +python imageio imageio@2.9.0 pillow@9.2.0 +python pyopenssl pyopenssl@23.1.0 cryptography@40.0.2 +python python-docx python-docx@0.8.11 lxml@4.9.0 +python django-filter django-filter@23.1 django@4.2.0 +python django-debug-toolbar django-debug-toolbar@3.8.1 django@4.1.7 sqlparse@0.4.3 diff --git a/test/projectgraph/receivers_test.go b/test/projectgraph/receivers_test.go index 051fa023..7b5b7b69 100644 --- a/test/projectgraph/receivers_test.go +++ b/test/projectgraph/receivers_test.go @@ -37,13 +37,15 @@ type receiverRow struct { // receiverUnits are the units of dependency code of the receiver rows, by // language: a directory under testdata/receivers and its unit. var receiverUnits = map[core.LanguageCode][2]string{ - core.LanguageCodeGo: {"go/dep", "go:example.com/dep@v1.0.0"}, - core.LanguageCodeRust: {"rust/dep", "cargo:dep@1.0.0"}, + core.LanguageCodeGo: {"go/dep", "go:example.com/dep@v1.0.0"}, + core.LanguageCodeRust: {"rust/dep", "cargo:dep@1.0.0"}, + core.LanguageCodePython: {"python/dep", "pypi:dep@1.0.0"}, } const ( - goReceiverManifest = "module app\n\ngo 1.22\n\nrequire example.com/dep v1.0.0\n" - rustReceiverManifest = "[package]\nname = \"app\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\ndep = \"1.0.0\"\n" + goReceiverManifest = "module app\n\ngo 1.22\n\nrequire example.com/dep v1.0.0\n" + rustReceiverManifest = "[package]\nname = \"app\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[dependencies]\ndep = \"1.0.0\"\n" + pythonReceiverManifest = "[project]\nname = \"app\"\nversion = \"0.1.0\"\ndependencies = [\"dep\"]\n" ) var goReceiverRows = []receiverRow{ @@ -209,6 +211,161 @@ var rustReceiverRows = []receiverRow{ }, } +// pythonTypeHints is the step of the Python plan that reads the type hints +// of a parameter, a local, a field and a return into the index, and gives +// a definition of a pypi unit its external key. Most Python rows wait on +// it. Two forms of today show the gap: a call on a variable that holds +// the result of an external call gets the false key dep//new_thing//do and +// no record, so the method is no-such-target, and a call on an unknown +// receiver is a record and a name edge, so the method is uncertain. +const pythonTypeHints = "the type-hint step of the Python plan" + +// pythonReceiverRows are the forms of a Python receiver. A type hint is +// the only declared type that Python has. +var pythonReceiverRows = []receiverRow{ + { + name: "declared variable", + source: "def run():\n t: Thing = dep.new_thing()\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "no-such-target", + }, + { + name: "parameter", + source: "def run(t: Thing):\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "optional parameter", + source: "def run(t: Optional[Thing]):\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "union with none parameter", + source: "def run(t: Thing | None):\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "untyped parameter", + source: "def run(t):\n t.do()", + target: "dep//Thing//do", + want: "uncertain/unknown-receiver", + }, + { + name: "typed instance attribute", + source: "class App:\n def __init__(self):\n self.t: Thing = dep.new_thing()\n\n def go(self):\n self.t.do()\n\n\ndef run():\n App().go()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/no-definition", + }, + { + name: "instance attribute from the constructor", + source: "class App:\n def __init__(self):\n self.t = Thing()\n\n def go(self):\n self.t.do()\n\n\ndef run():\n App().go()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/no-definition", + }, + { + name: "dataclass field", + source: "@dataclass\nclass App:\n t: Thing\n\n\ndef run(a: App):\n a.t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "first-party function return", + source: "def make() -> Thing:\n return dep.new_thing()\n\n\ndef run():\n t = make()\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "external function return", + source: "def run():\n t = dep.new_thing()\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "no-such-target", + }, + { + name: "tuple return", + source: "def run():\n t, n = dep.new_pair()\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "optional return", + source: "def run():\n t = dep.try_thing()\n if t is not None:\n t.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "no-such-target", + }, + { + name: "chained call", + source: "def run():\n dep.new_thing().do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "chained method call", + source: "def run():\n dep.new_thing().next().do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "uncertain/unknown-receiver", + }, + { + name: "instance from the constructor", + source: "def run():\n Thing().do()", + target: "dep//Thing//do", + want: "found", + }, + { + name: "abstract base return", + source: "def run():\n d = dep.new_doer()\n d.do()", + target: "dep//Thing//do", + want: "found", + gap: pythonTypeHints, + today: "no-such-target", + }, + { + name: "abstract base return with no implementation", + source: "def run():\n r = dep.new_runner()\n r.run()", + target: "other//Job//run", + want: "uncertain/interface-receiver", + gap: pythonTypeHints, + today: "no-such-target", + }, + { + name: "external function return with no dependency code", + source: "def run():\n t = dep.new_thing()\n t.do()", + target: "dep//Thing//do", + want: "uncertain/no-dependency-source", + noDeps: true, + gap: pythonTypeHints, + today: "no-such-target", + }, +} + func TestReceiverForms(t *testing.T) { languages := caseLanguages(t) for _, tc := range []struct { @@ -218,6 +375,7 @@ func TestReceiverForms(t *testing.T) { }{ {core.LanguageCodeGo, goReceiverRows, goReceiverProject}, {core.LanguageCodeRust, rustReceiverRows, rustReceiverProject}, + {core.LanguageCodePython, pythonReceiverRows, pythonReceiverProject}, } { for _, row := range tc.rows { t.Run(string(tc.language)+"/"+row.name, func(t *testing.T) { @@ -250,6 +408,15 @@ func rustReceiverProject(source string) map[string]string { } } +// pythonReceiverProject is a Python project whose main.py holds the source +// and the imports of the dependency. +func pythonReceiverProject(source string) map[string]string { + return map[string]string{ + "pyproject.toml": pythonReceiverManifest, + "main.py": "from dataclasses import dataclass\nfrom typing import Optional\n\nimport dep\nfrom dep import Doer, Thing\n\n\n" + source + "\n\n\nif __name__ == \"__main__\":\n run()\n", + } +} + // receiverAnswer links a project of the files with the dependency code of // the language and answers the target of the row. func receiverAnswer(t *testing.T, languages []core.Language, language core.LanguageCode, files map[string]string, row receiverRow) string { @@ -282,8 +449,11 @@ func receiverAnswer(t *testing.T, languages []core.Language, language core.Langu // which the search starts from. func receiverSource(t *testing.T, language core.LanguageCode) reach.Source { p := "main.go" - if language == core.LanguageCodeRust { + switch language { + case core.LanguageCodeRust: p = "src/main.rs" + case core.LanguageCodePython: + p = "main.py" } s, err := reach.SourceByKey(fileindex.Join(p, "run")) require.NoError(t, err) diff --git a/test/projectgraph/report.md b/test/projectgraph/report.md index d2c602a6..d261ee42 100644 --- a/test/projectgraph/report.md +++ b/test/projectgraph/report.md @@ -1037,3 +1037,116 @@ After FOU-555: After the fixes for the review of PR 53, the measure gives the same numbers. No project of the corpus renames a Cargo dependency, and no function of a dependency has two platform twins with two result types. + +### Python + +The Python rows of the corpus come from the first step of the Python +plan, on 2026-10-24, on the machine above, at the commit that adds them. +No fix of the Python link step is in this run. The numbers are the +baseline that the later steps of the plan must move. + +| Project | Dependencies with an advisory | +| --- | --- | +| mkdocs 1.1.2 | PyYAML 5.3.1, Jinja2 2.11.3, tornado 6.3.1 | +| httpie 3.2.1 | requests 2.28.2, urllib3 1.26.14, idna 3.6 | +| requests-toolbelt 0.9.1 | requests 2.30.0, urllib3 1.26.16 | +| flask-cors 3.0.10 | Flask 2.2.4, Werkzeug 2.2.2 | +| fabric 2.7.1 | paramiko 2.9.2 | +| imageio 2.9.0 | Pillow 9.2.0 | +| pyopenssl 23.1.0 | cryptography 40.0.2 | +| python-docx 0.8.11 | lxml 4.9.0 | +| django-filter 23.1 | Django 4.2.0 | +| django-debug-toolbar 3.8.1 | Django 4.1.7, sqlparse 0.4.3 | + +A Python dependency has a version that the requirement of the project +admits and that an advisory affects, as the Rust rows do. OSV lists no +symbols for a PyPI advisory, so `scripts/pypi_advisories.py` holds the +symbols of 72 advisories of the 14 packages by hand, as the text of each +advisory names them, with the ranges from OSV. The code of a dependency is +its wheel, unpacked, so the unit has the layout of site-packages. A +dependency whose code is compiled, as lxml, has only its `.so` files and +its Python wrappers in the unit. + +The receiver table of `receivers_test.go` has 18 Python rows. 2 rows give +their answer today: a call on an instance from a constructor is `found`, +and a call on an untyped parameter is `uncertain/unknown-receiver`. The +other 16 wait on the type-hint step of the plan. 8 of them are +`uncertain/unknown-receiver` today, 2 are `uncertain/no-definition`, and 6 +are `no-such-target`: a call on a variable that holds the result of an +external call, as `t = dep.new_thing()` then `t.do()`, gets the false key +`dep//new_thing//do` and no record, so the method of the real type is +`no-such-target`. A Python definition of a unit has no external key either, +because the Python profile has no `symbol` rule, so a target that only the +code of its unit calls cannot match a definition. + +| Language | Kind | Run | Targets | found | not-found | no-such-target | unknown-receiver | no-definition | dynamic-flag | budget | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Python | function | every language | 44 | 5 | 0 | 0 | 3 | 1 | 6 | 29 | +| Python | method | every language | 63 | 0 | 0 | 0 | 27 | 5 | 13 | 18 | +| Python | function | python alone | 44 | 5 | 0 | 0 | 3 | 13 | 14 | 9 | +| Python | method | python alone | 63 | 0 | 0 | 0 | 27 | 14 | 17 | 5 | + +| Language | Run | Method targets that end as unknown-receiver | +| --- | --- | --- | +| Python | every language | 0.429 (27/63) | +| Python | python alone | 0.429 (27/63) | + +The "every language" run indexes each language, as the Go and Rust rows +do. The "python alone" run sets `PROJECTGRAPH_METHOD_LANGUAGES=python`. +The 10 Python graphs hold 3,170 files in the first run and 2,956 in the +second. + +- No target is `not-found` and no target is `no-such-target`. Every + search meets an uncertainty that concerns every target, so the measure + cannot yet tell a target that the code reaches from one that it does + not. The 5 found function targets are `yaml.load` in mkdocs, + `requests.utils.get_netrc_auth` and `idna.encode` in httpie, and + `sqlparse.parse` and `sqlparse.format` in django-debug-toolbar, each a + call of the project on an import. +- `budget` is the largest cause of the first run, and most of it is + JavaScript. A Python project ships the JavaScript of its web pages, and + the index step truncates a minified file: `jquery-1.10.2.min.js`, + `bootstrap.min.js` and `lunr.js` in the themes of mkdocs, and + `jquery.js`, `jquery.min.js` and `select2.full.min.js` in the admin of + the Django unit. A truncated file of the project concerns every search, + and one of a unit concerns every search that enters the unit, so 32 of + the 32 Django targets of django-filter end as `budget`. No Python program + runs that JavaScript. The run of `python` alone has 14 budget targets + left: `_tifffile.py` of imageio, a module of 10,182 lines and 360 KiB + over a limit of the index, and the test files `tests/test_ssl.py` of pyopenssl and + `tests/opc/unitdata/rels.py` and `tests/oxml/test_xmlchemy.py` of + python-docx. The test paths mark them as test files, and a truncated + test file still concerns every search. A Werkzeug file, + `werkzeug/serving.py`, is truncated too. +- `dynamic-flag` is the next cause, and `decorator-wrap` is most of it. + The index flags a decorator that is not a bare name, as + `@click.option(...)`, `@app.route(...)` and `@functools.wraps(f)`, + because the decorator can wrap the function in code that the graph does + not see. Every click, Flask or Django project has such a decorator on + the path from its roots, so the flag concerns every target of the + project. The other flags are `monkey-patch` in httpie, python-docx and + the Django unit, `reflection` in fabric and `computed-attribute` in + httpie. The `computed-name` records of `click.option(...)` come from the + same decorators: the call of a call result has no name. +- `no-definition` is the first cause when no JavaScript is in the graph: a + call of a name that no file defines, as `http.client` in httpie and + `.packages.six.moves.http_client` in urllib3, both a module of the + standard library or a vendored copy that the graph has no unit for. The + Python standard library has no unit in this run. +- `unknown-receiver` decides 27 of the 63 method targets, and the share is + the same in both runs. The receiver forms that the rows name are the + cause: a typed parameter, a typed attribute, a function result, and a + call on a value that a `for` loop or a comprehension binds. +- fabric has no root of its own. It is a library, so every exported symbol + is a root, and the `root-fallback` uncertainty is on each result. +- lxml gives no target of its own kind. Its `iterwalk`, `canonicalize` and + `iterparse` are compiled, and the unit holds no Python definition for + them, so each one ends in the uncertainty of the project. + +The numbers say what the plan must change first. The type-hint step moves +`unknown-receiver`. Before it can show, a Python project needs three +rules of the index and the link step: a JavaScript file that a Python +project or a pypi unit ships for a web page is no code of the program, a +truncated test file concerns only a search that counts the tests, and a +decorator that a framework rule knows, as a route or a command, wraps the +function in code that the rule already reads. diff --git a/test/projectgraph/testdata/receivers/python/dep/dep/__init__.py b/test/projectgraph/testdata/receivers/python/dep/dep/__init__.py new file mode 100644 index 00000000..1a00c522 --- /dev/null +++ b/test/projectgraph/testdata/receivers/python/dep/dep/__init__.py @@ -0,0 +1,45 @@ +import abc +import subprocess + + +class Doer(abc.ABC): + @abc.abstractmethod + def do(self): + raise NotImplementedError + + +class Runner(abc.ABC): + @abc.abstractmethod + def run(self): + raise NotImplementedError + + +class Thing(Doer): + def __init__(self): + self.child = None + + def do(self): + subprocess.run(["true"]) + + def next(self) -> "Thing": + return self.child + + +def new_thing() -> Thing: + return Thing() + + +def new_pair() -> tuple[Thing, int]: + return Thing(), 0 + + +def try_thing() -> Thing | None: + return Thing() + + +def new_doer() -> Doer: + return Thing() + + +def new_runner() -> Runner: + raise NotImplementedError diff --git a/test/projectgraph/testdata/receivers/python/dep/pyproject.toml b/test/projectgraph/testdata/receivers/python/dep/pyproject.toml new file mode 100644 index 00000000..34e3dec0 --- /dev/null +++ b/test/projectgraph/testdata/receivers/python/dep/pyproject.toml @@ -0,0 +1,3 @@ +[project] +name = "dep" +version = "1.0.0" From 79cfa6ebbf3f9d2ae8e28c596450987ae9a73100 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 06:37:41 +0000 Subject: [PATCH 02/11] feat(projectgraph): count only the files, the tests and the decorators that the program runs The Python baseline of the method measure had every target uncertain. Three causes hid the receiver forms that the type-hint step must fix, and none of them was a call that the graph could not see. A Python project ships the JavaScript of its web pages, and the index step truncates a minified file of it. Every search counted the file as a budget. The index step now reports the language of each file, the link step keeps it on the module node, and Stats.Languages holds the languages of the files with a root other than a test root. Graph.Runs reads them, with the languages that the ecosystem of a unit installs. The reach query and the summary leave out the budget of a file that the program does not run. A truncated test file counts only for a search that includes the tests. The index flagged every decorator that was not a bare name as decorator-wrap, so each click or Flask project made every target uncertain. A decorator that is a name, a dotted name or a call of one is a call of the scope that defines the function, which the link step reads. The flag stays for a decorator that the graph cannot name, as @handlers[name]. A call of the value of a call, as click.option(...)(f), is no computed-name record of its own, unless the inner call reflects. An absolute Python import was first party when a file of the name was anywhere in the project, so import types in cryptography and import collections in Django were records that concerned every target. The first segment must now be a module under an import root or in the directory of the importer, as the import system reads it. Five cases cover the rules. flask-route now reads a dead call as not-found. pycg-micro keeps 0.954 precision and 0.708 recall, with the unresolved rate down from 0.340 to 0.312. The Python budget of the measure falls from 47 to 5 targets, and the run of every language gives the same numbers as the run of Python alone. report.md has the table. fileindex.Version 15, projectgraph.Version 17, reach.Version 4. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- docs/projectgraph.md | 37 +++++++++- examples/projectgraph/main.go | 3 +- fileindex/index.go | 10 ++- fileindex/testdata/index.json | 2 +- plugin/callgraph/python.go | 24 ++++++ plugin/callgraph/records_test.go | 7 +- plugin/indexer/flags.go | 27 ++++++- plugin/indexer/flags_test.go | 27 ++++--- plugin/indexer/indexer.go | 9 +++ plugin/indexer/testdata/flags.py | 5 ++ plugin/indexer/testdata/golden/App.cs.json | 2 +- plugin/indexer/testdata/golden/App.java.json | 2 +- plugin/indexer/testdata/golden/App.kt.json | 2 +- plugin/indexer/testdata/golden/app.go.json | 2 +- plugin/indexer/testdata/golden/app.js.json | 2 +- plugin/indexer/testdata/golden/app.php.json | 2 +- plugin/indexer/testdata/golden/app.py.json | 2 +- plugin/indexer/testdata/golden/app.rb.json | 2 +- plugin/indexer/testdata/golden/app.rs.json | 2 +- plugin/indexer/testdata/golden/app.ts.json | 2 +- .../indexer/testdata/golden/exports.js.json | 2 +- .../indexer/testdata/golden/exports.py.json | 2 +- .../indexer/testdata/golden/exports.ts.json | 2 +- plugin/indexer/testdata/golden/flags.js.json | 12 +-- plugin/indexer/testdata/golden/flags.py.json | 49 +++++++----- plugin/indexer/testdata/golden/flags.ts.json | 24 +----- projectgraph/definitions.go | 2 +- projectgraph/graph.go | 74 ++++++++++++++++--- projectgraph/link.go | 10 ++- projectgraph/modules/python_test.go | 24 ++++++ projectgraph/modules/resolver.go | 31 +++++++- projectgraph/packages.go | 8 ++ projectgraph/reach/reach.go | 2 +- projectgraph/reach/reach_test.go | 73 ++++++++++++++++++ projectgraph/reach/uncertainty.go | 37 ++++++++-- projectgraph/resolve.go | 12 ++- projectgraph/roots.go | 16 ++++ projectgraph/summary/summary.go | 5 +- .../javascript/barrel-chain/graph.golden.json | 19 +++-- .../javascript/barrel-chain/indexes.json | 8 +- .../commonjs-require/graph.golden.json | 16 ++-- .../javascript/commonjs-require/indexes.json | 6 +- .../default-import/graph.golden.json | 16 ++-- .../javascript/default-import/indexes.json | 6 +- .../esm-named-import/graph.golden.json | 16 ++-- .../javascript/esm-named-import/indexes.json | 6 +- .../javascript/export-star/graph.golden.json | 22 ++++-- .../javascript/export-star/indexes.json | 10 +-- .../exports-conditions/graph.golden.json | 19 +++-- .../exports-conditions/indexes.json | 8 +- .../unknown-receiver/graph.golden.json | 10 ++- .../javascript/unknown-receiver/indexes.json | 2 +- .../javascript/workspaces/graph.golden.json | 16 ++-- .../javascript/workspaces/indexes.json | 6 +- .../python/all-hides/graph.golden.json | 16 ++-- .../testdata/python/all-hides/indexes.json | 6 +- .../python/assignment/graph.golden.json | 10 ++- .../testdata/python/assignment/indexes.json | 2 +- .../python/class-body/graph.golden.json | 8 +- .../testdata/python/class-body/indexes.json | 2 +- .../python/decorators/graph.golden.json | 10 ++- .../testdata/python/decorators/indexes.json | 2 +- .../python/external/graph.golden.json | 10 ++- .../testdata/python/external/indexes.json | 2 +- .../python/local-receiver/graph.golden.json | 10 ++- .../python/local-receiver/indexes.json | 2 +- .../python/member-call/graph.golden.json | 16 ++-- .../testdata/python/member-call/indexes.json | 6 +- .../python/missing-file/graph.golden.json | 10 ++- .../testdata/python/missing-file/indexes.json | 2 +- .../python/package-init/graph.golden.json | 16 ++-- .../testdata/python/package-init/indexes.json | 6 +- .../python/reexport-cycle/graph.golden.json | 16 ++-- .../python/reexport-cycle/indexes.json | 6 +- .../python/relative-imports/graph.golden.json | 16 ++-- .../python/relative-imports/indexes.json | 6 +- .../python/sibling-import/graph.golden.json | 13 +++- .../python/sibling-import/indexes.json | 4 +- .../python/src-layout/graph.golden.json | 16 ++-- .../testdata/python/src-layout/indexes.json | 6 +- .../python/sys-path/graph.golden.json | 13 +++- .../testdata/python/sys-path/indexes.json | 4 +- .../python/wildcard/graph.golden.json | 13 +++- .../testdata/python/wildcard/indexes.json | 4 +- .../roots/extra-rules/graph.golden.json | 18 +++-- .../testdata/roots/extra-rules/indexes.json | 6 +- .../roots/library-surface/graph.golden.json | 13 +++- .../roots/library-surface/indexes.json | 4 +- .../roots/main-guard/graph.golden.json | 13 +++- .../testdata/roots/main-guard/indexes.json | 4 +- .../roots/package-json-bin/graph.golden.json | 13 +++- .../roots/package-json-bin/indexes.json | 4 +- .../roots/package-json-main/graph.golden.json | 19 +++-- .../roots/package-json-main/indexes.json | 8 +- .../procfile-dockerfile/graph.golden.json | 26 +++++-- .../roots/procfile-dockerfile/indexes.json | 12 +-- .../roots/pyproject-scripts/graph.golden.json | 22 ++++-- .../roots/pyproject-scripts/indexes.json | 10 +-- .../testdata/roots/setup-py/graph.golden.json | 13 +++- .../testdata/roots/setup-py/indexes.json | 4 +- .../roots/test-file/graph.golden.json | 10 ++- .../testdata/roots/test-file/indexes.json | 2 +- .../typescript/js-specifier/graph.golden.json | 13 +++- .../typescript/js-specifier/indexes.json | 4 +- .../tsconfig-paths/graph.golden.json | 16 ++-- .../typescript/tsconfig-paths/indexes.json | 6 +- .../typescript/type-only/graph.golden.json | 16 ++-- .../typescript/type-only/indexes.json | 6 +- test/projectgraph/bench_test.go | 4 +- .../go_github.com_acme_kit_v1.0.0.json | 4 +- .../go_github.com_acme_kit_v1.0.0.json | 4 +- .../go_github.com_acme_dep_v1.0.0.json | 4 +- .../go_github.com_acme_kit_v1.0.0.json | 4 +- .../go_github.com_acme_yamlish_v1.2.0.json | 4 +- .../go_github.com_acme_kit_v1.0.0.json | 4 +- .../go_github.com_acme_kit_v1.0.0.json | 4 +- .../go_example.com_markup_v1.0.0.json | 4 +- .../summaries/go_std_go1.25.1.json | 4 +- .../summaries/npm_store-kit_1.0.0.json | 4 +- .../summaries/npm_kit_1.0.0.json | 4 +- .../summaries/npm_kit_1.0.0.json | 4 +- .../summaries/npm_plug_1.0.0.json | 4 +- .../summaries/npm_util-kit_1.0.0.json | 4 +- .../npm_lodash._reinterpolate_3.0.0.json | 4 +- .../summaries/npm_lodash_4.17.21.json | 4 +- .../python/bundled-javascript/expect.yaml | 41 ++++++++++ .../python/bundled-javascript/project/main.py | 4 + .../project/static/vendor.min.js | 12 +++ .../python/decorator-computed/expect.yaml | 21 ++++++ .../python/decorator-computed/project/main.py | 13 ++++ .../python/decorator-factory/expect.yaml | 60 +++++++++++++++ .../python/decorator-factory/project/main.py | 17 +++++ .../summaries/pypi_PyYAML_6.0.1.json | 4 +- .../summaries/pypi_badpkg_1.0.0.json | 4 +- .../summaries/pypi_okpkg_1.0.0.json | 4 +- .../summaries/pypi_badpkg_1.0.0.json | 4 +- .../summaries/pypi_okpkg_1.0.0.json | 4 +- .../summaries/pypi_outer_1.0.0.json | 4 +- .../summaries/pypi_requests_2.31.0.json | 4 +- .../summaries/pypi_urllib3_2.0.7.json | 4 +- .../cases/python/flask-route/expect.yaml | 7 +- .../cases/python/stdlib-name-deep/expect.yaml | 51 +++++++++++++ .../stdlib-name-deep/project/app/__init__.py | 0 .../project/app/models/__init__.py | 0 .../project/app/models/types.py | 3 + .../python/stdlib-name-deep/project/main.py | 14 ++++ .../python/truncated-test-file/expect.yaml | 37 ++++++++++ .../truncated-test-file/project/main.py | 4 + .../project/tests/test_big.py | 35 +++++++++ .../summaries/cargo_kit_1.0.0.json | 4 +- .../summaries/cargo_kit_1.0.0.json | 4 +- .../summaries/cargo_kit_1.0.0.json | 4 +- .../summaries/cargo_kit_1.0.0.json | 4 +- test/projectgraph/cases_test.go | 2 +- test/projectgraph/helpers_test.go | 2 +- test/projectgraph/report.md | 56 ++++++++++++++ test/projectgraph/version_test.go | 3 + 157 files changed, 1327 insertions(+), 428 deletions(-) create mode 100644 test/projectgraph/cases/python/bundled-javascript/expect.yaml create mode 100644 test/projectgraph/cases/python/bundled-javascript/project/main.py create mode 100644 test/projectgraph/cases/python/bundled-javascript/project/static/vendor.min.js create mode 100644 test/projectgraph/cases/python/decorator-computed/expect.yaml create mode 100644 test/projectgraph/cases/python/decorator-computed/project/main.py create mode 100644 test/projectgraph/cases/python/decorator-factory/expect.yaml create mode 100644 test/projectgraph/cases/python/decorator-factory/project/main.py create mode 100644 test/projectgraph/cases/python/stdlib-name-deep/expect.yaml create mode 100644 test/projectgraph/cases/python/stdlib-name-deep/project/app/__init__.py create mode 100644 test/projectgraph/cases/python/stdlib-name-deep/project/app/models/__init__.py create mode 100644 test/projectgraph/cases/python/stdlib-name-deep/project/app/models/types.py create mode 100644 test/projectgraph/cases/python/stdlib-name-deep/project/main.py create mode 100644 test/projectgraph/cases/python/truncated-test-file/expect.yaml create mode 100644 test/projectgraph/cases/python/truncated-test-file/project/main.py create mode 100644 test/projectgraph/cases/python/truncated-test-file/project/tests/test_big.py diff --git a/docs/projectgraph.md b/docs/projectgraph.md index 078744d5..0d0359c3 100644 --- a/docs/projectgraph.md +++ b/docs/projectgraph.md @@ -130,6 +130,7 @@ order of the indexes. | `Root` | The manifests of the project, as `pyproject.toml`. A nil root reads no manifest | | `SysPath` | More Python import roots, relative to the project root | | `Files` | The status of each file of the index step. A file with no complete index becomes a module node with that status | +| `Languages` | The language of each file of the index step, as `indexer.Result.Languages` gives it. The module node of a file with no index gets its language from here, so the reach query can tell a file that the program does not run | | `TestPaths` | Globs that mark the test files. `fileindex.DefaultTestPaths` holds the common ones | | `Rules` | Root rules that go over the embedded ones by id | | `Units` | The files of each unit of dependency code, by unit. The link step links each unit as a project of its own | @@ -414,7 +415,7 @@ causes it. | --- | --- | | `unresolved-call` | A call that the link step could not resolve, on a path from a root | | `dynamic-flag` | A dynamic feature, as `eval` or a computed attribute, in code on a path from a root | -| `budget` | A missing, failed or truncated file, a limit of the link step, or a search that the depth cap or the deadline stopped | +| `budget` | A missing, failed or truncated file that the program runs, a limit of the link step, or a search that the depth cap or the deadline stopped | | `root-fallback` | No root rule matched, so every exported symbol is a root | An uncertainty concerns every target, unless its site says less. A call @@ -429,5 +430,39 @@ A file of dependency code that the index step did not complete counts only for a search that enters its package. A file of the project counts for every search, because it can hold a root. +Two kinds of incomplete file count for no search, because no search can +reach a caller in them. `Graph.Runs` decides the first. + +- A file of a language that the program does not run. The languages that + the program runs are the languages of the files that hold a root other + than a test root, in `Graph.Stats.Languages`, and for a unit of dependency code the + languages that its ecosystem installs, as Python for `pypi`. A Python + project ships the JavaScript of its web pages, and the index step + truncates a minified file of it. No Python program runs that file. A + file of an unknown language counts, and so does every file of a graph + with no root. +- A test file, which holds only test roots. It counts for a search with + `Query.IncludeTests`. + +An import that no file of the project or of the unit gives is an external +node, unless the module can still be first party: a relative module, or a +module whose first segment a search path root or a build step can give. +Such an import is a record with the reason `no-definition`, and it +concerns every target, because the module can hold any code. In Python +the first segment must be a module or a package under an import root or +in the directory of the importer, as the import system reads it. A file +of the name deep in a package, as a `types.py`, is no module of +`import types`, so that import names the standard library and is external. + +A decorator counts as a dynamic feature, with the flag `decorator-wrap`, +only when the graph cannot name its code, as `@handlers[name]`. A +decorator that is a name, a dotted name or a call of one, as +`@app.route("/")` or `@click.option("-v")`, is a call of the scope that +defines the function, and the link step reads it. A call of the value of +a call, as `click.option("-v")(f)` in a decorator factory, is no record +of its own: the graph reaches the code that the inner call gives through +the inner call. A reflective inner call, as `getattr(o, n)()`, keeps its +record. + A summary keeps every uncertainty of a symbol in a table, with the targets that its scope concerns, so `Status` applies the same rule with no graph. diff --git a/examples/projectgraph/main.go b/examples/projectgraph/main.go index 97cfb662..1e0da7a0 100644 --- a/examples/projectgraph/main.go +++ b/examples/projectgraph/main.go @@ -262,6 +262,7 @@ func printReach(ctx context.Context, cfg config, out io.Writer, root string, ind g, err := projectgraph.Build(ctx, indexed.Indexes, projectgraph.Options{ Root: os.DirFS(root), Files: indexed.Statuses(), + Languages: indexed.Languages, TestPaths: fileindex.DefaultTestPaths, Units: units, }) @@ -313,7 +314,7 @@ func printSummaries(ctx context.Context, out io.Writer, indexed *indexer.Result, if len(indexes) == 0 && len(files) == 0 { return fmt.Errorf("the unit %s has no file to summarize", unit) } - s, err := summary.Summarize(ctx, indexes, projectgraph.Options{Files: files, Units: map[string]iofs.FS{unit: units[unit]}}) + s, err := summary.Summarize(ctx, indexes, projectgraph.Options{Files: files, Languages: indexed.Languages, Units: map[string]iofs.FS{unit: units[unit]}}) if err != nil { return fmt.Errorf("failed to summarize %s: %w", unit, err) } diff --git a/fileindex/index.go b/fileindex/index.go index 53c8de84..0476dfdc 100644 --- a/fileindex/index.go +++ b/fileindex/index.go @@ -19,7 +19,7 @@ import ( // Version is the version of the index records and of the rules that fill // them. Bump it when a field, a kind, an order or a fill rule changes. A // store keys an index by Version, so the bump makes every older index miss. -const Version = "14" +const Version = "15" // DefaultTestPaths are the globs that mark the test files of the common // layouts, as projectgraph.Options.TestPaths takes them. A glob with no "/" @@ -108,8 +108,12 @@ const ( FlagComputedImport FlagKind = "computed-import" FlagComputedAttribute FlagKind = "computed-attribute" FlagMonkeyPatch FlagKind = "monkey-patch" - FlagDecoratorWrap FlagKind = "decorator-wrap" - FlagParseError FlagKind = "parse-error" + // FlagDecoratorWrap marks a decorator whose code the graph cannot + // name, as @handlers[name] or @make()(). A decorator that is a name, + // a dotted name or a call of one, as @app.route("/"), is a call of + // the scope that defines the function, and the link step reads it. + FlagDecoratorWrap FlagKind = "decorator-wrap" + FlagParseError FlagKind = "parse-error" // FlagInjection marks a value that a container sets, as a Java field // with @Autowired. The declared type of the value narrows a call on it. FlagInjection FlagKind = "injection" diff --git a/fileindex/testdata/index.json b/fileindex/testdata/index.json index bfd539ee..53e6bc85 100644 --- a/fileindex/testdata/index.json +++ b/fileindex/testdata/index.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "src/app.py", "language": "python", "content_hash": "a172cedcae47474b615c54d510a5d84a8dea3032e958587430b413538be3f333", diff --git a/plugin/callgraph/python.go b/plugin/callgraph/python.go index 1a29f8a2..fb1b59a0 100644 --- a/plugin/callgraph/python.go +++ b/plugin/callgraph/python.go @@ -224,6 +224,16 @@ func functionCallProcessor(functionCallNode *sitter.Node, argumentsNode *sitter. } break } + // A call of the value of a call, as click.option(...)(f) in a + // decorator factory, runs code that the inner call gives. The + // graph reaches that code through the inner call and the + // reference edges from its callee, so the outer call is no record + // of its own. A reflective inner call, as getattr(o, n), gives a + // value that no edge names, so it keeps the record. Any other + // callee computes its name. + if function.Type() == "call" && !pythonReflective(function, treeData) { + return result + } reason = UnresolvedComputedName } @@ -381,3 +391,17 @@ func dissectAttributeQualifierWithDepth(node *sitter.Node, treeData []byte, curr return objectSymbol, attributeQualifierNamespace, nil } + +// pythonReflectiveCallees name the calls whose value the graph cannot +// name: a lookup of an attribute or a module by a name, and a run of code +// in a string. +var pythonReflectiveCallees = map[string]bool{ + "getattr": true, "__import__": true, "importlib.import_module": true, "import_module": true, + "eval": true, "exec": true, "compile": true, +} + +// pythonReflective reports whether a call reads its value by reflection. +func pythonReflective(call *sitter.Node, treeData []byte) bool { + callee := call.ChildByFieldName("function") + return callee != nil && pythonReflectiveCallees[callee.Content(treeData)] +} diff --git a/plugin/callgraph/records_test.go b/plugin/callgraph/records_test.go index f760ca13..ec93f6cc 100644 --- a/plugin/callgraph/records_test.go +++ b/plugin/callgraph/records_test.go @@ -229,13 +229,16 @@ func TestUnresolvedCallSites(t *testing.T) { }, }, { + // A call of the value of a call, as f()(), runs code that the + // inner call gives, and the graph reaches it through the inner + // call. So it is no record, unless the inner call reflects. name: "python computed receiver and call result", language: pythonLanguage(t), file: "recv.py", - source: "handlers[name].run()\nf()()\n", + source: "handlers[name].run()\nf()()\nmake(x)(y)\ngetattr(o, n)()\n", expected: []site{ {"handlers[name].run", "handlers[name]", "run", UnresolvedComputedName}, - {"f()", "", "", UnresolvedComputedName}, + {"getattr(o, n)", "", "", UnresolvedComputedName}, }, }, { diff --git a/plugin/indexer/flags.go b/plugin/indexer/flags.go index 5762c014..baeb6a2b 100644 --- a/plugin/indexer/flags.go +++ b/plugin/indexer/flags.go @@ -107,7 +107,7 @@ func flagVisitor(p *projection) visitor { case syntax.assignment: p.flagMonkeyPatch(*syntax, node.ChildByFieldName("left")) case "decorator": - if node.NamedChildCount() > 0 && node.NamedChild(0).Type() != "identifier" { + if node.NamedChildCount() > 0 && !namedDecorator(node.NamedChild(0)) { p.flagAt(fileindex.FlagDecoratorWrap, decorated(node).StartByte(), lineOf(node)) } } @@ -138,6 +138,31 @@ func mainGuardVisitor(p *projection) visitor { } } +// namedDecorator reports whether the expression of a decorator names its +// code: a name, a dotted name, or a call of one, as @app.route("/"). The +// link step resolves such a decorator as a call of the scope that defines +// the function. Any other expression, as @handlers[name] or @make()(), +// gives code that the graph cannot name. +func namedDecorator(expression *sitter.Node) bool { + for expression != nil { + switch expression.Type() { + case "identifier": + return true + case "attribute", "member_expression": + expression = expression.ChildByFieldName("object") + case "call", "call_expression": + callee := expression.ChildByFieldName("function") + if callee == nil || callee.Type() == "call" || callee.Type() == "call_expression" { + return false + } + expression = callee + default: + return false + } + } + return false +} + // decorated returns the node that a decorator wraps. A TypeScript class // body holds the decorators of a member before the member. func decorated(decorator *sitter.Node) *sitter.Node { diff --git a/plugin/indexer/flags_test.go b/plugin/indexer/flags_test.go index d722dc7e..7caa1d07 100644 --- a/plugin/indexer/flags_test.go +++ b/plugin/indexer/flags_test.go @@ -1,6 +1,7 @@ package indexer import ( + "slices" "testing" "github.com/safedep/code/core" @@ -24,6 +25,11 @@ func TestDynamicSites(t *testing.T) { file string code core.LanguageCode sites []fileindex.Flag + // missing names the kinds that no line of the fixture can set. The + // JavaScript and the TypeScript grammars read a decorator as a + // name, a dotted name or a call of one, so no decorator of theirs + // wraps a function in code that the graph cannot name. + missing []fileindex.FlagKind }{ {"flags.js", core.LanguageCodeJavascript, []fileindex.Flag{ {Kind: fileindex.FlagComputedImport, Line: 2}, @@ -34,9 +40,8 @@ func TestDynamicSites(t *testing.T) { {Kind: fileindex.FlagReflection, Line: 14}, {Kind: fileindex.FlagComputedAttribute, Line: 15}, {Kind: fileindex.FlagMonkeyPatch, Line: 16}, - {Kind: fileindex.FlagDecoratorWrap, Line: 21}, {Kind: fileindex.FlagParseError, Line: 25}, - }}, + }, []fileindex.FlagKind{fileindex.FlagDecoratorWrap}}, {"flags.ts", core.LanguageCodeTypescript, []fileindex.Flag{ {Kind: fileindex.FlagComputedImport, Line: 2}, {Kind: fileindex.FlagComputedImport, Line: 6}, @@ -46,10 +51,8 @@ func TestDynamicSites(t *testing.T) { {Kind: fileindex.FlagComputedAttribute, Line: 13}, {Kind: fileindex.FlagMonkeyPatch, Line: 14}, {Kind: fileindex.FlagMonkeyPatch, Line: 15}, - {Kind: fileindex.FlagDecoratorWrap, Line: 19}, - {Kind: fileindex.FlagDecoratorWrap, Line: 21}, {Kind: fileindex.FlagParseError, Line: 25}, - }}, + }, []fileindex.FlagKind{fileindex.FlagDecoratorWrap}}, {"flags.py", core.LanguageCodePython, []fileindex.Flag{ {Kind: fileindex.FlagComputedImport, Line: 6}, {Kind: fileindex.FlagComputedImport, Line: 8}, @@ -58,10 +61,10 @@ func TestDynamicSites(t *testing.T) { {Kind: fileindex.FlagReflection, Line: 15}, {Kind: fileindex.FlagComputedAttribute, Line: 17}, {Kind: fileindex.FlagMonkeyPatch, Line: 18}, - {Kind: fileindex.FlagDecoratorWrap, Line: 21}, - {Kind: fileindex.FlagModuleGetattr, Line: 31}, - {Kind: fileindex.FlagParseError, Line: 35}, - }}, + {Kind: fileindex.FlagDecoratorWrap, Line: 31}, + {Kind: fileindex.FlagModuleGetattr, Line: 37}, + {Kind: fileindex.FlagParseError, Line: 41}, + }, nil}, } for _, tc := range cases { t.Run(tc.file, func(t *testing.T) { @@ -74,6 +77,9 @@ func TestDynamicSites(t *testing.T) { kinds[flag.Kind] = true } for _, kind := range flagKinds { + if slices.Contains(tc.missing, kind) { + continue + } assert.True(t, kinds[kind], "no fixture line sets %s", kind) } }) @@ -89,7 +95,8 @@ func TestFlagLandsOnTheEnclosingDefinition(t *testing.T) { flags[def.Key] = def.Flags } assert.Contains(t, flags["flags.py//run"], fileindex.Flag{Kind: fileindex.FlagEval, Line: 13}) - assert.Equal(t, []fileindex.Flag{{Kind: fileindex.FlagDecoratorWrap, Line: 21}}, flags["flags.py//index"]) + assert.Equal(t, []fileindex.Flag{{Kind: fileindex.FlagDecoratorWrap, Line: 31}}, flags["flags.py//picked"]) + assert.Empty(t, flags["flags.py//index"], "a decorator that names its code, as @app.route, is no dynamic feature") assert.Empty(t, flags["flags.py//plain"]) assert.Contains(t, ix.Flags, fileindex.Flag{Kind: fileindex.FlagEval, Line: 13}) diff --git a/plugin/indexer/indexer.go b/plugin/indexer/indexer.go index 2c8e4a7e..b4b4dd98 100644 --- a/plugin/indexer/indexer.go +++ b/plugin/indexer/indexer.go @@ -15,6 +15,7 @@ import ( "github.com/safedep/code/fileindex" "github.com/safedep/code/fileindex/store" "github.com/safedep/code/fs" + "github.com/safedep/code/langcode" "github.com/safedep/code/parser" "github.com/safedep/code/plugin" ) @@ -77,6 +78,12 @@ type Result struct { // two sets stay apart. Dependencies map[string]fileindex.FileStatus + // Languages holds the language of each file that has a status, by + // name, as projectgraph.Options.Languages takes them. The link step + // reads it for a file with no index, so it knows the language of a + // failed or a missing file. + Languages map[string]langcode.Code + // Report lists the files that the run parsed, skipped and failed. Report *plugin.Report @@ -203,6 +210,7 @@ func (ix *Indexer) Run(ctx context.Context, fsys core.ImportAwareFileSystem) (*R result: &Result{ Files: make(map[string]fileindex.FileStatus, len(files)), Dependencies: map[string]fileindex.FileStatus{}, + Languages: make(map[string]langcode.Code, len(files)), Report: &plugin.Report{Skipped: skipped}, }, strs: interner{}, @@ -273,6 +281,7 @@ func (r *run) index(ctx context.Context, files []core.File, byExtension map[stri r.exclude(file) continue } + r.result.Languages[file.Name()] = language.Meta().Code data, err := read(file, r.opts.MaxFileBytes) if err != nil { r.fail(ctx, file, core.StageRead, err) diff --git a/plugin/indexer/testdata/flags.py b/plugin/indexer/testdata/flags.py index 57d98898..1913fd86 100644 --- a/plugin/indexer/testdata/flags.py +++ b/plugin/indexer/testdata/flags.py @@ -28,6 +28,11 @@ def plain(): pass +@handlers[attr] +def picked(): + pass + + def __getattr__(name): return load diff --git a/plugin/indexer/testdata/golden/App.cs.json b/plugin/indexer/testdata/golden/App.cs.json index 7a21f6ce..aded0e45 100644 --- a/plugin/indexer/testdata/golden/App.cs.json +++ b/plugin/indexer/testdata/golden/App.cs.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "App.cs", "language": "csharp", "content_hash": "8991919e54610418b58a53fb6b26dc563c5ef5af684165b60e8af5b6e7e8773c", diff --git a/plugin/indexer/testdata/golden/App.java.json b/plugin/indexer/testdata/golden/App.java.json index 0ea48aa0..1d3d79b5 100644 --- a/plugin/indexer/testdata/golden/App.java.json +++ b/plugin/indexer/testdata/golden/App.java.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "App.java", "language": "java", "content_hash": "4b93f5c2c4ed8e1881cb857a9b7cd65c3a0d4cb4cd0929140539d0a418a27a95", diff --git a/plugin/indexer/testdata/golden/App.kt.json b/plugin/indexer/testdata/golden/App.kt.json index 220cf808..7c8f04a9 100644 --- a/plugin/indexer/testdata/golden/App.kt.json +++ b/plugin/indexer/testdata/golden/App.kt.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "App.kt", "language": "kotlin", "content_hash": "458147a6c6ebc2561187207bb3bca7e091914bfd2e52f0540f58887976983462", diff --git a/plugin/indexer/testdata/golden/app.go.json b/plugin/indexer/testdata/golden/app.go.json index 1ed55cf9..e8cbe556 100644 --- a/plugin/indexer/testdata/golden/app.go.json +++ b/plugin/indexer/testdata/golden/app.go.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.go", "language": "go", "content_hash": "3d65f36ed0edfd5e009c06d07d3d038337e9faebaca6c9762b5c473a14826166", diff --git a/plugin/indexer/testdata/golden/app.js.json b/plugin/indexer/testdata/golden/app.js.json index d35c1491..e96db400 100644 --- a/plugin/indexer/testdata/golden/app.js.json +++ b/plugin/indexer/testdata/golden/app.js.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.js", "language": "javascript", "content_hash": "247fdd512217fb5e7ce19cbf3f60b327e145abc15a26ee510e167cbc651b0a54", diff --git a/plugin/indexer/testdata/golden/app.php.json b/plugin/indexer/testdata/golden/app.php.json index 22c23723..5f5c421a 100644 --- a/plugin/indexer/testdata/golden/app.php.json +++ b/plugin/indexer/testdata/golden/app.php.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.php", "language": "php", "content_hash": "b57d8d52aafac8d7cbe2d19e5889e4e7b851ee71a48ea966cdd9cbfd14214466", diff --git a/plugin/indexer/testdata/golden/app.py.json b/plugin/indexer/testdata/golden/app.py.json index 67e1a583..06de79d4 100644 --- a/plugin/indexer/testdata/golden/app.py.json +++ b/plugin/indexer/testdata/golden/app.py.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.py", "language": "python", "content_hash": "a9cdb5fb012d2a29970c07deda9cc8360983015e2360cb9091bca7bfce5c801d", diff --git a/plugin/indexer/testdata/golden/app.rb.json b/plugin/indexer/testdata/golden/app.rb.json index ed1ac686..0a04df74 100644 --- a/plugin/indexer/testdata/golden/app.rb.json +++ b/plugin/indexer/testdata/golden/app.rb.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.rb", "language": "ruby", "content_hash": "f3a3e601660e0130750119df4267cac37972c412ccc8f5d8f16e3ba516f659d9", diff --git a/plugin/indexer/testdata/golden/app.rs.json b/plugin/indexer/testdata/golden/app.rs.json index a57077f2..0960c899 100644 --- a/plugin/indexer/testdata/golden/app.rs.json +++ b/plugin/indexer/testdata/golden/app.rs.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.rs", "language": "rust", "content_hash": "f07a751f9610f47e7b49679223d4ca80f85ecb21abde61488935d2f21a5c3603", diff --git a/plugin/indexer/testdata/golden/app.ts.json b/plugin/indexer/testdata/golden/app.ts.json index 9e409080..159d8555 100644 --- a/plugin/indexer/testdata/golden/app.ts.json +++ b/plugin/indexer/testdata/golden/app.ts.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "app.ts", "language": "typescript", "content_hash": "d3908948ab8ec7b3adc898e0707ce74bb3a2d6393a724dfbeb67147840397234", diff --git a/plugin/indexer/testdata/golden/exports.js.json b/plugin/indexer/testdata/golden/exports.js.json index 5e68f690..0d17a1b1 100644 --- a/plugin/indexer/testdata/golden/exports.js.json +++ b/plugin/indexer/testdata/golden/exports.js.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "exports.js", "language": "javascript", "content_hash": "765716b5977b05d7bbc4ef5e0192e196e0209400366036053df632d54fdb0f95", diff --git a/plugin/indexer/testdata/golden/exports.py.json b/plugin/indexer/testdata/golden/exports.py.json index 0acadd05..6321a428 100644 --- a/plugin/indexer/testdata/golden/exports.py.json +++ b/plugin/indexer/testdata/golden/exports.py.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "exports.py", "language": "python", "content_hash": "653115ddc22536763049e9209d4942bca5c6817b98bc01722879b69c04d8605b", diff --git a/plugin/indexer/testdata/golden/exports.ts.json b/plugin/indexer/testdata/golden/exports.ts.json index 78e7f72a..e4cd7a74 100644 --- a/plugin/indexer/testdata/golden/exports.ts.json +++ b/plugin/indexer/testdata/golden/exports.ts.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "exports.ts", "language": "typescript", "content_hash": "0edcb9324080b924f3ac45d57f5b5cbe90d625038a0dd7f99fa9e5fcbeee1edc", diff --git a/plugin/indexer/testdata/golden/flags.js.json b/plugin/indexer/testdata/golden/flags.js.json index 78abeb49..bd504c2f 100644 --- a/plugin/indexer/testdata/golden/flags.js.json +++ b/plugin/indexer/testdata/golden/flags.js.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "flags.js", "language": "javascript", "content_hash": "70731d0448bc5af9ecf5c81156eefbd39535a3fc674fffc250937d76c2ba4610", @@ -155,12 +155,6 @@ "parent": "flags.js//View", "decorators": [ "@observe" - ], - "flags": [ - { - "kind": "decorator-wrap", - "line": 21 - } ] } ], @@ -319,10 +313,6 @@ "kind": "monkey-patch", "line": 16 }, - { - "kind": "decorator-wrap", - "line": 21 - }, { "kind": "parse-error", "line": 25 diff --git a/plugin/indexer/testdata/golden/flags.py.json b/plugin/indexer/testdata/golden/flags.py.json index 3175a29a..9031c0c6 100644 --- a/plugin/indexer/testdata/golden/flags.py.json +++ b/plugin/indexer/testdata/golden/flags.py.json @@ -1,8 +1,8 @@ { - "version": "14", + "version": "15", "path": "flags.py", "language": "python", - "content_hash": "d3169e0c3c00508c3f9492866cf4db99ddd780d42dc9e4438558d1d4a22f2621", + "content_hash": "9d58772a61d2cb32039a7e81164d1cb50c3c63a536c2676c91e7a36fad954132", "imports": [ { "line": 1, @@ -43,12 +43,12 @@ "kind": "module", "name": "flags", "start_line": 1, - "end_line": 37, + "end_line": 42, "exported": true, "flags": [ { "kind": "module-getattr", - "line": 31 + "line": 36 } ] }, @@ -109,12 +109,6 @@ "exported": true, "decorators": [ "@app.route" - ], - "flags": [ - { - "kind": "decorator-wrap", - "line": 21 - } ] }, { @@ -128,25 +122,42 @@ "@staticmethod" ] }, + { + "key": "flags.py//picked", + "kind": "function", + "name": "picked", + "start_line": 32, + "end_line": 33, + "exported": true, + "decorators": [ + "@handlers[attr]" + ], + "flags": [ + { + "kind": "decorator-wrap", + "line": 31 + } + ] + }, { "key": "flags.py//__getattr__", "kind": "function", "name": "__getattr__", - "start_line": 31, - "end_line": 32, + "start_line": 36, + "end_line": 37, "exported": true }, { "key": "flags.py//broken", "kind": "function", "name": "broken", - "start_line": 35, - "end_line": 36, + "start_line": 40, + "end_line": 41, "exported": true, "flags": [ { "kind": "parse-error", - "line": 35 + "line": 40 } ] } @@ -294,7 +305,7 @@ "references": [ { "caller": "flags.py//__getattr__", - "line": 32, + "line": 37, "column": 12, "text": "load", "targets": [ @@ -376,15 +387,15 @@ }, { "kind": "decorator-wrap", - "line": 21 + "line": 31 }, { "kind": "module-getattr", - "line": 31 + "line": 36 }, { "kind": "parse-error", - "line": 35 + "line": 40 } ] } diff --git a/plugin/indexer/testdata/golden/flags.ts.json b/plugin/indexer/testdata/golden/flags.ts.json index 74721a1c..1adb53a4 100644 --- a/plugin/indexer/testdata/golden/flags.ts.json +++ b/plugin/indexer/testdata/golden/flags.ts.json @@ -1,5 +1,5 @@ { - "version": "14", + "version": "15", "path": "flags.ts", "language": "typescript", "content_hash": "a84c4ad21735c1c250e33317463237749489ea1f220a93f97e2bea3f07b040c4", @@ -130,13 +130,7 @@ "name": "View", "start_line": 18, "end_line": 23, - "exported": false, - "flags": [ - { - "kind": "decorator-wrap", - "line": 19 - } - ] + "exported": false }, { "key": "flags.ts//View//render", @@ -148,12 +142,6 @@ "parent": "flags.ts//View", "decorators": [ "@observe" - ], - "flags": [ - { - "kind": "decorator-wrap", - "line": 21 - } ] } ], @@ -306,14 +294,6 @@ "kind": "monkey-patch", "line": 15 }, - { - "kind": "decorator-wrap", - "line": 19 - }, - { - "kind": "decorator-wrap", - "line": 21 - }, { "kind": "parse-error", "line": 25 diff --git a/projectgraph/definitions.go b/projectgraph/definitions.go index 80037477..807c68c3 100644 --- a/projectgraph/definitions.go +++ b/projectgraph/definitions.go @@ -15,7 +15,7 @@ func (b *builder) addDefinitions(ix *fileindex.Index) { defFlags[f] = true } } - module := &Node{Key: ix.Path, Kind: NodeModule, Path: ix.Path, Line: 1, Exported: true, Test: b.test[ix.Path]} + module := &Node{Key: ix.Path, Kind: NodeModule, Path: ix.Path, Line: 1, Exported: true, Test: b.test[ix.Path], Language: ix.Language} for _, f := range ix.Flags { if !defFlags[f] { module.Flags = append(module.Flags, f) diff --git a/projectgraph/graph.go b/projectgraph/graph.go index 09f7bd51..b54702e8 100644 --- a/projectgraph/graph.go +++ b/projectgraph/graph.go @@ -11,13 +11,14 @@ import ( "github.com/safedep/code/fileindex" "github.com/safedep/code/fileindex/roots" + "github.com/safedep/code/langcode" "github.com/safedep/code/projectgraph/modules" ) // Version is the version of the link rules. Bump it when a rule that makes a // node, an edge, a root or an unresolved site changes, so a diff of two // reports shows the change. -const Version = "16" +const Version = "17" type NodeKind string @@ -166,7 +167,9 @@ func (s Site) String() string { // external target. An external node has only its key and kind. External // is the key that an importer of the package writes for a function, a // method or a constructor of a unit of dependency code, as -// golang.org//x//net//html//render1. +// golang.org//x//net//html//render1. Language is the language of the file +// of a module top level, also of a file with no index, when the index step +// told it. type Node struct { Key string `json:"key"` Kind NodeKind `json:"kind"` @@ -174,6 +177,7 @@ type Node struct { Line int `json:"line,omitempty"` Exported bool `json:"exported,omitempty"` Test bool `json:"test,omitempty"` + Language langcode.Code `json:"language,omitempty"` Flags []fileindex.Flag `json:"flags,omitempty"` External string `json:"external,omitempty"` } @@ -230,14 +234,18 @@ type Truncation struct { // Stats holds the counts of a graph. Duration is not in the JSON form, so two // builds of one input give one JSON form. type Stats struct { - Version string `json:"version"` - Files int `json:"files"` - Nodes int `json:"nodes"` - Edges int `json:"edges"` - Unresolved int `json:"unresolved"` - Roots int `json:"roots"` - RootFallback bool `json:"root_fallback,omitempty"` - Duration time.Duration `json:"-"` + Version string `json:"version"` + Files int `json:"files"` + Nodes int `json:"nodes"` + Edges int `json:"edges"` + Unresolved int `json:"unresolved"` + Roots int `json:"roots"` + RootFallback bool `json:"root_fallback,omitempty"` + // Languages are the languages of the files that hold a root other than + // a test root, in a fixed order: the languages that the program runs. + // Runs reads it. + Languages []langcode.Code `json:"languages,omitempty"` + Duration time.Duration `json:"-"` } // Graph is the call graph of one project. Edges, Unresolved, Roots and @@ -258,6 +266,52 @@ type Graph struct { edges *edgeSet } +// Runs reports whether the program of the graph runs a file, by the +// language of the file. A file of the project runs when a root of the +// graph is in a file of its language. A file of a unit of dependency code +// runs when the ecosystem of the unit installs code of its language, as +// pypi installs Python. A file with no known language runs, and so does +// every file of a graph with no root. A truncated or a failed file that +// does not run, as the JavaScript of a web page that a Python project +// ships, holds no caller that a search can reach, so its budget concerns +// no target. +func (g *Graph) Runs(path string) bool { + n := g.Nodes[path] + if n == nil || n.Language == "" { + return true + } + if unit := fileindex.PackageOf(path); unit != "" { + langs := unitLanguages(unit) + return langs == nil || slices.Contains(langs, n.Language) + } + return len(g.Stats.Languages) == 0 || slices.Contains(g.Stats.Languages, n.Language) +} + +// unitLanguages returns the languages of the code that the ecosystem of a +// unit installs, or nil for an ecosystem that the link step does not know. +func unitLanguages(unit string) []langcode.Code { + pkg, err := fileindex.ParseUnit(unit) + if err != nil { + return nil + } + return ecosystemLanguages[pkg.Ecosystem] +} + +// ecosystemLanguages are the languages of the code that a package of an +// ecosystem installs. A package can ship files of other languages, as the +// JavaScript of the admin pages of Django, and no program of the ecosystem +// runs them. +var ecosystemLanguages = map[string][]langcode.Code{ + "pypi": {langcode.Python}, + "npm": {langcode.Javascript, langcode.Typescript}, + "go": {langcode.Go}, + "cargo": {langcode.Rust}, + "maven": {langcode.Java, langcode.Kotlin}, + "nuget": {langcode.CSharp}, + "packagist": {langcode.PHP}, + "rubygems": {langcode.Ruby}, +} + // IsExternal reports whether the key names a node outside the project. func (g *Graph) IsExternal(key string) bool { if n, ok := g.Nodes[key]; ok { diff --git a/projectgraph/link.go b/projectgraph/link.go index 4f511f41..048b02e0 100644 --- a/projectgraph/link.go +++ b/projectgraph/link.go @@ -32,6 +32,13 @@ type Options struct { // with no index and a status that is not complete, as missing or // failed, becomes a module node with that status. Files map[string]fileindex.FileStatus + // Languages is the language of each file of the index step, by path, + // as indexer.Result.Languages gives it. A file with no index gets its + // language from here. The reach query leaves out the budget of a file + // of a language that the program does not run, as the JavaScript of a + // web page in a Python project. A nil map leaves the language of such + // a file unknown, and its budget counts. + Languages map[string]langcode.Code // TestPaths marks the test files. A glob with no "/" matches one part // of the path, as tests or test_*.py. TestPaths []string @@ -326,7 +333,7 @@ func (b *builder) linkAll(ctx context.Context) error { ix := b.byPath[p] if ix == nil { b.g.Files[p] = b.opts.Files[p] - b.addNode(&Node{Key: p, Kind: NodeModule, Path: p, Test: b.test[p]}) + b.addNode(&Node{Key: p, Kind: NodeModule, Path: p, Test: b.test[p], Language: b.opts.Languages[p]}) continue } b.g.Files[p] = b.status(ix) @@ -460,6 +467,7 @@ func (b *builder) finish() *Graph { } g.Problems = b.problems g.Stats.Version = Version + g.Stats.Languages = b.rootLanguages() g.Stats.Files, g.Stats.Nodes, g.Stats.Edges = len(g.Files), len(g.Nodes), len(g.Edges) g.Stats.Unresolved, g.Stats.Roots = len(g.Unresolved), len(g.Roots) return g diff --git a/projectgraph/modules/python_test.go b/projectgraph/modules/python_test.go index f925ccb9..17a46871 100644 --- a/projectgraph/modules/python_test.go +++ b/projectgraph/modules/python_test.go @@ -141,3 +141,27 @@ func TestPythonRelativePath(t *testing.T) { assert.Equal(t, want, PythonRelativePath(module), module) } } + +func TestPythonFirstPartyFrom(t *testing.T) { + p := NewPython(NewFiles([]string{"main.py", "conf/__init__.py", "app/models/types.py", "scripts/run.py", "scripts/helper.py"}), nil, nil) + cases := []struct { + name string + importer string + spec string + want bool + }{ + {name: "a relative module", importer: "main.py", spec: ".missing", want: true}, + {name: "a missing module of a package at a root", importer: "main.py", spec: "conf.loader", want: true}, + {name: "a module of the directory of the importer", importer: "scripts/run.py", spec: "helper.missing", want: true}, + {name: "a file of the name deep in the project", importer: "main.py", spec: "types", want: false}, + {name: "a module of another directory", importer: "main.py", spec: "helper", want: false}, + {name: "a dependency", importer: "main.py", spec: "yaml.loader", want: false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + assert.Equal(t, tc.want, p.FirstPartyFrom(tc.importer, tc.spec)) + }) + } + assert.False(t, p.FirstParty("helper"), "with no importer, no directory of a script is a root") + assert.True(t, p.FirstParty("conf.loader")) +} diff --git a/projectgraph/modules/resolver.go b/projectgraph/modules/resolver.go index 8d3a850e..97893a1b 100644 --- a/projectgraph/modules/resolver.go +++ b/projectgraph/modules/resolver.go @@ -14,6 +14,16 @@ type Resolver interface { FirstParty(spec string) bool } +// ImporterResolver is a Resolver whose first-party answer depends on the +// importer, as the directory of a Python file is on the path of its +// imports. +type ImporterResolver interface { + Resolver + // FirstPartyFrom reports whether a spec that names no module can still + // name a file of the project when the importer imports it. + FirstPartyFrom(importer, spec string) bool +} + // FormResolver is a Resolver whose answer depends on the form of the // import, as the exports map of a package.json does. type FormResolver interface { @@ -37,11 +47,26 @@ func (p *Python) Module(importer, spec string) (string, string, bool) { } // FirstParty reports whether a Python module is relative or starts with -// the name of a file or a directory of the project, or with the name of -// the package that the file set is. +// the name of a module under an import root, or with the name of the +// package that the file set is. It reads no importer, so the directory of +// the importer is no root here. func (p *Python) FirstParty(spec string) bool { + return p.FirstPartyFrom("", spec) +} + +// FirstPartyFrom reports whether a Python module is relative or starts +// with the name of a module or a package under an import root or in the +// directory of the importer, or with the name of the package that the +// file set is. A file of the name deep in the project, as a types.py in a +// package, is no module of an absolute import, so import types names the +// standard library and not the file. +func (p *Python) FirstPartyFrom(importer, spec string) bool { first, _, _ := strings.Cut(spec, ".") - return strings.HasPrefix(spec, ".") || p.files.HasName(first) || p.pkg != "" && first == p.pkg + if strings.HasPrefix(spec, ".") || p.pkg != "" && first == p.pkg { + return true + } + _, ok := p.Absolute(importer, first) + return ok } // For returns the resolver of the JavaScript or the TypeScript importers. diff --git a/projectgraph/packages.go b/projectgraph/packages.go index a84e88d4..b8675296 100644 --- a/projectgraph/packages.go +++ b/projectgraph/packages.go @@ -90,6 +90,14 @@ func (k spaceResolver) FirstParty(spec string) bool { return k.r.FirstParty(spec) } +// FirstPartyFrom reads the importer when the resolver of the paths does. +func (k spaceResolver) FirstPartyFrom(importer, spec string) bool { + if ir, ok := k.r.(modules.ImporterResolver); ok { + return ir.FirstPartyFrom(k.sp.inner(importer), spec) + } + return k.r.FirstParty(spec) +} + // addSpaces makes the space of the project and the space of each unit that // has a file. A pypi unit with an __init__.py at its root is its package // directory, and its root is the package of the name of the unit. diff --git a/projectgraph/reach/reach.go b/projectgraph/reach/reach.go index 873e77b4..54bc31ca 100644 --- a/projectgraph/reach/reach.go +++ b/projectgraph/reach/reach.go @@ -23,7 +23,7 @@ import ( // change. Version 2 added the root kind to the first hop of a path. // Version 3 matches a target against the external key of a definition of a // unit of dependency code. -const Version = "3" +const Version = "4" const ( defaultMaxDepth = 64 diff --git a/projectgraph/reach/reach_test.go b/projectgraph/reach/reach_test.go index 30b75d28..9b961022 100644 --- a/projectgraph/reach/reach_test.go +++ b/projectgraph/reach/reach_test.go @@ -93,6 +93,16 @@ func graphOf(t *testing.T, spec string) *projectgraph.Graph { n.Flags = append(n.Flags, fileindex.Flag{Kind: fileindex.FlagKind(f[2]), Line: l}) case "file": g.Files[f[1]] = fileindex.FileStatus(f[2]) + if _, ok := g.Nodes[f[1]]; !ok { + g.Nodes[f[1]] = &projectgraph.Node{Key: f[1], Kind: projectgraph.NodeModule, Path: f[1], Test: strings.HasPrefix(f[1], "tests/")} + } + if len(f) > 3 { + g.Nodes[f[1]].Language = langcode.Code(f[3]) + } + case "program": + for _, code := range f[1:] { + g.Stats.Languages = append(g.Stats.Languages, langcode.Code(code)) + } case "truncated": g.Truncated = append(g.Truncated, projectgraph.Truncation{Kind: projectgraph.TruncationKind(f[1]), Key: arg(f, 2, "")}) case "fallback": @@ -240,6 +250,69 @@ func TestReachUncertainties(t *testing.T) { } } +// TestReachFileBudgets checks which incomplete files count for a search: a +// file of a language that the program does not run holds no caller that a +// search can reach, a test file counts only with the tests, and a file of +// an unknown language counts. +func TestReachFileBudgets(t *testing.T) { + cases := []struct { + name string + spec string + includeTests bool + want []UncertaintyKind + }{ + { + name: "a failed file of a language that the program runs", + spec: "root main.py//main\nfile server.py failed python\nprogram python", + want: []UncertaintyKind{Budget}, + }, + { + name: "a failed file of a language that the program does not run", + spec: "root main.py//main\nfile static/vendor.js failed javascript\nprogram python", + }, + { + name: "a failed file of an unknown language", + spec: "root main.py//main\nfile static/vendor.js failed\nprogram python", + want: []UncertaintyKind{Budget}, + }, + { + name: "a failed file of a graph with no root language", + spec: "root main.py//main\nfile static/vendor.js failed javascript", + want: []UncertaintyKind{Budget}, + }, + { + name: "a failed test file without the tests", + spec: "root main.py//main\nfile tests/test_big.py failed python\nprogram python", + }, + { + name: "a failed test file with the tests", + spec: "root main.py//main\nfile tests/test_big.py failed python\nprogram python", + includeTests: true, + want: []UncertaintyKind{Budget}, + }, + { + name: "a failed file of a unit in a language that its ecosystem installs", + spec: "root main.py//main\nedge main.py//main pypi:dep@1.0.0//dep/a.py//f\nfile pypi:dep@1.0.0//dep/b.py failed python", + want: []UncertaintyKind{Budget}, + }, + { + name: "a failed file of a unit in another language", + spec: "root main.py//main\nedge main.py//main pypi:dep@1.0.0//dep/a.py//f\nfile pypi:dep@1.0.0//dep/static/app.js failed javascript", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + g := graphOf(t, tc.spec) + r := reachOne(t, g, Query{Targets: targetsOf(t, "os//system"), IncludeTests: tc.includeTests}) + if len(tc.want) == 0 { + assert.Empty(t, r.Uncertainties) + return + } + assert.Equal(t, tc.want, kinds(r.Uncertainties)) + }) + } +} + func TestReachMethodClosureSkipsTests(t *testing.T) { g := graphOf(t, basic+` root tests/test_app.py test diff --git a/projectgraph/reach/uncertainty.go b/projectgraph/reach/uncertainty.go index 64b7f9c3..e21e2076 100644 --- a/projectgraph/reach/uncertainty.go +++ b/projectgraph/reach/uncertainty.go @@ -47,14 +47,18 @@ type sources struct { // that no import names. They count for every search. global []Uncertainty // budgets holds the file budgets of each unit of dependency code. A - // file of the project is a global budget. - budgets map[string][]Uncertainty + // file of the project is a global budget. testGlobal and testBudgets + // hold the same for the test files, which count only for a search + // that includes the tests. + budgets map[string][]Uncertainty + testGlobal []Uncertainty + testBudgets map[string][]Uncertainty // units lists the units that a search can enter, in key order. units []string } func newSources(g *projectgraph.Graph) *sources { - s := &sources{unresolved: map[string][]projectgraph.UnresolvedSite{}, prefixes: map[projectgraph.Site]string{}, truncated: map[string][]projectgraph.Truncation{}, budgets: map[string][]Uncertainty{}} + s := &sources{unresolved: map[string][]projectgraph.UnresolvedSite{}, prefixes: map[projectgraph.Site]string{}, truncated: map[string][]projectgraph.Truncation{}, budgets: map[string][]Uncertainty{}, testBudgets: map[string][]Uncertainty{}} for _, u := range g.Unresolved { s.unresolved[u.Caller] = append(s.unresolved[u.Caller], u) if u.ImportPrefix != "" { @@ -77,16 +81,26 @@ func newSources(g *projectgraph.Graph) *sources { // does not know, so it counts where the search does not reach it. A // file of dependency code holds no root, and its code runs only when a // call enters its package. So it counts only for a search that - // reaches a node of its unit. + // reaches a node of its unit. A file that the program does not run, as + // the JavaScript of a web page in a Python project, holds no such + // caller. A test file holds only test roots, so it counts only for a + // search that includes the tests. for _, p := range slices.Sorted(maps.Keys(g.Files)) { status := g.Files[p] - if status.Complete() { + if status.Complete() || !g.Runs(p) { continue } u := Uncertainty{Kind: Budget, Key: p, Site: projectgraph.Site{Path: p}, Text: "the file is " + string(status)} - if unit := fileindex.PackageOf(p); unit != "" { + test := g.Nodes[p] != nil && g.Nodes[p].Test + unit := fileindex.PackageOf(p) + switch { + case unit != "" && test: + s.testBudgets[unit] = append(s.testBudgets[unit], u) + case unit != "": s.budgets[unit] = append(s.budgets[unit], u) - } else { + case test: + s.testGlobal = append(s.testGlobal, u) + default: s.global = append(s.global, u) } } @@ -173,6 +187,15 @@ func (a *adjacency) uncertainties(s *search, maxDepth int) *found { for _, u := range src.global { f.add(u) } + if a.includeTests { + for _, u := range src.testGlobal { + f.add(u) + } + f.units = maps.Clone(src.budgets) + for unit, us := range src.testBudgets { + f.units[unit] = append(f.units[unit], us...) + } + } for _, id := range s.order { key := a.keys[id] n := g.Nodes[key] diff --git a/projectgraph/resolve.go b/projectgraph/resolve.go index fd7e7b6f..d0725b4b 100644 --- a/projectgraph/resolve.go +++ b/projectgraph/resolve.go @@ -674,12 +674,16 @@ func (b *builder) resolveModules(ix *fileindex.Index, spec string, form modules. // mayBeFirstParty reports whether a module that resolves to no file can be // a file of the project or the unit of the file. A search path root, an // alias or a build step can resolve a relative module, a module whose first -// segment names a file or a directory of the project, or any module of a -// language with no resolver. A Go import path under the path of a module of -// the project, or a Java name under a package of the project, names a file -// of the project. +// segment names a module of the project, or any module of a language with +// no resolver. A Go import path under the path of a module of the project, +// or a Java name under a package of the project, names a file of the +// project. A resolver that reads the importer, as Python does for the +// directory of a script, gets it. func (b *builder) mayBeFirstParty(ix *fileindex.Index, spec string) bool { r := b.spaceOf(ix.Path).resolvers[ix.Language] + if ir, ok := r.(modules.ImporterResolver); ok { + return ir.FirstPartyFrom(ix.Path, spec) + } return r == nil || r.FirstParty(spec) } diff --git a/projectgraph/roots.go b/projectgraph/roots.go index 33483fad..c6aa0335 100644 --- a/projectgraph/roots.go +++ b/projectgraph/roots.go @@ -2,6 +2,7 @@ package projectgraph import ( "context" + "maps" "slices" "strings" @@ -169,3 +170,18 @@ func (b *builder) exportKeys(p string, seen map[string]bool) []string { slices.Sort(keys) return slices.Compact(keys) } + +// rootLanguages returns the languages of the files that hold a root other +// than a test root, in the order of the codes. A test root runs only a +// test, as the JavaScript test of a Python project, so it names no +// language of the program. A file with no index has the language that the +// index step told. With no root, the program runs every language. +func (b *builder) rootLanguages() []langcode.Code { + set := map[langcode.Code]bool{} + for _, r := range b.g.Roots { + if n := b.g.Nodes[fileindex.Unit(r.Key)]; r.Kind != RootTest && n != nil && n.Language != "" { + set[n.Language] = true + } + } + return slices.Sorted(maps.Keys(set)) +} diff --git a/projectgraph/summary/summary.go b/projectgraph/summary/summary.go index a17a1a31..392d74ea 100644 --- a/projectgraph/summary/summary.go +++ b/projectgraph/summary/summary.go @@ -111,8 +111,11 @@ func Build(ctx context.Context, indexes []*fileindex.Index, opts projectgraph.Op } s := &Summary{Version: Version, IndexVersion: fileindex.Version, LinkVersion: projectgraph.Version, Unit: unit, Symbols: []Symbol{}, Targets: []string{}, Problems: g.Problems} + // A file that the package does not run, as the JavaScript of a web + // page, and a test file of the package hold no code that a symbol + // reaches, so they are no budget of the summary. for _, p := range slices.Sorted(maps.Keys(g.Files)) { - if !g.Files[p].Complete() { + if n := g.Nodes[p]; !g.Files[p].Complete() && g.Runs(p) && (n == nil || !n.Test) { s.Incomplete = append(s.Incomplete, p) } } diff --git a/projectgraph/testdata/javascript/barrel-chain/graph.golden.json b/projectgraph/testdata/javascript/barrel-chain/graph.golden.json index 1a713c9f..b24dac40 100644 --- a/projectgraph/testdata/javascript/barrel-chain/graph.golden.json +++ b/projectgraph/testdata/javascript/barrel-chain/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "src/api/index.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/api/index.js//parse": { "key": "src/api/index.js//parse", @@ -19,7 +20,8 @@ "kind": "module", "path": "src/core/index.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/core/index.js//parse": { "key": "src/core/index.js//parse", @@ -33,7 +35,8 @@ "kind": "module", "path": "src/core/parse.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/core/parse.js//doParse": { "key": "src/core/parse.js//doParse", @@ -47,7 +50,8 @@ "kind": "module", "path": "src/main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" } }, "edges": [ @@ -174,12 +178,15 @@ "src/main.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 4, "nodes": 7, "edges": 6, "unresolved": 0, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/barrel-chain/indexes.json b/projectgraph/testdata/javascript/barrel-chain/indexes.json index d1cffad8..75c2b418 100644 --- a/projectgraph/testdata/javascript/barrel-chain/indexes.json +++ b/projectgraph/testdata/javascript/barrel-chain/indexes.json @@ -1,22 +1,22 @@ [ { - "version": "14", "path": "src/main.js", "language": "javascript", + "version": "15", "path": "src/main.js", "language": "javascript", "imports": [{"line": 1, "module": "./api", "item": "parse", "alias": "parse", "kind": "static", "namespace": ".//api//parse"}], "definitions": [{"key": "src/main.js", "kind": "module", "name": "main", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "src/main.js", "line": 2, "column": 1, "callee": "parse", "targets": [".//api//parse"], "kind": "call"}] }, { - "version": "14", "path": "src/api/index.js", "language": "javascript", + "version": "15", "path": "src/api/index.js", "language": "javascript", "exports": [{"name": "parse", "target": "", "from": "../core", "item": "parse", "line": 1}], "definitions": [{"key": "src/api/index.js", "kind": "module", "name": "index", "start_line": 1, "end_line": 1, "exported": true}] }, { - "version": "14", "path": "src/core/index.js", "language": "javascript", + "version": "15", "path": "src/core/index.js", "language": "javascript", "exports": [{"name": "parse", "target": "", "from": "./parse", "item": "doParse", "line": 1}], "definitions": [{"key": "src/core/index.js", "kind": "module", "name": "index", "start_line": 1, "end_line": 1, "exported": true}] }, { - "version": "14", "path": "src/core/parse.js", "language": "javascript", + "version": "15", "path": "src/core/parse.js", "language": "javascript", "exports": [{"name": "doParse", "target": "src/core/parse.js//doParse", "line": 1}], "definitions": [ {"key": "src/core/parse.js", "kind": "module", "name": "parse", "start_line": 1, "end_line": 3, "exported": true}, diff --git a/projectgraph/testdata/javascript/commonjs-require/graph.golden.json b/projectgraph/testdata/javascript/commonjs-require/graph.golden.json index b8f8c095..a245016f 100644 --- a/projectgraph/testdata/javascript/commonjs-require/graph.golden.json +++ b/projectgraph/testdata/javascript/commonjs-require/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "helper.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "helper.js//helper": { "key": "helper.js//helper", @@ -19,7 +20,8 @@ "kind": "module", "path": "lib/index.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/index.js//run": { "key": "lib/index.js//run", @@ -33,7 +35,8 @@ "kind": "module", "path": "main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" } }, "edges": [ @@ -138,12 +141,15 @@ "main.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 5, "edges": 4, "unresolved": 0, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/commonjs-require/indexes.json b/projectgraph/testdata/javascript/commonjs-require/indexes.json index cda79406..e4ec9795 100644 --- a/projectgraph/testdata/javascript/commonjs-require/indexes.json +++ b/projectgraph/testdata/javascript/commonjs-require/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.js", "language": "javascript", + "version": "15", "path": "main.js", "language": "javascript", "imports": [ {"line": 1, "module": "./lib", "alias": "utils", "kind": "static", "namespace": ".//lib"}, {"line": 2, "module": "./helper", "item": "helper", "alias": "helper", "kind": "static", "namespace": ".//helper//helper"} @@ -12,7 +12,7 @@ ] }, { - "version": "14", "path": "lib/index.js", "language": "javascript", + "version": "15", "path": "lib/index.js", "language": "javascript", "exports": [{"name": "run", "target": "lib/index.js//run", "line": 3}], "definitions": [ {"key": "lib/index.js", "kind": "module", "name": "index", "start_line": 1, "end_line": 3, "exported": true}, @@ -20,7 +20,7 @@ ] }, { - "version": "14", "path": "helper.js", "language": "javascript", + "version": "15", "path": "helper.js", "language": "javascript", "exports": [{"name": "helper", "target": "helper.js//helper", "line": 3}], "definitions": [ {"key": "helper.js", "kind": "module", "name": "helper", "start_line": 1, "end_line": 3, "exported": true}, diff --git a/projectgraph/testdata/javascript/default-import/graph.golden.json b/projectgraph/testdata/javascript/default-import/graph.golden.json index bc1b6927..a1b1f7ba 100644 --- a/projectgraph/testdata/javascript/default-import/graph.golden.json +++ b/projectgraph/testdata/javascript/default-import/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "log.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "log.js//log": { "key": "log.js//log", @@ -19,14 +20,16 @@ "kind": "module", "path": "main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "make.js": { "key": "make.js", "kind": "module", "path": "make.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "make.js//make": { "key": "make.js//make", @@ -138,12 +141,15 @@ "make.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 5, "edges": 4, "unresolved": 0, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/default-import/indexes.json b/projectgraph/testdata/javascript/default-import/indexes.json index fcd832a1..4a1637a8 100644 --- a/projectgraph/testdata/javascript/default-import/indexes.json +++ b/projectgraph/testdata/javascript/default-import/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.js", "language": "javascript", + "version": "15", "path": "main.js", "language": "javascript", "imports": [ {"line": 1, "module": "./log", "alias": "log", "kind": "static", "namespace": ".//log"}, {"line": 2, "module": "./make", "alias": "make", "kind": "static", "namespace": ".//make"} @@ -12,7 +12,7 @@ ] }, { - "version": "14", "path": "log.js", "language": "javascript", + "version": "15", "path": "log.js", "language": "javascript", "exports": [{"name": "default", "target": "log.js//log", "line": 1}], "definitions": [ {"key": "log.js", "kind": "module", "name": "log", "start_line": 1, "end_line": 1, "exported": true}, @@ -20,7 +20,7 @@ ] }, { - "version": "14", "path": "make.js", "language": "javascript", + "version": "15", "path": "make.js", "language": "javascript", "exports": [{"name": "default", "target": "make.js//make", "line": 1}], "definitions": [ {"key": "make.js", "kind": "module", "name": "make", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/javascript/esm-named-import/graph.golden.json b/projectgraph/testdata/javascript/esm-named-import/graph.golden.json index 0595798c..69342230 100644 --- a/projectgraph/testdata/javascript/esm-named-import/graph.golden.json +++ b/projectgraph/testdata/javascript/esm-named-import/graph.golden.json @@ -9,14 +9,16 @@ "kind": "module", "path": "src/main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/parser.js": { "key": "src/parser.js", "kind": "module", "path": "src/parser.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/parser.js//parse": { "key": "src/parser.js//parse", @@ -30,7 +32,8 @@ "kind": "module", "path": "src/util.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/util.js//fmt": { "key": "src/util.js//fmt", @@ -166,12 +169,15 @@ "src/util.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 6, "edges": 5, "unresolved": 1, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/esm-named-import/indexes.json b/projectgraph/testdata/javascript/esm-named-import/indexes.json index 60c79a4b..5341e191 100644 --- a/projectgraph/testdata/javascript/esm-named-import/indexes.json +++ b/projectgraph/testdata/javascript/esm-named-import/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "src/main.js", "language": "javascript", + "version": "15", "path": "src/main.js", "language": "javascript", "imports": [ {"line": 1, "module": "./parser.js", "item": "parse", "alias": "parse", "kind": "static", "namespace": ".//parser.js//parse"}, {"line": 2, "module": "./util", "item": "fmt", "alias": "fmt", "kind": "static", "namespace": ".//util//fmt"}, @@ -14,7 +14,7 @@ "unresolved": [{"caller": "src/main.js", "line": 5, "column": 1, "text": "import", "reason": "dynamic-import"}] }, { - "version": "14", "path": "src/parser.js", "language": "javascript", + "version": "15", "path": "src/parser.js", "language": "javascript", "imports": [{"line": 1, "module": "js-yaml", "alias": "yaml", "kind": "static", "namespace": "js-yaml"}], "exports": [{"name": "parse", "target": "src/parser.js//parse", "line": 2}], "definitions": [ @@ -24,7 +24,7 @@ "calls": [{"caller": "src/parser.js//parse", "line": 3, "column": 10, "callee": "yaml.load", "targets": ["js-yaml//load"], "receiver": "yaml", "method": "load", "kind": "call"}] }, { - "version": "14", "path": "src/util.js", "language": "javascript", + "version": "15", "path": "src/util.js", "language": "javascript", "exports": [{"name": "fmt", "target": "src/util.js//fmt", "line": 1}], "definitions": [ {"key": "src/util.js", "kind": "module", "name": "util", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/javascript/export-star/graph.golden.json b/projectgraph/testdata/javascript/export-star/graph.golden.json index cacbbf58..0987ca09 100644 --- a/projectgraph/testdata/javascript/export-star/graph.golden.json +++ b/projectgraph/testdata/javascript/export-star/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "a.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "a.js//a": { "key": "a.js//a", @@ -19,7 +20,8 @@ "kind": "module", "path": "all.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "all.js//a": { "key": "all.js//a", @@ -47,7 +49,8 @@ "kind": "module", "path": "b.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "b.js//b": { "key": "b.js//b", @@ -61,7 +64,8 @@ "kind": "module", "path": "c.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "c.js//c": { "key": "c.js//c", @@ -75,7 +79,8 @@ "kind": "module", "path": "main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" } }, "edges": [ @@ -276,12 +281,15 @@ "main.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 5, "nodes": 11, "edges": 10, "unresolved": 0, "roots": 8, - "root_fallback": true + "root_fallback": true, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/export-star/indexes.json b/projectgraph/testdata/javascript/export-star/indexes.json index 0ede83ab..e5926a2f 100644 --- a/projectgraph/testdata/javascript/export-star/indexes.json +++ b/projectgraph/testdata/javascript/export-star/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.js", "language": "javascript", + "version": "15", "path": "main.js", "language": "javascript", "imports": [ {"line": 1, "module": "./all", "item": "a", "alias": "a", "kind": "static", "namespace": ".//all//a"}, {"line": 1, "module": "./all", "item": "b", "alias": "b", "kind": "static", "namespace": ".//all//b"}, @@ -14,7 +14,7 @@ ] }, { - "version": "14", "path": "all.js", "language": "javascript", + "version": "15", "path": "all.js", "language": "javascript", "exports": [ {"name": "*", "target": "", "from": "./a", "item": "*", "line": 1}, {"name": "*", "target": "", "from": "./b", "item": "*", "line": 2}, @@ -23,7 +23,7 @@ "definitions": [{"key": "all.js", "kind": "module", "name": "all", "start_line": 1, "end_line": 3, "exported": true}] }, { - "version": "14", "path": "a.js", "language": "javascript", + "version": "15", "path": "a.js", "language": "javascript", "exports": [{"name": "a", "target": "a.js//a", "line": 1}], "definitions": [ {"key": "a.js", "kind": "module", "name": "a", "start_line": 1, "end_line": 1, "exported": true}, @@ -31,7 +31,7 @@ ] }, { - "version": "14", "path": "b.js", "language": "javascript", + "version": "15", "path": "b.js", "language": "javascript", "exports": [{"name": "b", "target": "b.js//b", "line": 1}], "definitions": [ {"key": "b.js", "kind": "module", "name": "b", "start_line": 1, "end_line": 1, "exported": true}, @@ -39,7 +39,7 @@ ] }, { - "version": "14", "path": "c.js", "language": "javascript", + "version": "15", "path": "c.js", "language": "javascript", "exports": [{"name": "c", "target": "c.js//c", "line": 1}], "definitions": [ {"key": "c.js", "kind": "module", "name": "c", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/javascript/exports-conditions/graph.golden.json b/projectgraph/testdata/javascript/exports-conditions/graph.golden.json index c48789e3..383f42a2 100644 --- a/projectgraph/testdata/javascript/exports-conditions/graph.golden.json +++ b/projectgraph/testdata/javascript/exports-conditions/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "app/main.mjs", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/cjs/index.cjs": { "key": "lib/cjs/index.cjs", "kind": "module", "path": "lib/cjs/index.cjs", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/cjs/index.cjs//x": { "key": "lib/cjs/index.cjs//x", @@ -26,7 +28,8 @@ "kind": "module", "path": "lib/esm/index.mjs", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/esm/index.mjs//x": { "key": "lib/esm/index.mjs//x", @@ -40,7 +43,8 @@ "kind": "module", "path": "lib/src/feature/y.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/src/feature/y.js//y": { "key": "lib/src/feature/y.js//y", @@ -144,11 +148,14 @@ "lib/src/feature/y.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 4, "nodes": 7, "edges": 4, "unresolved": 0, - "roots": 4 + "roots": 4, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/exports-conditions/indexes.json b/projectgraph/testdata/javascript/exports-conditions/indexes.json index c0aefd37..74577682 100644 --- a/projectgraph/testdata/javascript/exports-conditions/indexes.json +++ b/projectgraph/testdata/javascript/exports-conditions/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "app/main.mjs", "language": "javascript", + "version": "15", "path": "app/main.mjs", "language": "javascript", "imports": [ {"line": 1, "module": "@acme/lib", "item": "x", "alias": "x", "kind": "static", "namespace": "@acme//lib//x"}, {"line": 2, "module": "@acme/lib/feature/y", "item": "y", "alias": "y", "kind": "static", "namespace": "@acme//lib//feature//y//y"} @@ -12,7 +12,7 @@ ] }, { - "version": "14", "path": "lib/esm/index.mjs", "language": "javascript", + "version": "15", "path": "lib/esm/index.mjs", "language": "javascript", "exports": [{"name": "x", "target": "lib/esm/index.mjs//x", "line": 1}], "definitions": [ {"key": "lib/esm/index.mjs", "kind": "module", "name": "index", "start_line": 1, "end_line": 1, "exported": true}, @@ -20,7 +20,7 @@ ] }, { - "version": "14", "path": "lib/cjs/index.cjs", "language": "javascript", + "version": "15", "path": "lib/cjs/index.cjs", "language": "javascript", "exports": [{"name": "x", "target": "lib/cjs/index.cjs//x", "line": 1}], "definitions": [ {"key": "lib/cjs/index.cjs", "kind": "module", "name": "index", "start_line": 1, "end_line": 1, "exported": true}, @@ -28,7 +28,7 @@ ] }, { - "version": "14", "path": "lib/src/feature/y.js", "language": "javascript", + "version": "15", "path": "lib/src/feature/y.js", "language": "javascript", "exports": [{"name": "y", "target": "lib/src/feature/y.js//y", "line": 1}], "definitions": [ {"key": "lib/src/feature/y.js", "kind": "module", "name": "y", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/javascript/unknown-receiver/graph.golden.json b/projectgraph/testdata/javascript/unknown-receiver/graph.golden.json index 5fce3766..33a12cc9 100644 --- a/projectgraph/testdata/javascript/unknown-receiver/graph.golden.json +++ b/projectgraph/testdata/javascript/unknown-receiver/graph.golden.json @@ -9,7 +9,8 @@ "kind": "module", "path": "main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "main.js//handle": { "key": "main.js//handle", @@ -124,12 +125,15 @@ "main.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 5, "edges": 3, "unresolved": 3, "roots": 1, - "root_fallback": true + "root_fallback": true, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/unknown-receiver/indexes.json b/projectgraph/testdata/javascript/unknown-receiver/indexes.json index c4414fa2..6ab9056d 100644 --- a/projectgraph/testdata/javascript/unknown-receiver/indexes.json +++ b/projectgraph/testdata/javascript/unknown-receiver/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "main.js", "language": "javascript", "imports": [ diff --git a/projectgraph/testdata/javascript/workspaces/graph.golden.json b/projectgraph/testdata/javascript/workspaces/graph.golden.json index 7af8a2f2..6ae5010b 100644 --- a/projectgraph/testdata/javascript/workspaces/graph.golden.json +++ b/projectgraph/testdata/javascript/workspaces/graph.golden.json @@ -9,14 +9,16 @@ "kind": "module", "path": "packages/app/src/main.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "packages/core/lib/index.js": { "key": "packages/core/lib/index.js", "kind": "module", "path": "packages/core/lib/index.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "packages/core/lib/index.js//start": { "key": "packages/core/lib/index.js//start", @@ -30,7 +32,8 @@ "kind": "module", "path": "packages/core/util.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "packages/core/util.js//tidy": { "key": "packages/core/util.js//tidy", @@ -127,11 +130,14 @@ "packages/core/util.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 6, "edges": 5, "unresolved": 0, - "roots": 2 + "roots": 2, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/javascript/workspaces/indexes.json b/projectgraph/testdata/javascript/workspaces/indexes.json index 65319d4b..52b12038 100644 --- a/projectgraph/testdata/javascript/workspaces/indexes.json +++ b/projectgraph/testdata/javascript/workspaces/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "packages/app/src/main.js", "language": "javascript", + "version": "15", "path": "packages/app/src/main.js", "language": "javascript", "imports": [ {"line": 1, "module": "@acme/core", "item": "start", "alias": "start", "kind": "static", "namespace": "@acme//core//start"}, {"line": 2, "module": "@acme/core/util", "item": "tidy", "alias": "tidy", "kind": "static", "namespace": "@acme//core//util//tidy"}, @@ -14,7 +14,7 @@ ] }, { - "version": "14", "path": "packages/core/lib/index.js", "language": "javascript", + "version": "15", "path": "packages/core/lib/index.js", "language": "javascript", "exports": [{"name": "start", "target": "packages/core/lib/index.js//start", "line": 1}], "definitions": [ {"key": "packages/core/lib/index.js", "kind": "module", "name": "index", "start_line": 1, "end_line": 1, "exported": true}, @@ -22,7 +22,7 @@ ] }, { - "version": "14", "path": "packages/core/util.js", "language": "javascript", + "version": "15", "path": "packages/core/util.js", "language": "javascript", "exports": [{"name": "tidy", "target": "packages/core/util.js//tidy", "line": 1}], "definitions": [ {"key": "packages/core/util.js", "kind": "module", "name": "util", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/python/all-hides/graph.golden.json b/projectgraph/testdata/python/all-hides/graph.golden.json index a5519cf5..78a625e1 100644 --- a/projectgraph/testdata/python/all-hides/graph.golden.json +++ b/projectgraph/testdata/python/all-hides/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/__init__.py": { "key": "pkg/__init__.py", "kind": "module", "path": "pkg/__init__.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/__init__.py//f": { "key": "pkg/__init__.py//f", @@ -33,7 +35,8 @@ "kind": "module", "path": "pkg/a.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/a.py//f": { "key": "pkg/a.py//f", @@ -174,12 +177,15 @@ "pkg/a.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 7, "edges": 6, "unresolved": 0, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/all-hides/indexes.json b/projectgraph/testdata/python/all-hides/indexes.json index bf36a029..4bde401b 100644 --- a/projectgraph/testdata/python/all-hides/indexes.json +++ b/projectgraph/testdata/python/all-hides/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [ {"line": 1, "module": "pkg", "item": "f", "alias": "f", "kind": "static", "namespace": "pkg//f"}, {"line": 1, "module": "pkg", "item": "g", "alias": "g", "kind": "static", "namespace": "pkg//g"} @@ -12,7 +12,7 @@ ] }, { - "version": "14", "path": "pkg/__init__.py", "language": "python", + "version": "15", "path": "pkg/__init__.py", "language": "python", "imports": [ {"line": 1, "module": ".a", "item": "f", "alias": "f", "kind": "static", "namespace": "//a//f"}, {"line": 1, "module": ".a", "item": "g", "alias": "g", "kind": "static", "namespace": "//a//g"} @@ -21,7 +21,7 @@ "definitions": [{"key": "pkg/__init__.py", "kind": "module", "name": "__init__", "start_line": 1, "end_line": 2, "exported": true}] }, { - "version": "14", "path": "pkg/a.py", "language": "python", + "version": "15", "path": "pkg/a.py", "language": "python", "definitions": [ {"key": "pkg/a.py", "kind": "module", "name": "a", "start_line": 1, "end_line": 4, "exported": true}, {"key": "pkg/a.py//f", "kind": "function", "name": "f", "start_line": 1, "end_line": 2, "exported": true}, diff --git a/projectgraph/testdata/python/assignment/graph.golden.json b/projectgraph/testdata/python/assignment/graph.golden.json index 50c8c1d7..21e85fe1 100644 --- a/projectgraph/testdata/python/assignment/graph.golden.json +++ b/projectgraph/testdata/python/assignment/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "main.py//helper": { "key": "main.py//helper", @@ -70,12 +71,15 @@ "main.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 3, "edges": 2, "unresolved": 0, "roots": 2, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/assignment/indexes.json b/projectgraph/testdata/python/assignment/indexes.json index 4a129982..d9ed405e 100644 --- a/projectgraph/testdata/python/assignment/indexes.json +++ b/projectgraph/testdata/python/assignment/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "yaml", "alias": "yaml", "kind": "static", "namespace": "yaml"}], "definitions": [ {"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 7, "exported": true}, diff --git a/projectgraph/testdata/python/class-body/graph.golden.json b/projectgraph/testdata/python/class-body/graph.golden.json index 2d7290cb..e922a79d 100644 --- a/projectgraph/testdata/python/class-body/graph.golden.json +++ b/projectgraph/testdata/python/class-body/graph.golden.json @@ -6,6 +6,7 @@ "path": "config.py", "line": 1, "exported": true, + "language": "python", "flags": [ { "kind": "reflection", @@ -224,12 +225,15 @@ "config.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 12, "edges": 7, "unresolved": 1, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/class-body/indexes.json b/projectgraph/testdata/python/class-body/indexes.json index aa75873e..ed28f565 100644 --- a/projectgraph/testdata/python/class-body/indexes.json +++ b/projectgraph/testdata/python/class-body/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "config.py", "language": "python", "imports": [ diff --git a/projectgraph/testdata/python/decorators/graph.golden.json b/projectgraph/testdata/python/decorators/graph.golden.json index 69b4630e..361df88b 100644 --- a/projectgraph/testdata/python/decorators/graph.golden.json +++ b/projectgraph/testdata/python/decorators/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "app.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "app.py//Point": { "key": "app.py//Point", @@ -194,12 +195,15 @@ "app.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 9, "edges": 5, "unresolved": 1, "roots": 6, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/decorators/indexes.json b/projectgraph/testdata/python/decorators/indexes.json index b5f50d62..d706dded 100644 --- a/projectgraph/testdata/python/decorators/indexes.json +++ b/projectgraph/testdata/python/decorators/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "app.py", "language": "python", "imports": [ diff --git a/projectgraph/testdata/python/external/graph.golden.json b/projectgraph/testdata/python/external/graph.golden.json index 8e6116a8..71af1f03 100644 --- a/projectgraph/testdata/python/external/graph.golden.json +++ b/projectgraph/testdata/python/external/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "main.py//load_plugin": { "key": "main.py//load_plugin", @@ -108,12 +109,15 @@ "main.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 5, "edges": 3, "unresolved": 1, "roots": 2, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/external/indexes.json b/projectgraph/testdata/python/external/indexes.json index 8abec365..85e263a6 100644 --- a/projectgraph/testdata/python/external/indexes.json +++ b/projectgraph/testdata/python/external/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [ {"line": 1, "module": "requests", "alias": "requests", "kind": "static", "namespace": "requests"}, {"line": 2, "module": "yaml", "item": "load", "alias": "yl", "kind": "static", "namespace": "yaml//load"}, diff --git a/projectgraph/testdata/python/local-receiver/graph.golden.json b/projectgraph/testdata/python/local-receiver/graph.golden.json index 3a333db8..905b7c85 100644 --- a/projectgraph/testdata/python/local-receiver/graph.golden.json +++ b/projectgraph/testdata/python/local-receiver/graph.golden.json @@ -9,7 +9,8 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "main.py//load_plugin": { "key": "main.py//load_plugin", @@ -145,12 +146,15 @@ "main.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 5, "edges": 3, "unresolved": 3, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/local-receiver/indexes.json b/projectgraph/testdata/python/local-receiver/indexes.json index 09ba712c..fccc8671 100644 --- a/projectgraph/testdata/python/local-receiver/indexes.json +++ b/projectgraph/testdata/python/local-receiver/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "main.py", "language": "python", "imports": [ diff --git a/projectgraph/testdata/python/member-call/graph.golden.json b/projectgraph/testdata/python/member-call/graph.golden.json index f2d92a37..377756e7 100644 --- a/projectgraph/testdata/python/member-call/graph.golden.json +++ b/projectgraph/testdata/python/member-call/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "base.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "base.py//Base": { "key": "base.py//Base", @@ -30,14 +31,16 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "parser.py": { "key": "parser.py", "kind": "module", "path": "parser.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "parser.py//Parser": { "key": "parser.py//Parser", @@ -258,12 +261,15 @@ "parser.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 9, "edges": 10, "unresolved": 0, "roots": 8, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/member-call/indexes.json b/projectgraph/testdata/python/member-call/indexes.json index 0d0fd90f..5c4112b9 100644 --- a/projectgraph/testdata/python/member-call/indexes.json +++ b/projectgraph/testdata/python/member-call/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "parser", "item": "Parser", "alias": "Parser", "kind": "static", "namespace": "parser//Parser"}], "definitions": [{"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 4, "exported": true}], "calls": [ @@ -10,7 +10,7 @@ ] }, { - "version": "14", "path": "parser.py", "language": "python", + "version": "15", "path": "parser.py", "language": "python", "imports": [{"line": 1, "module": "base", "item": "Base", "alias": "Base", "kind": "static", "namespace": "base//Base"}], "definitions": [ {"key": "parser.py", "kind": "module", "name": "parser", "start_line": 1, "end_line": 7, "exported": true}, @@ -22,7 +22,7 @@ "calls": [{"caller": "parser.py//Parser//parse", "line": 6, "column": 9, "callee": "self.helper", "targets": ["parser.py//Parser//helper"], "receiver": "self", "method": "helper", "kind": "call"}] }, { - "version": "14", "path": "base.py", "language": "python", + "version": "15", "path": "base.py", "language": "python", "imports": [{"line": 1, "module": "io", "alias": "io", "kind": "static", "namespace": "io"}], "definitions": [ {"key": "base.py", "kind": "module", "name": "base", "start_line": 1, "end_line": 4, "exported": true}, diff --git a/projectgraph/testdata/python/missing-file/graph.golden.json b/projectgraph/testdata/python/missing-file/graph.golden.json index 249c0096..fbfcb273 100644 --- a/projectgraph/testdata/python/missing-file/graph.golden.json +++ b/projectgraph/testdata/python/missing-file/graph.golden.json @@ -10,7 +10,8 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" } }, "edges": [ @@ -55,12 +56,15 @@ "main.py": "cached" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 2, "edges": 2, "unresolved": 0, "roots": 1, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/missing-file/indexes.json b/projectgraph/testdata/python/missing-file/indexes.json index b42f5396..b2b9b228 100644 --- a/projectgraph/testdata/python/missing-file/indexes.json +++ b/projectgraph/testdata/python/missing-file/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "handlers", "item": "parse", "alias": "parse", "kind": "static", "namespace": "handlers//parse"}], "definitions": [{"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "main.py", "line": 2, "column": 1, "callee": "parse", "targets": ["handlers//parse"], "kind": "call"}] diff --git a/projectgraph/testdata/python/package-init/graph.golden.json b/projectgraph/testdata/python/package-init/graph.golden.json index 3ae14b1f..978468df 100644 --- a/projectgraph/testdata/python/package-init/graph.golden.json +++ b/projectgraph/testdata/python/package-init/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/__init__.py": { "key": "pkg/__init__.py", "kind": "module", "path": "pkg/__init__.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/__init__.py//load": { "key": "pkg/__init__.py//load", @@ -26,7 +28,8 @@ "kind": "module", "path": "pkg/loader.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/loader.py//load": { "key": "pkg/loader.py//load", @@ -144,12 +147,15 @@ "pkg/loader.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 6, "edges": 5, "unresolved": 0, "roots": 4, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/package-init/indexes.json b/projectgraph/testdata/python/package-init/indexes.json index 6e275227..e9981571 100644 --- a/projectgraph/testdata/python/package-init/indexes.json +++ b/projectgraph/testdata/python/package-init/indexes.json @@ -1,17 +1,17 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "pkg", "item": "load", "alias": "load", "kind": "static", "namespace": "pkg//load"}], "definitions": [{"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "main.py", "line": 2, "column": 1, "callee": "load", "targets": ["pkg//load"], "kind": "call"}] }, { - "version": "14", "path": "pkg/__init__.py", "language": "python", + "version": "15", "path": "pkg/__init__.py", "language": "python", "imports": [{"line": 1, "module": ".loader", "item": "load", "alias": "load", "kind": "static", "namespace": "//loader//load"}], "definitions": [{"key": "pkg/__init__.py", "kind": "module", "name": "__init__", "start_line": 1, "end_line": 1, "exported": true}] }, { - "version": "14", "path": "pkg/loader.py", "language": "python", + "version": "15", "path": "pkg/loader.py", "language": "python", "imports": [{"line": 1, "module": "yaml", "alias": "yaml", "kind": "static", "namespace": "yaml"}], "definitions": [ {"key": "pkg/loader.py", "kind": "module", "name": "loader", "start_line": 1, "end_line": 4, "exported": true}, diff --git a/projectgraph/testdata/python/reexport-cycle/graph.golden.json b/projectgraph/testdata/python/reexport-cycle/graph.golden.json index c9b069c0..104e88d7 100644 --- a/projectgraph/testdata/python/reexport-cycle/graph.golden.json +++ b/projectgraph/testdata/python/reexport-cycle/graph.golden.json @@ -5,21 +5,24 @@ "kind": "module", "path": "a/__init__.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "b/__init__.py": { "key": "b/__init__.py", "kind": "module", "path": "b/__init__.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "main.py": { "key": "main.py", "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" } }, "edges": [ @@ -105,12 +108,15 @@ "main.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 3, "edges": 3, "unresolved": 1, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/reexport-cycle/indexes.json b/projectgraph/testdata/python/reexport-cycle/indexes.json index d5dc08b8..a94c2482 100644 --- a/projectgraph/testdata/python/reexport-cycle/indexes.json +++ b/projectgraph/testdata/python/reexport-cycle/indexes.json @@ -1,17 +1,17 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "a", "item": "x", "alias": "x", "kind": "static", "namespace": "a//x"}], "definitions": [{"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "main.py", "line": 2, "column": 1, "callee": "x", "targets": ["a//x"], "kind": "call"}] }, { - "version": "14", "path": "a/__init__.py", "language": "python", + "version": "15", "path": "a/__init__.py", "language": "python", "imports": [{"line": 1, "module": "b", "item": "x", "alias": "x", "kind": "static", "namespace": "b//x"}], "definitions": [{"key": "a/__init__.py", "kind": "module", "name": "__init__", "start_line": 1, "end_line": 1, "exported": true}] }, { - "version": "14", "path": "b/__init__.py", "language": "python", + "version": "15", "path": "b/__init__.py", "language": "python", "imports": [{"line": 1, "module": "a", "item": "x", "alias": "x", "kind": "static", "namespace": "a//x"}], "definitions": [{"key": "b/__init__.py", "kind": "module", "name": "__init__", "start_line": 1, "end_line": 1, "exported": true}] } diff --git a/projectgraph/testdata/python/relative-imports/graph.golden.json b/projectgraph/testdata/python/relative-imports/graph.golden.json index fd51479a..4f84f186 100644 --- a/projectgraph/testdata/python/relative-imports/graph.golden.json +++ b/projectgraph/testdata/python/relative-imports/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "pkg/c.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/c.py//g": { "key": "pkg/c.py//g", @@ -19,7 +20,8 @@ "kind": "module", "path": "pkg/sub/a.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/sub/a.py//run": { "key": "pkg/sub/a.py//run", @@ -33,7 +35,8 @@ "kind": "module", "path": "pkg/sub/b.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "pkg/sub/b.py//f": { "key": "pkg/sub/b.py//f", @@ -154,12 +157,15 @@ "pkg/sub/b.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 6, "edges": 4, "unresolved": 0, "roots": 6, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/relative-imports/indexes.json b/projectgraph/testdata/python/relative-imports/indexes.json index db008b91..f682cd9d 100644 --- a/projectgraph/testdata/python/relative-imports/indexes.json +++ b/projectgraph/testdata/python/relative-imports/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "pkg/sub/a.py", "language": "python", + "version": "15", "path": "pkg/sub/a.py", "language": "python", "imports": [ {"line": 1, "module": ".b", "item": "f", "alias": "f", "kind": "static", "namespace": "//b//f"}, {"line": 2, "module": "..c", "item": "g", "alias": "g", "kind": "static", "namespace": "////c//g"} @@ -15,14 +15,14 @@ ] }, { - "version": "14", "path": "pkg/sub/b.py", "language": "python", + "version": "15", "path": "pkg/sub/b.py", "language": "python", "definitions": [ {"key": "pkg/sub/b.py", "kind": "module", "name": "b", "start_line": 1, "end_line": 2, "exported": true}, {"key": "pkg/sub/b.py//f", "kind": "function", "name": "f", "start_line": 1, "end_line": 2, "exported": true} ] }, { - "version": "14", "path": "pkg/c.py", "language": "python", + "version": "15", "path": "pkg/c.py", "language": "python", "definitions": [ {"key": "pkg/c.py", "kind": "module", "name": "c", "start_line": 1, "end_line": 2, "exported": true}, {"key": "pkg/c.py//g", "kind": "function", "name": "g", "start_line": 1, "end_line": 2, "exported": true} diff --git a/projectgraph/testdata/python/sibling-import/graph.golden.json b/projectgraph/testdata/python/sibling-import/graph.golden.json index cebc8e7d..9545e3bd 100644 --- a/projectgraph/testdata/python/sibling-import/graph.golden.json +++ b/projectgraph/testdata/python/sibling-import/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "handlers.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "handlers.py//parse": { "key": "handlers.py//parse", @@ -19,7 +20,8 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "yaml//load": { "key": "yaml//load", @@ -98,12 +100,15 @@ "main.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 4, "edges": 3, "unresolved": 0, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/sibling-import/indexes.json b/projectgraph/testdata/python/sibling-import/indexes.json index b3c87e12..bb9410bb 100644 --- a/projectgraph/testdata/python/sibling-import/indexes.json +++ b/projectgraph/testdata/python/sibling-import/indexes.json @@ -1,12 +1,12 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "handlers", "item": "parse", "alias": "parse", "kind": "static", "namespace": "handlers//parse"}], "definitions": [{"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 3, "exported": true}], "calls": [{"caller": "main.py", "line": 3, "column": 1, "callee": "parse", "targets": ["handlers//parse"], "kind": "call"}] }, { - "version": "14", "path": "handlers.py", "language": "python", + "version": "15", "path": "handlers.py", "language": "python", "imports": [{"line": 1, "module": "yaml", "alias": "yaml", "kind": "static", "namespace": "yaml"}], "definitions": [ {"key": "handlers.py", "kind": "module", "name": "handlers", "start_line": 1, "end_line": 4, "exported": true}, diff --git a/projectgraph/testdata/python/src-layout/graph.golden.json b/projectgraph/testdata/python/src-layout/graph.golden.json index 9f2d1eb9..77e1ff1b 100644 --- a/projectgraph/testdata/python/src-layout/graph.golden.json +++ b/projectgraph/testdata/python/src-layout/graph.golden.json @@ -5,21 +5,24 @@ "kind": "module", "path": "services/api/src/api/__init__.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "services/api/src/api/cli.py": { "key": "services/api/src/api/cli.py", "kind": "module", "path": "services/api/src/api/cli.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "services/api/src/api/core.py": { "key": "services/api/src/api/core.py", "kind": "module", "path": "services/api/src/api/core.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "services/api/src/api/core.py//run": { "key": "services/api/src/api/core.py//run", @@ -99,12 +102,15 @@ "services/api/src/api/core.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 4, "edges": 2, "unresolved": 0, "roots": 4, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/src-layout/indexes.json b/projectgraph/testdata/python/src-layout/indexes.json index 8dab6e9b..57d71afc 100644 --- a/projectgraph/testdata/python/src-layout/indexes.json +++ b/projectgraph/testdata/python/src-layout/indexes.json @@ -1,16 +1,16 @@ [ { - "version": "14", "path": "services/api/src/api/__init__.py", "language": "python", + "version": "15", "path": "services/api/src/api/__init__.py", "language": "python", "definitions": [{"key": "services/api/src/api/__init__.py", "kind": "module", "name": "__init__", "start_line": 1, "end_line": 1, "exported": true}] }, { - "version": "14", "path": "services/api/src/api/cli.py", "language": "python", + "version": "15", "path": "services/api/src/api/cli.py", "language": "python", "imports": [{"line": 1, "module": "api.core", "item": "run", "alias": "run", "kind": "static", "namespace": "api//core//run"}], "definitions": [{"key": "services/api/src/api/cli.py", "kind": "module", "name": "cli", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "services/api/src/api/cli.py", "line": 2, "column": 1, "callee": "run", "targets": ["api//core//run"], "kind": "call"}] }, { - "version": "14", "path": "services/api/src/api/core.py", "language": "python", + "version": "15", "path": "services/api/src/api/core.py", "language": "python", "definitions": [ {"key": "services/api/src/api/core.py", "kind": "module", "name": "core", "start_line": 1, "end_line": 2, "exported": true}, {"key": "services/api/src/api/core.py//run", "kind": "function", "name": "run", "start_line": 1, "end_line": 2, "exported": true} diff --git a/projectgraph/testdata/python/sys-path/graph.golden.json b/projectgraph/testdata/python/sys-path/graph.golden.json index 2ed50c1b..1d330ec4 100644 --- a/projectgraph/testdata/python/sys-path/graph.golden.json +++ b/projectgraph/testdata/python/sys-path/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "app/main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "lib/shared.py": { "key": "lib/shared.py", "kind": "module", "path": "lib/shared.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "lib/shared.py//go": { "key": "lib/shared.py//go", @@ -82,12 +84,15 @@ "lib/shared.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 3, "edges": 2, "unresolved": 0, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/sys-path/indexes.json b/projectgraph/testdata/python/sys-path/indexes.json index 372cc0fb..4e0ab812 100644 --- a/projectgraph/testdata/python/sys-path/indexes.json +++ b/projectgraph/testdata/python/sys-path/indexes.json @@ -1,12 +1,12 @@ [ { - "version": "14", "path": "app/main.py", "language": "python", + "version": "15", "path": "app/main.py", "language": "python", "imports": [{"line": 1, "module": "shared", "alias": "shared", "kind": "static", "namespace": "shared"}], "definitions": [{"key": "app/main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "app/main.py", "line": 2, "column": 1, "callee": "shared.go", "targets": ["shared//go"], "receiver": "shared", "method": "go", "kind": "call"}] }, { - "version": "14", "path": "lib/shared.py", "language": "python", + "version": "15", "path": "lib/shared.py", "language": "python", "definitions": [ {"key": "lib/shared.py", "kind": "module", "name": "shared", "start_line": 1, "end_line": 2, "exported": true}, {"key": "lib/shared.py//go", "kind": "function", "name": "go", "start_line": 1, "end_line": 2, "exported": true} diff --git a/projectgraph/testdata/python/wildcard/graph.golden.json b/projectgraph/testdata/python/wildcard/graph.golden.json index db310758..386358f9 100644 --- a/projectgraph/testdata/python/wildcard/graph.golden.json +++ b/projectgraph/testdata/python/wildcard/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "util.py": { "key": "util.py", "kind": "module", "path": "util.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "util.py//_private": { "key": "util.py//_private", @@ -112,12 +114,15 @@ "util.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 4, "edges": 3, "unresolved": 1, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/python/wildcard/indexes.json b/projectgraph/testdata/python/wildcard/indexes.json index 8c56f11b..a93d6767 100644 --- a/projectgraph/testdata/python/wildcard/indexes.json +++ b/projectgraph/testdata/python/wildcard/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "main.py", "language": "python", + "version": "15", "path": "main.py", "language": "python", "imports": [{"line": 1, "module": "util", "wildcard": true, "kind": "static", "namespace": "util"}], "definitions": [{"key": "main.py", "kind": "module", "name": "main", "start_line": 1, "end_line": 3, "exported": true}], "calls": [ @@ -9,7 +9,7 @@ ] }, { - "version": "14", "path": "util.py", "language": "python", + "version": "15", "path": "util.py", "language": "python", "definitions": [ {"key": "util.py", "kind": "module", "name": "util", "start_line": 1, "end_line": 4, "exported": true}, {"key": "util.py//helper", "kind": "function", "name": "helper", "start_line": 1, "end_line": 2, "exported": true}, diff --git a/projectgraph/testdata/roots/extra-rules/graph.golden.json b/projectgraph/testdata/roots/extra-rules/graph.golden.json index da675a69..b3287dc2 100644 --- a/projectgraph/testdata/roots/extra-rules/graph.golden.json +++ b/projectgraph/testdata/roots/extra-rules/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "app/users/route.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/handler.js": { "key": "src/handler.js", "kind": "module", "path": "src/handler.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "src/handler.js//Handler": { "key": "src/handler.js//Handler", @@ -40,7 +42,8 @@ "kind": "module", "path": "views.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "views.py//index": { "key": "views.py//index", @@ -105,11 +108,16 @@ "views.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 7, "edges": 0, "unresolved": 0, - "roots": 5 + "roots": 5, + "languages": [ + "javascript", + "python", + "typescript" + ] } } diff --git a/projectgraph/testdata/roots/extra-rules/indexes.json b/projectgraph/testdata/roots/extra-rules/indexes.json index 12b245f9..1844aae8 100644 --- a/projectgraph/testdata/roots/extra-rules/indexes.json +++ b/projectgraph/testdata/roots/extra-rules/indexes.json @@ -1,10 +1,10 @@ [ { - "version": "14", "path": "app/users/route.ts", "language": "typescript", + "version": "15", "path": "app/users/route.ts", "language": "typescript", "definitions": [{"key": "app/users/route.ts", "kind": "module", "name": "route", "start_line": 1, "end_line": 1, "exported": true}] }, { - "version": "14", "path": "src/handler.js", "language": "javascript", + "version": "15", "path": "src/handler.js", "language": "javascript", "definitions": [ {"key": "src/handler.js", "kind": "module", "name": "handler", "start_line": 1, "end_line": 4, "exported": true}, {"key": "src/handler.js//Handler", "kind": "class", "name": "Handler", "start_line": 1, "end_line": 4, "exported": true}, @@ -13,7 +13,7 @@ ] }, { - "version": "14", "path": "views.py", "language": "python", + "version": "15", "path": "views.py", "language": "python", "definitions": [ {"key": "views.py", "kind": "module", "name": "views", "start_line": 1, "end_line": 3, "exported": true}, {"key": "views.py//index", "kind": "function", "name": "index", "start_line": 2, "end_line": 3, "exported": true, "decorators": ["@app.route"]} diff --git a/projectgraph/testdata/roots/library-surface/graph.golden.json b/projectgraph/testdata/roots/library-surface/graph.golden.json index 5c8b68da..51665388 100644 --- a/projectgraph/testdata/roots/library-surface/graph.golden.json +++ b/projectgraph/testdata/roots/library-surface/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "lib/api.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "lib/api.py//_private": { "key": "lib/api.py//_private", @@ -26,7 +27,8 @@ "path": "lib/test_api.py", "line": 1, "exported": true, - "test": true + "test": true, + "language": "python" }, "lib/test_api.py//test_public": { "key": "lib/test_api.py//test_public", @@ -82,12 +84,15 @@ "lib/test_api.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 5, "edges": 0, "unresolved": 0, "roots": 4, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/roots/library-surface/indexes.json b/projectgraph/testdata/roots/library-surface/indexes.json index a89db48a..c52849e9 100644 --- a/projectgraph/testdata/roots/library-surface/indexes.json +++ b/projectgraph/testdata/roots/library-surface/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "lib/api.py", "language": "python", + "version": "15", "path": "lib/api.py", "language": "python", "definitions": [ {"key": "lib/api.py", "kind": "module", "name": "api", "start_line": 1, "end_line": 4, "exported": true}, {"key": "lib/api.py//public", "kind": "function", "name": "public", "start_line": 1, "end_line": 2, "exported": true}, @@ -8,7 +8,7 @@ ] }, { - "version": "14", "path": "lib/test_api.py", "language": "python", + "version": "15", "path": "lib/test_api.py", "language": "python", "definitions": [ {"key": "lib/test_api.py", "kind": "module", "name": "test_api", "start_line": 1, "end_line": 2, "exported": true}, {"key": "lib/test_api.py//test_public", "kind": "function", "name": "test_public", "start_line": 1, "end_line": 2, "exported": true} diff --git a/projectgraph/testdata/roots/main-guard/graph.golden.json b/projectgraph/testdata/roots/main-guard/graph.golden.json index d5b4b552..83944145 100644 --- a/projectgraph/testdata/roots/main-guard/graph.golden.json +++ b/projectgraph/testdata/roots/main-guard/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "app.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "app.py//run": { "key": "app.py//run", @@ -19,7 +20,8 @@ "kind": "module", "path": "lib.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "lib.py//lib": { "key": "lib.py//lib", @@ -60,11 +62,14 @@ "lib.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 4, "edges": 1, "unresolved": 0, - "roots": 1 + "roots": 1, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/roots/main-guard/indexes.json b/projectgraph/testdata/roots/main-guard/indexes.json index 890a4e4b..ecd29e9d 100644 --- a/projectgraph/testdata/roots/main-guard/indexes.json +++ b/projectgraph/testdata/roots/main-guard/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "app.py", "language": "python", "main_guard": true, + "version": "15", "path": "app.py", "language": "python", "main_guard": true, "definitions": [ {"key": "app.py", "kind": "module", "name": "app", "start_line": 1, "end_line": 5, "exported": true}, {"key": "app.py//run", "kind": "function", "name": "run", "start_line": 1, "end_line": 2, "exported": true} @@ -8,7 +8,7 @@ "calls": [{"caller": "app.py", "line": 5, "column": 5, "callee": "run", "targets": ["app.py//run"], "kind": "call"}] }, { - "version": "14", "path": "lib.py", "language": "python", + "version": "15", "path": "lib.py", "language": "python", "definitions": [ {"key": "lib.py", "kind": "module", "name": "lib", "start_line": 1, "end_line": 2, "exported": true}, {"key": "lib.py//lib", "kind": "function", "name": "lib", "start_line": 1, "end_line": 2, "exported": true} diff --git a/projectgraph/testdata/roots/package-json-bin/graph.golden.json b/projectgraph/testdata/roots/package-json-bin/graph.golden.json index 11a8e36f..9a15b3e7 100644 --- a/projectgraph/testdata/roots/package-json-bin/graph.golden.json +++ b/projectgraph/testdata/roots/package-json-bin/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "bin/cli.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/run.js": { "key": "lib/run.js", "kind": "module", "path": "lib/run.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "lib/run.js//run": { "key": "lib/run.js//run", @@ -64,11 +66,14 @@ "lib/run.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 3, "edges": 2, "unresolved": 0, - "roots": 1 + "roots": 1, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/roots/package-json-bin/indexes.json b/projectgraph/testdata/roots/package-json-bin/indexes.json index ef2b0a99..bf531a62 100644 --- a/projectgraph/testdata/roots/package-json-bin/indexes.json +++ b/projectgraph/testdata/roots/package-json-bin/indexes.json @@ -1,12 +1,12 @@ [ { - "version": "14", "path": "bin/cli.js", "language": "javascript", + "version": "15", "path": "bin/cli.js", "language": "javascript", "imports": [{"line": 1, "module": "../lib/run", "item": "run", "alias": "run", "kind": "static", "namespace": "..//lib//run//run"}], "definitions": [{"key": "bin/cli.js", "kind": "module", "name": "cli", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "bin/cli.js", "line": 2, "column": 1, "callee": "run", "targets": ["..//lib//run//run"], "kind": "call"}] }, { - "version": "14", "path": "lib/run.js", "language": "javascript", + "version": "15", "path": "lib/run.js", "language": "javascript", "exports": [{"name": "run", "target": "lib/run.js//run", "line": 1}], "definitions": [ {"key": "lib/run.js", "kind": "module", "name": "run", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/roots/package-json-main/graph.golden.json b/projectgraph/testdata/roots/package-json-main/graph.golden.json index dc63358f..4808ac57 100644 --- a/projectgraph/testdata/roots/package-json-main/graph.golden.json +++ b/projectgraph/testdata/roots/package-json-main/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "cli.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "cli.js//helper": { "key": "cli.js//helper", @@ -19,7 +20,8 @@ "kind": "module", "path": "fmt.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "fmt.js//format": { "key": "fmt.js//format", @@ -33,7 +35,8 @@ "kind": "module", "path": "index.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "index.js//parse": { "key": "index.js//parse", @@ -54,7 +57,8 @@ "kind": "module", "path": "parser.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "parser.js//parse": { "key": "parser.js//parse", @@ -154,11 +158,14 @@ "parser.js": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 4, "nodes": 9, "edges": 3, "unresolved": 0, - "roots": 5 + "roots": 5, + "languages": [ + "javascript" + ] } } diff --git a/projectgraph/testdata/roots/package-json-main/indexes.json b/projectgraph/testdata/roots/package-json-main/indexes.json index 56e5e660..abf83c27 100644 --- a/projectgraph/testdata/roots/package-json-main/indexes.json +++ b/projectgraph/testdata/roots/package-json-main/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "cli.js", "language": "javascript", "exports": [ @@ -30,7 +30,7 @@ ] }, { - "version": "14", + "version": "15", "path": "fmt.js", "language": "javascript", "exports": [ @@ -60,7 +60,7 @@ ] }, { - "version": "14", + "version": "15", "path": "index.js", "language": "javascript", "exports": [ @@ -104,7 +104,7 @@ ] }, { - "version": "14", + "version": "15", "path": "parser.js", "language": "javascript", "exports": [ diff --git a/projectgraph/testdata/roots/procfile-dockerfile/graph.golden.json b/projectgraph/testdata/roots/procfile-dockerfile/graph.golden.json index 4b86f86c..1c376666 100644 --- a/projectgraph/testdata/roots/procfile-dockerfile/graph.golden.json +++ b/projectgraph/testdata/roots/procfile-dockerfile/graph.golden.json @@ -5,21 +5,24 @@ "kind": "module", "path": "app/__init__.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "app/cli.py": { "key": "app/cli.py", "kind": "module", "path": "app/cli.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "app/wsgi.py": { "key": "app/wsgi.py", "kind": "module", "path": "app/wsgi.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "app/wsgi.py//app": { "key": "app/wsgi.py//app", @@ -33,21 +36,24 @@ "kind": "module", "path": "ignored.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "web/server.js": { "key": "web/server.js", "kind": "module", "path": "web/server.js", "line": 1, - "exported": true + "exported": true, + "language": "javascript" }, "worker.py": { "key": "worker.py", "kind": "module", "path": "worker.py", "line": 1, - "exported": true + "exported": true, + "language": "python" } }, "edges": [], @@ -108,11 +114,15 @@ "worker.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 6, "nodes": 7, "edges": 0, "unresolved": 0, - "roots": 5 + "roots": 5, + "languages": [ + "javascript", + "python" + ] } } diff --git a/projectgraph/testdata/roots/procfile-dockerfile/indexes.json b/projectgraph/testdata/roots/procfile-dockerfile/indexes.json index 3955f2b6..c1cd3d5f 100644 --- a/projectgraph/testdata/roots/procfile-dockerfile/indexes.json +++ b/projectgraph/testdata/roots/procfile-dockerfile/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "app/wsgi.py", "language": "python", "definitions": [ @@ -23,7 +23,7 @@ ] }, { - "version": "14", + "version": "15", "path": "app/cli.py", "language": "python", "definitions": [ @@ -38,7 +38,7 @@ ] }, { - "version": "14", + "version": "15", "path": "app/__init__.py", "language": "python", "definitions": [ @@ -53,7 +53,7 @@ ] }, { - "version": "14", + "version": "15", "path": "worker.py", "language": "python", "definitions": [ @@ -68,7 +68,7 @@ ] }, { - "version": "14", + "version": "15", "path": "ignored.py", "language": "python", "definitions": [ @@ -83,7 +83,7 @@ ] }, { - "version": "14", + "version": "15", "path": "web/server.js", "language": "javascript", "definitions": [ diff --git a/projectgraph/testdata/roots/pyproject-scripts/graph.golden.json b/projectgraph/testdata/roots/pyproject-scripts/graph.golden.json index 62d83799..600ba456 100644 --- a/projectgraph/testdata/roots/pyproject-scripts/graph.golden.json +++ b/projectgraph/testdata/roots/pyproject-scripts/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "src/app/cli.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "src/app/cli.py//main": { "key": "src/app/cli.py//main", @@ -19,7 +20,8 @@ "kind": "module", "path": "src/app/ignored.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "src/app/ignored.py//x": { "key": "src/app/ignored.py//x", @@ -33,14 +35,16 @@ "kind": "module", "path": "src/app/legacy.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "src/app/plugin.py": { "key": "src/app/plugin.py", "kind": "module", "path": "src/app/plugin.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "src/app/plugin.py//Plugin": { "key": "src/app/plugin.py//Plugin", @@ -61,7 +65,8 @@ "kind": "module", "path": "src/app/tool.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "src/app/tool.py//run": { "key": "src/app/tool.py//run", @@ -146,11 +151,14 @@ "src/app/tool.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 5, "nodes": 10, "edges": 0, "unresolved": 0, - "roots": 7 + "roots": 7, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/roots/pyproject-scripts/indexes.json b/projectgraph/testdata/roots/pyproject-scripts/indexes.json index 38244e1b..d020c4e9 100644 --- a/projectgraph/testdata/roots/pyproject-scripts/indexes.json +++ b/projectgraph/testdata/roots/pyproject-scripts/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "src/app/cli.py", "language": "python", "definitions": [ @@ -23,7 +23,7 @@ ] }, { - "version": "14", + "version": "15", "path": "src/app/tool.py", "language": "python", "definitions": [ @@ -46,7 +46,7 @@ ] }, { - "version": "14", + "version": "15", "path": "src/app/plugin.py", "language": "python", "definitions": [ @@ -78,7 +78,7 @@ ] }, { - "version": "14", + "version": "15", "path": "src/app/legacy.py", "language": "python", "definitions": [ @@ -93,7 +93,7 @@ ] }, { - "version": "14", + "version": "15", "path": "src/app/ignored.py", "language": "python", "definitions": [ diff --git a/projectgraph/testdata/roots/setup-py/graph.golden.json b/projectgraph/testdata/roots/setup-py/graph.golden.json index ab3a569a..11784999 100644 --- a/projectgraph/testdata/roots/setup-py/graph.golden.json +++ b/projectgraph/testdata/roots/setup-py/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "tool/admin.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "tool/admin.py//run": { "key": "tool/admin.py//run", @@ -19,7 +20,8 @@ "kind": "module", "path": "tool/main.py", "line": 1, - "exported": true + "exported": true, + "language": "python" }, "tool/main.py//cli": { "key": "tool/main.py//cli", @@ -74,11 +76,14 @@ "tool/main.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 4, "edges": 0, "unresolved": 0, - "roots": 4 + "roots": 4, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/roots/setup-py/indexes.json b/projectgraph/testdata/roots/setup-py/indexes.json index 5a20a349..159af41e 100644 --- a/projectgraph/testdata/roots/setup-py/indexes.json +++ b/projectgraph/testdata/roots/setup-py/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", + "version": "15", "path": "tool/main.py", "language": "python", "definitions": [ @@ -23,7 +23,7 @@ ] }, { - "version": "14", + "version": "15", "path": "tool/admin.py", "language": "python", "definitions": [ diff --git a/projectgraph/testdata/roots/test-file/graph.golden.json b/projectgraph/testdata/roots/test-file/graph.golden.json index 97089938..3fa5e725 100644 --- a/projectgraph/testdata/roots/test-file/graph.golden.json +++ b/projectgraph/testdata/roots/test-file/graph.golden.json @@ -6,7 +6,8 @@ "path": "tests/test_app.py", "line": 1, "exported": true, - "test": true + "test": true, + "language": "python" }, "tests/test_app.py//TestApp": { "key": "tests/test_app.py//TestApp", @@ -68,12 +69,15 @@ "tests/test_app.py": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 1, "nodes": 4, "edges": 0, "unresolved": 0, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "python" + ] } } diff --git a/projectgraph/testdata/roots/test-file/indexes.json b/projectgraph/testdata/roots/test-file/indexes.json index 4059f41f..443885be 100644 --- a/projectgraph/testdata/roots/test-file/indexes.json +++ b/projectgraph/testdata/roots/test-file/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "tests/test_app.py", "language": "python", + "version": "15", "path": "tests/test_app.py", "language": "python", "definitions": [ {"key": "tests/test_app.py", "kind": "module", "name": "test_app", "start_line": 1, "end_line": 6, "exported": true}, {"key": "tests/test_app.py//TestApp", "kind": "class", "name": "TestApp", "start_line": 1, "end_line": 4, "exported": true}, diff --git a/projectgraph/testdata/typescript/js-specifier/graph.golden.json b/projectgraph/testdata/typescript/js-specifier/graph.golden.json index b1ffb28d..a040c9fe 100644 --- a/projectgraph/testdata/typescript/js-specifier/graph.golden.json +++ b/projectgraph/testdata/typescript/js-specifier/graph.golden.json @@ -5,14 +5,16 @@ "kind": "module", "path": "src/main.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/util.ts": { "key": "src/util.ts", "kind": "module", "path": "src/util.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/util.ts//f": { "key": "src/util.ts//f", @@ -82,12 +84,15 @@ "src/util.ts": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 2, "nodes": 3, "edges": 2, "unresolved": 0, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "typescript" + ] } } diff --git a/projectgraph/testdata/typescript/js-specifier/indexes.json b/projectgraph/testdata/typescript/js-specifier/indexes.json index 680e8985..95c01af5 100644 --- a/projectgraph/testdata/typescript/js-specifier/indexes.json +++ b/projectgraph/testdata/typescript/js-specifier/indexes.json @@ -1,12 +1,12 @@ [ { - "version": "14", "path": "src/main.ts", "language": "typescript", + "version": "15", "path": "src/main.ts", "language": "typescript", "imports": [{"line": 1, "module": "./util.js", "item": "f", "alias": "f", "kind": "static", "namespace": ".//util.js//f"}], "definitions": [{"key": "src/main.ts", "kind": "module", "name": "main", "start_line": 1, "end_line": 2, "exported": true}], "calls": [{"caller": "src/main.ts", "line": 2, "column": 1, "callee": "f", "targets": [".//util.js//f"], "kind": "call"}] }, { - "version": "14", "path": "src/util.ts", "language": "typescript", + "version": "15", "path": "src/util.ts", "language": "typescript", "exports": [{"name": "f", "target": "src/util.ts//f", "line": 1}], "definitions": [ {"key": "src/util.ts", "kind": "module", "name": "util", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/typescript/tsconfig-paths/graph.golden.json b/projectgraph/testdata/typescript/tsconfig-paths/graph.golden.json index 81b607c7..63378f5d 100644 --- a/projectgraph/testdata/typescript/tsconfig-paths/graph.golden.json +++ b/projectgraph/testdata/typescript/tsconfig-paths/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "src/app/server.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/app/server.ts//serve": { "key": "src/app/server.ts//serve", @@ -19,7 +20,8 @@ "kind": "module", "path": "src/config.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/config.ts//cfg": { "key": "src/config.ts//cfg", @@ -33,7 +35,8 @@ "kind": "module", "path": "src/main.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" } }, "edges": [ @@ -138,12 +141,15 @@ "src/main.ts": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 5, "edges": 4, "unresolved": 0, "roots": 5, - "root_fallback": true + "root_fallback": true, + "languages": [ + "typescript" + ] } } diff --git a/projectgraph/testdata/typescript/tsconfig-paths/indexes.json b/projectgraph/testdata/typescript/tsconfig-paths/indexes.json index 8b8e5a16..4dd00b21 100644 --- a/projectgraph/testdata/typescript/tsconfig-paths/indexes.json +++ b/projectgraph/testdata/typescript/tsconfig-paths/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "src/main.ts", "language": "typescript", + "version": "15", "path": "src/main.ts", "language": "typescript", "imports": [ {"line": 1, "module": "@app/server", "item": "serve", "alias": "serve", "kind": "static", "namespace": "@app//server//serve"}, {"line": 2, "module": "src/config", "item": "cfg", "alias": "cfg", "kind": "static", "namespace": "src//config//cfg"} @@ -12,7 +12,7 @@ ] }, { - "version": "14", "path": "src/app/server.ts", "language": "typescript", + "version": "15", "path": "src/app/server.ts", "language": "typescript", "exports": [{"name": "serve", "target": "src/app/server.ts//serve", "line": 1}], "definitions": [ {"key": "src/app/server.ts", "kind": "module", "name": "server", "start_line": 1, "end_line": 1, "exported": true}, @@ -20,7 +20,7 @@ ] }, { - "version": "14", "path": "src/config.ts", "language": "typescript", + "version": "15", "path": "src/config.ts", "language": "typescript", "exports": [{"name": "cfg", "target": "src/config.ts//cfg", "line": 1}], "definitions": [ {"key": "src/config.ts", "kind": "module", "name": "config", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/projectgraph/testdata/typescript/type-only/graph.golden.json b/projectgraph/testdata/typescript/type-only/graph.golden.json index 327d10a3..a15569ab 100644 --- a/projectgraph/testdata/typescript/type-only/graph.golden.json +++ b/projectgraph/testdata/typescript/type-only/graph.golden.json @@ -5,7 +5,8 @@ "kind": "module", "path": "src/decl.d.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/decl.d.ts//decl": { "key": "src/decl.d.ts//decl", @@ -19,14 +20,16 @@ "kind": "module", "path": "src/main.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/types.ts": { "key": "src/types.ts", "kind": "module", "path": "src/types.ts", "line": 1, - "exported": true + "exported": true, + "language": "typescript" }, "src/types.ts//Opts": { "key": "src/types.ts//Opts", @@ -97,12 +100,15 @@ "src/types.ts": "indexed" }, "stats": { - "version": "16", + "version": "17", "files": 3, "nodes": 5, "edges": 2, "unresolved": 0, "roots": 3, - "root_fallback": true + "root_fallback": true, + "languages": [ + "typescript" + ] } } diff --git a/projectgraph/testdata/typescript/type-only/indexes.json b/projectgraph/testdata/typescript/type-only/indexes.json index 3dc37beb..6db64bb1 100644 --- a/projectgraph/testdata/typescript/type-only/indexes.json +++ b/projectgraph/testdata/typescript/type-only/indexes.json @@ -1,6 +1,6 @@ [ { - "version": "14", "path": "src/main.ts", "language": "typescript", + "version": "15", "path": "src/main.ts", "language": "typescript", "imports": [ {"line": 1, "module": "./types", "item": "Opts", "alias": "Opts", "kind": "type-only", "namespace": ".//types//Opts"}, {"line": 2, "module": "./decl", "item": "decl", "alias": "decl", "kind": "static", "namespace": ".//decl//decl"} @@ -9,7 +9,7 @@ "calls": [{"caller": "src/main.ts", "line": 3, "column": 1, "callee": "decl", "targets": [".//decl//decl"], "kind": "call"}] }, { - "version": "14", "path": "src/types.ts", "language": "typescript", + "version": "15", "path": "src/types.ts", "language": "typescript", "exports": [{"name": "Opts", "target": "src/types.ts//Opts", "line": 1}], "definitions": [ {"key": "src/types.ts", "kind": "module", "name": "types", "start_line": 1, "end_line": 1, "exported": true}, @@ -17,7 +17,7 @@ ] }, { - "version": "14", "path": "src/decl.d.ts", "language": "typescript", + "version": "15", "path": "src/decl.d.ts", "language": "typescript", "exports": [{"name": "decl", "target": "src/decl.d.ts//decl", "line": 1}], "definitions": [ {"key": "src/decl.d.ts", "kind": "module", "name": "decl", "start_line": 1, "end_line": 1, "exported": true}, diff --git a/test/projectgraph/bench_test.go b/test/projectgraph/bench_test.go index 1fa7ff84..351b8089 100644 --- a/test/projectgraph/bench_test.go +++ b/test/projectgraph/bench_test.go @@ -172,7 +172,7 @@ func benchProject(b *testing.B, code core.LanguageCode, files int) { require.Len(b, dependency.Indexes, written) var s *summary.Summary measure(b, written, func() any { - s, err = summary.Summarize(ctx, dependency.Indexes, projectgraph.Options{Files: dependency.Dependencies, Units: map[string]iofs.FS{unit: os.DirFS(root)}}) + s, err = summary.Summarize(ctx, dependency.Indexes, projectgraph.Options{Files: dependency.Dependencies, Languages: dependency.Languages, Units: map[string]iofs.FS{unit: os.DirFS(root)}}) require.NoError(b, err) return s }) @@ -200,7 +200,7 @@ func TestSummaryStatusMatchesReachOnASyntheticPackage(t *testing.T) { require.NoError(t, err) dependency := indexProject(t, unitFS, indexer.Options{Languages: []core.Language{language}, Workers: 4, Imports: true}) require.Len(t, dependency.Indexes, written) - r, err := summary.Build(ctx, dependency.Indexes, projectgraph.Options{Files: dependency.Dependencies, Units: map[string]iofs.FS{unit: os.DirFS(root)}}) + r, err := summary.Build(ctx, dependency.Indexes, projectgraph.Options{Files: dependency.Dependencies, Languages: dependency.Languages, Units: map[string]iofs.FS{unit: os.DirFS(root)}}) require.NoError(t, err) require.NotEmpty(t, r.Summary.Uncertainties, "the generated package has dynamic calls") summarytest.AssertStatusMatchesReach(t, r) diff --git a/test/projectgraph/cases/go/dependency-build-files/summaries/go_github.com_acme_kit_v1.0.0.json b/test/projectgraph/cases/go/dependency-build-files/summaries/go_github.com_acme_kit_v1.0.0.json index d75c8c99..3d125082 100644 --- a/test/projectgraph/cases/go/dependency-build-files/summaries/go_github.com_acme_kit_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-build-files/summaries/go_github.com_acme_kit_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/kit@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-generated-code/summaries/go_github.com_acme_kit_v1.0.0.json b/test/projectgraph/cases/go/dependency-generated-code/summaries/go_github.com_acme_kit_v1.0.0.json index bd84fe61..a4788f1c 100644 --- a/test/projectgraph/cases/go/dependency-generated-code/summaries/go_github.com_acme_kit_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-generated-code/summaries/go_github.com_acme_kit_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/kit@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-implements-interface/summaries/go_github.com_acme_dep_v1.0.0.json b/test/projectgraph/cases/go/dependency-implements-interface/summaries/go_github.com_acme_dep_v1.0.0.json index 69acb2cf..8ed13395 100644 --- a/test/projectgraph/cases/go/dependency-implements-interface/summaries/go_github.com_acme_dep_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-implements-interface/summaries/go_github.com_acme_dep_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/dep@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-internal-call/summaries/go_github.com_acme_kit_v1.0.0.json b/test/projectgraph/cases/go/dependency-internal-call/summaries/go_github.com_acme_kit_v1.0.0.json index e8235fda..92ceff86 100644 --- a/test/projectgraph/cases/go/dependency-internal-call/summaries/go_github.com_acme_kit_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-internal-call/summaries/go_github.com_acme_kit_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/kit@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-module/summaries/go_github.com_acme_yamlish_v1.2.0.json b/test/projectgraph/cases/go/dependency-module/summaries/go_github.com_acme_yamlish_v1.2.0.json index c27b2ee8..143f2e1d 100644 --- a/test/projectgraph/cases/go/dependency-module/summaries/go_github.com_acme_yamlish_v1.2.0.json +++ b/test/projectgraph/cases/go/dependency-module/summaries/go_github.com_acme_yamlish_v1.2.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/yamlish@v1.2.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-platform-files/summaries/go_github.com_acme_kit_v1.0.0.json b/test/projectgraph/cases/go/dependency-platform-files/summaries/go_github.com_acme_kit_v1.0.0.json index bd84fe61..a4788f1c 100644 --- a/test/projectgraph/cases/go/dependency-platform-files/summaries/go_github.com_acme_kit_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-platform-files/summaries/go_github.com_acme_kit_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/kit@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-platform-results/summaries/go_github.com_acme_kit_v1.0.0.json b/test/projectgraph/cases/go/dependency-platform-results/summaries/go_github.com_acme_kit_v1.0.0.json index bc6a5746..a43b032b 100644 --- a/test/projectgraph/cases/go/dependency-platform-results/summaries/go_github.com_acme_kit_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-platform-results/summaries/go_github.com_acme_kit_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:github.com/acme/kit@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-result-types/summaries/go_example.com_markup_v1.0.0.json b/test/projectgraph/cases/go/dependency-result-types/summaries/go_example.com_markup_v1.0.0.json index 3ecc19d6..0ad93c43 100644 --- a/test/projectgraph/cases/go/dependency-result-types/summaries/go_example.com_markup_v1.0.0.json +++ b/test/projectgraph/cases/go/dependency-result-types/summaries/go_example.com_markup_v1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:example.com/markup@v1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/go/dependency-result-types/summaries/go_std_go1.25.1.json b/test/projectgraph/cases/go/dependency-result-types/summaries/go_std_go1.25.1.json index a138a779..b3a973f0 100644 --- a/test/projectgraph/cases/go/dependency-result-types/summaries/go_std_go1.25.1.json +++ b/test/projectgraph/cases/go/dependency-result-types/summaries/go_std_go1.25.1.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "go:std@go1.25.1", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-candidates/summaries/npm_store-kit_1.0.0.json b/test/projectgraph/cases/javascript/dependency-candidates/summaries/npm_store-kit_1.0.0.json index 5e45d1ab..cd771458 100644 --- a/test/projectgraph/cases/javascript/dependency-candidates/summaries/npm_store-kit_1.0.0.json +++ b/test/projectgraph/cases/javascript/dependency-candidates/summaries/npm_store-kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:store-kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-exports-import/summaries/npm_kit_1.0.0.json b/test/projectgraph/cases/javascript/dependency-exports-import/summaries/npm_kit_1.0.0.json index c094a670..2c39dae7 100644 --- a/test/projectgraph/cases/javascript/dependency-exports-import/summaries/npm_kit_1.0.0.json +++ b/test/projectgraph/cases/javascript/dependency-exports-import/summaries/npm_kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-exports-require/summaries/npm_kit_1.0.0.json b/test/projectgraph/cases/javascript/dependency-exports-require/summaries/npm_kit_1.0.0.json index c094a670..2c39dae7 100644 --- a/test/projectgraph/cases/javascript/dependency-exports-require/summaries/npm_kit_1.0.0.json +++ b/test/projectgraph/cases/javascript/dependency-exports-require/summaries/npm_kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-method-name/summaries/npm_plug_1.0.0.json b/test/projectgraph/cases/javascript/dependency-method-name/summaries/npm_plug_1.0.0.json index 3840a85b..3d525bbd 100644 --- a/test/projectgraph/cases/javascript/dependency-method-name/summaries/npm_plug_1.0.0.json +++ b/test/projectgraph/cases/javascript/dependency-method-name/summaries/npm_plug_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:plug@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-same-path/summaries/npm_util-kit_1.0.0.json b/test/projectgraph/cases/javascript/dependency-same-path/summaries/npm_util-kit_1.0.0.json index 0ee53d7c..0839f61c 100644 --- a/test/projectgraph/cases/javascript/dependency-same-path/summaries/npm_util-kit_1.0.0.json +++ b/test/projectgraph/cases/javascript/dependency-same-path/summaries/npm_util-kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:util-kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash._reinterpolate_3.0.0.json b/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash._reinterpolate_3.0.0.json index 7d16d7d1..3265721f 100644 --- a/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash._reinterpolate_3.0.0.json +++ b/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash._reinterpolate_3.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:lodash._reinterpolate@3.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash_4.17.21.json b/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash_4.17.21.json index e177f536..766e9a82 100644 --- a/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash_4.17.21.json +++ b/test/projectgraph/cases/javascript/dependency-transitive/summaries/npm_lodash_4.17.21.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "npm:lodash@4.17.21", "symbols": [ { diff --git a/test/projectgraph/cases/python/bundled-javascript/expect.yaml b/test/projectgraph/cases/python/bundled-javascript/expect.yaml new file mode 100644 index 00000000..1fd8896d --- /dev/null +++ b/test/projectgraph/cases/python/bundled-javascript/expect.yaml @@ -0,0 +1,41 @@ +source: safedep:hand +description: >- + A Python project ships the JavaScript of a web page, and the index step fails static/vendor.min.js, which + is over the size limit of the case. The roots of the graph are in Python files alone, so no program of the + project runs JavaScript, and the failed file holds no caller that a search can reach. Its budget concerns + no target, and os.system is not found. Compare failed-file, where the failed file is Python. +max_file_bytes: 200 +failed: + - static/vendor.min.js +roots: + - key: main.py + kind: main + rule: python.main-guard +exact_edges: true +edges: + - from: main.py + to: yaml//load + kind: direct + confidence: high + site: main.py:4 + - from: main.py + to: open + kind: direct + confidence: high + site: main.py:4 +unresolved: [] +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: yaml//load + site: main.py:4 + kind: direct + confidence: high + call_sites: + - caller: main.py + site: main.py:4 + reached: true + - target: os//system + status: no-such-target diff --git a/test/projectgraph/cases/python/bundled-javascript/project/main.py b/test/projectgraph/cases/python/bundled-javascript/project/main.py new file mode 100644 index 00000000..b7dd1507 --- /dev/null +++ b/test/projectgraph/cases/python/bundled-javascript/project/main.py @@ -0,0 +1,4 @@ +import yaml + +if __name__ == "__main__": + yaml.load(open("c.yaml")) diff --git a/test/projectgraph/cases/python/bundled-javascript/project/static/vendor.min.js b/test/projectgraph/cases/python/bundled-javascript/project/static/vendor.min.js new file mode 100644 index 00000000..1d4bb67e --- /dev/null +++ b/test/projectgraph/cases/python/bundled-javascript/project/static/vendor.min.js @@ -0,0 +1,12 @@ +function f0(a){return a+0} +function f1(a){return a+1} +function f2(a){return a+2} +function f3(a){return a+3} +function f4(a){return a+4} +function f5(a){return a+5} +function f6(a){return a+6} +function f7(a){return a+7} +function f8(a){return a+8} +function f9(a){return a+9} +function f10(a){return a+10} +function f11(a){return a+11} diff --git a/test/projectgraph/cases/python/decorator-computed/expect.yaml b/test/projectgraph/cases/python/decorator-computed/expect.yaml new file mode 100644 index 00000000..cc01d569 --- /dev/null +++ b/test/projectgraph/cases/python/decorator-computed/expect.yaml @@ -0,0 +1,21 @@ +source: safedep:hand +description: >- + The decorator of run is a lookup in a dict by a name, so the graph cannot name the code that wraps run. The + index flags the decorator as decorator-wrap, and the flag concerns every target, so os.system is uncertain. +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py//run + to: yaml//load + kind: direct + confidence: high + site: main.py:9 +unresolved: [] +results: + - target: os//system + status: uncertain + uncertainties: + - kind: dynamic-flag + site: main.py:7 diff --git a/test/projectgraph/cases/python/decorator-computed/project/main.py b/test/projectgraph/cases/python/decorator-computed/project/main.py new file mode 100644 index 00000000..6b73e595 --- /dev/null +++ b/test/projectgraph/cases/python/decorator-computed/project/main.py @@ -0,0 +1,13 @@ +import yaml + +handlers = {"log": print} +name = "log" + + +@handlers[name] +def run(): + yaml.load("a: 1") + + +if __name__ == "__main__": + run() diff --git a/test/projectgraph/cases/python/decorator-factory/expect.yaml b/test/projectgraph/cases/python/decorator-factory/expect.yaml new file mode 100644 index 00000000..d3455f6b --- /dev/null +++ b/test/projectgraph/cases/python/decorator-factory/expect.yaml @@ -0,0 +1,60 @@ +source: safedep:hand +description: >- + run has three decorators: a call of click.command, the first-party verbose_option, and a call of + click.option. verbose_option returns the value of click.option(...) applied to f. Each decorator names its + code, so none is a dynamic feature, and the call of the value of click.option(...) is no record of its own, + because the graph reaches the code of click through the call of click.option. The module top level runs + each decorator when it defines run. os.system is no such target, which only holds with no record and no + flag that concerns every target. +roots: + - key: main.py + kind: main + rule: python.main-guard + - key: main.py//run + kind: framework + rule: python.click.command +edges: + - from: main.py + to: click//command + kind: direct + confidence: high + site: main.py:9 + - from: main.py + to: main.py//verbose_option + kind: direct + confidence: high + site: main.py:12 + - from: main.py + to: click//option + kind: direct + confidence: high + site: main.py:11 + - from: main.py//verbose_option + to: click//option + kind: direct + confidence: high + site: main.py:6 + - from: main.py//run + to: yaml//load + kind: direct + confidence: high + site: main.py:13 +no_edges: + - from: main.py + to: os//system +unresolved: [] +results: + - target: yaml//load + status: found + path: + - key: main.py//run + - key: yaml//load + site: main.py:13 + kind: direct + confidence: high + call_sites: + - caller: main.py//run + site: main.py:13 + reached: true + - target: os//system + status: no-such-target diff --git a/test/projectgraph/cases/python/decorator-factory/project/main.py b/test/projectgraph/cases/python/decorator-factory/project/main.py new file mode 100644 index 00000000..f168dd6e --- /dev/null +++ b/test/projectgraph/cases/python/decorator-factory/project/main.py @@ -0,0 +1,17 @@ +import click +import yaml + + +def verbose_option(f): + return click.option("-v", "--verbose")(f) + + +@click.command() +@verbose_option +@click.option("-c", "--config") +def run(config): + yaml.load(config) + + +if __name__ == "__main__": + run() diff --git a/test/projectgraph/cases/python/dependency-dark-unit/summaries/pypi_PyYAML_6.0.1.json b/test/projectgraph/cases/python/dependency-dark-unit/summaries/pypi_PyYAML_6.0.1.json index 5457ea6f..3f50d6a1 100644 --- a/test/projectgraph/cases/python/dependency-dark-unit/summaries/pypi_PyYAML_6.0.1.json +++ b/test/projectgraph/cases/python/dependency-dark-unit/summaries/pypi_PyYAML_6.0.1.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:PyYAML@6.0.1", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_badpkg_1.0.0.json b/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_badpkg_1.0.0.json index fb669126..c1762ee2 100644 --- a/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_badpkg_1.0.0.json +++ b/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_badpkg_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:badpkg@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_okpkg_1.0.0.json b/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_okpkg_1.0.0.json index 39710708..75940477 100644 --- a/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_okpkg_1.0.0.json +++ b/test/projectgraph/cases/python/dependency-failed-reached/summaries/pypi_okpkg_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:okpkg@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_badpkg_1.0.0.json b/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_badpkg_1.0.0.json index fb669126..c1762ee2 100644 --- a/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_badpkg_1.0.0.json +++ b/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_badpkg_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:badpkg@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_okpkg_1.0.0.json b/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_okpkg_1.0.0.json index 39710708..75940477 100644 --- a/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_okpkg_1.0.0.json +++ b/test/projectgraph/cases/python/dependency-failed-unreached/summaries/pypi_okpkg_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:okpkg@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-package-dir/summaries/pypi_outer_1.0.0.json b/test/projectgraph/cases/python/dependency-package-dir/summaries/pypi_outer_1.0.0.json index 060315ed..4d777061 100644 --- a/test/projectgraph/cases/python/dependency-package-dir/summaries/pypi_outer_1.0.0.json +++ b/test/projectgraph/cases/python/dependency-package-dir/summaries/pypi_outer_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:outer@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_requests_2.31.0.json b/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_requests_2.31.0.json index 064e485d..9f426c70 100644 --- a/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_requests_2.31.0.json +++ b/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_requests_2.31.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:requests@2.31.0", "symbols": [ { diff --git a/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_urllib3_2.0.7.json b/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_urllib3_2.0.7.json index 567fa307..620b0a77 100644 --- a/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_urllib3_2.0.7.json +++ b/test/projectgraph/cases/python/dependency-site-packages/summaries/pypi_urllib3_2.0.7.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "pypi:urllib3@2.0.7", "symbols": [ { diff --git a/test/projectgraph/cases/python/flask-route/expect.yaml b/test/projectgraph/cases/python/flask-route/expect.yaml index b4f09aae..ff70dd6b 100644 --- a/test/projectgraph/cases/python/flask-route/expect.yaml +++ b/test/projectgraph/cases/python/flask-route/expect.yaml @@ -1,5 +1,5 @@ source: safedep:hand -description: api is a Flask app under another name. The decorator @api.route resolves to flask//Flask//route, so config is a root of the rule python.flask.route. helper has no route. The decorator call of config can wrap the function, so the answer for yaml.load is uncertain and not a dead call. +description: api is a Flask app under another name. The decorator @api.route resolves to flask//Flask//route, so config is a root of the rule python.flask.route. helper has no route and no caller. The decorator of config names its code, so it is no dynamic feature, and yaml.load in helper is a dead call. roots: - key: app.py//config kind: framework @@ -26,11 +26,8 @@ results: site: app.py:9 reached: true - target: yaml//load - status: uncertain + status: not-found call_sites: - caller: app.py//helper site: app.py:13 reached: false - uncertainties: - - kind: dynamic-flag - site: app.py:7 diff --git a/test/projectgraph/cases/python/stdlib-name-deep/expect.yaml b/test/projectgraph/cases/python/stdlib-name-deep/expect.yaml new file mode 100644 index 00000000..027e8cd0 --- /dev/null +++ b/test/projectgraph/cases/python/stdlib-name-deep/expect.yaml @@ -0,0 +1,51 @@ +source: safedep:hand +description: >- + main.py imports types, a module of the standard library, and the project has a file app/models/types.py. An + absolute import resolves against the import roots and the directory of the importer, so the deep file is no + module named types, and the import is external and no record. os.system is then no such target. Compare + missing-module, where the package of the missing module is at a root. +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py + to: app/models/types.py + kind: import + confidence: high + site: main.py:5 + - from: main.py//run + to: types//SimpleNamespace + kind: direct + confidence: high + site: main.py:9 + - from: main.py//run + to: app/models/types.py//Record + kind: direct + confidence: high + site: main.py:10 + - from: main.py//run + to: yaml//load + kind: direct + confidence: high + site: main.py:10 +unresolved: [] +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: main.py//run + site: main.py:14 + kind: direct + confidence: high + - key: yaml//load + site: main.py:10 + kind: direct + confidence: high + call_sites: + - caller: main.py//run + site: main.py:10 + reached: true + - target: os//system + status: no-such-target diff --git a/test/projectgraph/cases/python/stdlib-name-deep/project/app/__init__.py b/test/projectgraph/cases/python/stdlib-name-deep/project/app/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/projectgraph/cases/python/stdlib-name-deep/project/app/models/__init__.py b/test/projectgraph/cases/python/stdlib-name-deep/project/app/models/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/projectgraph/cases/python/stdlib-name-deep/project/app/models/types.py b/test/projectgraph/cases/python/stdlib-name-deep/project/app/models/types.py new file mode 100644 index 00000000..0fecfb87 --- /dev/null +++ b/test/projectgraph/cases/python/stdlib-name-deep/project/app/models/types.py @@ -0,0 +1,3 @@ +class Record: + def __init__(self, ns): + self.text = str(ns) diff --git a/test/projectgraph/cases/python/stdlib-name-deep/project/main.py b/test/projectgraph/cases/python/stdlib-name-deep/project/main.py new file mode 100644 index 00000000..12219a5d --- /dev/null +++ b/test/projectgraph/cases/python/stdlib-name-deep/project/main.py @@ -0,0 +1,14 @@ +import types + +import yaml + +from app.models.types import Record + + +def run(): + ns = types.SimpleNamespace(a=1) + return yaml.load(Record(ns).text) + + +if __name__ == "__main__": + run() diff --git a/test/projectgraph/cases/python/truncated-test-file/expect.yaml b/test/projectgraph/cases/python/truncated-test-file/expect.yaml new file mode 100644 index 00000000..6ede35f8 --- /dev/null +++ b/test/projectgraph/cases/python/truncated-test-file/expect.yaml @@ -0,0 +1,37 @@ +source: safedep:hand +description: >- + The index step fails tests/test_big.py, which is over the size limit of the case. The file is a test file, + so it holds only test roots, and its budget counts only for a search that includes the tests. Without the + tests, os.system is no such target. With them, the failed file is a budget and os.system is uncertain. A + failed file has no index, so the test-path rule marks no root in it. +max_file_bytes: 200 +test_paths: + - tests +failed: + - tests/test_big.py +roots: + - key: main.py + kind: main + rule: python.main-guard +exact_edges: true +edges: + - from: main.py + to: yaml//load + kind: direct + confidence: high + site: main.py:4 + - from: main.py + to: open + kind: direct + confidence: high + site: main.py:4 +unresolved: [] +results: + - target: os//system + status: no-such-target + - target: os//system + include_tests: true + status: uncertain + uncertainties: + - kind: budget + site: tests/test_big.py diff --git a/test/projectgraph/cases/python/truncated-test-file/project/main.py b/test/projectgraph/cases/python/truncated-test-file/project/main.py new file mode 100644 index 00000000..b7dd1507 --- /dev/null +++ b/test/projectgraph/cases/python/truncated-test-file/project/main.py @@ -0,0 +1,4 @@ +import yaml + +if __name__ == "__main__": + yaml.load(open("c.yaml")) diff --git a/test/projectgraph/cases/python/truncated-test-file/project/tests/test_big.py b/test/projectgraph/cases/python/truncated-test-file/project/tests/test_big.py new file mode 100644 index 00000000..757714c6 --- /dev/null +++ b/test/projectgraph/cases/python/truncated-test-file/project/tests/test_big.py @@ -0,0 +1,35 @@ +import main + + +def test_0(): + assert main.yaml is not None + + +def test_1(): + assert main.yaml is not None + + +def test_2(): + assert main.yaml is not None + + +def test_3(): + assert main.yaml is not None + + +def test_4(): + assert main.yaml is not None + + +def test_5(): + assert main.yaml is not None + + +def test_6(): + assert main.yaml is not None + + +def test_7(): + assert main.yaml is not None + + diff --git a/test/projectgraph/cases/rust/dependency-crate/summaries/cargo_kit_1.0.0.json b/test/projectgraph/cases/rust/dependency-crate/summaries/cargo_kit_1.0.0.json index 4bc0b860..4991c200 100644 --- a/test/projectgraph/cases/rust/dependency-crate/summaries/cargo_kit_1.0.0.json +++ b/test/projectgraph/cases/rust/dependency-crate/summaries/cargo_kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "cargo:kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/rust/dependency-internal-call/summaries/cargo_kit_1.0.0.json b/test/projectgraph/cases/rust/dependency-internal-call/summaries/cargo_kit_1.0.0.json index d7c8d300..e43ee48b 100644 --- a/test/projectgraph/cases/rust/dependency-internal-call/summaries/cargo_kit_1.0.0.json +++ b/test/projectgraph/cases/rust/dependency-internal-call/summaries/cargo_kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "cargo:kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/rust/dependency-renamed/summaries/cargo_kit_1.0.0.json b/test/projectgraph/cases/rust/dependency-renamed/summaries/cargo_kit_1.0.0.json index d7c8d300..e43ee48b 100644 --- a/test/projectgraph/cases/rust/dependency-renamed/summaries/cargo_kit_1.0.0.json +++ b/test/projectgraph/cases/rust/dependency-renamed/summaries/cargo_kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "cargo:kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases/rust/library-build-script/summaries/cargo_kit_1.0.0.json b/test/projectgraph/cases/rust/library-build-script/summaries/cargo_kit_1.0.0.json index aa9775d4..1f075b50 100644 --- a/test/projectgraph/cases/rust/library-build-script/summaries/cargo_kit_1.0.0.json +++ b/test/projectgraph/cases/rust/library-build-script/summaries/cargo_kit_1.0.0.json @@ -1,7 +1,7 @@ { "version": "2", - "index_version": "14", - "link_version": "16", + "index_version": "15", + "link_version": "17", "unit": "cargo:kit@1.0.0", "symbols": [ { diff --git a/test/projectgraph/cases_test.go b/test/projectgraph/cases_test.go index 70b001e2..2ece6fec 100644 --- a/test/projectgraph/cases_test.go +++ b/test/projectgraph/cases_test.go @@ -301,7 +301,7 @@ func dumpCase(t *testing.T, dir string, g *projectgraph.Graph, results []caseRes func checkSummary(t *testing.T, dir, unit string, indexed *indexer.Result, roots map[string]iofs.FS) { t.Helper() indexes, files := indexed.Unit(unit) - r, err := summary.Build(context.Background(), indexes, projectgraph.Options{Files: files, Units: map[string]iofs.FS{unit: roots[unit]}}) + r, err := summary.Build(context.Background(), indexes, projectgraph.Options{Files: files, Languages: indexed.Languages, Units: map[string]iofs.FS{unit: roots[unit]}}) require.NoError(t, err) testutil.AssertGolden(t, filepath.Join(dir, strings.NewReplacer(":", "_", "/", "_", "@", "_").Replace(unit)+".json"), *update, r.Summary) summarytest.AssertStatusMatchesReach(t, r) diff --git a/test/projectgraph/helpers_test.go b/test/projectgraph/helpers_test.go index c5e19f28..7af3ebca 100644 --- a/test/projectgraph/helpers_test.go +++ b/test/projectgraph/helpers_test.go @@ -42,7 +42,7 @@ func targetsOf(t testing.TB, keys ...string) []reach.Target { // go to the link step, so a failed file of either kind is a budget cause. func linkProject(t testing.TB, root string, indexed *indexer.Result, opts projectgraph.Options) *projectgraph.Graph { t.Helper() - opts.Root, opts.Files = os.DirFS(root), indexed.Statuses() + opts.Root, opts.Files, opts.Languages = os.DirFS(root), indexed.Statuses(), indexed.Languages g, err := projectgraph.Build(context.Background(), indexed.Indexes, opts) require.NoError(t, err) return g diff --git a/test/projectgraph/report.md b/test/projectgraph/report.md index d261ee42..f93272f5 100644 --- a/test/projectgraph/report.md +++ b/test/projectgraph/report.md @@ -1150,3 +1150,59 @@ project or a pypi unit ships for a web page is no code of the program, a truncated test file concerns only a search that counts the tests, and a decorator that a framework rule knows, as a route or a command, wraps the function in code that the rule already reads. + +#### After the four rules of the budget, the decorators and the imports + +The commit after the baseline adds four rules. `docs/projectgraph.md` +holds each one. + +1. An incomplete file counts for a search only when the program runs its + language: a root other than a test root is in a file of that language, + or the ecosystem of its unit installs that language. `Graph.Runs` + decides it, from `Node.Language` and `Stats.Languages`. +2. An incomplete test file counts only for a search that includes the + tests. +3. A decorator is a dynamic feature only when the graph cannot name its + code, as `@handlers[name]`. A decorator that is a name, a dotted name + or a call of one is a call of the scope that defines the function. A + call of the value of a call, as `click.option("-v")(f)`, is no record + of its own, unless the inner call reflects. +4. A Python import whose first segment is no module under an import root + or in the directory of the importer is external, not a record. Before, + `import types` in cryptography was a record because a `types.py` is + deep in a package of it, and `from collections import defaultdict` in + Django for the same reason. Each such record concerned every target. + +The run of the two languages gives the same numbers as the run of +`python` alone now, so the table has one row per kind. + +| Language | Kind | Run | Targets | found | not-found | no-such-target | unknown-receiver | no-definition | ambiguous | computed-name | dynamic-flag | budget | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Python | function | four rules | 44 | 5 | 0 | 0 | 3 | 2 | 14 | 8 | 11 | 1 | +| Python | method | four rules | 63 | 0 | 0 | 0 | 27 | 6 | 14 | 4 | 8 | 4 | + +- `budget` falls from 47 to 5 targets, all of imageio: `_tifffile.py` is a + module of 10,182 lines that the index truncates, and it is Python code + that the program runs. +- `dynamic-flag` falls from 19 to 19 targets in the first run and from 31 + to 19 in the run of `python` alone. What is left is real: `getattr` on a + computed name in flask-cors, fabric and pyopenssl, and the monkey patch + of python-docx. +- `no-definition` falls from 31 to 8 in the run of `python` alone. The + rest is `.packages.six.moves.http_client` in urllib3, a module that six + makes at run time, and three imports of imageio. +- `ambiguous` and `computed-name` are the first cause now that nothing + hides them. Django has `template.render` on an unknown receiver with + more than 20 candidate methods, and a definition named `render` reaches + most of Django, so the record concerns every target of the corpus that + Django code reaches. mkdocs has `config['plugins'].run_event`, a call on + a subscript, which computes its name. +- `found` stays at 5 and `unknown-receiver` at 27 of 63 methods. The + rules remove uncertainty that hid these causes. They add no path, as + the type-hint step will. +- pycg-micro keeps its precision of 0.954 and its recall of 0.708. Its + unresolved rate falls from 0.340 to 0.312, because an import whose first + segment names a file deep in a case is external now. govulncheck stays + at 9 of 9 stacks. +- The receiver table does not change. The 16 Python rows still wait on + the type-hint step. diff --git a/test/projectgraph/version_test.go b/test/projectgraph/version_test.go index 4fd1d2e5..5a7506ff 100644 --- a/test/projectgraph/version_test.go +++ b/test/projectgraph/version_test.go @@ -31,6 +31,7 @@ var graphHashes = map[string]string{ "14/14": "0f3c9c38b39f4de1835f34e8a58bcea85461f37e7aa2d5a2dd484569e670ff48", "14/15": "bbc708aba767ccf554c75a7e06816abda3fc1804ad6d0876ad8ce9575f4c3a88", "14/16": "823b1f82c4aed8ba27d19275505c8d1dee755086be48fcda64d323f55b4c62a5", + "15/17": "f2a715762dee048c50d5a6d7201adf462c89025d6b7f92632f4f85fc83275860", } // resultHashes records the hash of the reach results of every case for @@ -46,6 +47,7 @@ var resultHashes = map[string]string{ "14/14/2": "a832ea709bf3df852814375a56e6329c35e2ff4c49ae34ee49afe042e9502d17", "14/15/3": "77dfd0ea8ab71869ef612d41b8c2e46852d4fb67a49458cb645321692af77ea0", "14/16/3": "13b6eecb686ddec3c2ba470fc9752ea66cff445d5e18e0772a71b15ae7fe1c9c", + "15/17/4": "84e2c97bb8b24d95a2bbc908c8a2d4352f6270d103fcad3cacc165085c9a845d", } // summaryHashes records the hash of the golden summaries of the cases for @@ -62,6 +64,7 @@ var summaryHashes = map[string]string{ "14/14/2": "27424c4e2525450479faba78421959738748811d70b9a93b81e2bbdf1733bf0d", "14/15/2": "5969f3cda15b666cbd902bbc2db953cbdef8c0ce48d5e77446cf243cdd0bb0e7", "14/16/2": "fb678202978212aae932e50a56a0d3dc7f0f794ce908988bf605ce9a6e04bc51", + "15/17/2": "d9430fbc80d91e922184db91241eb22c29725b43ac88bfb5dd6f99548e7ed43f", } func TestGraphHashFollowsVersion(t *testing.T) { From e5d11366d0fcb97f18c272910ca453a70c5a9531 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 06:40:15 +0000 Subject: [PATCH 03/11] test(projectgraph): fix the fixture lines and the golden rows of the rules commit The rules commit went out with three tests red: the module-getattr line of the Python flag fixture moved with the new decorator, the index goldens had no hash row for fileindex.Version 15, and the golden graph of a project with test roots alone still listed python as a program language. go test ./... passes with exit 0 now. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- plugin/indexer/flags_test.go | 6 +++--- plugin/indexer/plugin_test.go | 1 + projectgraph/testdata/roots/test-file/graph.golden.json | 5 +---- 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/plugin/indexer/flags_test.go b/plugin/indexer/flags_test.go index 7caa1d07..1a1b77c2 100644 --- a/plugin/indexer/flags_test.go +++ b/plugin/indexer/flags_test.go @@ -62,8 +62,8 @@ func TestDynamicSites(t *testing.T) { {Kind: fileindex.FlagComputedAttribute, Line: 17}, {Kind: fileindex.FlagMonkeyPatch, Line: 18}, {Kind: fileindex.FlagDecoratorWrap, Line: 31}, - {Kind: fileindex.FlagModuleGetattr, Line: 37}, - {Kind: fileindex.FlagParseError, Line: 41}, + {Kind: fileindex.FlagModuleGetattr, Line: 36}, + {Kind: fileindex.FlagParseError, Line: 40}, }, nil}, } for _, tc := range cases { @@ -100,7 +100,7 @@ func TestFlagLandsOnTheEnclosingDefinition(t *testing.T) { assert.Empty(t, flags["flags.py//plain"]) assert.Contains(t, ix.Flags, fileindex.Flag{Kind: fileindex.FlagEval, Line: 13}) - getattr := fileindex.Flag{Kind: fileindex.FlagModuleGetattr, Line: 31} + getattr := fileindex.Flag{Kind: fileindex.FlagModuleGetattr, Line: 36} assert.Contains(t, flags["flags.py"], getattr, "a module __getattr__ flags the module") assert.Empty(t, flags["flags.py//__getattr__"]) assert.Contains(t, ix.Flags, getattr) diff --git a/plugin/indexer/plugin_test.go b/plugin/indexer/plugin_test.go index 175c5883..45e06271 100644 --- a/plugin/indexer/plugin_test.go +++ b/plugin/indexer/plugin_test.go @@ -103,6 +103,7 @@ var goldenHashes = map[string]string{ "12": "974943df4185f9c3070dcfdfedff8e91912e257d255a936f7c27608bb63a69f0", "13": "97614d8b96782e65813ee409e4a3f33c4655157b5dfa58baf17cd3677958c619", "14": "26f22c80fd4120cd04f228a53188accb1695434d0ed1ff5654b110981f8f44dd", + "15": "327ac05cb53e44b88a8e5e0185bdd1fb32ac0cdb9e8a8eb27b9804843bdca48d", } func TestFileIndexGoldenHashFollowsVersion(t *testing.T) { diff --git a/projectgraph/testdata/roots/test-file/graph.golden.json b/projectgraph/testdata/roots/test-file/graph.golden.json index 3fa5e725..6453dc54 100644 --- a/projectgraph/testdata/roots/test-file/graph.golden.json +++ b/projectgraph/testdata/roots/test-file/graph.golden.json @@ -75,9 +75,6 @@ "edges": 0, "unresolved": 0, "roots": 3, - "root_fallback": true, - "languages": [ - "python" - ] + "root_fallback": true } } From 9494392f0bc1a49a4222d144baa57c3ac2d71789 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 07:55:40 +0000 Subject: [PATCH 04/11] feat(projectgraph): read the .dist-info of a pypi unit and give its definitions external keys A pypi unit knew only the files that the index step read. The link step named its import name from the package name, so PyYAML gave no yaml, an extension module was an external edge that hid its code, every test file of the package was in the graph, and a target that only the code of the package called matched nothing. modules.PythonDistributions reads the .dist-info directories at the root of a unit, as a site-packages directory or an unpacked wheel holds them: the name and the version of METADATA, the import names of top_level.txt or RECORD, the files of RECORD and its extension modules. The link step matches the one of the unit by the normalized name and gives the Python resolver its import names and its extension modules. An import of an extension module is a record with the reason no-dependency-source. The Python profile gets the symbol rule that Go and Rust have, so each definition of a unit carries the key that an importer of its module writes. The index step leaves out the test files and the wheel metadata of a pypi unit, as it leaves out the test files of a Go unit. examples/projectgraph takes -site-packages dir and adds each distribution of the directory as a unit, by the roots that its RECORD names, so a namespace package that two distributions share and a single-file module both get their units. The measure and the receiver rows say found/low for a path with a hop of low confidence. The external keys move 54 of 107 Python targets from uncertain to found/low, and unknown-receiver falls from 27 to 4 of 63 methods. report.md has the table. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- docs/projectgraph.md | 30 +++ examples/projectgraph/main.go | 40 +++ examples/projectgraph/main_test.go | 11 + plugin/indexer/indexer.go | 10 +- plugin/indexer/pyunit.go | 33 +++ plugin/indexer/pyunit_test.go | 60 +++++ projectgraph/imports.go | 46 +++- projectgraph/modules/pydist.go | 234 ++++++++++++++++++ projectgraph/modules/pydist_test.go | 60 +++++ projectgraph/modules/python.go | 39 +++ projectgraph/modules/resolver.go | 9 +- projectgraph/packages.go | 67 ++++- projectgraph/python.go | 46 +++- test/projectgraph/README.md | 4 +- .../dependency-compiled-module/expect.yaml | 66 +++++ .../project/main.py | 4 + .../speedy-1.0.0.dist-info/METADATA | 3 + .../speedy-1.0.0.dist-info/RECORD | 5 + .../venv/site-packages/speedy/__init__.py | 4 + .../project/venv/site-packages/speedy/pure.py | 5 + .../summaries/pypi_speedy_1.0.0.json | 40 +++ .../dependency-record-import-name/expect.yaml | 31 +++ .../project/main.py | 4 + .../PyYAML-6.0.1.dist-info/METADATA | 3 + .../PyYAML-6.0.1.dist-info/RECORD | 3 + .../venv/site-packages/yaml/__init__.py | 5 + .../summaries/pypi_PyYAML_6.0.1.json | 20 ++ .../dependency-site-packages/expect.yaml | 34 +++ test/projectgraph/receivers_test.go | 43 ++-- test/projectgraph/report.md | 50 ++++ test/projectgraph/version_test.go | 6 +- 31 files changed, 959 insertions(+), 56 deletions(-) create mode 100644 plugin/indexer/pyunit.go create mode 100644 plugin/indexer/pyunit_test.go create mode 100644 projectgraph/modules/pydist.go create mode 100644 projectgraph/modules/pydist_test.go create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/expect.yaml create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/project/main.py create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/METADATA create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/RECORD create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/__init__.py create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/pure.py create mode 100644 test/projectgraph/cases/python/dependency-compiled-module/summaries/pypi_speedy_1.0.0.json create mode 100644 test/projectgraph/cases/python/dependency-record-import-name/expect.yaml create mode 100644 test/projectgraph/cases/python/dependency-record-import-name/project/main.py create mode 100644 test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/METADATA create mode 100644 test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/RECORD create mode 100644 test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/yaml/__init__.py create mode 100644 test/projectgraph/cases/python/dependency-record-import-name/summaries/pypi_PyYAML_6.0.1.json diff --git a/docs/projectgraph.md b/docs/projectgraph.md index 0d0359c3..25d63c23 100644 --- a/docs/projectgraph.md +++ b/docs/projectgraph.md @@ -229,6 +229,36 @@ The standard library is a unit too. Pass `GOROOT/src` as the unit each import path whose first element has no dot, as `net/http`. The index step leaves out the `cmd` module of it, which no program imports. +A `pypi:` unit is a distribution, as a site-packages directory or an +unpacked wheel holds it. These rules apply to it. + +- The root of the unit is the directory that holds its code, so a path + inside the unit is the path that Python imports, as `yaml/__init__.py`. + A unit whose root is its package directory, with an `__init__.py` at the + root, is that package. +- The link step reads the `.dist-info` directory of the package at the + root, when there is one, and matches it by the normalized name. The + `top_level.txt` or the `RECORD` gives the import names, so `PyYAML` + gives `yaml` and a distribution with several top-level packages gives + each one. Without it, the import name is the package name in lower + case, with `_` for `-` and `.`. +- The `RECORD` lists the extension modules, as `yaml/_yaml.so`. Such a + module has no Python source, so an import of it is a record with the + reason `no-dependency-source`, and the record concerns every target + where a search reaches the import. +- A definition of the unit carries the external key that an importer of + its module writes, as `requests//api//get` for `requests/api.py//get`, + so a target of a module of the package matches the definition also when + only the code of the package calls it. +- The index step leaves out the test files of the unit: a file under a + `tests` directory, a `test_*.py` or `*_test.py` file, a `conftest.py`, + and the files of the `.dist-info`, `.egg-info` and `.data` directories + of the wheel. A `test` or a `testing` directory stays, because + `django.test` and `numpy.testing` are modules that programs import. + +`examples/projectgraph` takes `-site-packages ` and adds each +distribution of the directory as a unit, by its `.dist-info`. + ## Overlay An overlay is a file of `llm` edges that resolve unresolved sites of a diff --git a/examples/projectgraph/main.go b/examples/projectgraph/main.go index 1e0da7a0..9a21f28d 100644 --- a/examples/projectgraph/main.go +++ b/examples/projectgraph/main.go @@ -17,6 +17,11 @@ // too, as -deps $(go env GOROOT)/src=go:std@go1.25.1, and so is a Rust // crate, as -deps vendor/smallvec=cargo:smallvec@1.13.2. With -summary it // prints the summary of each such unit as JSON. +// +// -site-packages dir reads the .dist-info of each distribution in a Python +// site-packages directory and adds each one as a unit, with the import +// names that its RECORD lists. It stands for one -import-dir and one -deps +// per distribution. package main import ( @@ -41,6 +46,7 @@ import ( "github.com/safedep/code/lang" "github.com/safedep/code/plugin/indexer" "github.com/safedep/code/projectgraph" + "github.com/safedep/code/projectgraph/modules" "github.com/safedep/code/projectgraph/overlay" "github.com/safedep/code/projectgraph/reach" "github.com/safedep/code/projectgraph/summary" @@ -111,6 +117,35 @@ func (l *sourceList) Set(value string) error { return nil } +// addSitePackages adds each distribution of each -site-packages directory +// to the -deps and the import directories. A distribution with no Python +// code, as one that installs only data, adds no unit. The .dist-info of a +// distribution names the directories and the files that hold its code, so +// a distribution whose import name differs from its package name, as yaml +// for PyYAML, and a namespace package that several distributions share +// both get their units. +func addSitePackages(cfg config, importDirs *[]string) error { + for _, dir := range cfg.sitePackages { + abs, err := filepath.Abs(dir) + if err != nil { + return err + } + dists, problems := modules.PythonDistributions(os.DirFS(abs)) + for _, p := range problems { + if _, err := fmt.Fprintln(os.Stderr, "site-packages:", p.Path, p.Text); err != nil { + return err + } + } + for _, d := range dists { + for _, root := range d.Roots() { + cfg.deps[filepath.Join(abs, filepath.FromSlash(root))] = d.Unit() + } + } + *importDirs = append(*importDirs, abs) + } + return nil +} + // depList is the -deps flags: the unit of each directory of dependency // code. type depList map[string]string @@ -141,6 +176,7 @@ type config struct { sourcesOnly bool importDirs stringList deps depList + sitePackages stringList summary bool includeTests bool json bool @@ -157,6 +193,7 @@ func main() { flag.BoolVar(&cfg.sourcesOnly, "sources-only", false, "search from the -source entry points alone, not from the roots that the rules mark") flag.Var(cfg.deps, "deps", "directory of a dependency and its unit, as node_modules/lodash=npm:lodash@4.17.21 (repeatable)") flag.Var(&cfg.importDirs, "import-dir", "directory that holds -deps directories, as venv/site-packages (repeatable)") + flag.Var(&cfg.sitePackages, "site-packages", "Python site-packages directory whose distributions become units, by their .dist-info (repeatable)") flag.BoolVar(&cfg.summary, "summary", false, "print the summary of each unit of -deps as JSON") flag.BoolVar(&cfg.includeTests, "include-tests", false, "count the test files as roots") flag.BoolVar(&cfg.json, "json", false, "print the reach results as JSON") @@ -193,6 +230,9 @@ func run(ctx context.Context, cfg config, out io.Writer) (err error) { } importDirs = append(importDirs, abs) } + if err := addSitePackages(cfg, &importDirs); err != nil { + return err + } units := map[string]iofs.FS{} for dir, unit := range cfg.deps { abs, err := filepath.Abs(dir) diff --git a/examples/projectgraph/main_test.go b/examples/projectgraph/main_test.go index 8e9622bb..af277cb4 100644 --- a/examples/projectgraph/main_test.go +++ b/examples/projectgraph/main_test.go @@ -19,6 +19,7 @@ const ( overlayCase = "../../test/projectgraph/cases/python/llm-overlay" deadCallCase = "../../test/projectgraph/cases/python/dead-call/project" dependencyCase = "../../test/projectgraph/cases/javascript/dependency-transitive/project" + compiledCase = "../../test/projectgraph/cases/python/dependency-compiled-module/project" ) // dependencyFlags name the two packages of the dependency case. @@ -183,3 +184,13 @@ func TestDependencyFlag(t *testing.T) { assert.Error(t, deps.Set("node_modules/lodash")) assert.Error(t, deps.Set("=npm:lodash@4.17.21")) } + +func TestExampleReadsASitePackagesDirectory(t *testing.T) { + out, err := exec.Command(buildExample(t), "-dir", compiledCase, "-site-packages", compiledCase+"/venv/site-packages", "-target", "json//loads", "-target", "os//system").CombinedOutput() + require.NoError(t, err, string(out)) + text := string(out) + assert.Contains(t, text, "json//loads: found\n") + assert.Contains(t, text, "pypi:speedy@1.0.0//speedy/pure.py", "the path goes into the unit that the .dist-info names") + assert.Contains(t, text, "os//system: uncertain\n") + assert.Contains(t, text, "no-dependency-source", "the extension module of the unit is a record") +} diff --git a/plugin/indexer/indexer.go b/plugin/indexer/indexer.go index b4b4dd98..08df7449 100644 --- a/plugin/indexer/indexer.go +++ b/plugin/indexer/indexer.go @@ -87,10 +87,10 @@ type Result struct { // Report lists the files that the run parsed, skipped and failed. Report *plugin.Report - // Excluded lists the Go files of dependency code that no Go build - // reads by default, in walk order: test files, files with //go:build - // ignore and files that need a tag the go command never sets. They have - // no status, as no program runs their code. + // Excluded lists the files of dependency code that no program runs, in + // walk order: the Go test files, the Go files with //go:build ignore or + // a tag the go command never sets, and the test files and the wheel + // metadata of a pypi unit. They have no status. Excluded []string } @@ -277,7 +277,7 @@ func (r *run) index(ctx context.Context, files []core.File, byExtension map[stri r.result.Report.Skipped = append(r.result.Report.Skipped, file.Name()) continue } - if goOutOfBuild(file) { + if goOutOfBuild(file) || pythonOutOfBuild(file) { r.exclude(file) continue } diff --git a/plugin/indexer/pyunit.go b/plugin/indexer/pyunit.go new file mode 100644 index 00000000..15d0a157 --- /dev/null +++ b/plugin/indexer/pyunit.go @@ -0,0 +1,33 @@ +package indexer + +import ( + "path" + "slices" + "strings" + + "github.com/safedep/code/core" + "github.com/safedep/code/fileindex" +) + +// pythonOutOfBuild reports whether no program imports a file of a pypi +// unit, by its path: a file under a tests directory, a test file by its +// name, a conftest.py, a file under a __pycache__, and a file under the +// .dist-info, .egg-info or .data directory of the wheel. Such a file holds +// no code that an importer runs, and a test file of a package adds test +// roots and budget to a graph that no search of the package needs. A test +// or a testing directory stays, because django.test and numpy.testing are +// modules that programs import. +func pythonOutOfBuild(file core.File) bool { + unit, inner := fileindex.SplitUnit(file.Name()) + pkg, err := fileindex.ParseUnit(unit) + if !file.IsImport() || err != nil || pkg.Ecosystem != "pypi" || path.Ext(inner) != ".py" { + return false + } + base := path.Base(inner) + if base == "conftest.py" || strings.HasPrefix(base, "test_") || strings.HasSuffix(base, "_test.py") { + return true + } + return slices.ContainsFunc(strings.Split(path.Dir(inner), "/"), func(d string) bool { + return d == "tests" || d == "__pycache__" || strings.HasSuffix(d, ".dist-info") || strings.HasSuffix(d, ".egg-info") || strings.HasSuffix(d, ".data") + }) +} diff --git a/plugin/indexer/pyunit_test.go b/plugin/indexer/pyunit_test.go new file mode 100644 index 00000000..7e98581e --- /dev/null +++ b/plugin/indexer/pyunit_test.go @@ -0,0 +1,60 @@ +package indexer + +import ( + "context" + "path/filepath" + "regexp" + "testing" + + "github.com/safedep/code/core" + "github.com/safedep/code/fileindex" + "github.com/safedep/code/fileindex/store/memstore" + "github.com/safedep/code/fs" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestIndexerLeavesOutTheTestsOfAPypiUnit(t *testing.T) { + dir := t.TempDir() + writeFiles(t, dir, map[string]string{"main.py": "import kit\n"}) + site := filepath.Join(dir, "venv", "site-packages") + writeFiles(t, site, map[string]string{ + "kit/__init__.py": "from .core import run\n", + "kit/core.py": "def run():\n pass\n", + "kit/test/client.py": "def client():\n pass\n", + "kit/tests/test_core.py": "def test_run():\n pass\n", + "kit/conftest.py": "x = 1\n", + "kit/core_test.py": "x = 1\n", + "kit/__pycache__/core.py": "x = 1\n", + "kit-1.0.0.dist-info/setup.py": "x = 1\n", + "kit-1.0.0.data/scripts/kit.py": "x = 1\n", + "kit-1.0.0.dist-info/METADATA": "Name: kit\nVersion: 1.0.0\n", + }) + fsys, err := fs.NewLocalFileSystem(fs.LocalFileSystemConfig{ + AppDirectories: []string{dir}, + ImportDirectories: []string{site}, + ImportUnits: map[string]string{filepath.Join(site, "kit"): "pypi:kit@1.0.0"}, + ExcludePatterns: []*regexp.Regexp{fs.SkipPattern(dir)}, + }) + require.NoError(t, err) + result, err := runIndexer(t, context.Background(), fsys, Options{Languages: languages(t, core.LanguageCodePython), Store: memstore.New(), Imports: true}) + require.NoError(t, err) + + var indexed []string + for p, status := range result.Dependencies { + _, inner := fileindex.SplitUnit(p) + assert.Equal(t, fileindex.FileIndexed, status, p) + indexed = append(indexed, inner) + } + var excluded []string + for _, p := range result.Excluded { + _, inner := fileindex.SplitUnit(p) + excluded = append(excluded, inner) + } + assert.ElementsMatch(t, []string{"kit/__init__.py", "kit/core.py", "kit/test/client.py"}, indexed) + assert.ElementsMatch(t, []string{"kit/tests/test_core.py", "kit/conftest.py", "kit/core_test.py", "kit/__pycache__/core.py"}, excluded) + require.Len(t, result.Files, 1, "a project file keeps its status") + for _, status := range result.Files { + assert.Equal(t, fileindex.FileIndexed, status) + } +} diff --git a/projectgraph/imports.go b/projectgraph/imports.go index 85a930d0..58ece9cf 100644 --- a/projectgraph/imports.go +++ b/projectgraph/imports.go @@ -5,6 +5,7 @@ import ( "strings" "github.com/safedep/code/fileindex" + "github.com/safedep/code/langcode" "github.com/safedep/code/projectgraph/modules" ) @@ -20,13 +21,14 @@ func (b *builder) linkImports(ix *fileindex.Index) { from := b.runner(sc.enclosing(imp.Line)) site := Site{Path: ix.Path, Line: imp.Line} mods, missing := b.importedModules(ix, imp.Module, formOf(imp)) - for _, unit := range missing { - b.addUnresolved(UnresolvedSite{Caller: from, Site: site, Text: imp.Module, Reason: fileindex.UnresolvedNoDefinition, Unit: unit}) + for _, pv := range missing { + b.addUnresolved(UnresolvedSite{Caller: from, Site: site, Text: imp.Module, Reason: missingReason(pv.compiled), Unit: pv.sp.unit}) } // The record has no method, so it makes every target uncertain - // where the search reaches the import. + // where the search reaches the import. An extension module of the + // unit of the file has no source that the graph reads. if len(mods) == 0 && len(missing) == 0 && b.mayBeFirstParty(ix, imp.Module) { - b.addUnresolved(UnresolvedSite{Caller: from, Site: site, Text: imp.Module, Reason: fileindex.UnresolvedNoDefinition}) + b.addUnresolved(UnresolvedSite{Caller: from, Site: site, Text: imp.Module, Reason: missingReason(b.compiledModule(ix, imp.Module))}) } for _, m := range mods { for _, to := range b.moduleFiles(m.key) { @@ -54,6 +56,36 @@ func (b *builder) linkImports(ix *fileindex.Index) { } } +// missingReason returns the reason of the record of an import that no file +// gives: no-dependency-source for an extension module, whose code the +// graph cannot read, and no-definition for any other module. +func missingReason(compiled bool) fileindex.UnresolvedReason { + if compiled { + return fileindex.UnresolvedNoDependencySource + } + return fileindex.UnresolvedNoDefinition +} + +// compiledModule reports whether an import of a file names an extension +// module of its own space, as yaml/_yaml.so for from yaml import _yaml. +func (b *builder) compiledModule(ix *fileindex.Index, spec string) bool { + sp := b.spaceOf(ix.Path) + if ix.Language != langcode.Python || sp.python == nil { + return false + } + module := spec + if strings.HasPrefix(spec, ".") { + // A relative module names a path from the directory of the file. + rel := modules.PythonRelativePath(spec) + base, ok := modules.Join(modules.Dir(sp.inner(ix.Path)), rel) + if !ok { + return false + } + module = strings.ReplaceAll(base, "/", ".") + } + return sp.python.Compiled(sp.inner(ix.Path), module) +} + // moduleRef is a module that an import names, and how the link step found // it. type moduleRef struct{ key, via string } @@ -65,7 +97,7 @@ type moduleRef struct{ key, via string } // JavaScript module is also the key of its default export, and the import // does not call it. missing holds each unit that gives the package and not // the module. -func (b *builder) importedModules(ix *fileindex.Index, spec string, form modules.Form) ([]moduleRef, []string) { +func (b *builder) importedModules(ix *fileindex.Index, spec string, form modules.Form) ([]moduleRef, []provision) { if mods := b.resolveModules(ix, spec, form); len(mods) > 0 { return mods, nil } @@ -73,10 +105,10 @@ func (b *builder) importedModules(ix *fileindex.Index, spec string, form modules return nil, nil } var mods []moduleRef - var missing []string + var missing []provision for _, pv := range b.providers(origin{lang: ix.Language, unit: ix.Unit, module: b.packageModule(ix, spec), form: form}) { if pv.module == "" { - missing = append(missing, pv.sp.unit) + missing = append(missing, pv) continue } b.named[pv.sp.unit] = true diff --git a/projectgraph/modules/pydist.go b/projectgraph/modules/pydist.go new file mode 100644 index 00000000..abe275ae --- /dev/null +++ b/projectgraph/modules/pydist.go @@ -0,0 +1,234 @@ +package modules + +import ( + "bufio" + "bytes" + "encoding/csv" + "errors" + "io" + "io/fs" + "path" + "regexp" + "slices" + "strings" +) + +// Distribution is one installed Python distribution, as the .dist-info +// directory of its wheel describes it. Dir is that directory, relative to +// the root. TopLevel holds the import names that the distribution installs, +// from top_level.txt, or else from the files of RECORD. Files holds the +// files of RECORD under the root, and Compiled the module paths with no +// Python source, as lxml/etree for a .so or a .pyd file. +type Distribution struct { + Name string + Version string + Dir string + TopLevel []string + Files []string + Compiled []string +} + +// Unit returns the package unit of the distribution, as pypi:PyYAML@6.0.1. +func (d Distribution) Unit() string { return "pypi:" + d.Name + "@" + d.Version } + +// Roots returns the paths under the root that hold the Python code of the +// distribution, as the keys of an import unit take them: a package +// directory with an __init__.py, a module file, or, for a namespace +// package that other distributions share, the packages and the modules +// under it. +func (d Distribution) Roots() []string { + files := map[string]bool{} + for _, f := range d.Files { + if path.Ext(f) == ".py" { + files[f] = true + } + } + var roots []string + var walk func(dir string) + walk = func(dir string) { + if files[path.Join(dir, "__init__.py")] { + roots = append(roots, dir) + return + } + seen := map[string]bool{} + for f := range files { + rel, ok := strings.CutPrefix(f, dir+"/") + if dir == "" { + rel, ok = f, true + } + if !ok { + continue + } + first, _, nested := strings.Cut(rel, "/") + if seen[first] { + continue + } + seen[first] = true + if nested { + walk(path.Join(dir, first)) + } else { + roots = append(roots, path.Join(dir, first)) + } + } + } + for _, top := range d.TopLevel { + if files[top+".py"] { + roots = append(roots, top+".py") + } + if slices.ContainsFunc(d.Files, func(f string) bool { return strings.HasPrefix(f, top+"/") }) { + walk(top) + } + } + slices.Sort(roots) + return slices.Compact(roots) +} + +var pythonNameRun = regexp.MustCompile(`[-_.]+`) + +// NormalizePythonName returns the name of a PyPI package as PEP 503 +// compares it: in lower case, with one - for each run of -, _ and ".". +func NormalizePythonName(name string) string { + return strings.ToLower(pythonNameRun.ReplaceAllString(name, "-")) +} + +// compiledExts are the extensions of a Python extension module. +var compiledExts = map[string]bool{".so": true, ".pyd": true, ".dylib": true} + +// PythonDistributions reads the .dist-info directories under a root, as a +// site-packages directory or an unpacked wheel holds them, and returns the +// distributions that they describe, in the order of the directories. A +// directory with no METADATA describes no distribution and is left out. It +// returns the files that it could not read. +func PythonDistributions(root fs.FS) ([]Distribution, []Problem) { + if root == nil { + return nil, nil + } + entries, err := fs.ReadDir(root, ".") + if err != nil { + return nil, []Problem{{Path: ".", Text: err.Error()}} + } + var out []Distribution + var ps problems + for _, e := range entries { + if !e.IsDir() || !strings.HasSuffix(e.Name(), ".dist-info") { + continue + } + d, err := readDistribution(root, e.Name()) + switch { + case errors.Is(err, fs.ErrNotExist): + continue + case err != nil: + ps.add(e.Name(), err) + continue + } + out = append(out, d) + } + return out, ps +} + +var errNoDistributionName = errors.New("METADATA has no Name") + +// readDistribution reads the METADATA, the RECORD and the top_level.txt of +// one .dist-info directory. +func readDistribution(root fs.FS, dir string) (Distribution, error) { + d := Distribution{Dir: dir} + meta, err := fs.ReadFile(root, path.Join(dir, "METADATA")) + if err != nil { + return d, err + } + for _, line := range strings.Split(string(meta), "\n") { + if line == "" { + break + } + if name, ok := strings.CutPrefix(line, "Name: "); ok && d.Name == "" { + d.Name = strings.TrimSpace(name) + } + if version, ok := strings.CutPrefix(line, "Version: "); ok && d.Version == "" { + d.Version = strings.TrimSpace(version) + } + } + if d.Name == "" { + return d, errNoDistributionName + } + record, err := fs.ReadFile(root, path.Join(dir, "RECORD")) + if err != nil && !errors.Is(err, fs.ErrNotExist) { + return d, err + } + if err == nil { + if d.Files, d.Compiled, err = readRecord(record); err != nil { + return d, err + } + } + top, err := fs.ReadFile(root, path.Join(dir, "top_level.txt")) + if err != nil && !errors.Is(err, fs.ErrNotExist) { + return d, err + } + if err == nil { + scanner := bufio.NewScanner(bytes.NewReader(top)) + for scanner.Scan() { + if name := strings.TrimSpace(scanner.Text()); name != "" { + d.TopLevel = append(d.TopLevel, name) + } + } + } + if len(d.TopLevel) == 0 { + d.TopLevel = topLevelOf(d.Files, d.Compiled) + } + return d, nil +} + +// readRecord returns the files of a RECORD under the root, and the module +// paths of its extension modules. A path that leaves the root, as a script +// in ../../bin, is no file of the package. +func readRecord(data []byte) ([]string, []string, error) { + r := csv.NewReader(bytes.NewReader(data)) + r.FieldsPerRecord = -1 + var files, compiled []string + for { + row, err := r.Read() + if errors.Is(err, io.EOF) { + return files, compiled, nil + } + if err != nil { + return nil, nil, err + } + if len(row) == 0 { + continue + } + p, ok := Join(row[0]) + if !ok || p == "" { + continue + } + files = append(files, p) + if compiledExts[path.Ext(p)] { + compiled = append(compiled, compiledModule(p)) + } + } +} + +// compiledModule returns the module path of an extension module, with the +// ABI tag and the extension of its file name cut, as lxml/etree for +// lxml/etree.cpython-311-x86_64-linux-gnu.so. +func compiledModule(p string) string { + dir, base := path.Split(p) + name, _, _ := strings.Cut(base, ".") + return dir + name +} + +// topLevelOf returns the first path element of each Python module and +// extension module, in order, each once. +func topLevelOf(files, compiled []string) []string { + var names []string + for _, f := range files { + if path.Ext(f) != ".py" { + continue + } + first, _, _ := strings.Cut(f, "/") + names = append(names, strings.TrimSuffix(first, ".py")) + } + for _, m := range compiled { + first, _, _ := strings.Cut(m, "/") + names = append(names, first) + } + return compactStable(names) +} diff --git a/projectgraph/modules/pydist_test.go b/projectgraph/modules/pydist_test.go new file mode 100644 index 00000000..86f202cf --- /dev/null +++ b/projectgraph/modules/pydist_test.go @@ -0,0 +1,60 @@ +package modules + +import ( + "testing" + "testing/fstest" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestPythonDistributions(t *testing.T) { + root := fstest.MapFS{ + "PyYAML-6.0.1.dist-info/METADATA": {Data: []byte("Metadata-Version: 2.1\nName: PyYAML\nVersion: 6.0.1\nSummary: YAML\n\nThe body.\nName: not this\n")}, + "PyYAML-6.0.1.dist-info/RECORD": {Data: []byte("PyYAML-6.0.1.dist-info/METADATA,sha256=abc,1730\n" + + "_yaml/__init__.py,sha256=abc,10\n" + + "yaml/__init__.py,sha256=abc,13340\n" + + "yaml/_yaml.cpython-311-x86_64-linux-gnu.so,sha256=abc,222208\n" + + "yaml/loader.py,sha256=abc,100\n" + + "../../bin/yamlc,sha256=abc,10\n")}, + "six-1.16.0.dist-info/METADATA": {Data: []byte("Name: six\nVersion: 1.16.0\n")}, + "six-1.16.0.dist-info/RECORD": {Data: []byte("six.py,sha256=abc,34549\nsix-1.16.0.dist-info/RECORD,,\n")}, + "six-1.16.0.dist-info/top_level.txt": {Data: []byte("six\n")}, + "google_auth-2.0.0.dist-info/METADATA": {Data: []byte("Name: google-auth\nVersion: 2.0.0\n")}, + "google_auth-2.0.0.dist-info/RECORD": {Data: []byte("google/auth/__init__.py,,\ngoogle/auth/crypt.py,,\ngoogle/oauth2/__init__.py,,\n")}, + "nometa.dist-info/RECORD": {Data: []byte("x.py,,\n")}, + "broken.dist-info/METADATA": {Data: []byte("Version: 1\n")}, + "notes.txt": {Data: []byte("")}, + } + dists, problems := PythonDistributions(root) + require.Len(t, problems, 1, "a .dist-info with no METADATA is no distribution, and one with no Name is a problem") + assert.Equal(t, "broken.dist-info", problems[0].Path) + require.Len(t, dists, 3) + + yaml := dists[0] + assert.Equal(t, "PyYAML", yaml.Name) + assert.Equal(t, "6.0.1", yaml.Version) + assert.Equal(t, "pypi:PyYAML@6.0.1", yaml.Unit()) + assert.Equal(t, []string{"_yaml", "yaml"}, yaml.TopLevel, "the top-level names come from RECORD when top_level.txt is missing") + assert.Equal(t, []string{"yaml/_yaml"}, yaml.Compiled) + assert.NotContains(t, yaml.Files, "../../bin/yamlc", "a path that leaves the root is no file of the package") + assert.Equal(t, []string{"_yaml", "yaml"}, yaml.Roots()) + + google := dists[1] + assert.Equal(t, []string{"google"}, google.TopLevel) + assert.Equal(t, []string{"google/auth", "google/oauth2"}, google.Roots(), "a namespace package gives the packages under it") + + six := dists[2] + assert.Equal(t, []string{"six"}, six.TopLevel) + assert.Equal(t, []string{"six.py"}, six.Roots()) + + assert.Equal(t, "pyyaml", NormalizePythonName("PyYAML")) + assert.Equal(t, "typing-extensions", NormalizePythonName("typing_extensions")) + assert.Equal(t, "zope-interface", NormalizePythonName("zope.interface")) +} + +func TestPythonDistributionsWithNoRoot(t *testing.T) { + dists, problems := PythonDistributions(nil) + assert.Empty(t, dists) + assert.Empty(t, problems) +} diff --git a/projectgraph/modules/python.go b/projectgraph/modules/python.go index 3b660ba3..a3822cf6 100644 --- a/projectgraph/modules/python.go +++ b/projectgraph/modules/python.go @@ -19,6 +19,11 @@ type Python struct { roots []string pkg string problems problems + // topLevel holds the import names that the distribution of the file + // set installs, and compiled its modules with no Python source, by + // module path, as the .dist-info of a wheel lists them. + topLevel []string + compiled map[string]bool } // NewPython returns the resolver of a project. The roots are the project @@ -69,6 +74,40 @@ func (p *Python) SetRootPackage(name string) { p.pkg = name } // RootPackage returns the name of the package that the file set is, or "". func (p *Python) RootPackage() string { return p.pkg } +// SetDistribution gives the resolver the import names and the extension +// modules of the distribution that the file set is. +func (p *Python) SetDistribution(d Distribution) { + p.topLevel = d.TopLevel + p.compiled = make(map[string]bool, len(d.Compiled)) + for _, m := range d.Compiled { + p.compiled[m] = true + } +} + +// TopLevel returns the import names that the distribution of the file set +// installs, or nil when no .dist-info told them. +func (p *Python) TopLevel() []string { return p.topLevel } + +// Compiled reports whether an absolute module that the importer names is +// an extension module of the distribution, with no Python source. Such a +// module resolves to no file, and the code that runs in it is unknown. +func (p *Python) Compiled(importer, module string) bool { + if len(p.compiled) == 0 || module == "" || strings.HasPrefix(module, ".") { + return false + } + rel := strings.ReplaceAll(module, ".", "/") + roots := p.searchOrder(importer) + if dir := Dir(importer); !slices.Contains(roots, dir) { + roots = append(roots, dir) + } + for _, r := range roots { + if base, ok := Join(r, rel); ok && p.compiled[base] { + return true + } + } + return false +} + // Problems returns the manifests that the resolver could not read. func (p *Python) Problems() []Problem { return p.problems } diff --git a/projectgraph/modules/resolver.go b/projectgraph/modules/resolver.go index 97893a1b..d157c670 100644 --- a/projectgraph/modules/resolver.go +++ b/projectgraph/modules/resolver.go @@ -57,7 +57,8 @@ func (p *Python) FirstParty(spec string) bool { // FirstPartyFrom reports whether a Python module is relative or starts // with the name of a module or a package under an import root or in the // directory of the importer, or with the name of the package that the -// file set is. A file of the name deep in the project, as a types.py in a +// file set is. An extension module of the distribution counts as a module +// too. A file of the name deep in the project, as a types.py in a // package, is no module of an absolute import, so import types names the // standard library and not the file. func (p *Python) FirstPartyFrom(importer, spec string) bool { @@ -65,8 +66,10 @@ func (p *Python) FirstPartyFrom(importer, spec string) bool { if strings.HasPrefix(spec, ".") || p.pkg != "" && first == p.pkg { return true } - _, ok := p.Absolute(importer, first) - return ok + if _, ok := p.Absolute(importer, first); ok { + return true + } + return p.Compiled(importer, first) } // For returns the resolver of the JavaScript or the TypeScript importers. diff --git a/projectgraph/packages.go b/projectgraph/packages.go index b8675296..b20d9570 100644 --- a/projectgraph/packages.go +++ b/projectgraph/packages.go @@ -123,6 +123,9 @@ func (b *builder) addSpaces() error { if sp.pkg.Ecosystem == pypi && sp.files.Has(pythonInit) { sp.python.SetRootPackage(pythonImportName(sp.pkg.Name)) } + if sp.pkg.Ecosystem == pypi { + b.readDistribution(sp) + } sp.js = modules.NewJavaScript(sp.files, sp.root) b.addProblems(sp, sp.python.Problems()) b.addProblems(sp, sp.js.Problems()) @@ -141,15 +144,50 @@ const ( // pythonImportName returns the import name that the name of a pypi package // gives with no other knowledge: in lower case, with _ for - and for ., as // typing_extensions for typing-extensions. A package whose import name -// differs, as yaml for PyYAML, needs its package directory under an -// import directory. +// differs, as yaml for PyYAML, needs the .dist-info of its wheel at the +// root of the unit, or its package directory under an import directory. func pythonImportName(name string) string { return strings.NewReplacer("-", "_", ".", "_").Replace(strings.ToLower(name)) } +// readDistribution reads the .dist-info directories at the root of a pypi +// unit, as a site-packages directory or an unpacked wheel holds them, and +// gives the resolver of the unit the one of its package: the import names +// that it installs and its extension modules. A root with several +// distributions, as site-packages, matches by the normalized name, and by +// the version when two share the name. +func (b *builder) readDistribution(sp *space) { + dists, problems := modules.PythonDistributions(sp.root) + b.addProblems(sp, problems) + name := modules.NormalizePythonName(sp.pkg.Name) + var found *modules.Distribution + for i := range dists { + d := &dists[i] + if modules.NormalizePythonName(d.Name) != name { + continue + } + if found == nil || d.Version == sp.pkg.Version { + found = d + } + } + if found != nil { + sp.python.SetDistribution(*found) + } +} + +// pythonTopLevel returns the import names that a pypi unit installs: the +// names of its .dist-info, or else the name of the package as an import +// writes it. +func pythonTopLevel(sp *space) []string { + if names := sp.python.TopLevel(); len(names) > 0 { + return names + } + return []string{pythonImportName(sp.pkg.Name)} +} + // checkUnits records a problem and a record of each pypi unit that no -// import names and that gives no module for its package name. The link -// step cannot tell which import would load such a unit, so the record +// import names and that gives no module for any of its import names. The +// link step cannot tell which import would load such a unit, so the record // holds for every search, and it makes uncertain every target that the // code of the unit reaches. A build with no project file, as a summary, // has no importer, so it skips the check. @@ -161,12 +199,15 @@ func (b *builder) checkUnits() { if sp.unit == "" || sp.pkg.Ecosystem != pypi || b.named[sp.unit] { continue } - name := pythonImportName(sp.pkg.Name) - if slices.ContainsFunc(b.provide(sp, langcode.Python, name, modules.FormImport), func(pv provision) bool { return pv.module != "" }) { + names := pythonTopLevel(sp) + if slices.ContainsFunc(names, func(name string) bool { + return slices.ContainsFunc(b.provide(sp, langcode.Python, name, modules.FormImport), func(pv provision) bool { return pv.module != "" }) + }) { continue } - b.problems = append(b.problems, modules.Problem{Path: sp.unit, Text: "no import names the unit, and it gives no module for the package " + name}) - b.addUnresolved(UnresolvedSite{Site: Site{Path: sp.unit}, Text: name, Reason: fileindex.UnresolvedNoDefinition, Unit: sp.unit}) + text := strings.Join(names, ", ") + b.problems = append(b.problems, modules.Problem{Path: sp.unit, Text: "no import names the unit, and it gives no module for the package " + text}) + b.addUnresolved(UnresolvedSite{Site: Site{Path: sp.unit}, Text: text, Reason: fileindex.UnresolvedNoDefinition, Unit: sp.unit}) } } @@ -276,11 +317,13 @@ func (b *builder) deriveKey(parent string, names []string) string { // provision is the module of a unit that gives the module of an origin. An // empty module is a unit that gives the package but not the module, as a -// subpath that the package does not have. +// subpath that the package does not have, or an extension module with no +// Python source, which compiled marks. type provision struct { - sp *space - module string - via string + sp *space + module string + via string + compiled bool } // provisionKey is what the providers of an origin depend on. diff --git a/projectgraph/python.go b/projectgraph/python.go index faf5d4e0..666f2588 100644 --- a/projectgraph/python.go +++ b/projectgraph/python.go @@ -18,6 +18,7 @@ var pythonProfile = profile{ provide: pythonProvide, narrowProviders: pythonNarrowProviders, surface: pythonSurface, + symbol: pythonSymbol, dependencies: pythonDependencies, instance: []string{"self", "cls"}, objectBase: "object", @@ -38,11 +39,15 @@ func pythonDependencies(b *builder) ([]string, []modules.Problem) { } // pythonProvide returns the module of a pypi unit that an absolute import -// names. +// names. An extension module of the unit has no Python source, so the unit +// gives the package and not the module. func pythonProvide(_ *builder, sp *space, module string, _ modules.Form) []provision { if strings.HasPrefix(module, ".") { return nil } + if sp.python.Compiled("", module) { + return []provision{{sp: sp, compiled: true}} + } if _, ok := sp.python.Absolute("", firstDotted(module)); !ok { return nil } @@ -53,6 +58,17 @@ func pythonProvide(_ *builder, sp *space, module string, _ modules.Form) []provi return []provision{{sp: sp, module: sp.key(mod), via: "absolute"}} } +// pythonSymbol returns the dotted module of a definition of a pypi unit, as +// an importer writes it, and the names of the definition under it, as +// requests.api and get for requests/api.py//get. +func pythonSymbol(_ *builder, sp *space, def string) (string, []string, bool) { + module, ok := pythonModulePath(sp.inner(fileindex.Unit(def)), sp.python.RootPackage()) + if !ok { + return "", nil, false + } + return module, fileindex.Segments(def), true +} + // pythonNarrowProviders keeps the units that give the module when one // does. The distributions of one namespace package, as google, share its // top-level directory. Only one of them gives a module under it, so the @@ -81,11 +97,29 @@ func pythonSurface(s *surfacer) { } } -// pythonModuleName returns the dotted name of a Python module of a package, -// as requests.api for requests/api.py, under the package that the file set -// is, if any. A module with a part that starts with an underscore is -// private, and a __main__ module is a script. +// pythonModuleName returns the dotted name of a public Python module of a +// package, as requests.api for requests/api.py, under the package that the +// file set is, if any. A module with a part that starts with an underscore +// is private, and a __main__ module is a script. func pythonModuleName(p, pkg string) (string, bool) { + module, ok := pythonModulePath(p, pkg) + if !ok { + return "", false + } + own := strings.Split(module, ".") + if pkg != "" { + own = own[1:] + } + if slices.ContainsFunc(own, func(part string) bool { return strings.HasPrefix(part, "_") }) { + return "", false + } + return module, true +} + +// pythonModulePath returns the dotted name of a Python module of a package, +// private or not, as yaml._internal for yaml/_internal.py, under the +// package that the file set is, if any. +func pythonModulePath(p, pkg string) (string, bool) { rel, ok := strings.CutSuffix(p, ".py") if !ok { return "", false @@ -98,7 +132,7 @@ func pythonModuleName(p, pkg string) (string, bool) { if rel != "" { parts = append(parts, strings.Split(rel, "/")...) } - if len(parts) == 0 || slices.ContainsFunc(parts[len(parts)-len(strings.Split(rel, "/")):], func(part string) bool { return strings.HasPrefix(part, "_") }) { + if len(parts) == 0 { return "", false } return strings.Join(parts, "."), true diff --git a/test/projectgraph/README.md b/test/projectgraph/README.md index 18cbbcc9..2b7a473e 100644 --- a/test/projectgraph/README.md +++ b/test/projectgraph/README.md @@ -134,7 +134,9 @@ go test -tags projectgraph -run TestMethodTargets -count=1 -v ./test/projectgrap ``` The test logs the count of each answer per language and per symbol kind, -and the same per project. Set `PROJECTGRAPH_METHOD_REPORT` to a file path +and the same per project. An answer is `found`, `found/low` for a path with +a hop of low confidence, `not-found`, `no-such-target`, or `uncertain` with +its cause. Set `PROJECTGRAPH_METHOD_REPORT` to a file path to write the block to that file. Set `PROJECTGRAPH_METHOD_LANGUAGES` to a list of languages, as `python`, to index those alone. A Python project bundles the JavaScript of its web pages, and a run of `python` alone diff --git a/test/projectgraph/cases/python/dependency-compiled-module/expect.yaml b/test/projectgraph/cases/python/dependency-compiled-module/expect.yaml new file mode 100644 index 00000000..327fe1ea --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/expect.yaml @@ -0,0 +1,66 @@ +source: safedep:hand +description: >- + The unit speedy has an extension module _speedups, which its RECORD lists as a .so file, and a pure Python + fallback. The import of the extension module resolves to no file, and the graph cannot read its code, so + the import is a record with the reason no-dependency-source, where the import of a module that no file + gives would be no-definition. The record concerns every target where a search reaches the import, so + os.system is uncertain and not no-such-target. The fallback gives loads, and json.loads is found through + it. +units: + venv/site-packages/speedy: pypi:speedy@1.0.0 +import_dirs: + - venv/site-packages +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py + to: pypi:speedy@1.0.0//speedy/__init__.py + kind: import + confidence: high + site: main.py:1 + - from: main.py + to: speedy//loads + kind: direct + confidence: high + site: main.py:4 + - from: pypi:speedy@1.0.0//speedy/__init__.py + to: pypi:speedy@1.0.0//speedy/pure.py + kind: import + confidence: high + site: pypi:speedy@1.0.0//speedy/__init__.py:4 +unresolved: + - caller: pypi:speedy@1.0.0//speedy/__init__.py + site: pypi:speedy@1.0.0//speedy/__init__.py:2 + reason: no-dependency-source +results: + - target: json//loads + status: found + path: + - key: main.py + - key: speedy//loads + site: main.py:4 + kind: direct + confidence: high + - key: pypi:speedy@1.0.0//speedy/__init__.py//loads + site: pypi:speedy@1.0.0//speedy/__init__.py:4 + kind: package + confidence: high + - key: pypi:speedy@1.0.0//speedy/pure.py//loads + site: pypi:speedy@1.0.0//speedy/__init__.py:4 + kind: export + confidence: high + - key: json//loads + site: pypi:speedy@1.0.0//speedy/pure.py:5 + kind: direct + confidence: high + call_sites: + - caller: pypi:speedy@1.0.0//speedy/pure.py//loads + site: pypi:speedy@1.0.0//speedy/pure.py:5 + reached: true + - target: os//system + status: uncertain + uncertainties: + - kind: unresolved-call + site: pypi:speedy@1.0.0//speedy/__init__.py:2 diff --git a/test/projectgraph/cases/python/dependency-compiled-module/project/main.py b/test/projectgraph/cases/python/dependency-compiled-module/project/main.py new file mode 100644 index 00000000..53f779e6 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/project/main.py @@ -0,0 +1,4 @@ +import speedy + +if __name__ == "__main__": + speedy.loads("{}") diff --git a/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/METADATA b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/METADATA new file mode 100644 index 00000000..6ee8fcdc --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/METADATA @@ -0,0 +1,3 @@ +Metadata-Version: 2.1 +Name: speedy +Version: 1.0.0 diff --git a/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/RECORD b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/RECORD new file mode 100644 index 00000000..b2aad705 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy-1.0.0.dist-info/RECORD @@ -0,0 +1,5 @@ +speedy/__init__.py,sha256=a,70 +speedy/_speedups.cpython-311-x86_64-linux-gnu.so,sha256=b,40000 +speedy/pure.py,sha256=c,60 +speedy-1.0.0.dist-info/METADATA,sha256=d,50 +speedy-1.0.0.dist-info/RECORD,, diff --git a/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/__init__.py b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/__init__.py new file mode 100644 index 00000000..6bae2a9d --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/__init__.py @@ -0,0 +1,4 @@ +try: + from ._speedups import loads +except ImportError: + from .pure import loads diff --git a/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/pure.py b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/pure.py new file mode 100644 index 00000000..2413dd0a --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/project/venv/site-packages/speedy/pure.py @@ -0,0 +1,5 @@ +import json + + +def loads(text): + return json.loads(text) diff --git a/test/projectgraph/cases/python/dependency-compiled-module/summaries/pypi_speedy_1.0.0.json b/test/projectgraph/cases/python/dependency-compiled-module/summaries/pypi_speedy_1.0.0.json new file mode 100644 index 00000000..22a34a9f --- /dev/null +++ b/test/projectgraph/cases/python/dependency-compiled-module/summaries/pypi_speedy_1.0.0.json @@ -0,0 +1,40 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "pypi:speedy@1.0.0", + "symbols": [ + { + "key": "speedy//loads", + "definitions": [ + "pypi:speedy@1.0.0//speedy/__init__.py//loads" + ], + "reaches": [ + "json//loads" + ] + }, + { + "key": "speedy//pure//loads", + "definitions": [ + "pypi:speedy@1.0.0//speedy/pure.py//loads" + ], + "reaches": [ + "json//loads" + ] + } + ], + "targets": [ + "json//loads" + ], + "uncertainties": [ + { + "kind": "unresolved-call", + "key": "pypi:speedy@1.0.0//speedy/__init__.py", + "site": { + "path": "pypi:speedy@1.0.0//speedy/__init__.py", + "line": 2 + }, + "text": "._speedups (no-dependency-source)" + } + ] +} diff --git a/test/projectgraph/cases/python/dependency-record-import-name/expect.yaml b/test/projectgraph/cases/python/dependency-record-import-name/expect.yaml new file mode 100644 index 00000000..0c4dbd62 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-record-import-name/expect.yaml @@ -0,0 +1,31 @@ +source: safedep:hand +description: >- + The unit PyYAML holds the package yaml, and no import of the project names it. Its RECORD lists yaml as + the import name, so the link step knows that import yaml would load the unit, and no file imports yaml. + Compare dependency-dark-unit, where the unit has no RECORD and the link step records a problem. Here the + unit is linked and not reached, so marshal.loads is not found, and the definition yaml.load carries its + external key, so the target yaml//load names it and is not found either. +units: + venv/site-packages/yaml: pypi:PyYAML@6.0.1 +import_dirs: + - venv/site-packages +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: pypi:PyYAML@6.0.1//yaml/__init__.py//load + to: marshal//loads + kind: direct + confidence: high + site: pypi:PyYAML@6.0.1//yaml/__init__.py:5 +unresolved: [] +results: + - target: marshal//loads + status: not-found + call_sites: + - caller: pypi:PyYAML@6.0.1//yaml/__init__.py//load + site: pypi:PyYAML@6.0.1//yaml/__init__.py:5 + reached: false + - target: yaml//load + status: not-found diff --git a/test/projectgraph/cases/python/dependency-record-import-name/project/main.py b/test/projectgraph/cases/python/dependency-record-import-name/project/main.py new file mode 100644 index 00000000..aef04c45 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-record-import-name/project/main.py @@ -0,0 +1,4 @@ +import json + +if __name__ == "__main__": + json.loads("{}") diff --git a/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/METADATA b/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/METADATA new file mode 100644 index 00000000..b39f6f33 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/METADATA @@ -0,0 +1,3 @@ +Metadata-Version: 2.1 +Name: PyYAML +Version: 6.0.1 diff --git a/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/RECORD b/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/RECORD new file mode 100644 index 00000000..585b64f9 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/PyYAML-6.0.1.dist-info/RECORD @@ -0,0 +1,3 @@ +PyYAML-6.0.1.dist-info/METADATA,sha256=a,50 +PyYAML-6.0.1.dist-info/RECORD,, +yaml/__init__.py,sha256=b,80 diff --git a/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/yaml/__init__.py b/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/yaml/__init__.py new file mode 100644 index 00000000..569bbb4f --- /dev/null +++ b/test/projectgraph/cases/python/dependency-record-import-name/project/venv/site-packages/yaml/__init__.py @@ -0,0 +1,5 @@ +import marshal + + +def load(stream): + return marshal.loads(stream) diff --git a/test/projectgraph/cases/python/dependency-record-import-name/summaries/pypi_PyYAML_6.0.1.json b/test/projectgraph/cases/python/dependency-record-import-name/summaries/pypi_PyYAML_6.0.1.json new file mode 100644 index 00000000..3f50d6a1 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-record-import-name/summaries/pypi_PyYAML_6.0.1.json @@ -0,0 +1,20 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "pypi:PyYAML@6.0.1", + "symbols": [ + { + "key": "yaml//load", + "definitions": [ + "pypi:PyYAML@6.0.1//yaml/__init__.py//load" + ], + "reaches": [ + "marshal//loads" + ] + } + ], + "targets": [ + "marshal//loads" + ] +} diff --git a/test/projectgraph/cases/python/dependency-site-packages/expect.yaml b/test/projectgraph/cases/python/dependency-site-packages/expect.yaml index b29b7bbf..f446feba 100644 --- a/test/projectgraph/cases/python/dependency-site-packages/expect.yaml +++ b/test/projectgraph/cases/python/dependency-site-packages/expect.yaml @@ -93,3 +93,37 @@ results: reached: true - target: os//system status: no-such-target + - target: urllib3//poolmanager//request + status: found + path: + - key: main.py + - key: requests//get + site: main.py:3 + kind: direct + confidence: high + - key: pypi:requests@2.31.0//requests/__init__.py//get + site: pypi:requests@2.31.0//requests/__init__.py:1 + kind: package + confidence: high + - key: pypi:requests@2.31.0//requests/api.py//get + site: pypi:requests@2.31.0//requests/__init__.py:1 + kind: export + confidence: high + - key: urllib3//request + site: pypi:requests@2.31.0//requests/api.py:5 + kind: direct + confidence: high + - key: pypi:urllib3@2.0.7//urllib3/__init__.py//request + site: pypi:urllib3@2.0.7//urllib3/__init__.py:1 + kind: package + confidence: high + - key: pypi:urllib3@2.0.7//urllib3/poolmanager.py//request + site: pypi:urllib3@2.0.7//urllib3/__init__.py:1 + kind: export + confidence: high + call_sites: + - caller: pypi:urllib3@2.0.7//urllib3/__init__.py//request + site: pypi:urllib3@2.0.7//urllib3/__init__.py:1 + reached: true + - target: requests//api//post + status: not-found diff --git a/test/projectgraph/receivers_test.go b/test/projectgraph/receivers_test.go index 7b5b7b69..f7578bc9 100644 --- a/test/projectgraph/receivers_test.go +++ b/test/projectgraph/receivers_test.go @@ -6,6 +6,7 @@ import ( "os" "path/filepath" "regexp" + "slices" "testing" "github.com/safedep/code/core" @@ -212,12 +213,11 @@ var rustReceiverRows = []receiverRow{ } // pythonTypeHints is the step of the Python plan that reads the type hints -// of a parameter, a local, a field and a return into the index, and gives -// a definition of a pypi unit its external key. Most Python rows wait on -// it. Two forms of today show the gap: a call on a variable that holds -// the result of an external call gets the false key dep//new_thing//do and -// no record, so the method is no-such-target, and a call on an unknown -// receiver is a record and a name edge, so the method is uncertain. +// of a parameter, a local, a field and a return into the index. Most +// Python rows wait on it. Today a row that reaches the method reaches it +// through a low hop: the override rule joins the class Thing to its do, +// so the constructor call in the unit reaches do, and a call on an unknown +// receiver gets a name edge. The declared type must give the member edge. const pythonTypeHints = "the type-hint step of the Python plan" // pythonReceiverRows are the forms of a Python receiver. A type hint is @@ -229,7 +229,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "no-such-target", + today: "found/low", }, { name: "parameter", @@ -267,7 +267,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "uncertain/no-definition", + today: "found/low", }, { name: "instance attribute from the constructor", @@ -275,7 +275,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "uncertain/no-definition", + today: "found/low", }, { name: "dataclass field", @@ -291,7 +291,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "uncertain/unknown-receiver", + today: "found/low", }, { name: "external function return", @@ -299,7 +299,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "no-such-target", + today: "found/low", }, { name: "tuple return", @@ -307,7 +307,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "uncertain/unknown-receiver", + today: "found/low", }, { name: "optional return", @@ -315,7 +315,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "no-such-target", + today: "found/low", }, { name: "chained call", @@ -323,7 +323,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "uncertain/unknown-receiver", + today: "found/low", }, { name: "chained method call", @@ -331,7 +331,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "uncertain/unknown-receiver", + today: "found/low", }, { name: "instance from the constructor", @@ -345,7 +345,7 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "found", gap: pythonTypeHints, - today: "no-such-target", + today: "found/low", }, { name: "abstract base return with no implementation", @@ -460,10 +460,15 @@ func receiverSource(t *testing.T, language core.LanguageCode) reach.Source { return s } -// outcome is the answer for one target: found, not-found, no-such-target, -// or uncertain with its cause. +// outcome is the answer for one target: found, found/low when a hop of the +// path has low confidence, as a reference edge of the override rule, +// not-found, no-such-target, or uncertain with its cause. A row that wants +// found asks for a path that the declared types and the calls prove. func outcome(r reach.Result, methods map[string]string) string { - if r.Status != reach.Uncertain { + switch { + case r.Status == reach.Found && slices.ContainsFunc(r.Path, func(h reach.Hop) bool { return h.Confidence == projectgraph.ConfidenceLow }): + return "found/low" + case r.Status != reach.Uncertain: return string(r.Status) } return "uncertain/" + uncertainCause(r, methods) diff --git a/test/projectgraph/report.md b/test/projectgraph/report.md index f93272f5..3167e4ca 100644 --- a/test/projectgraph/report.md +++ b/test/projectgraph/report.md @@ -1206,3 +1206,53 @@ The run of the two languages gives the same numbers as the run of at 9 of 9 stacks. - The receiver table does not change. The 16 Python rows still wait on the type-hint step. + +#### After the distribution reader, the compiled-module record and the external keys + +The commit after the four rules adds the rest of step 2 of the plan. + +1. The index step leaves out the test files and the wheel metadata of a + pypi unit, as it leaves out the test files of a Go unit. +2. The link step reads the `.dist-info` at the root of a pypi unit: the + import names of `top_level.txt` or `RECORD`, and the extension modules + of `RECORD`. An import of an extension module is a record with the + reason `no-dependency-source`, because the graph cannot read its code. +3. A definition of a pypi unit carries the external key that an importer + of its module writes, as `requests//api//get`, through the `symbol` + rule that Go and Rust had. A target that only the code of its package + calls matches the definition. +4. `outcome` of the measure and of the receiver rows now says `found/low` + for a path with a hop of low confidence, as a `name` edge or a + `reference` edge of the override rule, and `found` only for a path + that the calls and the declared types prove. + +| Language | Kind | Run | Targets | found | found/low | not-found | no-such-target | unknown-receiver | no-definition | ambiguous | computed-name | dynamic-flag | budget | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Python | function | step 2 | 44 | 1 | 18 | 0 | 0 | 0 | 0 | 10 | 4 | 10 | 1 | +| Python | method | step 2 | 63 | 0 | 36 | 0 | 0 | 4 | 1 | 11 | 2 | 6 | 3 | + +| Language | Run | Method targets that end as unknown-receiver | +| --- | --- | --- | +| Python | step 2 | 0.063 (4/63) | + +- The external keys move 54 targets from uncertain to `found/low`. As for + Go after FOU-555, most symbols of an advisory are methods that the code + of the dependency calls inside its own package, and the match on the + definition finds them. Each of these paths has a `name` edge or a + `reference` edge. The `name` edges come from calls on unknown receivers + inside the unit, as `self.request.GET.get` in Django, which the type + hints of step 3 and the instance attributes of the unit must narrow + before a path reads as `found`. +- `unknown-receiver` falls from 27 to 4 method targets, because a reached + record with the method name of the target no longer decides the answer + when a path reaches the definition. The 4 that stay are requests + methods in httpie and requests-toolbelt whose paths the search does not + find. +- The 10 Python receiver rows that reach their method today reach it as + `found/low`: the override rule joins the class `Thing` to its `do`, so + the constructor call in the unit reaches `do`. The rows keep their gap, + because they want the member edge of the declared type. +- `no-such-target` stays at 0, and `not-found` at 0. No project of the + corpus has a dependency it never imports. +- pycg-micro keeps 0.954 precision and 0.708 recall, with the unresolved + rate at 0.312. govulncheck stays at 9 of 9 stacks. diff --git a/test/projectgraph/version_test.go b/test/projectgraph/version_test.go index 5a7506ff..07c354cd 100644 --- a/test/projectgraph/version_test.go +++ b/test/projectgraph/version_test.go @@ -31,7 +31,7 @@ var graphHashes = map[string]string{ "14/14": "0f3c9c38b39f4de1835f34e8a58bcea85461f37e7aa2d5a2dd484569e670ff48", "14/15": "bbc708aba767ccf554c75a7e06816abda3fc1804ad6d0876ad8ce9575f4c3a88", "14/16": "823b1f82c4aed8ba27d19275505c8d1dee755086be48fcda64d323f55b4c62a5", - "15/17": "f2a715762dee048c50d5a6d7201adf462c89025d6b7f92632f4f85fc83275860", + "15/17": "ca24a262671d86221ad25aad86efbf27c7efa3750335dc0f6c8999ba23ff2cf3", } // resultHashes records the hash of the reach results of every case for @@ -47,7 +47,7 @@ var resultHashes = map[string]string{ "14/14/2": "a832ea709bf3df852814375a56e6329c35e2ff4c49ae34ee49afe042e9502d17", "14/15/3": "77dfd0ea8ab71869ef612d41b8c2e46852d4fb67a49458cb645321692af77ea0", "14/16/3": "13b6eecb686ddec3c2ba470fc9752ea66cff445d5e18e0772a71b15ae7fe1c9c", - "15/17/4": "84e2c97bb8b24d95a2bbc908c8a2d4352f6270d103fcad3cacc165085c9a845d", + "15/17/4": "96f5a284da37b626dda6eb66c6d031145d9ee2b2bc737a1ec2da402b51b28c5b", } // summaryHashes records the hash of the golden summaries of the cases for @@ -64,7 +64,7 @@ var summaryHashes = map[string]string{ "14/14/2": "27424c4e2525450479faba78421959738748811d70b9a93b81e2bbdf1733bf0d", "14/15/2": "5969f3cda15b666cbd902bbc2db953cbdef8c0ce48d5e77446cf243cdd0bb0e7", "14/16/2": "fb678202978212aae932e50a56a0d3dc7f0f794ce908988bf605ce9a6e04bc51", - "15/17/2": "d9430fbc80d91e922184db91241eb22c29725b43ac88bfb5dd6f99548e7ed43f", + "15/17/2": "69d9b0f80c4f5811cc07608a7a1f224499e5f4e3b3eb5b62d2c2cf842df142e7", } func TestGraphHashFollowsVersion(t *testing.T) { From 1fc999896052a562fb983f479d191da4b80560c3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 08:00:17 +0000 Subject: [PATCH 05/11] fix(callgraph): keep the record of a call of an operator reflection helper The record of a call of the value of a call went away when the inner callee was no reflection. operator.methodcaller("system", cmd)(os) calls a method by a name in a string, as getattr(os, name)() does, and no edge names that method. So os.system read as no-such-target. methodcaller and attrgetter of the operator module count as reflective callees now, and the outer call keeps its computed-name record. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- plugin/callgraph/python.go | 11 +++++++---- plugin/callgraph/records_test.go | 3 ++- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/plugin/callgraph/python.go b/plugin/callgraph/python.go index fb1b59a0..b586e732 100644 --- a/plugin/callgraph/python.go +++ b/plugin/callgraph/python.go @@ -228,8 +228,9 @@ func functionCallProcessor(functionCallNode *sitter.Node, argumentsNode *sitter. // decorator factory, runs code that the inner call gives. The // graph reaches that code through the inner call and the // reference edges from its callee, so the outer call is no record - // of its own. A reflective inner call, as getattr(o, n), gives a - // value that no edge names, so it keeps the record. Any other + // of its own. A reflective inner call, as getattr(o, n) or + // operator.methodcaller("system"), gives a callable that runs a + // name that no edge names, so it keeps the record. Any other // callee computes its name. if function.Type() == "call" && !pythonReflective(function, treeData) { return result @@ -393,11 +394,13 @@ func dissectAttributeQualifierWithDepth(node *sitter.Node, treeData []byte, curr } // pythonReflectiveCallees name the calls whose value the graph cannot -// name: a lookup of an attribute or a module by a name, and a run of code -// in a string. +// name: a lookup of an attribute or a module by a name, a run of code in +// a string, and a callable of the operator module that calls a method or +// reads an attribute by a name, as operator.methodcaller("system")(os). var pythonReflectiveCallees = map[string]bool{ "getattr": true, "__import__": true, "importlib.import_module": true, "import_module": true, "eval": true, "exec": true, "compile": true, + "operator.methodcaller": true, "methodcaller": true, "operator.attrgetter": true, "attrgetter": true, } // pythonReflective reports whether a call reads its value by reflection. diff --git a/plugin/callgraph/records_test.go b/plugin/callgraph/records_test.go index ec93f6cc..974d3a97 100644 --- a/plugin/callgraph/records_test.go +++ b/plugin/callgraph/records_test.go @@ -235,10 +235,11 @@ func TestUnresolvedCallSites(t *testing.T) { name: "python computed receiver and call result", language: pythonLanguage(t), file: "recv.py", - source: "handlers[name].run()\nf()()\nmake(x)(y)\ngetattr(o, n)()\n", + source: "import operator\nhandlers[name].run()\nf()()\nmake(x)(y)\ngetattr(o, n)()\noperator.methodcaller(\"system\", cmd)(os)\n", expected: []site{ {"handlers[name].run", "handlers[name]", "run", UnresolvedComputedName}, {"getattr(o, n)", "", "", UnresolvedComputedName}, + {"operator.methodcaller(\"system\", cmd)", "", "", UnresolvedComputedName}, }, }, { From ad595e20b638572958d4cabe2c2d98916ef633d8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 13:41:09 +0000 Subject: [PATCH 06/11] Read the type hints of Python into the graph The link step did not know the class of a Python receiver that a type hint names. A call on a typed parameter, on a local that holds the result of a function with a return hint, or on a field of a class had a record of an unknown receiver and low name edges. The receiver rows of Python marked 16 forms as gaps. The index step records the hints: the class of a parameter, with Optional, Union and `| None` unwrapped, the return type and the tuple positions of a function, an annotated local, a name that holds a call result, a typed name of the top level of a module, and the fields of a class from its annotations and from the attributes that its methods set. A call on a typed name gets the key of the name in its scope as its target, as the Go index does, so the link step reads the type. A call that the call graph bound to a method of a class of the file keeps that target. A result of a reflective call, as import_module, has no type. The link step trusts the hints as it trusts TypeScript types. A hint of a dependency class gives the external key of the method, the class hierarchy analysis adds the overrides in the project and in the unit, and an abstract method with no override is a record with the reason interface-receiver. The walk now sets the class of an instance of an external class, so the analysis runs for it. All 16 Python receiver rows give their wanted answer. Six cases cover the forms. The external-base-attribute case gets a member edge in place of a name edge. The method measure adds a detail file per target. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- docs/projectgraph.md | 35 ++ plugin/callgraph/python.go | 8 +- plugin/indexer/calls.go | 62 +++- plugin/indexer/golang.go | 19 +- plugin/indexer/languages.go | 5 + plugin/indexer/plugin_test.go | 2 +- plugin/indexer/python.go | 326 +++++++++++++++++- plugin/indexer/python_test.go | 114 ++++++ plugin/indexer/testdata/golden/app.py.json | 7 + plugin/indexer/types_test.go | 6 +- projectgraph/cha.go | 36 +- projectgraph/declared.go | 49 ++- projectgraph/profile.go | 5 + projectgraph/python.go | 2 + projectgraph/resolve.go | 34 ++ projectgraph/walk.go | 6 + .../python/call-result-local/expect.yaml | 46 +++ .../call-result-local/project/factory.py | 10 + .../python/call-result-local/project/main.py | 10 + .../dependency-typed-return/expect.yaml | 61 ++++ .../dependency-typed-return/project/main.py | 12 + .../venv/site-packages/dep/__init__.py | 31 ++ .../summaries/pypi_dep_1.0.0.json | 83 +++++ .../external-base-attribute/expect.yaml | 16 +- .../python/typed-base-subclass/expect.yaml | 61 ++++ .../typed-base-subclass/project/main.py | 21 ++ .../venv/site-packages/dep/__init__.py | 31 ++ .../summaries/pypi_dep_1.0.0.json | 83 +++++ .../cases/python/typed-field/expect.yaml | 41 +++ .../python/typed-field/project/factory.py | 6 + .../cases/python/typed-field/project/main.py | 26 ++ .../cases/python/typed-parameter/expect.yaml | 56 +++ .../python/typed-parameter/project/factory.py | 6 + .../python/typed-parameter/project/main.py | 14 + .../cases/python/typed-return/expect.yaml | 48 +++ .../python/typed-return/project/Procfile | 1 + .../python/typed-return/project/factory.py | 20 ++ .../cases/python/typed-return/project/main.py | 9 + test/projectgraph/methods_test.go | 25 +- test/projectgraph/receivers_test.go | 40 --- test/projectgraph/version_test.go | 6 +- 41 files changed, 1392 insertions(+), 87 deletions(-) create mode 100644 plugin/indexer/python_test.go create mode 100644 test/projectgraph/cases/python/call-result-local/expect.yaml create mode 100644 test/projectgraph/cases/python/call-result-local/project/factory.py create mode 100644 test/projectgraph/cases/python/call-result-local/project/main.py create mode 100644 test/projectgraph/cases/python/dependency-typed-return/expect.yaml create mode 100644 test/projectgraph/cases/python/dependency-typed-return/project/main.py create mode 100644 test/projectgraph/cases/python/dependency-typed-return/project/venv/site-packages/dep/__init__.py create mode 100644 test/projectgraph/cases/python/dependency-typed-return/summaries/pypi_dep_1.0.0.json create mode 100644 test/projectgraph/cases/python/typed-base-subclass/expect.yaml create mode 100644 test/projectgraph/cases/python/typed-base-subclass/project/main.py create mode 100644 test/projectgraph/cases/python/typed-base-subclass/project/venv/site-packages/dep/__init__.py create mode 100644 test/projectgraph/cases/python/typed-base-subclass/summaries/pypi_dep_1.0.0.json create mode 100644 test/projectgraph/cases/python/typed-field/expect.yaml create mode 100644 test/projectgraph/cases/python/typed-field/project/factory.py create mode 100644 test/projectgraph/cases/python/typed-field/project/main.py create mode 100644 test/projectgraph/cases/python/typed-parameter/expect.yaml create mode 100644 test/projectgraph/cases/python/typed-parameter/project/factory.py create mode 100644 test/projectgraph/cases/python/typed-parameter/project/main.py create mode 100644 test/projectgraph/cases/python/typed-return/expect.yaml create mode 100644 test/projectgraph/cases/python/typed-return/project/Procfile create mode 100644 test/projectgraph/cases/python/typed-return/project/factory.py create mode 100644 test/projectgraph/cases/python/typed-return/project/main.py diff --git a/docs/projectgraph.md b/docs/projectgraph.md index 25d63c23..ed26f333 100644 --- a/docs/projectgraph.md +++ b/docs/projectgraph.md @@ -256,6 +256,41 @@ unpacked wheel holds it. These rules apply to it. of the wheel. A `test` or a `testing` directory stays, because `django.test` and `numpy.testing` are modules that programs import. +A type hint is the only declared type that Python has. The link step +trusts it as it trusts a TypeScript type, because both are erased at run +time and a wrong hint gives a wrong edge, not a crash. These rules apply. + +- The index records the class name of a hint of a parameter, as `Parser` + in `run(p: Parser)` or `run(p: Parser = None)`. `Optional[Parser]`, + `Union[Parser, None]` and `Parser | None` give `Parser`, and so does the + forward reference `"Parser"`. A builtin, as `int` or `str`, a container, + as `list[Parser]`, and a union of two classes give no type. +- A return hint gives the result type of the function. A tuple hint, as + `tuple[Parser, int]`, gives the type of each position, so `p, n = + pair()` binds `Parser` to `p`. +- A local with a hint, as `t: Thing = make()`, has that type. A name that + holds the result of a call, as `t = make()` or `a, b = make()`, has the + result type of the call at its position, as a Go short variable + declaration does. A name of the top level of a module that is no + variable of its own, as `q` in `q, n = pair()`, has it too. A call that + reads its value by reflection, as `importlib.import_module(name)` or + `getattr(o, name)`, gives no type. +- A class has the fields that its body declares with a hint, as the field + of a dataclass, and the attributes of the instance that its methods set, + as `self.parser = Parser()` or `self.parser: Parser = p`. The first + setting of a name wins. +- A call on a typed name is a `member` call of the method of its class, + with no record. A hint that names a class of a dependency gives the + external key of the method, as `dep//Thing//do`, and the class hierarchy + analysis adds the `cha` edges to the overrides in the project and in the + unit of the dependency. A method that `@abstractmethod` marks, with no + override in the graph, is a record with the reason `interface-receiver`, + as a Go interface with no implementation. +- A call on the result of a function with no return hint keeps the rules + above the table: a record with the reason `unknown-receiver` and the + `name` edges of the method name. A function of a dependency that no unit + gives is a record with the reason `no-dependency-source`. + `examples/projectgraph` takes `-site-packages ` and adds each distribution of the directory as a unit, by its `.dist-info`. diff --git a/plugin/callgraph/python.go b/plugin/callgraph/python.go index b586e732..d4dc7018 100644 --- a/plugin/callgraph/python.go +++ b/plugin/callgraph/python.go @@ -406,5 +406,11 @@ var pythonReflectiveCallees = map[string]bool{ // pythonReflective reports whether a call reads its value by reflection. func pythonReflective(call *sitter.Node, treeData []byte) bool { callee := call.ChildByFieldName("function") - return callee != nil && pythonReflectiveCallees[callee.Content(treeData)] + return callee != nil && PythonReflectiveCallee(callee.Content(treeData)) +} + +// PythonReflectiveCallee reports whether a call of the callee text reads +// its value by reflection, so the value has no type that the code names. +func PythonReflectiveCallee(callee string) bool { + return pythonReflectiveCallees[callee] } diff --git a/plugin/indexer/calls.go b/plugin/indexer/calls.go index 209aa607..828598e0 100644 --- a/plugin/indexer/calls.go +++ b/plugin/indexer/calls.go @@ -11,6 +11,25 @@ import ( "github.com/safedep/code/plugin/callgraph" ) +// retargetTyped gives a call on a receiver that a declared type of the +// scope names the key of the name in its scope as its target, in a language +// with typedReceivers. The call graph bound the receiver to the value of an +// assignment, as dep//new_thing for t = dep.new_thing(), and a call on that +// value, as dep//new_thing//do, names no method. The declared type or the +// result type of the name does. A call that the call graph bound to a +// method of a class of the file, as Parser//render for p = Parser(), keeps +// that target. +func (p *projection) retargetTyped(call *fileindex.Call) { + for _, t := range call.Targets { + if def, ok := p.defs[fileindex.Parent(t)]; ok && def.Kind == fileindex.DefClass { + return + } + } + if target, ok := p.typedTarget(call.Caller, call.Receiver, call.Method); ok { + call.Targets = []string{target} + } +} + func (p *projection) addCalls() { for key, node := range p.cg.Nodes { for _, ref := range node.CallsTo { @@ -33,7 +52,7 @@ func (p *projection) addCalls() { targets = local p.members[[2]uint32{ref.Site.StartLine, ref.Site.StartColumn}] = true } - p.ix.Calls = append(p.ix.Calls, fileindex.Call{ + call := fileindex.Call{ Caller: p.caller(key, ref.Site), Line: int(ref.Site.StartLine), Column: int(ref.Site.StartColumn), @@ -43,7 +62,9 @@ func (p *projection) addCalls() { Method: ref.Method, Kind: kind, Args: p.arguments(key, ref.Site), - }) + } + p.retargetTyped(&call) + p.ix.Calls = append(p.ix.Calls, call) } } } @@ -210,8 +231,22 @@ func (p *projection) addUnresolved() { if p.members[[2]uint32{u.Site.StartLine, u.Site.StartColumn}] && u.Reason == callgraph.UnresolvedUnknownReceiver { continue } + caller := p.caller(u.Caller, u.Site) + if target, ok := p.typedTarget(caller, u.Receiver, u.Method); ok && u.Reason == callgraph.UnresolvedUnknownReceiver { + p.ix.Calls = append(p.ix.Calls, fileindex.Call{ + Caller: caller, + Line: int(u.Site.StartLine), + Column: int(u.Site.StartColumn), + Callee: u.Text, + Targets: []string{target}, + Receiver: u.Receiver, + Method: u.Method, + Kind: fileindex.CallCall, + }) + continue + } p.ix.Unresolved = append(p.ix.Unresolved, fileindex.Unresolved{ - Caller: p.caller(u.Caller, u.Site), + Caller: caller, Line: int(u.Site.StartLine), Column: int(u.Site.StartColumn), Text: u.Text, @@ -222,6 +257,27 @@ func (p *projection) addUnresolved() { } } +// typedTarget returns the target of a call on a receiver whose first name a +// declared type of the scope chain of the caller types, as main.py//run//t//do +// for t.do() in run(t: Thing), in a language with typedReceivers. The link +// step reads the type of the name from the scope, as it does for Go. +func (p *projection) typedTarget(caller, receiver, method string) (string, bool) { + if !p.rules().typedReceivers || receiver == "" || method == "" { + return "", false + } + names := strings.Split(receiver, ".") + for _, name := range names { + if !isIdentifier(name) { + return "", false + } + } + scope, ok := p.typedScope(caller, names[0]) + if !ok { + return "", false + } + return fileindex.Join(scope, append(names, method)...), true +} + func (p *projection) addTruncations() { for _, t := range p.cg.Truncations() { p.truncate(fileindex.TruncationKind(t.Kind), t.Namespace, t.Line) diff --git a/plugin/indexer/golang.go b/plugin/indexer/golang.go index fcb30db1..01b7d88d 100644 --- a/plugin/indexer/golang.go +++ b/plugin/indexer/golang.go @@ -424,18 +424,19 @@ func childrenByField(node *sitter.Node, field string) []*sitter.Node { // typedScope returns the definition in the scope chain of the caller that // declares the type of a parameter or a local variable of the name. func (p *projection) typedScope(caller, name string) (string, bool) { - for scope := caller; fileindex.HasKeyPrefix(scope, p.cg.FileName) && scope != p.cg.FileName; scope = fileindex.Parent(scope) { - def, ok := p.defs[scope] - if !ok { - continue - } - for _, typed := range [][]fileindex.Typed{def.Params, def.Locals} { - for _, t := range typed { - if t.Name == name { - return scope, true + for scope := caller; fileindex.HasKeyPrefix(scope, p.cg.FileName); scope = fileindex.Parent(scope) { + if def, ok := p.defs[scope]; ok { + for _, typed := range [][]fileindex.Typed{def.Params, def.Locals} { + for _, t := range typed { + if t.Name == name { + return scope, true + } } } } + if scope == p.cg.FileName { + break + } } return "", false } diff --git a/plugin/indexer/languages.go b/plugin/indexer/languages.go index 71120a58..34e0dae0 100644 --- a/plugin/indexer/languages.go +++ b/plugin/indexer/languages.go @@ -24,6 +24,11 @@ type languageRules struct { // not define keeps the name as its target, as Box for new Box<>() in // Java. rawUnbound bool + // typedReceivers marks a language whose call graph does not type a + // receiver that a declared type of the scope names, as Python. A call + // on such a receiver gets the key of the name in its scope as its + // target, as main.py//run//t//do, so the link step reads the type. + typedReceivers bool // packageOf returns the package or the namespace of the file, which the // files of one unit share. diff --git a/plugin/indexer/plugin_test.go b/plugin/indexer/plugin_test.go index 45e06271..14cdb360 100644 --- a/plugin/indexer/plugin_test.go +++ b/plugin/indexer/plugin_test.go @@ -103,7 +103,7 @@ var goldenHashes = map[string]string{ "12": "974943df4185f9c3070dcfdfedff8e91912e257d255a936f7c27608bb63a69f0", "13": "97614d8b96782e65813ee409e4a3f33c4655157b5dfa58baf17cd3677958c619", "14": "26f22c80fd4120cd04f228a53188accb1695434d0ed1ff5654b110981f8f44dd", - "15": "327ac05cb53e44b88a8e5e0185bdd1fb32ac0cdb9e8a8eb27b9804843bdca48d", + "15": "c4b949e72be5b2b4474e06f20ae18b0cc4e080b6c5fe20ccaf712c3714bf2a12", } func TestFileIndexGoldenHashFollowsVersion(t *testing.T) { diff --git a/plugin/indexer/python.go b/plugin/indexer/python.go index 968f33c1..6ced7d44 100644 --- a/plugin/indexer/python.go +++ b/plugin/indexer/python.go @@ -1,10 +1,13 @@ package indexer import ( + "github.com/safedep/code/plugin/callgraph" "regexp" "slices" + "strings" "github.com/safedep/code/fileindex" + "github.com/safedep/code/langcode" "github.com/safedep/code/pkg/ts" sitter "github.com/smacker/go-tree-sitter" ) @@ -14,12 +17,15 @@ var pythonRules = languageRules{ "class_definition": fileindex.DefClass, "function_definition": fileindex.DefFunction, }, - constructor: "__init__", - bases: pythonBases, - decorators: decorators, - assigned: pythonAssigned, - exports: pythonExports, - visitors: []func(p *projection) visitor{flagVisitor, mainGuardVisitor, referenceVisitor}, + constructor: "__init__", + typedReceivers: true, + bases: pythonBases, + fields: pythonFields, + decorators: decorators, + signature: pythonSignature, + assigned: pythonAssigned, + exports: pythonExports, + visitors: []func(p *projection) visitor{flagVisitor, mainGuardVisitor, referenceVisitor}, dynamic: &dynamicSyntax{ call: "call", assignment: "assignment", @@ -132,3 +138,311 @@ func (p *projection) addAll(value *sitter.Node, line int) { p.export(name, name, line) } } + +// pythonSignature fills the typed parameters, the return type and the typed +// locals of a Python function from its type hints. A hint is the only +// declared type that Python has, and the link step trusts it as it trusts +// a TypeScript type: both are erased at run time. A module gets the typed +// names of its top level that are no variable of their own, as q in +// q, n = pair(). +func pythonSignature(p *projection, def *fileindex.Definition, node *sitter.Node) { + if def.Kind == fileindex.DefModule { + for _, t := range p.pythonLocals(node, "") { + if _, defined := p.defs[fileindex.Join(def.Key, t.Name)]; !defined { + def.Locals = append(def.Locals, t) + } + } + return + } + if def.Kind != fileindex.DefFunction && def.Kind != fileindex.DefMethod && def.Kind != fileindex.DefConstructor { + return + } + def.Params = p.pythonParams(node.ChildByFieldName("parameters")) + if ret := node.ChildByFieldName("return_type"); ret != nil { + def.Returns = p.pythonTypeName(ret) + if results := p.pythonTupleTypes(ret); len(results) > 1 { + def.Results = results + } + } + def.Locals = p.pythonLocals(node.ChildByFieldName("body"), "") +} + +// pythonParams returns the parameters of a function that declare a type +// name, as parser in run(parser: Parser) or run(parser: Parser = None). +func (p *projection) pythonParams(list *sitter.Node) []fileindex.Typed { + if list == nil { + return nil + } + var typed []fileindex.Typed + for _, param := range ts.NamedChildren(list) { + var name *sitter.Node + switch param.Type() { + case "typed_parameter": + if param.NamedChildCount() > 0 && param.NamedChild(0).Type() == "identifier" { + name = param.NamedChild(0) + } + case "typed_default_parameter": + name = param.ChildByFieldName("name") + default: + continue + } + if name == nil { + continue + } + if typ := p.pythonTypeName(param.ChildByFieldName("type")); typ != "" { + typed = append(typed, fileindex.Typed{Name: p.text(name), Type: typ}) + } + } + return typed +} + +// pythonOptional names the hints that wrap one type and let it be None. +var pythonOptional = map[string]bool{"Optional": true, "typing.Optional": true, "Union": true, "typing.Union": true} + +// pythonTuples names the hints of a tuple, whose members a tuple return +// gives by position. +var pythonTuples = map[string]bool{"tuple": true, "Tuple": true, "typing.Tuple": true} + +// pythonTypeName returns the class name that a type hint names, as written: +// Parser, dep.Parser, the Parser of Optional[Parser], Parser | None or +// "Parser", and "" for a builtin, a container or a union of two classes. +func (p *projection) pythonTypeName(node *sitter.Node) string { + if node == nil { + return "" + } + switch node.Type() { + case "type": + if node.NamedChildCount() == 0 { + return "" + } + return p.pythonTypeName(node.NamedChild(0)) + case "identifier", "attribute": + if name := p.text(node); !langcode.IsBuiltin(langcode.Python, name) { + return name + } + case "string": + for _, child := range ts.NamedChildren(node) { + if child.Type() == "string_content" { + if name := p.text(child); isDottedName(name) && !langcode.IsBuiltin(langcode.Python, name) { + return name + } + } + } + case "generic_type", "subscript": + head, args := p.pythonGenericParts(node) + if !pythonOptional[head] { + return "" + } + return p.pythonOneOf(args) + case "binary_operator": + return p.pythonOneOf([]*sitter.Node{node.ChildByFieldName("left"), node.ChildByFieldName("right")}) + } + return "" +} + +// pythonOneOf returns the one class among the members of a union, with +// None left out, and "" for two classes or none. +func (p *projection) pythonOneOf(members []*sitter.Node) string { + name := "" + for _, m := range members { + if m == nil || m.Type() == "none" { + continue + } + if m.Type() == "binary_operator" { + if inner := p.pythonOneOf([]*sitter.Node{m.ChildByFieldName("left"), m.ChildByFieldName("right")}); inner != "" && name == "" { + name = inner + continue + } + return "" + } + t := p.pythonTypeName(m) + if t == "" || name != "" { + return "" + } + name = t + } + return name +} + +// pythonGenericParts returns the head of a hint with type arguments and +// the argument nodes, as Optional and Parser for Optional[Parser]. The +// grammar writes the hint as a generic_type, or as a subscript in an +// older version. +func (p *projection) pythonGenericParts(node *sitter.Node) (string, []*sitter.Node) { + if node.Type() == "subscript" { + return p.text(node.ChildByFieldName("value")), childrenByField(node, "subscript") + } + head := "" + var args []*sitter.Node + for _, child := range ts.NamedChildren(node) { + switch child.Type() { + case "identifier", "attribute": + if head == "" { + head = p.text(child) + } + case "type_parameter": + args = append(args, ts.NamedChildren(child)...) + } + } + return head, args +} + +// pythonTupleTypes returns the class name of each member of a tuple hint, +// as a return of tuple[Parser, int] gives them by position, and nil for a +// hint that is no tuple. +func (p *projection) pythonTupleTypes(node *sitter.Node) []string { + if node != nil && node.Type() == "type" && node.NamedChildCount() > 0 { + node = node.NamedChild(0) + } + if node == nil || node.Type() != "generic_type" && node.Type() != "subscript" { + return nil + } + head, args := p.pythonGenericParts(node) + if !pythonTuples[head] { + return nil + } + out := make([]string, 0, len(args)) + for _, arg := range args { + out = append(out, p.pythonTypeName(arg)) + } + return out +} + +// pythonLocals returns the names that a function body binds with a type: +// an annotated assignment, as t: Thing = make(), and a name that holds the +// result of a call, as t = make() or a, b = make(). With an instance name, +// as self, it returns the attributes of the instance that the body sets +// instead, as the fields of the class. +func (p *projection) pythonLocals(body *sitter.Node, instance string) []fileindex.Typed { + var typed []fileindex.Typed + stop := map[string]bool{"function_definition": true, "class_definition": true, "lambda": true} + walkBody(body, stop, func(node *sitter.Node) { + if node.Type() != "assignment" { + return + } + left := node.ChildByFieldName("left") + if left == nil { + return + } + var names []*sitter.Node + switch { + case instance != "" && left.Type() == "attribute": + object, attr := left.ChildByFieldName("object"), left.ChildByFieldName("attribute") + if object == nil || attr == nil || object.Type() != "identifier" || p.text(object) != instance { + return + } + names = []*sitter.Node{attr} + case instance == "" && left.Type() == "identifier": + names = []*sitter.Node{left} + case instance == "" && (left.Type() == "pattern_list" || left.Type() == "tuple_pattern"): + names = ts.NamedChildren(left) + default: + return + } + if typ := p.pythonTypeName(node.ChildByFieldName("type")); typ != "" && len(names) == 1 { + typed = append(typed, fileindex.Typed{Name: p.text(names[0]), Type: typ}) + return + } + typed = append(typed, p.pythonAssignedTypes(names, node.ChildByFieldName("right"))...) + }) + return typed +} + +// pythonAssignedTypes returns the names of an assignment that hold the +// result of a call, each at its position, as a and b in a, b = make(). A +// call that reads its value by reflection, as importlib.import_module(name), +// gives no type. +func (p *projection) pythonAssignedTypes(names []*sitter.Node, right *sitter.Node) []fileindex.Typed { + if right == nil || right.Type() != "call" { + return nil + } + function := right.ChildByFieldName("function") + if function == nil || function.Type() != "identifier" && function.Type() != "attribute" || callgraph.PythonReflectiveCallee(p.text(function)) { + return nil + } + var typed []fileindex.Typed + for i, name := range names { + if name.Type() == "identifier" && p.text(name) != "_" { + typed = append(typed, fileindex.Typed{Name: p.text(name), Type: p.text(function), Result: true, Index: i}) + } + } + return typed +} + +// pythonFields returns the fields of a Python class that declare a type or +// hold the result of a call: an annotated name of the class body, as the +// field of a dataclass, and an attribute of the instance that a method +// sets, as self.parser: Parser = Parser() or self.parser = Parser(). The +// first setting of a name wins. +func pythonFields(p *projection, class *sitter.Node) []fileindex.Typed { + body := class.ChildByFieldName("body") + if class.Type() != "class_definition" || body == nil { + return nil + } + var fields []fileindex.Typed + seen := map[string]bool{} + add := func(ts []fileindex.Typed) { + for _, t := range ts { + if !seen[t.Name] { + seen[t.Name] = true + fields = append(fields, t) + } + } + } + for _, statement := range ts.NamedChildren(body) { + if statement.Type() != "expression_statement" || statement.NamedChildCount() == 0 { + continue + } + assignment := statement.NamedChild(0) + left := assignment.ChildByFieldName("left") + if assignment.Type() != "assignment" || left == nil || left.Type() != "identifier" { + continue + } + if typ := p.pythonTypeName(assignment.ChildByFieldName("type")); typ != "" { + add([]fileindex.Typed{{Name: p.text(left), Type: typ}}) + } + } + for _, member := range ts.NamedChildren(body) { + if member.Type() == "decorated_definition" { + member = member.ChildByFieldName("definition") + } + if member == nil || member.Type() != "function_definition" { + continue + } + params := member.ChildByFieldName("parameters") + if params == nil || params.NamedChildCount() == 0 || params.NamedChild(0).Type() != "identifier" { + continue + } + add(p.pythonLocals(member.ChildByFieldName("body"), p.text(params.NamedChild(0)))) + } + return fields +} + +// isDottedName reports whether a text is a name or a dotted name, as a +// forward reference in a string hint writes it. +func isDottedName(text string) bool { + if text == "" { + return false + } + for _, part := range strings.Split(text, ".") { + if !isIdentifier(part) { + return false + } + } + return true +} + +// isIdentifier reports whether a text is one name. +func isIdentifier(text string) bool { + if text == "" { + return false + } + for i, r := range text { + letter := r == '_' || r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' + digit := i > 0 && r >= '0' && r <= '9' + if !letter && !digit { + return false + } + } + return true +} diff --git a/plugin/indexer/python_test.go b/plugin/indexer/python_test.go new file mode 100644 index 00000000..7ea1d6ec --- /dev/null +++ b/plugin/indexer/python_test.go @@ -0,0 +1,114 @@ +package indexer + +import ( + "strings" + "testing" + + "github.com/safedep/code/core" + "github.com/safedep/code/fileindex" + "github.com/stretchr/testify/assert" +) + +func TestFileIndexPythonTypeHints(t *testing.T) { + source := strings.Join([]string{ + "from dataclasses import dataclass", + "from typing import Optional", + "import dep", + "from dep import Thing", + "", + "def typed(t: Thing, o: Optional[Thing], u: Thing | None, d: dep.Doer, n: int, s: str = 'x', *args, **kwargs):", + " t.do()", + "", + "def two(a: Thing | Runner, lst: list[Thing]):", + " pass", + "", + "def returns() -> Thing:", + " return Thing()", + "", + "def forward() -> 'Thing':", + " return Thing()", + "", + "def optional() -> Optional[Thing]:", + " return None", + "", + "def pair() -> tuple[Thing, int]:", + " return Thing(), 1", + "", + "def builtin() -> int:", + " return 1", + "", + "def locals_():", + " a: Thing = make()", + " b = dep.new_thing()", + " c, n = dep.new_pair()", + " d = Thing()", + " e = 1", + " b.do()", + "", + "class App:", + " worker: Thing", + " count: int = 0", + "", + " def __init__(self, t: Thing):", + " self.t = dep.new_thing()", + " self.n = 1", + " self.k: Thing = t", + "", + " def go(self):", + " self.t.do()", + "", + "@dataclass", + "class Job:", + " thing: Thing", + " name: str", + }, "\n") + "\n" + ix := indexSource(t, core.LanguageCodePython, "a.py", source) + + t.Run("a parameter with a type hint has the type", func(t *testing.T) { + assert.Equal(t, []fileindex.Typed{{Name: "t", Type: "Thing"}, {Name: "o", Type: "Thing"}, {Name: "u", Type: "Thing"}, {Name: "d", Type: "dep.Doer"}}, + definitionOf(t, ix, "a.py//typed").Params, "a builtin type, a default of a builtin type and a star parameter name no type") + }) + t.Run("a union of two classes and a generic container name no type", func(t *testing.T) { + assert.Empty(t, definitionOf(t, ix, "a.py//two").Params) + }) + t.Run("a return hint gives the result type", func(t *testing.T) { + assert.Equal(t, "Thing", definitionOf(t, ix, "a.py//returns").Returns) + assert.Equal(t, "Thing", definitionOf(t, ix, "a.py//forward").Returns, "a forward reference in a string") + assert.Equal(t, "Thing", definitionOf(t, ix, "a.py//optional").Returns, "an Optional unwraps") + assert.Equal(t, "", definitionOf(t, ix, "a.py//builtin").Returns) + }) + t.Run("a tuple return gives the type of each position", func(t *testing.T) { + def := definitionOf(t, ix, "a.py//pair") + assert.Equal(t, "", def.Returns) + assert.Equal(t, []string{"Thing", ""}, def.Results) + }) + t.Run("a local has its annotation or the call that gives its value", func(t *testing.T) { + assert.Equal(t, []fileindex.Typed{ + {Name: "a", Type: "Thing"}, + {Name: "b", Type: "dep.new_thing", Result: true}, + {Name: "c", Type: "dep.new_pair", Result: true}, + {Name: "n", Type: "dep.new_pair", Result: true, Index: 1}, + {Name: "d", Type: "Thing", Result: true}, + }, definitionOf(t, ix, "a.py//locals_").Locals) + }) + t.Run("a call on a typed local names the local in its scope", func(t *testing.T) { + assert.Contains(t, callTargets(ix, "a.py//locals_"), "a.py//locals_//b//do") + assert.Contains(t, callTargets(ix, "a.py//typed"), "a.py//typed//t//do") + assert.NotContains(t, callTargets(ix, "a.py//locals_"), "dep//new_thing//do") + }) + t.Run("the fields of a class come from its annotations and its methods", func(t *testing.T) { + assert.Equal(t, []fileindex.Typed{{Name: "worker", Type: "Thing"}, {Name: "t", Type: "dep.new_thing", Result: true}, {Name: "k", Type: "Thing"}}, + classOf(t, ix, "a.py//App").Fields) + assert.Equal(t, []fileindex.Typed{{Name: "thing", Type: "Thing"}}, classOf(t, ix, "a.py//Job").Fields) + }) +} + +func callTargets(ix *fileindex.Index, caller string) []string { + var out []string + for _, c := range ix.Calls { + if c.Caller == caller { + out = append(out, c.Targets...) + } + } + return out +} diff --git a/plugin/indexer/testdata/golden/app.py.json b/plugin/indexer/testdata/golden/app.py.json index 06de79d4..5f4e785a 100644 --- a/plugin/indexer/testdata/golden/app.py.json +++ b/plugin/indexer/testdata/golden/app.py.json @@ -124,6 +124,13 @@ "start_line": 23, "end_line": 35, "exported": true, + "locals": [ + { + "name": "p", + "type": "Parser", + "result": true + } + ], "flags": [ { "kind": "reflection", diff --git a/plugin/indexer/types_test.go b/plugin/indexer/types_test.go index e2ffeabb..a87e3208 100644 --- a/plugin/indexer/types_test.go +++ b/plugin/indexer/types_test.go @@ -79,9 +79,9 @@ func TestFileIndexDeclaredTypes(t *testing.T) { }) } -func TestFileIndexDeclaredTypesOnlyInTypeScript(t *testing.T) { - ix := indexSource(t, core.LanguageCodePython, "a.py", "def run(p: Parser):\n p.parse()\n") - assert.Empty(t, definitionOf(t, ix, "a.py//run").Params) +func TestFileIndexDeclaredTypesOnlyInTypedLanguages(t *testing.T) { + ix := indexSource(t, core.LanguageCodeRuby, "a.rb", "def run(p)\n p.parse\nend\n") + assert.Empty(t, definitionOf(t, ix, "a.rb//run").Params) } func TestFileIndexPropertyFunctions(t *testing.T) { diff --git a/projectgraph/cha.go b/projectgraph/cha.go index c26ff585..4a0e91d4 100644 --- a/projectgraph/cha.go +++ b/projectgraph/cha.go @@ -255,6 +255,16 @@ func (b *builder) subclasses(class string) []string { for _, base := range b.typeBases(c.Key) { b.children[base] = append(b.children[base], c.Key) } + // A class with a base of a dependency, as Worker(Thread), is + // a subclass of the external key of the base, so a call on + // a value of that declared type reaches its overrides. + if rec, ok := b.classes[c.Key]; ok { + for _, base := range b.bases(rec) { + if !b.isType(base) && b.g.IsExternal(base) && !slices.Contains(b.children[base], c.Key) { + b.children[base] = append(b.children[base], c.Key) + } + } + } } } } @@ -284,10 +294,12 @@ func (b *builder) overrides(class string, call fileindex.Call, found string) []s return b.implementations(class, call) } var out []string - for _, sub := range b.subclasses(class) { - key, fail := b.method(sub, call.Method) - if fail == nil && key != found && b.isMethod(key) && !slices.Contains(out, key) { - out = append(out, key) + for _, c := range b.classDefinitions(class) { + for _, sub := range b.subclasses(c) { + key, fail := b.method(sub, call.Method) + if fail == nil && key != found && b.isMethod(key) && !slices.Contains(out, key) { + out = append(out, key) + } } } family := b.languageOf(class).Family() @@ -303,6 +315,22 @@ func (b *builder) overrides(class string, call fileindex.Call, found string) []s return out } +// classDefinitions returns the class itself, and for the external key of a +// class of a unit of dependency code, as dep//Doer, the definitions that +// the units give for it, so the subclasses inside the unit count too. +func (b *builder) classDefinitions(class string) []string { + out := []string{class} + if b.g.IsExternal(class) { + defs, _ := b.unitDefinitions(b.origins[class]) + for _, d := range defs { + if b.isType(d) { + out = append(out, d) + } + } + } + return out +} + // openClasses returns the first-party classes of Python, JavaScript and // TypeScript with a base that names no class, in path order: a call, as // Loading(Base) in class App extends Loading(Base), or a parameter, as B in diff --git a/projectgraph/declared.go b/projectgraph/declared.go index 6dbb6c11..6d774041 100644 --- a/projectgraph/declared.go +++ b/projectgraph/declared.go @@ -9,14 +9,18 @@ import ( // typedName returns the class or interface that the declared type of a // parameter or a typed local variable of the name names. The name belongs -// to the scope or to a function that holds the scope, as the parameter of a -// method in an arrow function of the method. A Go local that holds a +// to the scope, to a function that holds the scope, as the parameter of a +// method in an arrow function of the method, or to the module, as a name +// that a Python module binds at its top level. A Go local that holds a // result of a call has the type of the result at its position. A name // with no known type gives the reason of the record of a call on it. func (b *builder) typedName(sc *scope, scopeKey, name string) (string, fileindex.UnresolvedReason, bool) { - for key := scopeKey; fileindex.HasKeyPrefix(key, sc.ix.Path) && key != sc.ix.Path; key = fileindex.Parent(key) { + for key := scopeKey; fileindex.HasKeyPrefix(key, sc.ix.Path); key = fileindex.Parent(key) { def := b.defs[key] if def == nil { + if key == sc.ix.Path { + break + } continue } for _, typed := range [][]fileindex.Typed{def.Params, def.Locals} { @@ -32,6 +36,11 @@ func (b *builder) typedName(sc *scope, scopeKey, name string) (string, fileindex if !ok { return "", fileindex.UnresolvedUnknownReceiver, false } + // A call of a class makes an instance of it, as + // t = Thing() in Python. + if class, ok := b.constructedType(sc, fn); ok && t.Index == 0 { + return class, "", true + } res, reason, ok := b.resultType(fileindex.PackageOf(sc.ix.Path), fn, t.Index) // A name that holds a whole Option or Result has no // type of a method of the value inside. @@ -41,6 +50,9 @@ func (b *builder) typedName(sc *scope, scopeKey, name string) (string, fileindex return res.class, reason, ok } } + if key == sc.ix.Path { + break + } } return "", fileindex.UnresolvedUnknownReceiver, false } @@ -77,6 +89,28 @@ func (b *builder) fieldType(class, name string) (string, bool) { return "", false } +// constructedType returns the class whose instance a call of the key +// makes, in a language where a call of a class constructs it, as Python: a +// first-party class, or the type that a unit of dependency code gives for +// an external key, or the external key itself with no unit. +func (b *builder) constructedType(sc *scope, key string) (string, bool) { + if !sc.profile.declaredTypes || sc.profile.typeConversions { + return "", false + } + switch { + case b.kindOf(key) == NodeClass: + return key, true + case b.g.IsExternal(key) && strings.Contains(key, fileindex.Separator): + if t, ok := b.unitType(key); ok { + return t, true + } + if _, provided := b.unitDefinitions(b.origins[key]); !provided && startsUpper(fileindex.Last(key)) { + return key, true + } + } + return "", false +} + // resolveType resolves a type name as written at a key of the file to a // first-party class or interface. A type of a dependency narrows nothing, as an // instance of a project class can extend it. A Go value holds its declared @@ -85,22 +119,25 @@ func (b *builder) fieldType(class, name string) (string, bool) { // dependency code gives narrows in any language with no subclass, as a // Rust struct: a class gives its external key, and an interface or a trait // gives its definition, whose implementations the class hierarchy analysis -// reads. +// reads. A language whose declared types the link step trusts, as the +// type hints of Python, gives the external key too, and the class +// hierarchy analysis adds the first-party subclasses. func (b *builder) resolveType(sc *scope, from, text string) (string, bool) { if sc == nil { return "", false } key, ok := b.resolveNameIn(sc, from, text) + trusted := sc.profile.implicitInterfaces || sc.profile.unitTypes || sc.profile.declaredTypes switch { case !ok: return "", false case b.isType(key): return key, true case b.g.IsExternal(key) && strings.Contains(key, fileindex.Separator): - if t, ok := b.unitType(key); ok && (sc.profile.implicitInterfaces || sc.profile.unitTypes) { + if t, ok := b.unitType(key); ok && trusted { return t, true } - return key, sc.profile.implicitInterfaces && b.kindOf(key) == "" + return key, (sc.profile.implicitInterfaces || sc.profile.declaredTypes) && b.kindOf(key) == "" } return "", false } diff --git a/projectgraph/profile.go b/projectgraph/profile.go index eb2579df..e5e5c2fb 100644 --- a/projectgraph/profile.go +++ b/projectgraph/profile.go @@ -60,6 +60,11 @@ type profile struct { // crate, as Rust, so a declared type that a unit of dependency code // gives narrows a call to the method of the type. unitTypes bool + // declaredTypes marks a language whose declared types the link step + // trusts although the run time erases them, as the type hints of + // Python. A declared type of a dependency gives the method of the type + // and, with openBases, the cha edges to the first-party subclasses. + declaredTypes bool // sameTypeMethods name the methods that give a value of the type of // the receiver, as clone of Rust, also when the type derives them and // the code defines none. diff --git a/projectgraph/python.go b/projectgraph/python.go index 666f2588..3f723575 100644 --- a/projectgraph/python.go +++ b/projectgraph/python.go @@ -20,6 +20,8 @@ var pythonProfile = profile{ surface: pythonSurface, symbol: pythonSymbol, dependencies: pythonDependencies, + declaredTypes: true, + abstractReceiver: fileindex.UnresolvedInterfaceReceiver, instance: []string{"self", "cls"}, objectBase: "object", literals: map[string]bool{"None": true, "True": true, "False": true, "Ellipsis": true, "NotImplemented": true}, diff --git a/projectgraph/resolve.go b/projectgraph/resolve.go index d0725b4b..0e76429a 100644 --- a/projectgraph/resolve.go +++ b/projectgraph/resolve.go @@ -97,6 +97,12 @@ func (b *builder) linkCalls(ix *fileindex.Index) { if r.declared && (r.fail == nil || r.fail.name == call.Method) && (r.fail == nil || !b.forwards(r.class)) { b.narrow(site, call.Callee) } + // A call of an abstract method of a declared type with no + // override in the graph runs the method of a class elsewhere, + // as a call of a Go interface with no implementation. + if r.fail == nil && r.declared && overrides == 0 && sc.profile.abstractReceiver != "" && b.abstractMethod(r.key) { + b.addUnresolved(UnresolvedSite{Caller: from, Site: site, Text: call.Callee, Receiver: call.Receiver, Method: call.Method, Reason: sc.profile.abstractReceiver}) + } if r.fail != nil { reason := r.fail.reason // A value of an interface or a trait that no type of the @@ -554,6 +560,15 @@ func (b *builder) resolveLocal(sc *scope, caller, target string) resolution { r.rule = cmp.Or(r.rule, ruleLocal) return r } + // A typed name of the top level of the module, as q in q, n = pair(), + // has no node of its own, so its declared type gives its class. + if unit == sc.ix.Path && len(segs) > 1 { + if class, _, ok := b.typedName(sc, unit, segs[0]); ok { + r := b.walkTyped(class, segs[1:]) + r.rule = cmp.Or(r.rule, ruleLocal) + return r + } + } return b.walkAs(unit, segs, ruleLocal) } @@ -687,6 +702,25 @@ func (b *builder) mayBeFirstParty(ix *fileindex.Index, spec string) bool { return r == nil || r.FirstParty(spec) } +// pythonAbstract names the decorators of a Python method with no body of +// its own, which a subclass must give. +var pythonAbstract = map[string]bool{"@abstractmethod": true, "@abc.abstractmethod": true} + +// abstractMethod reports whether a key names a method that a decorator +// marks abstract, in the project or in a unit of dependency code. +func (b *builder) abstractMethod(key string) bool { + defs := []string{key} + if b.defs[key] == nil && b.g.IsExternal(key) { + defs, _ = b.unitDefinitions(b.origins[key]) + } + for _, d := range defs { + if def := b.defs[d]; def != nil && slices.ContainsFunc(def.Decorators, func(d string) bool { return pythonAbstract[d] }) { + return true + } + } + return false +} + func (b *builder) kindOf(key string) NodeKind { if n := b.g.Nodes[key]; n != nil { return n.Kind diff --git a/projectgraph/walk.go b/projectgraph/walk.go index d0d32b73..9855829b 100644 --- a/projectgraph/walk.go +++ b/projectgraph/walk.go @@ -91,6 +91,12 @@ func (b *builder) walkFrom(r resolution, names []string) resolution { continue } } + // The last name on an instance of an external class is a + // method of the class, whose subclasses the class hierarchy + // analysis reads. + if r.virtual && i+1 == len(names) { + r.class = r.key + } r.key = b.deriveExternal(r.key, names[i:]) r.rule = cmp.Or(r.rule, ruleExternal) return r diff --git a/test/projectgraph/cases/python/call-result-local/expect.yaml b/test/projectgraph/cases/python/call-result-local/expect.yaml new file mode 100644 index 00000000..a9dabb67 --- /dev/null +++ b/test/projectgraph/cases/python/call-result-local/expect.yaml @@ -0,0 +1,46 @@ +source: safedep:hand +description: >- + run binds p to the result of make(), and make declares no return type. The index does not know the + value of the result, so the call on p has a record of an unknown receiver and a name edge joins it to + Parser.parse. +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py//run + to: factory.py//make + kind: direct + confidence: high + site: main.py:5 + - from: main.py//run + to: factory.py//Parser//parse + kind: name + confidence: low + site: main.py:6 +unresolved: + - caller: main.py//run + site: main.py:6 + reason: unknown-receiver + method: parse +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: main.py//run + site: main.py:10 + kind: direct + confidence: high + - key: factory.py//Parser//parse + site: main.py:6 + kind: name + confidence: low + - key: yaml//load + site: factory.py:6 + kind: direct + confidence: high + call_sites: + - caller: factory.py//Parser//parse + site: factory.py:6 + reached: true diff --git a/test/projectgraph/cases/python/call-result-local/project/factory.py b/test/projectgraph/cases/python/call-result-local/project/factory.py new file mode 100644 index 00000000..26a37d73 --- /dev/null +++ b/test/projectgraph/cases/python/call-result-local/project/factory.py @@ -0,0 +1,10 @@ +import yaml + + +class Parser: + def parse(self, text): + return yaml.load(text) + + +def make(): + return Parser() diff --git a/test/projectgraph/cases/python/call-result-local/project/main.py b/test/projectgraph/cases/python/call-result-local/project/main.py new file mode 100644 index 00000000..5cc72764 --- /dev/null +++ b/test/projectgraph/cases/python/call-result-local/project/main.py @@ -0,0 +1,10 @@ +from factory import make + + +def run(): + p = make() + p.parse("a: 1") + + +if __name__ == "__main__": + run() diff --git a/test/projectgraph/cases/python/dependency-typed-return/expect.yaml b/test/projectgraph/cases/python/dependency-typed-return/expect.yaml new file mode 100644 index 00000000..ac2920f8 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-typed-return/expect.yaml @@ -0,0 +1,61 @@ +source: safedep:hand +description: >- + The dependency dep declares the return type of new_thing as Thing and of new_doer as the abstract base + Doer. A call on the result of new_thing is a call of Thing.do. A call on the result of new_doer is a call + of Doer.do, and the class hierarchy analysis adds Thing.do, the one override in the dependency. Neither + site has a record. +units: + venv/site-packages/dep: pypi:dep@1.0.0 +import_dirs: + - venv/site-packages +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py//run + to: dep//new_thing + kind: direct + confidence: high + site: main.py:5 + - from: main.py//run + to: dep//Thing//do + kind: direct + confidence: high + site: main.py:6 + - from: main.py//run + to: dep//Doer//do + kind: direct + confidence: high + site: main.py:8 + - from: main.py//run + to: dep//Thing//do + kind: cha + confidence: medium + site: main.py:8 +unresolved: [] +results: + - target: subprocess//run + status: found + path: + - key: main.py + - key: main.py//run + site: main.py:12 + kind: direct + confidence: high + - key: dep//Thing//do + site: main.py:6 + kind: direct + confidence: high + - key: pypi:dep@1.0.0//dep/__init__.py//Thing//do + site: pypi:dep@1.0.0//dep/__init__.py:18 + kind: package + confidence: high + - key: subprocess//run + site: pypi:dep@1.0.0//dep/__init__.py:19 + kind: direct + confidence: high + call_sites: + - caller: pypi:dep@1.0.0//dep/__init__.py//Thing//do + site: pypi:dep@1.0.0//dep/__init__.py:19 + reached: true diff --git a/test/projectgraph/cases/python/dependency-typed-return/project/main.py b/test/projectgraph/cases/python/dependency-typed-return/project/main.py new file mode 100644 index 00000000..c8a13128 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-typed-return/project/main.py @@ -0,0 +1,12 @@ +import dep + + +def run(): + t = dep.new_thing() + t.do() + d = dep.new_doer() + d.do() + + +if __name__ == "__main__": + run() diff --git a/test/projectgraph/cases/python/dependency-typed-return/project/venv/site-packages/dep/__init__.py b/test/projectgraph/cases/python/dependency-typed-return/project/venv/site-packages/dep/__init__.py new file mode 100644 index 00000000..c1b1e823 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-typed-return/project/venv/site-packages/dep/__init__.py @@ -0,0 +1,31 @@ +import abc +import subprocess + + +class Doer(abc.ABC): + @abc.abstractmethod + def do(self): + raise NotImplementedError + + +class Runner(abc.ABC): + @abc.abstractmethod + def run(self): + raise NotImplementedError + + +class Thing(Doer): + def do(self): + subprocess.run(["ls"]) + + +def new_thing() -> Thing: + return Thing() + + +def new_doer() -> Doer: + return Thing() + + +def new_runner() -> Runner: + raise NotImplementedError diff --git a/test/projectgraph/cases/python/dependency-typed-return/summaries/pypi_dep_1.0.0.json b/test/projectgraph/cases/python/dependency-typed-return/summaries/pypi_dep_1.0.0.json new file mode 100644 index 00000000..657312ba --- /dev/null +++ b/test/projectgraph/cases/python/dependency-typed-return/summaries/pypi_dep_1.0.0.json @@ -0,0 +1,83 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "pypi:dep@1.0.0", + "symbols": [ + { + "key": "dep//Doer", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Doer" + ], + "reaches": [] + }, + { + "key": "dep//Doer//do", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Doer//do" + ], + "reaches": [] + }, + { + "key": "dep//Runner", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Runner" + ], + "reaches": [] + }, + { + "key": "dep//Runner//run", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Runner//run" + ], + "reaches": [] + }, + { + "key": "dep//Thing", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Thing" + ], + "reaches": [ + "subprocess//run" + ] + }, + { + "key": "dep//Thing//do", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Thing//do" + ], + "reaches": [ + "subprocess//run" + ] + }, + { + "key": "dep//new_doer", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//new_doer" + ], + "reaches": [ + "subprocess//run" + ] + }, + { + "key": "dep//new_runner", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//new_runner" + ], + "reaches": [] + }, + { + "key": "dep//new_thing", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//new_thing" + ], + "reaches": [ + "subprocess//run" + ] + } + ], + "targets": [ + "abc//abstractmethod", + "subprocess//run" + ] +} diff --git a/test/projectgraph/cases/python/external-base-attribute/expect.yaml b/test/projectgraph/cases/python/external-base-attribute/expect.yaml index 90a6965e..1c8c1246 100644 --- a/test/projectgraph/cases/python/external-base-attribute/expect.yaml +++ b/test/projectgraph/cases/python/external-base-attribute/expect.yaml @@ -1,5 +1,5 @@ source: safedep:hand -description: Worker extends threading.Thread and sets self.parser in its constructor. In self.parser.parse() the name parser is an attribute of the instance, not a method that the dependency base gives. So the call is a call on an unknown receiver, and a name edge joins it to Parser.parse. A call that names a method last, as super().__init__(), still falls to the dependency base. +description: Worker extends threading.Thread and sets self.parser to Parser() in its constructor. The constructor call gives the field parser its type, so self.parser.parse() is a member call of Parser.parse with no record. A call that names a method last, as super().__init__(), still falls to the dependency base. roots: - key: main.py kind: main @@ -12,17 +12,13 @@ edges: site: main.py:8 - from: main.py//Worker//work to: parser.py//Parser//parse - kind: name - confidence: low + kind: member + confidence: high site: main.py:12 no_edges: - from: main.py//Worker//work to: threading//Thread//parser//parse -unresolved: - - caller: main.py//Worker//work - site: main.py:12 - reason: unknown-receiver - method: parse +unresolved: [] results: - target: yaml//safe_load status: found @@ -34,8 +30,8 @@ results: confidence: high - key: parser.py//Parser//parse site: main.py:12 - kind: name - confidence: low + kind: member + confidence: high - key: yaml//safe_load site: parser.py:6 kind: direct diff --git a/test/projectgraph/cases/python/typed-base-subclass/expect.yaml b/test/projectgraph/cases/python/typed-base-subclass/expect.yaml new file mode 100644 index 00000000..716c603e --- /dev/null +++ b/test/projectgraph/cases/python/typed-base-subclass/expect.yaml @@ -0,0 +1,61 @@ +source: safedep:hand +description: >- + Mine extends the abstract base Doer of the dependency. run types its parameter as Doer, so d.do() is a + call of Doer.do, and the class hierarchy analysis adds the override Mine.do of the project and Thing.do of + the dependency. Runner has no implementation in the project or in the dependency, so r.run() in other has + a record of an interface receiver, as a call of a Go interface with no implementation. +units: + venv/site-packages/dep: pypi:dep@1.0.0 +import_dirs: + - venv/site-packages +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py//run + to: dep//Doer//do + kind: direct + confidence: high + site: main.py:12 + - from: main.py//run + to: main.py//Mine//do + kind: cha + confidence: medium + site: main.py:12 + - from: main.py//run + to: dep//Thing//do + kind: cha + confidence: medium + site: main.py:12 + - from: main.py//other + to: dep//Runner//run + kind: direct + confidence: high + site: main.py:16 +unresolved: + - caller: main.py//other + site: main.py:16 + reason: interface-receiver + method: run +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: main.py//run + site: main.py:20 + kind: direct + confidence: high + - key: main.py//Mine//do + site: main.py:12 + kind: cha + confidence: medium + - key: yaml//load + site: main.py:8 + kind: direct + confidence: high + call_sites: + - caller: main.py//Mine//do + site: main.py:8 + reached: true diff --git a/test/projectgraph/cases/python/typed-base-subclass/project/main.py b/test/projectgraph/cases/python/typed-base-subclass/project/main.py new file mode 100644 index 00000000..ce99c1b5 --- /dev/null +++ b/test/projectgraph/cases/python/typed-base-subclass/project/main.py @@ -0,0 +1,21 @@ +import dep +import yaml +from dep import Doer + + +class Mine(Doer): + def do(self): + yaml.load("a: 1") + + +def run(d: Doer): + d.do() + + +def other(r: dep.Runner): + r.run() + + +if __name__ == "__main__": + run(Mine()) + other(dep.new_runner()) diff --git a/test/projectgraph/cases/python/typed-base-subclass/project/venv/site-packages/dep/__init__.py b/test/projectgraph/cases/python/typed-base-subclass/project/venv/site-packages/dep/__init__.py new file mode 100644 index 00000000..c1b1e823 --- /dev/null +++ b/test/projectgraph/cases/python/typed-base-subclass/project/venv/site-packages/dep/__init__.py @@ -0,0 +1,31 @@ +import abc +import subprocess + + +class Doer(abc.ABC): + @abc.abstractmethod + def do(self): + raise NotImplementedError + + +class Runner(abc.ABC): + @abc.abstractmethod + def run(self): + raise NotImplementedError + + +class Thing(Doer): + def do(self): + subprocess.run(["ls"]) + + +def new_thing() -> Thing: + return Thing() + + +def new_doer() -> Doer: + return Thing() + + +def new_runner() -> Runner: + raise NotImplementedError diff --git a/test/projectgraph/cases/python/typed-base-subclass/summaries/pypi_dep_1.0.0.json b/test/projectgraph/cases/python/typed-base-subclass/summaries/pypi_dep_1.0.0.json new file mode 100644 index 00000000..657312ba --- /dev/null +++ b/test/projectgraph/cases/python/typed-base-subclass/summaries/pypi_dep_1.0.0.json @@ -0,0 +1,83 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "pypi:dep@1.0.0", + "symbols": [ + { + "key": "dep//Doer", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Doer" + ], + "reaches": [] + }, + { + "key": "dep//Doer//do", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Doer//do" + ], + "reaches": [] + }, + { + "key": "dep//Runner", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Runner" + ], + "reaches": [] + }, + { + "key": "dep//Runner//run", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Runner//run" + ], + "reaches": [] + }, + { + "key": "dep//Thing", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Thing" + ], + "reaches": [ + "subprocess//run" + ] + }, + { + "key": "dep//Thing//do", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//Thing//do" + ], + "reaches": [ + "subprocess//run" + ] + }, + { + "key": "dep//new_doer", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//new_doer" + ], + "reaches": [ + "subprocess//run" + ] + }, + { + "key": "dep//new_runner", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//new_runner" + ], + "reaches": [] + }, + { + "key": "dep//new_thing", + "definitions": [ + "pypi:dep@1.0.0//dep/__init__.py//new_thing" + ], + "reaches": [ + "subprocess//run" + ] + } + ], + "targets": [ + "abc//abstractmethod", + "subprocess//run" + ] +} diff --git a/test/projectgraph/cases/python/typed-field/expect.yaml b/test/projectgraph/cases/python/typed-field/expect.yaml new file mode 100644 index 00000000..3805b9f4 --- /dev/null +++ b/test/projectgraph/cases/python/typed-field/expect.yaml @@ -0,0 +1,41 @@ +source: safedep:hand +description: >- + The dataclass Job declares the field parser with a hint, and App declares it in the class body and sets + it in the constructor. A call on self.parser in a method is a member call of Parser.parse with no record. +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py//Job//work + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:11 + - from: main.py//App//go + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:21 +unresolved: [] +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: main.py//Job//work + site: main.py:25 + kind: member + confidence: high + - key: factory.py//Parser//parse + site: main.py:11 + kind: member + confidence: high + - key: yaml//load + site: factory.py:6 + kind: direct + confidence: high + call_sites: + - caller: factory.py//Parser//parse + site: factory.py:6 + reached: true diff --git a/test/projectgraph/cases/python/typed-field/project/factory.py b/test/projectgraph/cases/python/typed-field/project/factory.py new file mode 100644 index 00000000..bf42c881 --- /dev/null +++ b/test/projectgraph/cases/python/typed-field/project/factory.py @@ -0,0 +1,6 @@ +import yaml + + +class Parser: + def parse(self, text): + return yaml.load(text) diff --git a/test/projectgraph/cases/python/typed-field/project/main.py b/test/projectgraph/cases/python/typed-field/project/main.py new file mode 100644 index 00000000..c25147a5 --- /dev/null +++ b/test/projectgraph/cases/python/typed-field/project/main.py @@ -0,0 +1,26 @@ +from dataclasses import dataclass + +from factory import Parser + + +@dataclass +class Job: + parser: Parser + + def work(self): + return self.parser.parse("a: 1") + + +class App: + parser: Parser + + def __init__(self): + self.parser = Parser() + + def go(self): + return self.parser.parse("b: 2") + + +if __name__ == "__main__": + Job(Parser()).work() + App().go() diff --git a/test/projectgraph/cases/python/typed-parameter/expect.yaml b/test/projectgraph/cases/python/typed-parameter/expect.yaml new file mode 100644 index 00000000..5353469b --- /dev/null +++ b/test/projectgraph/cases/python/typed-parameter/expect.yaml @@ -0,0 +1,56 @@ +source: safedep:hand +description: >- + run types three parameters with a hint, as Parser, Optional[Parser] and Parser | None, so a call on each + one is a member call of Parser.parse with no record. The parameter raw has no hint, so the call on it has + a record of an unknown receiver and a name edge. +roots: + - key: main.py + kind: main + rule: python.main-guard +edges: + - from: main.py//run + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:7 + - from: main.py//run + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:8 + - from: main.py//run + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:9 + - from: main.py//run + to: factory.py//Parser//parse + kind: name + confidence: low + site: main.py:10 +unresolved: + - caller: main.py//run + site: main.py:10 + reason: unknown-receiver + method: parse +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: main.py//run + site: main.py:14 + kind: direct + confidence: high + - key: factory.py//Parser//parse + site: main.py:7 + kind: member + confidence: high + - key: yaml//load + site: factory.py:6 + kind: direct + confidence: high + call_sites: + - caller: factory.py//Parser//parse + site: factory.py:6 + reached: true diff --git a/test/projectgraph/cases/python/typed-parameter/project/factory.py b/test/projectgraph/cases/python/typed-parameter/project/factory.py new file mode 100644 index 00000000..bf42c881 --- /dev/null +++ b/test/projectgraph/cases/python/typed-parameter/project/factory.py @@ -0,0 +1,6 @@ +import yaml + + +class Parser: + def parse(self, text): + return yaml.load(text) diff --git a/test/projectgraph/cases/python/typed-parameter/project/main.py b/test/projectgraph/cases/python/typed-parameter/project/main.py new file mode 100644 index 00000000..e8865cd6 --- /dev/null +++ b/test/projectgraph/cases/python/typed-parameter/project/main.py @@ -0,0 +1,14 @@ +from typing import Optional + +from factory import Parser + + +def run(p: Parser, o: Optional[Parser], u: Parser | None, raw): + p.parse("a: 1") + o.parse("b: 2") + u.parse("c: 3") + raw.parse("d: 4") + + +if __name__ == "__main__": + run(Parser(), None, None, Parser()) diff --git a/test/projectgraph/cases/python/typed-return/expect.yaml b/test/projectgraph/cases/python/typed-return/expect.yaml new file mode 100644 index 00000000..ab1b78e6 --- /dev/null +++ b/test/projectgraph/cases/python/typed-return/expect.yaml @@ -0,0 +1,48 @@ +source: safedep:hand +description: >- + make, maybe and pair declare a return type. A call on the result of make(), on a name that holds the + result of make() or maybe(), and on the first name of a tuple from pair() is a member call of + Parser.parse. The Optional and the tuple hint give the class of the value. No site has a record. +roots: + - key: main.py + kind: manifest + rule: procfile +edges: + - from: main.py + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:3 + - from: main.py + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:5 + - from: main.py + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:7 + - from: main.py + to: factory.py//Parser//parse + kind: member + confidence: high + site: main.py:9 +unresolved: [] +results: + - target: yaml//load + status: found + path: + - key: main.py + - key: factory.py//Parser//parse + site: main.py:3 + kind: member + confidence: high + - key: yaml//load + site: factory.py:8 + kind: direct + confidence: high + call_sites: + - caller: factory.py//Parser//parse + site: factory.py:8 + reached: true diff --git a/test/projectgraph/cases/python/typed-return/project/Procfile b/test/projectgraph/cases/python/typed-return/project/Procfile new file mode 100644 index 00000000..629b83ad --- /dev/null +++ b/test/projectgraph/cases/python/typed-return/project/Procfile @@ -0,0 +1 @@ +web: python main.py diff --git a/test/projectgraph/cases/python/typed-return/project/factory.py b/test/projectgraph/cases/python/typed-return/project/factory.py new file mode 100644 index 00000000..e4d7a9b6 --- /dev/null +++ b/test/projectgraph/cases/python/typed-return/project/factory.py @@ -0,0 +1,20 @@ +from typing import Optional + +import yaml + + +class Parser: + def parse(self, text): + return yaml.load(text) + + +def make() -> Parser: + return Parser() + + +def maybe() -> Optional[Parser]: + return None + + +def pair() -> tuple[Parser, int]: + return Parser(), 1 diff --git a/test/projectgraph/cases/python/typed-return/project/main.py b/test/projectgraph/cases/python/typed-return/project/main.py new file mode 100644 index 00000000..2b7f4ac0 --- /dev/null +++ b/test/projectgraph/cases/python/typed-return/project/main.py @@ -0,0 +1,9 @@ +from factory import make, maybe, pair + +make().parse("a: 1") +p = make() +p.parse("b: 2") +m = maybe() +m.parse("c: 3") +q, n = pair() +q.parse("d: 4") diff --git a/test/projectgraph/methods_test.go b/test/projectgraph/methods_test.go index 23727a7a..4163f7ff 100644 --- a/test/projectgraph/methods_test.go +++ b/test/projectgraph/methods_test.go @@ -278,7 +278,7 @@ func TestMethodTargets(t *testing.T) { projects, advisories := readMethodCorpus(t) counts := methodCounts{} - var lines []string + var lines, details []string for _, p := range projects { dir, _ := methodDir(t, corpus, p.language, p.source) cfg := codefs.LocalFileSystemConfig{AppDirectories: []string{dir}, ImportUnits: map[string]string{}, ExcludePatterns: codefs.DefaultSkipPatterns(dir)} @@ -311,6 +311,7 @@ func TestMethodTargets(t *testing.T) { o := outcome(r, methods) counts.add(p.language, targets[i].kind, o) project.add(p.language, targets[i].kind, o) + details = append(details, methodDetail(p.name, targets[i].key, o, r)) } lines = append(lines, "project="+p.name+" files="+strconv.Itoa(len(g.Files))+" truncated="+strconv.Itoa(len(g.Truncated))+" problems="+strconv.Itoa(len(g.Problems))) for _, row := range project.rows() { @@ -323,4 +324,26 @@ func TestMethodTargets(t *testing.T) { if out := os.Getenv("PROJECTGRAPH_METHOD_REPORT"); out != "" { require.NoError(t, os.WriteFile(out, []byte(strings.Join(rows, "\n")+"\n"), 0o644)) } + if out := os.Getenv(methodDetailEnv); out != "" { + require.NoError(t, os.WriteFile(out, []byte(strings.Join(details, "\n")+"\n"), 0o644)) + } +} + +// methodDetailEnv names the file that gets one line per target: the +// project, the target, the outcome, and the sites of the path or of the +// uncertainties, so a reader can find the receiver form behind a number +// of the report. +const methodDetailEnv = "PROJECTGRAPH_METHOD_DETAIL" + +func methodDetail(project, target, outcome string, r reach.Result) string { + var sites []string + for _, h := range r.Path { + if h.Site.Path != "" { + sites = append(sites, h.Site.String()+":"+string(h.Confidence)) + } + } + for _, u := range r.Uncertainties { + sites = append(sites, u.Site.String()+":"+string(u.Kind)) + } + return "project=" + project + " target=" + target + " outcome=" + outcome + " sites=" + strings.Join(sites, ",") } diff --git a/test/projectgraph/receivers_test.go b/test/projectgraph/receivers_test.go index f7578bc9..4c5fc597 100644 --- a/test/projectgraph/receivers_test.go +++ b/test/projectgraph/receivers_test.go @@ -212,14 +212,6 @@ var rustReceiverRows = []receiverRow{ }, } -// pythonTypeHints is the step of the Python plan that reads the type hints -// of a parameter, a local, a field and a return into the index. Most -// Python rows wait on it. Today a row that reaches the method reaches it -// through a low hop: the override rule joins the class Thing to its do, -// so the constructor call in the unit reaches do, and a call on an unknown -// receiver gets a name edge. The declared type must give the member edge. -const pythonTypeHints = "the type-hint step of the Python plan" - // pythonReceiverRows are the forms of a Python receiver. A type hint is // the only declared type that Python has. var pythonReceiverRows = []receiverRow{ @@ -228,32 +220,24 @@ var pythonReceiverRows = []receiverRow{ source: "def run():\n t: Thing = dep.new_thing()\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "parameter", source: "def run(t: Thing):\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "uncertain/unknown-receiver", }, { name: "optional parameter", source: "def run(t: Optional[Thing]):\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "uncertain/unknown-receiver", }, { name: "union with none parameter", source: "def run(t: Thing | None):\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "uncertain/unknown-receiver", }, { name: "untyped parameter", @@ -266,72 +250,54 @@ var pythonReceiverRows = []receiverRow{ source: "class App:\n def __init__(self):\n self.t: Thing = dep.new_thing()\n\n def go(self):\n self.t.do()\n\n\ndef run():\n App().go()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "instance attribute from the constructor", source: "class App:\n def __init__(self):\n self.t = Thing()\n\n def go(self):\n self.t.do()\n\n\ndef run():\n App().go()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "dataclass field", source: "@dataclass\nclass App:\n t: Thing\n\n\ndef run(a: App):\n a.t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "uncertain/unknown-receiver", }, { name: "first-party function return", source: "def make() -> Thing:\n return dep.new_thing()\n\n\ndef run():\n t = make()\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "external function return", source: "def run():\n t = dep.new_thing()\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "tuple return", source: "def run():\n t, n = dep.new_pair()\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "optional return", source: "def run():\n t = dep.try_thing()\n if t is not None:\n t.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "chained call", source: "def run():\n dep.new_thing().do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "chained method call", source: "def run():\n dep.new_thing().next().do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "instance from the constructor", @@ -344,16 +310,12 @@ var pythonReceiverRows = []receiverRow{ source: "def run():\n d = dep.new_doer()\n d.do()", target: "dep//Thing//do", want: "found", - gap: pythonTypeHints, - today: "found/low", }, { name: "abstract base return with no implementation", source: "def run():\n r = dep.new_runner()\n r.run()", target: "other//Job//run", want: "uncertain/interface-receiver", - gap: pythonTypeHints, - today: "no-such-target", }, { name: "external function return with no dependency code", @@ -361,8 +323,6 @@ var pythonReceiverRows = []receiverRow{ target: "dep//Thing//do", want: "uncertain/no-dependency-source", noDeps: true, - gap: pythonTypeHints, - today: "no-such-target", }, } diff --git a/test/projectgraph/version_test.go b/test/projectgraph/version_test.go index 07c354cd..e666b4d6 100644 --- a/test/projectgraph/version_test.go +++ b/test/projectgraph/version_test.go @@ -31,7 +31,7 @@ var graphHashes = map[string]string{ "14/14": "0f3c9c38b39f4de1835f34e8a58bcea85461f37e7aa2d5a2dd484569e670ff48", "14/15": "bbc708aba767ccf554c75a7e06816abda3fc1804ad6d0876ad8ce9575f4c3a88", "14/16": "823b1f82c4aed8ba27d19275505c8d1dee755086be48fcda64d323f55b4c62a5", - "15/17": "ca24a262671d86221ad25aad86efbf27c7efa3750335dc0f6c8999ba23ff2cf3", + "15/17": "ffa293e72af1682407e900e442fddb0c18f6174eba38cf412d1b7f362d9338ac", } // resultHashes records the hash of the reach results of every case for @@ -47,7 +47,7 @@ var resultHashes = map[string]string{ "14/14/2": "a832ea709bf3df852814375a56e6329c35e2ff4c49ae34ee49afe042e9502d17", "14/15/3": "77dfd0ea8ab71869ef612d41b8c2e46852d4fb67a49458cb645321692af77ea0", "14/16/3": "13b6eecb686ddec3c2ba470fc9752ea66cff445d5e18e0772a71b15ae7fe1c9c", - "15/17/4": "96f5a284da37b626dda6eb66c6d031145d9ee2b2bc737a1ec2da402b51b28c5b", + "15/17/4": "d3a7367dacb56d7e6dbff3b203d5b5de583abe70c1aad31898b7d6659629654c", } // summaryHashes records the hash of the golden summaries of the cases for @@ -64,7 +64,7 @@ var summaryHashes = map[string]string{ "14/14/2": "27424c4e2525450479faba78421959738748811d70b9a93b81e2bbdf1733bf0d", "14/15/2": "5969f3cda15b666cbd902bbc2db953cbdef8c0ce48d5e77446cf243cdd0bb0e7", "14/16/2": "fb678202978212aae932e50a56a0d3dc7f0f794ce908988bf605ce9a6e04bc51", - "15/17/2": "69d9b0f80c4f5811cc07608a7a1f224499e5f4e3b3eb5b62d2c2cf842df142e7", + "15/17/2": "2b27be7d0d105d84922201a60cda828d37723ca9e131d69dc71c55bd43d2e750", } func TestGraphHashFollowsVersion(t *testing.T) { From 125cebac6de37ee780000b7022479cf4511e03f5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 13:43:28 +0000 Subject: [PATCH 07/11] Report the type-hint step of the Python plan The method report had no section for step 3. This adds the numbers of the run after the hints, the two targets that move, and why the rest do not: the paths of the corpus cross dependency code with no hints. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- test/projectgraph/report.md | 65 +++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/test/projectgraph/report.md b/test/projectgraph/report.md index 3167e4ca..b9cc671c 100644 --- a/test/projectgraph/report.md +++ b/test/projectgraph/report.md @@ -1256,3 +1256,68 @@ The commit after the four rules adds the rest of step 2 of the plan. corpus has a dependency it never imports. - pycg-micro keeps 0.954 precision and 0.708 recall, with the unresolved rate at 0.312. govulncheck stays at 9 of 9 stacks. + +#### After the type hints + +The commit after step 2 adds step 3 of the plan: the index reads the +type hints of Python, and the link step trusts them as it trusts +TypeScript types. `docs/projectgraph.md` holds the rules. + +1. The index records the class of a hint of a parameter, with + `Optional`, `Union` and `| None` unwrapped, the return type and the + tuple positions of a function, an annotated local, a name that holds + the result of a call, a typed name of the top level of a module, and + the fields of a class from its annotations and from the attributes + that its methods set. +2. A call on a typed name gets the key of the name in its scope as its + target, as the Go index does, so the link step reads the type. A call + that the call graph bound to a method of a class of the file keeps + that target. A result of a reflective call, as `import_module`, has + no type. +3. A hint that names a dependency class gives the external key of the + method. The class hierarchy analysis adds the overrides in the project + and in the unit, and an abstract method with no override is a record + with the reason `interface-receiver`. + +| Language | Kind | Run | Targets | found | found/low | not-found | no-such-target | unknown-receiver | no-definition | ambiguous | computed-name | dynamic-flag | budget | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Python | function | step 3 | 44 | 2 | 17 | 0 | 0 | 0 | 0 | 10 | 4 | 10 | 1 | +| Python | method | step 3 | 63 | 0 | 35 | 0 | 0 | 4 | 1 | 12 | 2 | 6 | 3 | + +| Language | Run | Method targets that end as unknown-receiver | +| --- | --- | --- | +| Python | step 3 | 0.063 (4/63) | + +- All 16 Python receiver rows that waited on this step give their wanted + answer: 14 are `found` with a `member` edge of high confidence or a + `cha` edge of medium confidence, the abstract base with no + implementation is `uncertain/interface-receiver`, and the external + result with no dependency code is `uncertain/no-dependency-source`. + The rows have no gap left. +- The corpus moves two targets. `requests.utils.get_netrc_auth` in httpie + goes from `found/low` to `found`. `MultiPartParser.parse` in + django-filter goes from `found/low` to `uncertain/ambiguous`: the path + before went through `parser.parse()` in `django/template/base.py`, + where the call graph did not bind `parser` to the `Parser(...)` call + over several lines, so a `name` edge joined the site to every `parse` + method of Django. The hint-free assignment now types `parser` as the + template `Parser`, and the false path is gone. +- The other numbers do not move, and the reason is the corpus, not the + rules. The paths of the Python targets run through requests, Django + 4.2 and urllib3 1.26, and that code has no type hints on the receivers + that the paths cross: `conn.urlopen(...)` in `requests/adapters.py` + binds `conn` to the result of an untyped method, and Django reads + `self.request` with no annotation. A project of the corpus that types + its own code, as httpie, calls these dependencies through their + untyped API. The receiver rows and the six cases of this step prove the + mechanism, and the corpus shows how much of the Python ecosystem of + 2023 the mechanism can read: little, until a dependency ships hints. +- The 4 `unknown-receiver` method targets of httpie and requests-toolbelt + are `urlopen` of urllib3, which requests calls on `conn`, a result of + `self.get_connection_with_tls_context(...)` with no hint. +- pycg-micro keeps 0.954 precision and 0.708 recall, with the unresolved + rate at 0.312. Its cases have no type hints. govulncheck stays at 9 of 9 + stacks. +- `PROJECTGRAPH_METHOD_DETAIL` now writes one line per target with its + outcome and the sites of its path or of its uncertainties, so the next + step can read the cause of each number. From 61893b7a81c5b33d6cf81e513096f0cd4b40879c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 13:56:00 +0000 Subject: [PATCH 08/11] Add the distribution graph of a Python project A caller of the project graph had no way to tell which installed Python distribution imports which, as modgraph tells it for the modules of a Go build. Without the edges, a caller cannot trust that no code imports a distribution, and a vulnerable distribution hides behind a false unused. distgraph builds the graph from the code on the disk. It reads the .dist-info directories of each site-packages directory and the import statements of the Python files of the project and of each distribution that an import reaches. It runs no Python. The graph never misses an edge that a run has: a distribution with an extension module, a computed import, an import that nothing holds, a file that does not parse, or a file past the budget is open, and a path that meets it reaches each distribution. An import in a try statement that catches an ImportError opens nothing. A distribution with no Python file gets its Requires-Dist lines as evidence of edges. The distribution reader gains the Requires-Dist names and the test-path rule, which the index step shares. examples/distgraph prints the graph, and docs/distgraph.md holds the rules. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- README.md | 3 + distgraph/files.go | 87 ++++ distgraph/graph.go | 609 ++++++++++++++++++++++++++++ distgraph/graph_test.go | 324 +++++++++++++++ distgraph/imports.go | 157 +++++++ distgraph/imports_test.go | 70 ++++ distgraph/stdlib.go | 71 ++++ docs/distgraph.md | 124 ++++++ examples/distgraph/main.go | 90 ++++ examples/distgraph/main_test.go | 50 +++ plugin/indexer/pyunit.go | 21 +- projectgraph/modules/pydist.go | 41 ++ projectgraph/modules/pydist_test.go | 22 +- 13 files changed, 1652 insertions(+), 17 deletions(-) create mode 100644 distgraph/files.go create mode 100644 distgraph/graph.go create mode 100644 distgraph/graph_test.go create mode 100644 distgraph/imports.go create mode 100644 distgraph/imports_test.go create mode 100644 distgraph/stdlib.go create mode 100644 docs/distgraph.md create mode 100644 examples/distgraph/main.go create mode 100644 examples/distgraph/main_test.go diff --git a/README.md b/README.md index 8d76b42c..13a0ecc7 100644 --- a/README.md +++ b/README.md @@ -16,3 +16,6 @@ summary steps. See [docs/modgraph.md](./docs/modgraph.md) for the module graph of a Go module: which module imports which, from the code on the disk, with no Go toolchain. + +See [docs/distgraph.md](./docs/distgraph.md) for the distribution graph of a Python project: which +installed distribution imports which, from the code on the disk, with no Python. diff --git a/distgraph/files.go b/distgraph/files.go new file mode 100644 index 00000000..f83b3922 --- /dev/null +++ b/distgraph/files.go @@ -0,0 +1,87 @@ +package distgraph + +import ( + "errors" + "io/fs" + "path" + "slices" + "strings" + + "github.com/safedep/code/projectgraph/modules" +) + +// compiledExts are the extensions of a Python extension module. +var compiledExts = map[string]bool{".so": true, ".pyd": true, ".dylib": true} + +// pythonFiles returns the Python files of a node, in path order, and its +// extension modules. The project gives each .py file under its root, except +// under a hidden directory, a __pycache__, a node_modules, a site-packages +// or a virtual environment, which holds a pyvenv.cfg. A distribution gives +// the files under each of its top-level names, except its test files, +// which no program imports. +func (b *builder) pythonFiles(d *dist) ([]string, []string, error) { + var files, compiled []string + visit := func(p string, entry fs.DirEntry, err error) error { + if err != nil { + if errors.Is(err, fs.ErrNotExist) { + return nil + } + return err + } + if entry.IsDir() { + if b.skipDir(d, p) { + return fs.SkipDir + } + return nil + } + switch ext := path.Ext(p); { + case ext == ".py" && (d.meta.Name == "" || !modules.PythonTestPath(p)): + files = append(files, p) + case compiledExts[ext]: + compiled = append(compiled, p) + } + return nil + } + roots := []string{"."} + if d.meta.Name != "" { + roots = nil + for _, top := range d.meta.TopLevel { + if _, err := fs.Stat(d.fsys, top+".py"); err == nil { + files = append(files, top+".py") + } + roots = append(roots, top) + } + compiled = append(compiled, d.meta.Compiled...) + } + for _, root := range roots { + if root != "." { + if info, err := fs.Stat(d.fsys, root); err != nil || !info.IsDir() { + continue + } + } + if err := fs.WalkDir(d.fsys, root, visit); err != nil { + return nil, nil, err + } + } + slices.Sort(files) + slices.Sort(compiled) + return slices.Compact(files), slices.Compact(compiled), nil +} + +// skipDir reports whether no file of a directory is code of the node. +func (b *builder) skipDir(d *dist, p string) bool { + if p == "." { + return false + } + base := path.Base(p) + switch { + case strings.HasPrefix(base, ".") || base == "__pycache__" || base == "node_modules": + return true + case d.meta.Name != "": + return false + case base == "site-packages": + return true + } + _, err := fs.Stat(d.fsys, path.Join(p, "pyvenv.cfg")) + return err == nil +} diff --git a/distgraph/graph.go b/distgraph/graph.go new file mode 100644 index 00000000..ae54d8b1 --- /dev/null +++ b/distgraph/graph.go @@ -0,0 +1,609 @@ +// Package distgraph builds the distribution graph of a Python project from +// the code on the disk: which installed distribution of a site-packages +// directory imports which. It reads the import statements of the Python +// files of the project and of each distribution, and the Requires-Dist +// lines of a distribution whose code it cannot read. It runs no Python, +// opens no network connection and reads no byte code. +// +// The graph may hold an edge that a run of the program does not have, and +// it never misses an edge that a run has. A false absence of an edge hides +// a vulnerable distribution behind a false "unused". So a distribution +// whose imports Build cannot read in full is open: it may import any +// distribution. +package distgraph + +import ( + "context" + "errors" + "fmt" + "io/fs" + "path" + "slices" + "strings" + "sync" + + "github.com/safedep/code/projectgraph/modules" +) + +// Version changes when Build gives another graph for the same input. +const Version = "1" + +// StdName is the name of the node of the standard library. The standard +// library imports no distribution, so the node is a leaf. +const StdName = "std" + +// defaultMaxFiles is the file budget when Input.MaxFiles is zero. +const defaultMaxFiles = 200_000 + +// Distribution is one node of the graph: the project, an installed +// distribution, or the standard library. Name is the name that METADATA +// gives, and two names are one distribution when PEP 503 normalizes them +// to one. +type Distribution struct { + Name string + Version string +} + +func (d Distribution) String() string { + if d.Version == "" { + return d.Name + } + return d.Name + "@" + d.Version +} + +// key returns the normalized name of the distribution, which the graph +// indexes its nodes by. +func (d Distribution) key() string { return modules.NormalizePythonName(d.Name) } + +// Input is the project that Build reads. +type Input struct { + // ProjectFS is the root of the project. + ProjectFS fs.FS + // ProjectName is the name of the node of the project. An empty name + // gives "project". + ProjectName string + // SitePackages holds each site-packages directory of the environment + // of the project, as the .dist-info directories describe its + // distributions. + SitePackages []fs.FS + // MaxFiles caps the Python files that Build reads. Zero selects 200,000. + MaxFiles int + // TrustCompiled keeps a distribution with an extension module closed. + // The code of a .so or a .pyd file can import any module, so Build + // opens its distribution. A caller that reads the graph for a path + // into a distribution, and not for the absence of one, can trust the + // Python files alone. + TrustCompiled bool +} + +// State says what Build knows of the imports of a distribution. +type State string + +const ( + // StateImports is a distribution whose imports Build read, in each of + // its Python files. + StateImports State = "imports" + // StateRequires is a distribution with no Python file that Build could + // read. Its Requires-Dist lines show distributions that it needs, but a + // requirement is no import and an import needs no requirement. So the + // node can import any distribution, as an open node. + StateRequires State = "requires" + // StateOpen is a distribution whose imports Build could not read in + // full. It can import any distribution. + StateOpen State = "open" +) + +// Node is one distribution of the graph. +type Node struct { + Distribution Distribution + State State + // Reached is true when a path from the project can reach the + // distribution. A path through an open node can reach each one. + Reached bool + // Reason says why the node is not StateImports. + Reason string + // Walked is true when Build read the files of the distribution. The + // edges of a distribution that Build did not walk are not known. + Walked bool +} + +// Open reports whether the node can import a distribution that it has no +// edge to. +func (n Node) Open() bool { return n.State != StateImports } + +// Evidence names one import that gives an edge: the file, relative to the +// root of the project or of the site-packages directory, and the line. +type Evidence struct { + File string + Line int +} + +// Edge joins a distribution to a distribution that it imports. +type Edge struct { + From, To Distribution + // Kind is StateImports for an import that Build read, and + // StateRequires for a Requires-Dist line of a distribution whose code + // Build could not read. + Kind State + // TestOnly marks an edge from the project that only its test files + // give. + TestOnly bool + Evidence Evidence +} + +// Stats counts the work of Build. +type Stats struct { + // Files is the count of Python files that Build read. + Files int + // Distributions is the count of distributions of the site-packages + // directories. + Distributions int + // Truncated is true when the file budget ran out, so some node is open. + Truncated bool +} + +// Graph is the distribution graph of one project. +type Graph struct { + Version string + // Main is the project. Its Version is "". + Main Distribution + // Nodes holds the project, each distribution of the site-packages + // directories, and the standard library when the code imports it, + // sorted by name. + Nodes []Node + // Edges is sorted by the name of From, then of To. + Edges []Edge + // Problems names each .dist-info directory that Build could not read. + // It describes no distribution, so the graph has no node for it. + Problems []modules.Problem + Stats Stats + + index sync.Once + next map[string][]string + nodeIndex map[string]int +} + +func (g *Graph) build() { + g.index.Do(func() { + g.next = map[string][]string{} + for _, e := range g.Edges { + g.next[e.From.key()] = append(g.next[e.From.key()], e.To.key()) + } + g.nodeIndex = make(map[string]int, len(g.Nodes)) + for i, n := range g.Nodes { + g.nodeIndex[n.Distribution.key()] = i + } + }) +} + +// Node returns the node of a distribution name, in any spelling that PEP +// 503 normalizes to the name of the node. +func (g *Graph) Node(name string) (Node, bool) { + g.build() + i, ok := g.nodeIndex[modules.NormalizePythonName(name)] + if !ok { + return Node{}, false + } + return g.Nodes[i], true +} + +// Reaches reports whether a path from one of the distributions can reach a +// distribution. See Path. +func (g *Graph) Reaches(from []string, to string) bool { return g.Path(from, to) != nil } + +// Path returns a shortest path of distribution names from one of the +// distributions to a distribution, or nil. A path that meets a node whose +// edges are not known, an open node or a distribution that Build did not +// walk, can reach each distribution: the path ends at that node, and to +// follows it. A distribution is not on a path from itself. +func (g *Graph) Path(from []string, to string) []string { + g.build() + to = modules.NormalizePythonName(to) + prev := map[string]string{} + var queue []string + for _, f := range from { + f = modules.NormalizePythonName(f) + if _, ok := prev[f]; !ok && f != to { + prev[f] = "" + queue = append(queue, f) + } + } + name := func(key string) string { + if i, ok := g.nodeIndex[key]; ok { + return g.Nodes[i].Distribution.Name + } + return key + } + back := func(p string) []string { + var out []string + for ; p != ""; p = prev[p] { + out = append(out, name(p)) + } + slices.Reverse(out) + return out + } + for len(queue) > 0 { + p := queue[0] + queue = queue[1:] + if n, ok := g.Node(p); !ok || n.Open() || !n.Walked { + return append(back(p), name(to)) + } + for _, q := range g.next[p] { + if q == to { + return append(back(p), name(to)) + } + if _, seen := prev[q]; !seen { + prev[q] = p + queue = append(queue, q) + } + } + } + return nil +} + +// Build reads the distribution graph of the project in ProjectFS. +func Build(ctx context.Context, in Input) (*Graph, error) { + if in.ProjectFS == nil { + return nil, errors.New("distgraph: no project") + } + b, err := newBuilder(ctx, in) + if err != nil { + return nil, err + } + return b.run() +} + +// dist is the state of one node during a build. +type dist struct { + node Node + // fsys is the site-packages directory that holds the distribution, or + // the project for the project node. + fsys fs.FS + // meta is the distribution as its .dist-info describes it. The project + // has none. + meta modules.Distribution + // files holds the Python files of the distribution that RECORD lists, + // for the choice between two distributions of one namespace package. + files map[string]bool +} + +type edgeKey struct{ from, to string } + +type builder struct { + ctx context.Context + in Input + main string + std string + dists map[string]*dist + order []string + byTop map[string][]string + project map[string]bool + edges map[edgeKey]*Edge + problems []modules.Problem + budget int + stats Stats + parser *importReader +} + +func newBuilder(ctx context.Context, in Input) (*builder, error) { + reader, err := newImportReader() + if err != nil { + return nil, err + } + b := &builder{ + ctx: ctx, + in: in, + dists: map[string]*dist{}, + byTop: map[string][]string{}, + project: map[string]bool{}, + edges: map[edgeKey]*Edge{}, + budget: in.MaxFiles, + parser: reader, + } + if b.budget <= 0 { + b.budget = defaultMaxFiles + } + name := in.ProjectName + if name == "" { + name = "project" + } + b.main = Distribution{Name: name}.key() + b.std = Distribution{Name: StdName}.key() + if b.main == b.std { + return nil, fmt.Errorf("distgraph: the project holds the reserved name %q", StdName) + } + b.add(&dist{node: Node{Distribution: Distribution{Name: name}, State: StateImports}, fsys: in.ProjectFS}) + for _, sp := range in.SitePackages { + found, problems := modules.PythonDistributions(sp) + b.problems = append(b.problems, problems...) + for _, d := range found { + b.addDistribution(sp, d) + } + } + b.stats.Distributions = len(b.order) - 1 + b.readProjectNames() + return b, nil +} + +func (b *builder) add(d *dist) { + key := d.node.Distribution.key() + b.dists[key] = d + b.order = append(b.order, key) +} + +// addDistribution adds a node for a distribution. Two .dist-info +// directories of one name, as a stale one next to the current one, keep +// the first and name the second as a problem: Build cannot tell which code +// the program runs. +func (b *builder) addDistribution(sp fs.FS, meta modules.Distribution) { + d := &dist{node: Node{Distribution: Distribution{Name: meta.Name, Version: meta.Version}, State: StateImports}, fsys: sp, meta: meta, files: map[string]bool{}} + key := d.node.Distribution.key() + if prev, dup := b.dists[key]; dup { + b.problems = append(b.problems, modules.Problem{Path: meta.Dir, Text: fmt.Sprintf("a second .dist-info of %s, next to %s", meta.Name, prev.meta.Dir)}) + prev.node.State, prev.node.Reason = StateOpen, "two .dist-info directories describe the distribution" + return + } + for _, f := range meta.Files { + if path.Ext(f) == ".py" { + d.files[f] = true + } + } + b.add(d) + for _, top := range meta.TopLevel { + b.byTop[top] = append(b.byTop[top], key) + } +} + +// readProjectNames reads the top-level names of the project: a module or a +// package at the root, and under a src directory. +func (b *builder) readProjectNames() { + for _, dir := range []string{".", "src"} { + entries, err := fs.ReadDir(b.in.ProjectFS, dir) + if err != nil { + continue + } + for _, e := range entries { + name := e.Name() + switch { + case e.IsDir(): + if _, err := fs.Stat(b.in.ProjectFS, path.Join(dir, name, "__init__.py")); err == nil || dir == "src" { + b.project[name] = true + } + case strings.HasSuffix(name, ".py"): + b.project[strings.TrimSuffix(name, ".py")] = true + } + } + } +} + +func (b *builder) run() (*Graph, error) { + if err := b.walk(b.main); err != nil { + return nil, err + } + reached := b.reach() + // A reached open node can import any distribution, so each node is + // reached, and the edges of each one matter. + if slices.ContainsFunc(b.order, func(key string) bool { return reached[key] && b.dists[key].node.Open() }) { + for _, key := range b.order { + if err := b.walk(key); err != nil { + return nil, err + } + reached[key] = true + } + } + b.requires() + return b.graph(reached), nil +} + +// reach marks each node that a path from the project reaches, through the +// edges that Build has read so far. +func (b *builder) reach() map[string]bool { + reached := map[string]bool{b.main: true} + queue := []string{b.main} + for len(queue) > 0 { + from := queue[0] + queue = queue[1:] + for _, e := range b.edges { + to := e.To.key() + if e.From.key() == from && !reached[to] { + reached[to] = true + queue = append(queue, to) + } + } + } + return reached +} + +// walk reads the Python files of a node and resolves its imports. A +// distribution that an import reaches is walked in turn, until no new +// node is reached. +func (b *builder) walk(key string) error { + queue := []string{key} + for len(queue) > 0 { + key := queue[0] + queue = queue[1:] + d := b.dists[key] + if d == nil || d.node.Walked { + continue + } + d.node.Walked = true + targets, err := b.readNode(d) + if err != nil { + return err + } + queue = append(queue, targets...) + } + return nil +} + +// open marks a node open, with the first reason. +func (b *builder) open(d *dist, reason string) { + if d.node.State == StateImports { + d.node.State, d.node.Reason = StateOpen, reason + } +} + +// readNode reads the Python files of a node and returns the keys of the +// distributions that its imports reach. +func (b *builder) readNode(d *dist) ([]string, error) { + files, compiled, err := b.pythonFiles(d) + if err != nil { + return nil, err + } + if len(compiled) > 0 && !b.in.TrustCompiled { + b.open(d, "the extension module "+compiled[0]+" can import any module") + } + if len(files) == 0 && d.node.Distribution.key() != b.main { + d.node.State, d.node.Reason = StateRequires, "no Python file" + } + var targets []string + for _, file := range files { + if err := b.ctx.Err(); err != nil { + return nil, err + } + if b.stats.Files >= b.budget { + b.stats.Truncated = true + b.open(d, "the file budget ran out") + break + } + b.stats.Files++ + data, err := fs.ReadFile(d.fsys, file) + if err != nil { + b.open(d, "cannot read "+file) + continue + } + imports, parsed := b.parser.read(b.ctx, data) + if !parsed { + b.open(d, file+" does not parse") + } + testOnly := d.node.Distribution.key() == b.main && modules.PythonTestPath(file) + for _, imp := range imports { + if imp.dynamic { + b.open(d, fmt.Sprintf("%s:%d imports a computed name", file, imp.line)) + continue + } + for _, to := range b.resolve(d, imp, file) { + b.edge(d, to, testOnly, Evidence{File: file, Line: imp.line}) + targets = append(targets, to) + } + } + } + return targets, nil +} + +// resolve returns the keys of the nodes that an import names. The first +// segment of the module path decides: a name of the project, a top-level +// name of a distribution, or a module of the standard library. A name of +// a namespace package that two distributions share goes to the one whose +// RECORD lists a file under the module path, else to each one. An import +// that nothing holds opens its importer, unless a try statement guards it +// against an ImportError. +func (b *builder) resolve(d *dist, imp importRef, file string) []string { + first, _, _ := strings.Cut(imp.module, ".") + self := d.node.Distribution.key() + switch { + case first == "": + return nil + case self == b.main && b.project[first]: + return nil + case slices.Contains(d.meta.TopLevel, first): + return nil + } + if keys := b.byTop[first]; len(keys) > 0 { + return b.namespaceOwners(keys, imp.module) + } + if self != b.main && b.project[first] { + return []string{b.main} + } + if stdlibModules[first] { + b.addStd() + return []string{b.std} + } + if !imp.guarded { + b.open(d, fmt.Sprintf("%s:%d imports %s, which no distribution holds", file, imp.line, imp.module)) + } + return nil +} + +// namespaceOwners returns the distributions of a top-level name that hold +// a module path, as google.cloud.storage: the ones whose RECORD lists a +// file under the longest prefix of the path that any of them holds. With +// no RECORD, each one may hold it. +func (b *builder) namespaceOwners(keys []string, module string) []string { + if len(keys) == 1 { + return keys + } + parts := strings.Split(module, ".") + for n := len(parts); n > 1; n-- { + prefix := path.Join(parts[:n]...) + var owners []string + for _, key := range keys { + files := b.dists[key].files + if files[prefix+".py"] || files[prefix+"/__init__.py"] { + owners = append(owners, key) + } + } + if len(owners) > 0 { + return owners + } + } + return keys +} + +// addStd adds the node of the standard library on its first import. +func (b *builder) addStd() { + if _, ok := b.dists[b.std]; !ok { + b.add(&dist{node: Node{Distribution: Distribution{Name: StdName}, State: StateImports, Walked: true}}) + } +} + +func (b *builder) edge(from *dist, to string, testOnly bool, ev Evidence) { + key := edgeKey{from.node.Distribution.key(), to} + if e, ok := b.edges[key]; ok { + if e.TestOnly && !testOnly { + e.TestOnly, e.Evidence = false, ev + } + return + } + b.edges[key] = &Edge{From: from.node.Distribution, To: b.dists[to].node.Distribution, Kind: StateImports, TestOnly: testOnly, Evidence: ev} +} + +// requires adds the Requires-Dist lines of each open distribution as +// evidence of edges that Build could not read. +func (b *builder) requires() { + for _, key := range b.order { + d := b.dists[key] + if !d.node.Open() { + continue + } + for _, name := range d.meta.Requires { + to := Distribution{Name: name}.key() + if _, ok := b.dists[to]; !ok || to == key { + continue + } + if _, ok := b.edges[edgeKey{key, to}]; !ok { + b.edges[edgeKey{key, to}] = &Edge{From: d.node.Distribution, To: b.dists[to].node.Distribution, Kind: StateRequires, Evidence: Evidence{File: path.Join(d.meta.Dir, "METADATA")}} + } + } + } +} + +func (b *builder) graph(reached map[string]bool) *Graph { + g := &Graph{Version: Version, Main: b.dists[b.main].node.Distribution, Problems: b.problems, Stats: b.stats} + for _, key := range b.order { + n := b.dists[key].node + n.Reached = reached[key] + g.Nodes = append(g.Nodes, n) + } + slices.SortFunc(g.Nodes, func(a, c Node) int { return strings.Compare(a.Distribution.key(), c.Distribution.key()) }) + for _, e := range b.edges { + g.Edges = append(g.Edges, *e) + } + slices.SortFunc(g.Edges, func(a, c Edge) int { + if r := strings.Compare(a.From.key(), c.From.key()); r != 0 { + return r + } + return strings.Compare(a.To.key(), c.To.key()) + }) + return g +} diff --git a/distgraph/graph_test.go b/distgraph/graph_test.go new file mode 100644 index 00000000..c2edf3ac --- /dev/null +++ b/distgraph/graph_test.go @@ -0,0 +1,324 @@ +package distgraph + +import ( + "context" + "io/fs" + "strings" + "testing" + "testing/fstest" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// py returns a Python file with one import statement per module. +func py(imports ...string) *fstest.MapFile { + var b strings.Builder + for _, imp := range imports { + b.WriteString("import " + imp + "\n") + } + return &fstest.MapFile{Data: []byte(b.String())} +} + +func text(s string) *fstest.MapFile { return &fstest.MapFile{Data: []byte(s)} } + +// sitePackages builds a site-packages directory. Each distribution is a +// name, a version, the files of its RECORD and the Requires-Dist lines of +// its METADATA. The files get the content of the files map, or an empty +// file. +func sitePackages(files fstest.MapFS, dists ...distInfo) fstest.MapFS { + out := fstest.MapFS{} + for k, v := range files { + out[k] = v + } + for _, d := range dists { + dir := strings.ReplaceAll(d.name, "-", "_") + "-" + d.version + ".dist-info" + meta := "Name: " + d.name + "\nVersion: " + d.version + "\n" + for _, r := range d.requires { + meta += "Requires-Dist: " + r + "\n" + } + out[dir+"/METADATA"] = text(meta) + record := "" + for _, f := range d.files { + record += f + ",,\n" + if _, ok := out[f]; !ok && strings.HasSuffix(f, ".py") { + out[f] = py() + } + } + out[dir+"/RECORD"] = text(record) + } + return out +} + +type distInfo struct { + name, version string + files []string + requires []string +} + +func build(t *testing.T, in Input) *Graph { + t.Helper() + g, err := Build(context.Background(), in) + require.NoError(t, err) + return g +} + +// edgeList writes each edge as "from -> to" with a mark for a test-only +// edge or a requires edge. +func edgeList(g *Graph) []string { + var out []string + for _, e := range g.Edges { + s := e.From.Name + " -> " + e.To.Name + if e.TestOnly { + s += " (test)" + } + if e.Kind == StateRequires { + s += " (requires)" + } + out = append(out, s) + } + return out +} + +func node(t *testing.T, g *Graph, name string) Node { + t.Helper() + n, ok := g.Node(name) + require.True(t, ok, "no node %s", name) + return n +} + +func TestBuildFollowsTheImportsOfTheReachedDistributions(t *testing.T) { + project := fstest.MapFS{ + "app/__init__.py": py(), + "app/main.py": py("os", "requests", "app.util", "yaml"), + "app/util.py": py("json"), + } + sp := sitePackages(fstest.MapFS{ + "requests/__init__.py": py("urllib3", "requests.api"), + "requests/api.py": text("try:\n import simplejson as json\nexcept ImportError:\n import json\n"), + "urllib3/__init__.py": py("socket"), + "six.py": py("yaml"), + "yaml/__init__.py": py(), + }, + distInfo{name: "requests", version: "2.31.0", files: []string{"requests/__init__.py", "requests/api.py"}}, + distInfo{name: "urllib3", version: "2.0.7", files: []string{"urllib3/__init__.py"}}, + distInfo{name: "six", version: "1.16.0", files: []string{"six.py"}}, + distInfo{name: "PyYAML", version: "6.0.1", files: []string{"yaml/__init__.py"}}, + ) + g := build(t, Input{ProjectFS: project, ProjectName: "app", SitePackages: []fs.FS{sp}}) + + assert.Equal(t, Distribution{Name: "app"}, g.Main) + assert.Equal(t, []string{"app -> PyYAML", "app -> requests", "app -> std", "requests -> std", "requests -> urllib3", "urllib3 -> std"}, edgeList(g), + "a guarded import of a module that nothing holds gives no edge, and a module of the project gives none") + for _, name := range []string{"app", "requests", "urllib3", "PyYAML", "std"} { + n := node(t, g, name) + assert.Equal(t, StateImports, n.State, name) + assert.True(t, n.Reached, name) + assert.True(t, n.Walked, name) + } + six := node(t, g, "six") + assert.False(t, six.Reached, "no import reaches six") + assert.False(t, six.Walked, "Build does not read a distribution that no import reaches") + assert.Equal(t, StateImports, six.State) + assert.Equal(t, 4, g.Stats.Distributions) + assert.Equal(t, 7, g.Stats.Files, "the three files of the project and the four of the reached distributions") + assert.Equal(t, Evidence{File: "app/main.py", Line: 2}, g.Edges[1].Evidence) + assert.Equal(t, []string{"app", "requests", "urllib3"}, g.Path([]string{"app"}, "urllib3")) + assert.Nil(t, g.Path([]string{"app"}, "six")) + assert.Equal(t, "2.0.7", node(t, g, "URLLIB3").Distribution.Version, "a node is found by any spelling of its name") +} + +func TestBuildMarksTheTestOnlyEdgesOfTheProject(t *testing.T) { + project := fstest.MapFS{ + "app.py": py("requests"), + "tests/test_app.py": py("pytest", "requests"), + "tests/conftest.py": py("six"), + "src/pkg/__init__.py": py("pkg.inner"), + "src/pkg/inner.py": py(), + } + sp := sitePackages(nil, + distInfo{name: "requests", version: "2.31.0", files: []string{"requests/__init__.py"}}, + distInfo{name: "pytest", version: "7.0.0", files: []string{"pytest/__init__.py"}}, + distInfo{name: "six", version: "1.16.0", files: []string{"six.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + assert.Equal(t, []string{"project -> pytest (test)", "project -> requests", "project -> six (test)"}, edgeList(g), + "an edge that a test file and a program file give is no test-only edge, and a package under src is a name of the project") + assert.Equal(t, "project", g.Main.Name, "the project node has a default name") +} + +func TestBuildResolvesANamespacePackage(t *testing.T) { + project := fstest.MapFS{ + "main.py": py("google.cloud.storage", "google.auth.crypt"), + "other.py": py("google"), + } + sp := sitePackages(nil, + distInfo{name: "google-auth", version: "2.0.0", files: []string{"google/auth/__init__.py", "google/auth/crypt.py"}}, + distInfo{name: "google-cloud-storage", version: "2.1.0", files: []string{"google/cloud/storage/__init__.py"}}, + distInfo{name: "google-cloud-core", version: "2.3.0", files: []string{"google/cloud/__init__.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + assert.Equal(t, []string{"project -> google-auth", "project -> google-cloud-core", "project -> google-cloud-storage"}, edgeList(g)) + assert.Equal(t, Evidence{File: "main.py", Line: 1}, g.Edges[2].Evidence, "google.cloud.storage goes to the distribution whose RECORD holds the module") + assert.Equal(t, Evidence{File: "other.py", Line: 1}, g.Edges[1].Evidence, "a bare import of the namespace goes to each distribution that shares it") +} + +func TestBuildOpensADistributionWithAnExtensionModule(t *testing.T) { + project := fstest.MapFS{"main.py": py("lxml")} + sp := sitePackages(fstest.MapFS{"lxml/etree.cpython-311-x86_64-linux-gnu.so": text("")}, + distInfo{name: "lxml", version: "4.9.0", files: []string{"lxml/__init__.py", "lxml/etree.cpython-311-x86_64-linux-gnu.so"}, requires: []string{"six", "cssselect ; extra == 'cssselect'"}}, + distInfo{name: "six", version: "1.16.0", files: []string{"six.py"}}, + distInfo{name: "cssselect", version: "1.2.0", files: []string{"cssselect/__init__.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + lxml := node(t, g, "lxml") + assert.Equal(t, StateOpen, lxml.State) + assert.Equal(t, "the extension module lxml/etree can import any module", lxml.Reason) + assert.Equal(t, []string{"lxml -> six (requires)", "project -> lxml"}, edgeList(g), "the Requires-Dist lines of an open node are evidence of edges") + assert.True(t, node(t, g, "cssselect").Reached, "a reached open node reaches each distribution") + assert.True(t, node(t, g, "cssselect").Walked, "and Build reads each one") + assert.Equal(t, []string{"project", "lxml", "cssselect"}, g.Path([]string{"project"}, "cssselect")) + + g = build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}, TrustCompiled: true}) + assert.Equal(t, StateImports, node(t, g, "lxml").State) + assert.Equal(t, []string{"project -> lxml"}, edgeList(g)) + assert.False(t, node(t, g, "six").Reached) +} + +func TestBuildOpensADistributionThatImportsAComputedName(t *testing.T) { + project := fstest.MapFS{"main.py": py("plugins", "loader")} + sp := sitePackages(fstest.MapFS{ + "plugins/__init__.py": text("import importlib\n\ndef load(name):\n return importlib.import_module(name)\n"), + "loader/__init__.py": text("import importlib\n\nyaml = importlib.import_module(\"yaml\")\nsix = __import__('six')\n"), + }, + distInfo{name: "plugins", version: "1.0.0", files: []string{"plugins/__init__.py"}}, + distInfo{name: "loader", version: "1.0.0", files: []string{"loader/__init__.py"}}, + distInfo{name: "PyYAML", version: "6.0.1", files: []string{"yaml/__init__.py"}}, + distInfo{name: "six", version: "1.16.0", files: []string{"six.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + plugins := node(t, g, "plugins") + assert.Equal(t, StateOpen, plugins.State) + assert.Equal(t, "plugins/__init__.py:4 imports a computed name", plugins.Reason) + assert.Equal(t, StateImports, node(t, g, "loader").State, "an import of a string names its module") + assert.Equal(t, []string{"loader -> PyYAML", "loader -> six", "loader -> std", "plugins -> std", "project -> loader", "project -> plugins"}, edgeList(g)) +} + +func TestBuildOpensADistributionThatImportsWhatNothingHolds(t *testing.T) { + project := fstest.MapFS{"main.py": py("dep")} + sp := sitePackages(fstest.MapFS{ + "dep/__init__.py": text("import missing\ntry:\n import optional\nexcept (ImportError, OSError):\n optional = None\ntry:\n import other\nexcept:\n pass\n"), + }, + distInfo{name: "dep", version: "1.0.0", files: []string{"dep/__init__.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + dep := node(t, g, "dep") + assert.Equal(t, StateOpen, dep.State) + assert.Equal(t, "dep/__init__.py:1 imports missing, which no distribution holds", dep.Reason, "the guarded imports open nothing") + assert.Equal(t, []string{"project -> dep"}, edgeList(g)) +} + +func TestBuildOpensADistributionWithAFileThatDoesNotParse(t *testing.T) { + project := fstest.MapFS{"main.py": py("dep")} + sp := sitePackages(fstest.MapFS{"dep/__init__.py": text("import yaml\ndef (:\n")}, + distInfo{name: "dep", version: "1.0.0", files: []string{"dep/__init__.py"}}, + distInfo{name: "PyYAML", version: "6.0.1", files: []string{"yaml/__init__.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + dep := node(t, g, "dep") + assert.Equal(t, StateOpen, dep.State) + assert.Equal(t, "dep/__init__.py does not parse", dep.Reason) + assert.Equal(t, []string{"dep -> PyYAML", "project -> dep"}, edgeList(g), "the imports that the tree shows still give edges") +} + +func TestBuildOpensANodePastTheFileBudget(t *testing.T) { + project := fstest.MapFS{"a.py": py("dep"), "b.py": py(), "c.py": py()} + sp := sitePackages(nil, distInfo{name: "dep", version: "1.0.0", files: []string{"dep/__init__.py"}}) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}, MaxFiles: 2}) + assert.True(t, g.Stats.Truncated) + assert.Equal(t, StateOpen, node(t, g, "project").State) + assert.Equal(t, "the file budget ran out", node(t, g, "project").Reason) + assert.Equal(t, 2, g.Stats.Files) +} + +func TestBuildMarksADistributionWithNoPythonFileAsRequires(t *testing.T) { + project := fstest.MapFS{"main.py": py("fast")} + sp := sitePackages(fstest.MapFS{"fast.cpython-311-x86_64-linux-gnu.so": text("")}, + distInfo{name: "fast", version: "1.0.0", files: []string{"fast.cpython-311-x86_64-linux-gnu.so"}, requires: []string{"six>=1.0", "fast"}}, + distInfo{name: "six", version: "1.16.0", files: []string{"six.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}, TrustCompiled: true}) + fast := node(t, g, "fast") + assert.Equal(t, StateRequires, fast.State) + assert.Equal(t, "no Python file", fast.Reason) + assert.Equal(t, []string{"fast -> six (requires)", "project -> fast"}, edgeList(g), "a requirement on itself is no edge") + assert.Equal(t, Evidence{File: "fast-1.0.0.dist-info/METADATA"}, g.Edges[0].Evidence) +} + +func TestBuildNamesASecondDistInfoOfOneDistribution(t *testing.T) { + project := fstest.MapFS{"main.py": py("six")} + sp := sitePackages(nil, + distInfo{name: "six", version: "1.16.0", files: []string{"six.py"}}, + distInfo{name: "six", version: "1.15.0", files: []string{"six.py"}}, + ) + sp["broken.dist-info/METADATA"] = text("Version: 1\n") + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + six := node(t, g, "six") + assert.Equal(t, "1.15.0", six.Distribution.Version, "the first .dist-info in directory order stays") + assert.Equal(t, StateOpen, six.State) + assert.Equal(t, "two .dist-info directories describe the distribution", six.Reason) + require.Len(t, g.Problems, 2) + assert.Equal(t, "broken.dist-info", g.Problems[0].Path) + assert.Equal(t, "six-1.16.0.dist-info", g.Problems[1].Path) + assert.Equal(t, 1, g.Stats.Distributions) +} + +func TestBuildSkipsTheEnvironmentInsideTheProject(t *testing.T) { + project := fstest.MapFS{ + "main.py": py("requests"), + ".venv/pyvenv.cfg": text("home = /usr/bin\n"), + ".venv/lib/python3.12/site-packages/x.py": py("yaml"), + "venv/pyvenv.cfg": text(""), + "venv/lib/site-packages/y.py": py("yaml"), + "node_modules/pkg/z.py": py("yaml"), + "build/__pycache__/w.py": py("yaml"), + } + sp := sitePackages(nil, + distInfo{name: "requests", version: "2.31.0", files: []string{"requests/__init__.py"}}, + distInfo{name: "PyYAML", version: "6.0.1", files: []string{"yaml/__init__.py"}}, + ) + g := build(t, Input{ProjectFS: project, SitePackages: []fs.FS{sp}}) + assert.Equal(t, []string{"project -> requests"}, edgeList(g)) + assert.Equal(t, 2, g.Stats.Files) +} + +func TestBuildRejectsABadInput(t *testing.T) { + _, err := Build(context.Background(), Input{}) + assert.EqualError(t, err, "distgraph: no project") + _, err = Build(context.Background(), Input{ProjectFS: fstest.MapFS{}, ProjectName: "std"}) + assert.EqualError(t, err, `distgraph: the project holds the reserved name "std"`) +} + +func TestGraphPath(t *testing.T) { + g := &Graph{ + Nodes: []Node{ + {Distribution: Distribution{Name: "a"}, State: StateImports, Walked: true}, + {Distribution: Distribution{Name: "b"}, State: StateImports, Walked: true}, + {Distribution: Distribution{Name: "c"}, State: StateOpen, Walked: true}, + {Distribution: Distribution{Name: "d"}, State: StateImports}, + {Distribution: Distribution{Name: "Z_z"}, State: StateImports, Walked: true}, + }, + Edges: []Edge{ + {From: Distribution{Name: "a"}, To: Distribution{Name: "b"}}, + {From: Distribution{Name: "b"}, To: Distribution{Name: "c"}}, + {From: Distribution{Name: "a"}, To: Distribution{Name: "d"}}, + }, + } + assert.Equal(t, []string{"a", "b"}, g.Path([]string{"a"}, "b")) + assert.Equal(t, []string{"b", "c", "Z_z"}, g.Path([]string{"b"}, "z-z"), "an open node reaches each distribution, in any spelling of its name") + assert.Equal(t, []string{"a", "d", "Z_z"}, g.Path([]string{"a"}, "Z.Z"), "a node that Build did not walk reaches each distribution") + assert.Equal(t, []string{"b", "c", "a"}, g.Path([]string{"b"}, "a")) + assert.Nil(t, g.Path([]string{"Z_z"}, "a")) + assert.Nil(t, g.Path([]string{"a"}, "a"), "a distribution is not on a path from itself") + assert.True(t, g.Reaches([]string{"a"}, "d")) +} diff --git a/distgraph/imports.go b/distgraph/imports.go new file mode 100644 index 00000000..754da378 --- /dev/null +++ b/distgraph/imports.go @@ -0,0 +1,157 @@ +package distgraph + +import ( + "context" + "fmt" + + sitter "github.com/smacker/go-tree-sitter" + + "github.com/safedep/code/core" + "github.com/safedep/code/lang" + "github.com/safedep/code/parser" + "github.com/safedep/code/pkg/ts" +) + +// importRef is one import of a Python file. A dynamic import names a +// module that the code computes, so the file can import any module. A +// guarded import sits in a try statement that catches an ImportError, so +// the program runs without the module. +type importRef struct { + module string + line int + dynamic bool + guarded bool +} + +// importers are the functions that import a module by a name at run time. +var importers = map[string]bool{"importlib.import_module": true, "import_module": true, "__import__": true} + +// importErrors are the exceptions that a try statement catches to run +// without a module. +var importErrors = map[string]bool{"ImportError": true, "ModuleNotFoundError": true, "Exception": true, "BaseException": true} + +// importReader reads the imports of Python files with the tree-sitter +// grammar of Python. +type importReader struct { + lang core.Language +} + +func newImportReader() (*importReader, error) { + l, err := lang.NewPythonLanguage() + if err != nil { + return nil, fmt.Errorf("distgraph: %w", err) + } + return &importReader{lang: l}, nil +} + +// read returns the imports of a Python file, and whether the whole file +// parses. A file with a syntax error can hold an import that the tree does +// not show. +func (r *importReader) read(ctx context.Context, data []byte) ([]importRef, bool) { + tree, err := parser.ParseBytes(ctx, r.lang, data) + if err != nil { + return nil, false + } + defer tree.Close() + root := tree.Tree().RootNode() + var out []importRef + text := func(n *sitter.Node) string { return n.Content(data) } + var walk func(n *sitter.Node, guarded bool) + walk = func(n *sitter.Node, guarded bool) { + switch n.Type() { + case "import_statement": + for _, child := range ts.NamedChildren(n) { + name := child + if child.Type() == "aliased_import" { + name = child.ChildByFieldName("name") + } + if name != nil && name.Type() == "dotted_name" { + out = append(out, importRef{module: text(name), line: int(name.StartPoint().Row) + 1, guarded: guarded}) + } + } + return + case "import_from_statement": + if name := n.ChildByFieldName("module_name"); name != nil && name.Type() == "dotted_name" { + out = append(out, importRef{module: text(name), line: int(name.StartPoint().Row) + 1, guarded: guarded}) + } + return + case "future_import_statement": + return + case "call": + if fn := n.ChildByFieldName("function"); fn != nil && importers[text(fn)] { + out = append(out, r.dynamicImport(n, data, guarded)) + } + case "try_statement": + guarded = guarded || catchesImportError(n, data) + } + for _, child := range ts.NamedChildren(n) { + walk(child, guarded) + } + } + walk(root, false) + return out, !root.HasError() +} + +// dynamicImport returns the import of a call of an importer: the module of +// a plain string argument, or a dynamic import for any other argument. +func (r *importReader) dynamicImport(call *sitter.Node, data []byte, guarded bool) importRef { + ref := importRef{line: int(call.StartPoint().Row) + 1, dynamic: true, guarded: guarded} + args := call.ChildByFieldName("arguments") + if args == nil || args.NamedChildCount() == 0 { + return ref + } + arg := args.NamedChild(0) + if arg.Type() != "string" { + return ref + } + var content string + for _, part := range ts.NamedChildren(arg) { + switch part.Type() { + case "string_content": + content += part.Content(data) + case "string_start", "string_end": + default: + return ref + } + } + ref.module, ref.dynamic = content, false + return ref +} + +// catchesImportError reports whether a try statement has an except clause +// for an ImportError: a bare except, or one that names the error or a base +// of it. +func catchesImportError(try *sitter.Node, data []byte) bool { + for _, child := range ts.NamedChildren(try) { + if child.Type() != "except_clause" { + continue + } + var caught *sitter.Node + for _, part := range ts.NamedChildren(child) { + if part.Type() != "block" { + caught = part + break + } + } + if caught == nil || namesImportError(caught, data) { + return true + } + } + return false +} + +// namesImportError reports whether an exception expression names an +// ImportError, alone or in a tuple, as except (ImportError, OSError). +func namesImportError(n *sitter.Node, data []byte) bool { + switch n.Type() { + case "identifier": + return importErrors[n.Content(data)] + case "tuple", "parenthesized_expression", "as_pattern": + for _, child := range ts.NamedChildren(n) { + if namesImportError(child, data) { + return true + } + } + } + return false +} diff --git a/distgraph/imports_test.go b/distgraph/imports_test.go new file mode 100644 index 00000000..99e53c4f --- /dev/null +++ b/distgraph/imports_test.go @@ -0,0 +1,70 @@ +package distgraph + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestImportReaderReadsEachFormOfAnImport(t *testing.T) { + reader, err := newImportReader() + require.NoError(t, err) + source := `from __future__ import annotations +import os, sys.path +import yaml as y +from a.b import c, d as e +from . import sibling +from ..up import thing +import importlib +m = importlib.import_module("yaml.loader") +n = importlib.import_module(name) +o = __import__('six') +p = __import__(f"pkg.{sub}") +try: + import simplejson +except ImportError: + simplejson = None +try: + import ujson +except (ValueError, ModuleNotFoundError) as e: + pass +try: + import orjson +except: + pass +try: + import nope +except ValueError: + pass +def f(): + import inner +` + imports, parsed := reader.read(context.Background(), []byte(source)) + assert.True(t, parsed) + assert.Equal(t, []importRef{ + {module: "os", line: 2}, + {module: "sys.path", line: 2}, + {module: "yaml", line: 3}, + {module: "a.b", line: 4}, + {module: "importlib", line: 7}, + {module: "yaml.loader", line: 8}, + {line: 9, dynamic: true}, + {module: "six", line: 10}, + {line: 11, dynamic: true}, + {module: "simplejson", line: 13, guarded: true}, + {module: "ujson", line: 17, guarded: true}, + {module: "orjson", line: 21, guarded: true}, + {module: "nope", line: 25}, + {module: "inner", line: 29}, + }, imports, "a relative import and a future import give nothing") +} + +func TestImportReaderReportsAFileThatDoesNotParse(t *testing.T) { + reader, err := newImportReader() + require.NoError(t, err) + imports, parsed := reader.read(context.Background(), []byte("import yaml\ndef (:\n")) + assert.False(t, parsed) + assert.Equal(t, []importRef{{module: "yaml", line: 1}}, imports) +} diff --git a/distgraph/stdlib.go b/distgraph/stdlib.go new file mode 100644 index 00000000..275f6314 --- /dev/null +++ b/distgraph/stdlib.go @@ -0,0 +1,71 @@ +package distgraph + +// stdlibModules names the top-level modules of the Python standard library: +// sys.stdlib_module_names of CPython 3.13, and the modules that 3.12 and +// 3.13 removed, which a program on an older Python still imports. An +// import of one of them is an edge to the node of the standard library. +var stdlibModules = map[string]bool{ + "__future__": true, "_abc": true, "_aix_support": true, "_android_support": true, + "_apple_support": true, "_ast": true, "_asyncio": true, "_bisect": true, "_blake2": true, + "_bootlocale": true, "_bz2": true, "_codecs": true, "_codecs_cn": true, "_codecs_hk": true, + "_codecs_iso2022": true, "_codecs_jp": true, "_codecs_kr": true, "_codecs_tw": true, + "_collections": true, "_collections_abc": true, "_colorize": true, "_compat_pickle": true, + "_compression": true, "_contextvars": true, "_csv": true, "_ctypes": true, "_curses": true, + "_curses_panel": true, "_datetime": true, "_dbm": true, "_decimal": true, "_dummy_thread": true, + "_elementtree": true, "_frozen_importlib": true, "_frozen_importlib_external": true, + "_functools": true, "_gdbm": true, "_hashlib": true, "_heapq": true, "_imp": true, + "_interpchannels": true, "_interpqueues": true, "_interpreters": true, "_io": true, + "_ios_support": true, "_json": true, "_locale": true, "_lsprof": true, "_lzma": true, + "_markupbase": true, "_md5": true, "_multibytecodec": true, "_multiprocessing": true, + "_opcode": true, "_opcode_metadata": true, "_operator": true, "_osx_support": true, + "_overlapped": true, "_pickle": true, "_posixshmem": true, "_posixsubprocess": true, + "_py_abc": true, "_pydatetime": true, "_pydecimal": true, "_pyio": true, "_pylong": true, + "_pyrepl": true, "_queue": true, "_random": true, "_scproxy": true, "_sha1": true, "_sha2": true, + "_sha3": true, "_signal": true, "_sitebuiltins": true, "_socket": true, "_sqlite3": true, + "_sre": true, "_ssl": true, "_stat": true, "_statistics": true, "_string": true, + "_strptime": true, "_struct": true, "_suggestions": true, "_symtable": true, "_sysconfig": true, + "_thread": true, "_threading_local": true, "_tkinter": true, "_tokenize": true, + "_tracemalloc": true, "_typing": true, "_uuid": true, "_warnings": true, "_weakref": true, + "_weakrefset": true, "_winapi": true, "_wmi": true, "_zoneinfo": true, "abc": true, "aifc": true, + "antigravity": true, "argparse": true, "array": true, "ast": true, "asynchat": true, + "asyncio": true, "asyncore": true, "atexit": true, "audioop": true, "base64": true, "bdb": true, + "binascii": true, "binhex": true, "bisect": true, "builtins": true, "bz2": true, "cProfile": true, + "calendar": true, "cgi": true, "cgitb": true, "chunk": true, "cmath": true, "cmd": true, + "code": true, "codecs": true, "codeop": true, "collections": true, "colorsys": true, + "compileall": true, "concurrent": true, "configparser": true, "contextlib": true, + "contextvars": true, "copy": true, "copyreg": true, "crypt": true, "csv": true, "ctypes": true, + "curses": true, "dataclasses": true, "datetime": true, "dbm": true, "decimal": true, + "difflib": true, "dis": true, "distutils": true, "doctest": true, "dummy_threading": true, + "email": true, "encodings": true, "ensurepip": true, "enum": true, "errno": true, + "faulthandler": true, "fcntl": true, "filecmp": true, "fileinput": true, "fnmatch": true, + "formatter": true, "fractions": true, "ftplib": true, "functools": true, "gc": true, + "genericpath": true, "getopt": true, "getpass": true, "gettext": true, "glob": true, + "graphlib": true, "grp": true, "gzip": true, "hashlib": true, "heapq": true, "hmac": true, + "html": true, "http": true, "idlelib": true, "imaplib": true, "imghdr": true, "imp": true, + "importlib": true, "inspect": true, "io": true, "ipaddress": true, "itertools": true, + "json": true, "keyword": true, "lib2to3": true, "linecache": true, "locale": true, + "logging": true, "lzma": true, "macpath": true, "mailbox": true, "mailcap": true, "marshal": true, + "math": true, "mimetypes": true, "mmap": true, "modulefinder": true, "msilib": true, + "msvcrt": true, "multiprocessing": true, "netrc": true, "nis": true, "nntplib": true, "nt": true, + "ntpath": true, "nturl2path": true, "numbers": true, "opcode": true, "operator": true, + "optparse": true, "os": true, "ossaudiodev": true, "parser": true, "pathlib": true, "pdb": true, + "pickle": true, "pickletools": true, "pipes": true, "pkgutil": true, "platform": true, + "plistlib": true, "poplib": true, "posix": true, "posixpath": true, "pprint": true, + "profile": true, "pstats": true, "pty": true, "pwd": true, "py_compile": true, "pyclbr": true, + "pydoc": true, "pydoc_data": true, "pyexpat": true, "queue": true, "quopri": true, "random": true, + "re": true, "readline": true, "reprlib": true, "resource": true, "rlcompleter": true, + "runpy": true, "sched": true, "secrets": true, "select": true, "selectors": true, "shelve": true, + "shlex": true, "shutil": true, "signal": true, "site": true, "smtpd": true, "smtplib": true, + "sndhdr": true, "socket": true, "socketserver": true, "spwd": true, "sqlite3": true, + "sre_compile": true, "sre_constants": true, "sre_parse": true, "ssl": true, "stat": true, + "statistics": true, "string": true, "stringprep": true, "struct": true, "subprocess": true, + "sunau": true, "symbol": true, "symtable": true, "sys": true, "sysconfig": true, "syslog": true, + "tabnanny": true, "tarfile": true, "telnetlib": true, "tempfile": true, "termios": true, + "textwrap": true, "this": true, "threading": true, "time": true, "timeit": true, "tkinter": true, + "token": true, "tokenize": true, "tomllib": true, "trace": true, "traceback": true, + "tracemalloc": true, "tty": true, "turtle": true, "turtledemo": true, "types": true, + "typing": true, "unicodedata": true, "unittest": true, "urllib": true, "uu": true, "uuid": true, + "venv": true, "warnings": true, "wave": true, "weakref": true, "webbrowser": true, "winreg": true, + "winsound": true, "wsgiref": true, "xdrlib": true, "xml": true, "xmlrpc": true, "zipapp": true, + "zipfile": true, "zipimport": true, "zlib": true, "zoneinfo": true, +} diff --git a/docs/distgraph.md b/docs/distgraph.md new file mode 100644 index 00000000..1dd29071 --- /dev/null +++ b/docs/distgraph.md @@ -0,0 +1,124 @@ +# Distribution graph + +`distgraph` builds the distribution graph of a Python project: which +installed distribution of its environment imports which. A lockfile lists +the distributions of an environment, but not the edges between them. A +caller needs the edges to tell that no code imports a distribution, or +that no other distribution can call one. + +The package reads the code on the disk: the Python files of the project +and of each distribution in the site-packages directories that the caller +names. It runs no Python, opens no network connection and reads no byte +code. It parses each file with the tree-sitter grammar of Python and reads +the import statements and the calls of `importlib.import_module` and +`__import__`. It is the analog of [`modgraph`](modgraph.md) for Python. + +## The rule + +The graph may hold an edge that a run of the program does not have. It +never misses an edge that a run has. A caller can then trust the absence +of a path: no run of the program can reach a distribution that the graph +does not reach. + +So a distribution whose imports `Build` cannot read in full is **open**: +it may import any distribution, and a path that meets it reaches each +distribution. + +| State | The distribution | Edges | +| --- | --- | --- | +| `imports` | `Build` read each Python file of it | The imports of those files | +| `requires` | Has no Python file, as a distribution of one extension module | Its `Requires-Dist` lines, as evidence only. A requirement is no import, and an import needs no requirement. So the node is open | +| `open` | Has a file that `Build` could not read in full, or an import that `Build` cannot resolve | The imports that `Build` read, and its `Requires-Dist` lines as evidence | + +A distribution is open when: + +- it has an extension module, a `.so`, a `.pyd` or a `.dylib` file. Its + code can import any module. `Input.TrustCompiled` keeps such a node + closed, for a caller that reads the graph for a path into a distribution + and not for the absence of one. Most environments hold such a + distribution, as numpy, pydantic-core or cryptography; +- a file of it imports a computed name, as `importlib.import_module(name)` + or `__import__(f"pkg.{sub}")`. An import of a plain string, as + `importlib.import_module("yaml")`, is an import of that module; +- a file of it imports a module that no distribution of the site-packages + directories, no module of the project and no module of the standard + library holds. The module can be in a path that a `.pth` file or + `PYTHONPATH` adds, or in a system site-packages that the caller did not + name. An import inside a `try` statement whose `except` clause catches + `ImportError`, `ModuleNotFoundError`, `Exception` or every exception + opens nothing: the program runs without the module. An optional import + outside such a guard, as in a `contrib` module, opens its distribution; +- a file of it does not parse; +- the file budget ran out, `Input.MaxFiles` or 200,000 files; +- two `.dist-info` directories describe it, as a stale install next to the + current one. `Build` keeps the first in directory order, and cannot tell + which code the program runs. + +`Node.Reason` names the first cause, with the file and the line. + +## The walk + +1. `Build` reads the `.dist-info` directories of each site-packages + directory with `modules.PythonDistributions`: the name and the version + of `METADATA`, the import names of `top_level.txt` or `RECORD`, the + extension modules of `RECORD` and the `Requires-Dist` lines with no + `extra` marker. A `.dist-info` that it cannot read is a problem of the + graph, not a node. +2. It reads each Python file of the project, its test files too, except + under a hidden directory, a `__pycache__`, a `node_modules`, a + `site-packages` or a virtual environment, which holds a `pyvenv.cfg`. + An edge that only a test file gives is `TestOnly`. The top-level names + of the project are the modules and the packages at its root and under + `src`. +3. For each import, the first segment of the module path decides. A name + of the project gives no edge. A top-level name of a distribution gives + an edge to it. A name that two distributions share, as `google`, goes + to the distribution whose `RECORD` lists a file under the longest + prefix of the module path that any of them holds, as + `google/cloud/storage/__init__.py` for `google.cloud.storage`, else to + each one. A module of the standard library, by the table of + `sys.stdlib_module_names`, gives an edge to the node `std`. A relative + import stays inside its distribution. +4. The walk reads each distribution that an import reaches. The files of a + distribution are the files under its top-level names, except its test + files, which no program imports. A distribution that no import reaches + adds nothing, and `Node.Walked` is false for it. +5. Once a reached node is open, it can import any distribution. So each + node counts as reached, and `Build` reads each distribution. + +## Use + +```go +g, err := distgraph.Build(ctx, distgraph.Input{ + ProjectFS: os.DirFS(projectDir), + ProjectName: "app", + SitePackages: []fs.FS{os.DirFS(sitePackagesDir)}, +}) +if err != nil { + return err +} +path := g.Path([]string{"app"}, "urllib3") // nil when no path reaches it +``` + +`Graph.Path` and `Graph.Reaches` take a set of start distributions, in any +spelling that PEP 503 normalizes to the name of the node. A path that +meets an open node, or a distribution that `Build` did not walk, can reach +each distribution: the path ends at that node. + +The caller finds the site-packages directories. A virtual environment has +one under `lib/python3.X/site-packages`, and `pyvenv.cfg` says whether the +system site-packages count too. `examples/distgraph` takes each one as +`-site-packages`. Install the project first, as `pip install -e .` or +`uv sync`: an import of a distribution that is not installed opens its +importer, and one open node that the project reaches makes each +distribution reached. + +## What the graph does not read + +- Byte code, `.pth` files and `PYTHONPATH`. A module that only they give is + an import that nothing holds. +- Imports under `if TYPE_CHECKING:`. They give an edge, which a run does + not have. The rule allows the edge. +- The version of Python. The table of the standard library is the union of + the modules of CPython 3.13 and of the modules that 3.12 and 3.13 + removed, so a program on an older Python resolves them too. diff --git a/examples/distgraph/main.go b/examples/distgraph/main.go new file mode 100644 index 00000000..adcb3364 --- /dev/null +++ b/examples/distgraph/main.go @@ -0,0 +1,90 @@ +// Command distgraph prints the distribution graph of a Python project: +// each distribution of its site-packages directories, what Build knows of +// its imports, and the edges between them. It runs no Python and reads no +// byte code. Install the project first, as pip install -e . or uv sync, so +// each distribution that the code imports is in a site-packages directory: +// an import that no distribution holds opens its importer, and one open +// node that the project reaches makes each distribution reached. +// +// go run ./examples/distgraph -dir ../app -site-packages ../app/.venv/lib/python3.12/site-packages +// go run ./examples/distgraph -dir ../app -site-packages ../app/.venv/lib/python3.12/site-packages -json +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "io" + "os" + "path/filepath" + + "github.com/safedep/code/distgraph" +) + +type dirList []string + +func (d *dirList) String() string { return fmt.Sprint(*d) } +func (d *dirList) Set(v string) error { *d = append(*d, v); return nil } + +func main() { + dir := flag.String("dir", ".", "the root directory of the project") + name := flag.String("name", "", "the name of the project node, else the name of its directory") + var sitePackages dirList + flag.Var(&sitePackages, "site-packages", "a site-packages directory of the environment of the project (repeatable)") + trustCompiled := flag.Bool("trust-compiled", false, "keep a distribution with an extension module closed") + asJSON := flag.Bool("json", false, "print the graph as JSON") + flag.Parse() + if err := run(os.Stdout, *dir, *name, sitePackages, *trustCompiled, *asJSON); err != nil { + fmt.Fprintln(os.Stderr, "distgraph:", err) + os.Exit(1) + } +} + +func run(w io.Writer, dir, name string, sitePackages []string, trustCompiled, asJSON bool) error { + abs, err := filepath.Abs(dir) + if err != nil { + return err + } + if name == "" { + name = filepath.Base(abs) + } + in := distgraph.Input{ProjectFS: os.DirFS(abs), ProjectName: name, TrustCompiled: trustCompiled} + for _, sp := range sitePackages { + in.SitePackages = append(in.SitePackages, os.DirFS(sp)) + } + g, err := distgraph.Build(context.Background(), in) + if err != nil { + return err + } + if asJSON { + enc := json.NewEncoder(w) + enc.SetIndent("", " ") + return enc.Encode(g) + } + return print(w, g) +} + +func print(w io.Writer, g *distgraph.Graph) error { + for _, p := range g.Problems { + if _, err := fmt.Fprintf(w, "problem %s: %s\n", p.Path, p.Text); err != nil { + return err + } + } + for _, n := range g.Nodes { + reached := "unreached" + if n.Reached { + reached = "reached" + } + if _, err := fmt.Fprintf(w, "%-9s %-9s %s %s\n", reached, n.State, n.Distribution, n.Reason); err != nil { + return err + } + } + for _, e := range g.Edges { + if _, err := fmt.Fprintf(w, "%s -> %s (%s, %s:%d)\n", e.From.Name, e.To.Name, e.Kind, e.Evidence.File, e.Evidence.Line); err != nil { + return err + } + } + _, err := fmt.Fprintf(w, "%d files, %d distributions\n", g.Stats.Files, g.Stats.Distributions) + return err +} diff --git a/examples/distgraph/main_test.go b/examples/distgraph/main_test.go new file mode 100644 index 00000000..f8736c48 --- /dev/null +++ b/examples/distgraph/main_test.go @@ -0,0 +1,50 @@ +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/safedep/code/distgraph" +) + +func TestRunPrintsTheGraph(t *testing.T) { + dir := t.TempDir() + sp := filepath.Join(dir, ".venv", "lib", "python3.12", "site-packages") + for name, data := range map[string]string{ + filepath.Join(dir, "main.py"): "import requests\n", + filepath.Join(dir, ".venv", "pyvenv.cfg"): "home = /usr/bin\n", + filepath.Join(sp, "requests", "__init__.py"): "import urllib3\n", + filepath.Join(sp, "requests-2.31.0.dist-info", "METADATA"): "Name: requests\nVersion: 2.31.0\n", + filepath.Join(sp, "requests-2.31.0.dist-info", "RECORD"): "requests/__init__.py,,\n", + filepath.Join(sp, "urllib3", "__init__.py"): "import socket\n", + filepath.Join(sp, "urllib3-2.0.7.dist-info", "METADATA"): "Name: urllib3\nVersion: 2.0.7\n", + filepath.Join(sp, "urllib3-2.0.7.dist-info", "RECORD"): "urllib3/__init__.py,,\n", + } { + require.NoError(t, os.MkdirAll(filepath.Dir(name), 0o755)) + require.NoError(t, os.WriteFile(name, []byte(data), 0o600)) + } + + var out strings.Builder + require.NoError(t, run(&out, dir, "app", []string{sp}, false, false)) + assert.Contains(t, out.String(), "reached imports requests@2.31.0") + assert.Contains(t, out.String(), "app -> requests (imports, main.py:1)") + assert.Contains(t, out.String(), "urllib3 -> std (imports, urllib3/__init__.py:1)") + assert.Contains(t, out.String(), "3 files, 2 distributions") + + out.Reset() + require.NoError(t, run(&out, dir, "", []string{sp}, false, true)) + var g distgraph.Graph + require.NoError(t, json.Unmarshal([]byte(out.String()), &g)) + assert.Equal(t, filepath.Base(dir), g.Main.Name, "the project takes the name of its directory") + urllib3, ok := g.Node("urllib3") + require.True(t, ok) + assert.True(t, urllib3.Reached) + assert.Equal(t, distgraph.StateImports, urllib3.State) + assert.Equal(t, []string{g.Main.Name, "requests", "urllib3"}, g.Path([]string{g.Main.Name}, "urllib3")) +} diff --git a/plugin/indexer/pyunit.go b/plugin/indexer/pyunit.go index 15d0a157..4a239ebb 100644 --- a/plugin/indexer/pyunit.go +++ b/plugin/indexer/pyunit.go @@ -2,32 +2,21 @@ package indexer import ( "path" - "slices" - "strings" "github.com/safedep/code/core" "github.com/safedep/code/fileindex" + "github.com/safedep/code/projectgraph/modules" ) // pythonOutOfBuild reports whether no program imports a file of a pypi -// unit, by its path: a file under a tests directory, a test file by its -// name, a conftest.py, a file under a __pycache__, and a file under the -// .dist-info, .egg-info or .data directory of the wheel. Such a file holds -// no code that an importer runs, and a test file of a package adds test -// roots and budget to a graph that no search of the package needs. A test -// or a testing directory stays, because django.test and numpy.testing are -// modules that programs import. +// unit, by its path. Such a file holds no code that an importer runs, and a +// test file of a package adds test roots and budget to a graph that no +// search of the package needs. modules.PythonTestPath holds the rule. func pythonOutOfBuild(file core.File) bool { unit, inner := fileindex.SplitUnit(file.Name()) pkg, err := fileindex.ParseUnit(unit) if !file.IsImport() || err != nil || pkg.Ecosystem != "pypi" || path.Ext(inner) != ".py" { return false } - base := path.Base(inner) - if base == "conftest.py" || strings.HasPrefix(base, "test_") || strings.HasSuffix(base, "_test.py") { - return true - } - return slices.ContainsFunc(strings.Split(path.Dir(inner), "/"), func(d string) bool { - return d == "tests" || d == "__pycache__" || strings.HasSuffix(d, ".dist-info") || strings.HasSuffix(d, ".egg-info") || strings.HasSuffix(d, ".data") - }) + return modules.PythonTestPath(inner) } diff --git a/projectgraph/modules/pydist.go b/projectgraph/modules/pydist.go index abe275ae..92f6a218 100644 --- a/projectgraph/modules/pydist.go +++ b/projectgraph/modules/pydist.go @@ -26,6 +26,9 @@ type Distribution struct { TopLevel []string Files []string Compiled []string + // Requires holds the names of the distributions that the Requires-Dist + // lines of METADATA name with no extra marker, in order, each once. + Requires []string } // Unit returns the package unit of the distribution, as pypi:PyYAML@6.0.1. @@ -83,6 +86,39 @@ func (d Distribution) Roots() []string { return slices.Compact(roots) } +// requiredName returns the distribution name of a Requires-Dist value, as +// requests of "requests (>=2.0); python_version >= '3.8'" or of +// "requests[socks]>=2.0". A requirement with an extra marker, as +// "; extra == 'socks'", is no requirement of a plain install. +func requiredName(req string) (string, bool) { + spec, marker, _ := strings.Cut(req, ";") + if strings.Contains(marker, "extra") { + return "", false + } + end := strings.IndexAny(spec, " [(<>=!~;@") + if end < 0 { + end = len(spec) + } + name := strings.TrimSpace(spec[:end]) + return name, name != "" +} + +// PythonTestPath reports whether no program imports a Python file of a +// package, by its path under the root of the package: a file under a tests +// directory, a test file by its name, a conftest.py, a file under a +// __pycache__, and a file under the .dist-info, .egg-info or .data +// directory of a wheel. A test or a testing directory stays, because +// django.test and numpy.testing are modules that programs import. +func PythonTestPath(p string) bool { + base := path.Base(p) + if base == "conftest.py" || strings.HasPrefix(base, "test_") || strings.HasSuffix(base, "_test.py") { + return true + } + return slices.ContainsFunc(strings.Split(path.Dir(p), "/"), func(d string) bool { + return d == "tests" || d == "__pycache__" || strings.HasSuffix(d, ".dist-info") || strings.HasSuffix(d, ".egg-info") || strings.HasSuffix(d, ".data") + }) +} + var pythonNameRun = regexp.MustCompile(`[-_.]+`) // NormalizePythonName returns the name of a PyPI package as PEP 503 @@ -146,6 +182,11 @@ func readDistribution(root fs.FS, dir string) (Distribution, error) { if version, ok := strings.CutPrefix(line, "Version: "); ok && d.Version == "" { d.Version = strings.TrimSpace(version) } + if req, ok := strings.CutPrefix(line, "Requires-Dist: "); ok { + if name, ok := requiredName(req); ok && !slices.Contains(d.Requires, name) { + d.Requires = append(d.Requires, name) + } + } } if d.Name == "" { return d, errNoDistributionName diff --git a/projectgraph/modules/pydist_test.go b/projectgraph/modules/pydist_test.go index 86f202cf..fc80e201 100644 --- a/projectgraph/modules/pydist_test.go +++ b/projectgraph/modules/pydist_test.go @@ -10,7 +10,9 @@ import ( func TestPythonDistributions(t *testing.T) { root := fstest.MapFS{ - "PyYAML-6.0.1.dist-info/METADATA": {Data: []byte("Metadata-Version: 2.1\nName: PyYAML\nVersion: 6.0.1\nSummary: YAML\n\nThe body.\nName: not this\n")}, + "PyYAML-6.0.1.dist-info/METADATA": {Data: []byte("Metadata-Version: 2.1\nName: PyYAML\nVersion: 6.0.1\nSummary: YAML\n" + + "Requires-Dist: six (>=1.0)\nRequires-Dist: typing_extensions>=4; python_version < \"3.11\"\nRequires-Dist: six\n" + + "Requires-Dist: pytest ; extra == 'test'\nRequires-Dist: google-auth[aiohttp]\n\nThe body.\nName: not this\n")}, "PyYAML-6.0.1.dist-info/RECORD": {Data: []byte("PyYAML-6.0.1.dist-info/METADATA,sha256=abc,1730\n" + "_yaml/__init__.py,sha256=abc,10\n" + "yaml/__init__.py,sha256=abc,13340\n" + @@ -39,6 +41,7 @@ func TestPythonDistributions(t *testing.T) { assert.Equal(t, []string{"yaml/_yaml"}, yaml.Compiled) assert.NotContains(t, yaml.Files, "../../bin/yamlc", "a path that leaves the root is no file of the package") assert.Equal(t, []string{"_yaml", "yaml"}, yaml.Roots()) + assert.Equal(t, []string{"six", "typing_extensions", "google-auth"}, yaml.Requires, "a requirement with an extra marker is no requirement of a plain install") google := dists[1] assert.Equal(t, []string{"google"}, google.TopLevel) @@ -58,3 +61,20 @@ func TestPythonDistributionsWithNoRoot(t *testing.T) { assert.Empty(t, dists) assert.Empty(t, problems) } + +func TestPythonTestPath(t *testing.T) { + for p, want := range map[string]bool{ + "yaml/loader.py": false, + "django/test/client.py": false, + "numpy/testing/__init__.py": false, + "requests/tests/test_api.py": true, + "pkg/test_util.py": true, + "pkg/util_test.py": true, + "pkg/conftest.py": true, + "pkg/__pycache__/mod.py": true, + "six-1.16.0.dist-info/METADATA": true, + "pkg-1.0.data/scripts/run.py": true, + } { + assert.Equal(t, want, PythonTestPath(p), p) + } +} From e99ef9fd4ddd161e7bd29fc7fa0e633be3c1e721 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 14:02:19 +0000 Subject: [PATCH 09/11] Link the standard library of Python as a unit A call of the standard library on self, as self.run in Thread.start, reached the override of the project only through the reference edge of the override rule, because the graph had no code of the library. A path into a module of the library ended at its external key. The Lib directory of CPython is now the unit python:std@, as GOROOT/src is go:std. The index step leaves out its test suite, the IDLE, the turtle demos and lib2to3. The link step knows that the python ecosystem installs Python, so a truncated file of the unit counts only for a Python program. A class of a unit gives its external key to the class hierarchy analysis, because a class of the project names that key as its base, so Thread.run reaches Worker.run with a cha edge. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- docs/projectgraph.md | 12 +++ examples/projectgraph/main.go | 7 +- plugin/indexer/pyunit.go | 23 ++++- plugin/indexer/pyunit_test.go | 42 ++++++++ projectgraph/cha.go | 8 +- projectgraph/graph.go | 4 +- .../python/dependency-stdlib/expect.yaml | 95 +++++++++++++++++++ .../project/python/Lib/http/__init__.py | 0 .../project/python/Lib/subprocess.py | 10 ++ .../project/python/Lib/test/test_threading.py | 1 + .../project/python/Lib/threading.py | 12 +++ .../dependency-stdlib/project/server.py | 14 +++ .../summaries/python_std_3.12.3.json | 53 +++++++++++ test/projectgraph/version_test.go | 6 +- 14 files changed, 275 insertions(+), 12 deletions(-) create mode 100644 test/projectgraph/cases/python/dependency-stdlib/expect.yaml create mode 100644 test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/http/__init__.py create mode 100644 test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/subprocess.py create mode 100644 test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/test/test_threading.py create mode 100644 test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/threading.py create mode 100644 test/projectgraph/cases/python/dependency-stdlib/project/server.py create mode 100644 test/projectgraph/cases/python/dependency-stdlib/summaries/python_std_3.12.3.json diff --git a/docs/projectgraph.md b/docs/projectgraph.md index ed26f333..3d0412ed 100644 --- a/docs/projectgraph.md +++ b/docs/projectgraph.md @@ -294,6 +294,18 @@ time and a wrong hint gives a wrong edge, not a crash. These rules apply. `examples/projectgraph` takes `-site-packages ` and adds each distribution of the directory as a unit, by its `.dist-info`. +The standard library of Python is a unit too. Pass the `Lib` directory of +CPython as the unit `python:std@`, as +`-deps /usr/lib/python3.12=python:std@3.12.3`. A virtual environment names +the installation in the `home` line of its `pyvenv.cfg`. The unit gives +each module of the standard library that has Python source, as +`threading` or `http.server`, so a call of the library on `self`, as +`Thread.run` in `Thread.start`, reaches the override of the project with a +`cha` edge, and a path into the library goes on to its calls. A module +that only C gives, as `_thread`, stays an external key. The index step +leaves out the test suite under `test`, the IDLE, the turtle demos and +`lib2to3`, which no program imports. + ## Overlay An overlay is a file of `llm` edges that resolve unresolved sites of a diff --git a/examples/projectgraph/main.go b/examples/projectgraph/main.go index 9a21f28d..1166a5cf 100644 --- a/examples/projectgraph/main.go +++ b/examples/projectgraph/main.go @@ -14,9 +14,10 @@ // keeps its path under the import directory, as yaml/__init__.py. Any // other -deps directory is an import directory of its own, and a pypi // package given that way is its package. The Go standard library is a unit -// too, as -deps $(go env GOROOT)/src=go:std@go1.25.1, and so is a Rust -// crate, as -deps vendor/smallvec=cargo:smallvec@1.13.2. With -summary it -// prints the summary of each such unit as JSON. +// too, as -deps $(go env GOROOT)/src=go:std@go1.25.1, and so are the Lib +// directory of CPython, as -deps /usr/lib/python3.12=python:std@3.12.3, and +// a Rust crate, as -deps vendor/smallvec=cargo:smallvec@1.13.2. With +// -summary it prints the summary of each such unit as JSON. // // -site-packages dir reads the .dist-info of each distribution in a Python // site-packages directory and adds each one as a unit, with the import diff --git a/plugin/indexer/pyunit.go b/plugin/indexer/pyunit.go index 4a239ebb..ec9a95b8 100644 --- a/plugin/indexer/pyunit.go +++ b/plugin/indexer/pyunit.go @@ -2,21 +2,36 @@ package indexer import ( "path" + "strings" "github.com/safedep/code/core" "github.com/safedep/code/fileindex" "github.com/safedep/code/projectgraph/modules" ) -// pythonOutOfBuild reports whether no program imports a file of a pypi +// pythonOutOfBuild reports whether no program imports a file of a Python // unit, by its path. Such a file holds no code that an importer runs, and a // test file of a package adds test roots and budget to a graph that no -// search of the package needs. modules.PythonTestPath holds the rule. +// search of the package needs. modules.PythonTestPath holds the rule of a +// pypi unit. The unit of the standard library, as python:std@3.12.3 for +// the Lib directory of CPython, also leaves out its test suite under test, +// the IDLE, the turtle demos and lib2to3, which no program imports. func pythonOutOfBuild(file core.File) bool { unit, inner := fileindex.SplitUnit(file.Name()) pkg, err := fileindex.ParseUnit(unit) - if !file.IsImport() || err != nil || pkg.Ecosystem != "pypi" || path.Ext(inner) != ".py" { + if !file.IsImport() || err != nil || path.Ext(inner) != ".py" { return false } - return modules.PythonTestPath(inner) + switch pkg.Ecosystem { + case "pypi": + return modules.PythonTestPath(inner) + case "python": + first, _, _ := strings.Cut(inner, "/") + return modules.PythonTestPath(inner) || pythonStdlibOutOfBuild[first] + } + return false } + +// pythonStdlibOutOfBuild names the directories of the Lib directory of +// CPython that no program imports. +var pythonStdlibOutOfBuild = map[string]bool{"test": true, "idlelib": true, "turtledemo": true, "lib2to3": true, "__phello__": true} diff --git a/plugin/indexer/pyunit_test.go b/plugin/indexer/pyunit_test.go index 7e98581e..5ca36edf 100644 --- a/plugin/indexer/pyunit_test.go +++ b/plugin/indexer/pyunit_test.go @@ -58,3 +58,45 @@ func TestIndexerLeavesOutTheTestsOfAPypiUnit(t *testing.T) { assert.Equal(t, fileindex.FileIndexed, status) } } + +func TestIndexerLeavesOutTheTestSuiteOfTheStandardLibrary(t *testing.T) { + dir := t.TempDir() + writeFiles(t, dir, map[string]string{"main.py": "import threading\n"}) + lib := filepath.Join(dir, "python", "Lib") + writeFiles(t, lib, map[string]string{ + "threading.py": "class Thread:\n pass\n", + "http/__init__.py": "", + "http/server.py": "x = 1\n", + "test/test_threading.py": "x = 1\n", + "test/support/__init__.py": "x = 1\n", + "idlelib/editor.py": "x = 1\n", + "turtledemo/clock.py": "x = 1\n", + "lib2to3/main.py": "x = 1\n", + "unittest/test/test_case.py": "x = 1\n", + "unittest/__init__.py": "", + "ensurepip/__init__.py": "", + "__phello__/__init__.py": "", + }) + fsys, err := fs.NewLocalFileSystem(fs.LocalFileSystemConfig{ + AppDirectories: []string{dir}, + ImportDirectories: []string{lib}, + ImportUnits: map[string]string{lib: "python:std@3.12.3"}, + ExcludePatterns: []*regexp.Regexp{fs.SkipPattern(dir)}, + }) + require.NoError(t, err) + result, err := runIndexer(t, context.Background(), fsys, Options{Languages: languages(t, core.LanguageCodePython), Store: memstore.New(), Imports: true}) + require.NoError(t, err) + + var indexed, excluded []string + for p := range result.Dependencies { + _, inner := fileindex.SplitUnit(p) + indexed = append(indexed, inner) + } + for _, p := range result.Excluded { + _, inner := fileindex.SplitUnit(p) + excluded = append(excluded, inner) + } + assert.ElementsMatch(t, []string{"threading.py", "http/__init__.py", "http/server.py", "unittest/__init__.py", "ensurepip/__init__.py"}, indexed) + assert.ElementsMatch(t, []string{"test/test_threading.py", "test/support/__init__.py", "idlelib/editor.py", "turtledemo/clock.py", "lib2to3/main.py", "unittest/test/test_case.py", "__phello__/__init__.py"}, excluded, + "the test directory of a package of the standard library is a test suite, as unittest/test") +} diff --git a/projectgraph/cha.go b/projectgraph/cha.go index 4a0e91d4..0cfba62f 100644 --- a/projectgraph/cha.go +++ b/projectgraph/cha.go @@ -317,7 +317,10 @@ func (b *builder) overrides(class string, call fileindex.Call, found string) []s // classDefinitions returns the class itself, and for the external key of a // class of a unit of dependency code, as dep//Doer, the definitions that -// the units give for it, so the subclasses inside the unit count too. +// the units give for it, so the subclasses inside the unit count too. A +// class of a unit gives its external key, which a class of the project +// names as its base, so a call of the unit on self reaches the override +// of the project, as Thread.run of threading reaches the run of a Worker. func (b *builder) classDefinitions(class string) []string { out := []string{class} if b.g.IsExternal(class) { @@ -328,6 +331,9 @@ func (b *builder) classDefinitions(class string) []string { } } } + if key, _, ok := b.unitSymbol(class); ok && key != class { + out = append(out, key) + } return out } diff --git a/projectgraph/graph.go b/projectgraph/graph.go index b54702e8..d25631c9 100644 --- a/projectgraph/graph.go +++ b/projectgraph/graph.go @@ -300,9 +300,11 @@ func unitLanguages(unit string) []langcode.Code { // ecosystemLanguages are the languages of the code that a package of an // ecosystem installs. A package can ship files of other languages, as the // JavaScript of the admin pages of Django, and no program of the ecosystem -// runs them. +// runs them. The python ecosystem is the standard library of CPython, as +// the unit python:std@3.12.3 of its Lib directory. var ecosystemLanguages = map[string][]langcode.Code{ "pypi": {langcode.Python}, + "python": {langcode.Python}, "npm": {langcode.Javascript, langcode.Typescript}, "go": {langcode.Go}, "cargo": {langcode.Rust}, diff --git a/test/projectgraph/cases/python/dependency-stdlib/expect.yaml b/test/projectgraph/cases/python/dependency-stdlib/expect.yaml new file mode 100644 index 00000000..060c63a4 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-stdlib/expect.yaml @@ -0,0 +1,95 @@ +source: safedep:hand +description: >- + The Lib directory of CPython is the unit python:std@3.12.3. Worker extends threading.Thread, and the + main guard calls start, which the unit gives. start runs _bootstrap, _bootstrap calls self.run, and the + class hierarchy analysis adds the override Worker.run of the project, through the external key of + Thread that Worker names as its base. The override rule keeps its reference edge from Worker to run, and + the search reports that shorter path. subprocess.run resolves into the unit, and its path goes on to + os.execvp, a module that the unit does not hold. The test suite of the unit is left out. +units: + python/Lib: python:std@3.12.3 +roots: + - key: server.py + kind: main + rule: python.main-guard +edges: + - from: server.py + to: threading//Thread//start + kind: member + confidence: high + site: server.py:13 + - from: threading//Thread//start + to: python:std@3.12.3//threading.py//Thread//start + kind: package + confidence: high + site: python:std@3.12.3//threading.py:5 + - from: python:std@3.12.3//threading.py//Thread//_bootstrap + to: python:std@3.12.3//threading.py//Thread//run + kind: member + confidence: high + site: python:std@3.12.3//threading.py:9 + - from: python:std@3.12.3//threading.py//Thread//_bootstrap + to: server.py//Worker//run + kind: cha + confidence: medium + site: python:std@3.12.3//threading.py:9 + - from: server.py + to: subprocess//run + kind: direct + confidence: high + site: server.py:14 + - from: python:std@3.12.3//subprocess.py//Popen//__init__ + to: os//execvp + kind: direct + confidence: high + site: python:std@3.12.3//subprocess.py:6 +unresolved: [] +results: + - target: yaml//safe_load + status: found + path: + - key: server.py + - key: server.py//Worker + site: server.py:13 + kind: direct + confidence: high + - key: server.py//Worker//run + site: server.py:8 + kind: reference + confidence: low + - key: yaml//safe_load + site: server.py:9 + kind: direct + confidence: high + call_sites: + - caller: server.py//Worker//run + site: server.py:9 + reached: true + - target: os//execvp + status: found + path: + - key: server.py + - key: subprocess//run + site: server.py:14 + kind: direct + confidence: high + - key: python:std@3.12.3//subprocess.py//run + site: python:std@3.12.3//subprocess.py:9 + kind: package + confidence: high + - key: python:std@3.12.3//subprocess.py//Popen + site: python:std@3.12.3//subprocess.py:10 + kind: direct + confidence: high + - key: python:std@3.12.3//subprocess.py//Popen//__init__ + site: python:std@3.12.3//subprocess.py:5 + kind: direct + confidence: high + - key: os//execvp + site: python:std@3.12.3//subprocess.py:6 + kind: direct + confidence: high + call_sites: + - caller: python:std@3.12.3//subprocess.py//Popen//__init__ + site: python:std@3.12.3//subprocess.py:6 + reached: true diff --git a/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/http/__init__.py b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/http/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/subprocess.py b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/subprocess.py new file mode 100644 index 00000000..0c8f01a2 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/subprocess.py @@ -0,0 +1,10 @@ +import os + + +class Popen: + def __init__(self, args): + os.execvp(args[0], args) + + +def run(args): + return Popen(args) diff --git a/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/test/test_threading.py b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/test/test_threading.py new file mode 100644 index 00000000..7d4290a1 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/test/test_threading.py @@ -0,0 +1 @@ +x = 1 diff --git a/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/threading.py b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/threading.py new file mode 100644 index 00000000..f59a2eeb --- /dev/null +++ b/test/projectgraph/cases/python/dependency-stdlib/project/python/Lib/threading.py @@ -0,0 +1,12 @@ +import _thread + + +class Thread: + def start(self): + _thread.start_new_thread(self._bootstrap, ()) + + def _bootstrap(self): + self.run() + + def run(self): + pass diff --git a/test/projectgraph/cases/python/dependency-stdlib/project/server.py b/test/projectgraph/cases/python/dependency-stdlib/project/server.py new file mode 100644 index 00000000..768e796f --- /dev/null +++ b/test/projectgraph/cases/python/dependency-stdlib/project/server.py @@ -0,0 +1,14 @@ +import subprocess +import threading + +import yaml + + +class Worker(threading.Thread): + def run(self): + yaml.safe_load("a: 1") + + +if __name__ == "__main__": + Worker().start() + subprocess.run(["ls"]) diff --git a/test/projectgraph/cases/python/dependency-stdlib/summaries/python_std_3.12.3.json b/test/projectgraph/cases/python/dependency-stdlib/summaries/python_std_3.12.3.json new file mode 100644 index 00000000..0c1cb890 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-stdlib/summaries/python_std_3.12.3.json @@ -0,0 +1,53 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "python:std@3.12.3", + "symbols": [ + { + "key": "subprocess//Popen", + "definitions": [ + "python:std@3.12.3//subprocess.py//Popen" + ], + "reaches": [ + "os//execvp" + ] + }, + { + "key": "subprocess//run", + "definitions": [ + "python:std@3.12.3//subprocess.py//run" + ], + "reaches": [ + "os//execvp" + ] + }, + { + "key": "threading//Thread", + "definitions": [ + "python:std@3.12.3//threading.py//Thread" + ], + "reaches": [] + }, + { + "key": "threading//Thread//run", + "definitions": [ + "python:std@3.12.3//threading.py//Thread//run" + ], + "reaches": [] + }, + { + "key": "threading//Thread//start", + "definitions": [ + "python:std@3.12.3//threading.py//Thread//start" + ], + "reaches": [ + "_thread//start_new_thread" + ] + } + ], + "targets": [ + "_thread//start_new_thread", + "os//execvp" + ] +} diff --git a/test/projectgraph/version_test.go b/test/projectgraph/version_test.go index e666b4d6..7b02b001 100644 --- a/test/projectgraph/version_test.go +++ b/test/projectgraph/version_test.go @@ -31,7 +31,7 @@ var graphHashes = map[string]string{ "14/14": "0f3c9c38b39f4de1835f34e8a58bcea85461f37e7aa2d5a2dd484569e670ff48", "14/15": "bbc708aba767ccf554c75a7e06816abda3fc1804ad6d0876ad8ce9575f4c3a88", "14/16": "823b1f82c4aed8ba27d19275505c8d1dee755086be48fcda64d323f55b4c62a5", - "15/17": "ffa293e72af1682407e900e442fddb0c18f6174eba38cf412d1b7f362d9338ac", + "15/17": "7535543fc7ace2a4833b71de9ffac5a1dff1f349d508c2fe9bd9144ee2a2ae96", } // resultHashes records the hash of the reach results of every case for @@ -47,7 +47,7 @@ var resultHashes = map[string]string{ "14/14/2": "a832ea709bf3df852814375a56e6329c35e2ff4c49ae34ee49afe042e9502d17", "14/15/3": "77dfd0ea8ab71869ef612d41b8c2e46852d4fb67a49458cb645321692af77ea0", "14/16/3": "13b6eecb686ddec3c2ba470fc9752ea66cff445d5e18e0772a71b15ae7fe1c9c", - "15/17/4": "d3a7367dacb56d7e6dbff3b203d5b5de583abe70c1aad31898b7d6659629654c", + "15/17/4": "5b005738c979fc81b1e4617a0d922f5ab4c6e149c5f739271052180bdbbfd4ae", } // summaryHashes records the hash of the golden summaries of the cases for @@ -64,7 +64,7 @@ var summaryHashes = map[string]string{ "14/14/2": "27424c4e2525450479faba78421959738748811d70b9a93b81e2bbdf1733bf0d", "14/15/2": "5969f3cda15b666cbd902bbc2db953cbdef8c0ce48d5e77446cf243cdd0bb0e7", "14/16/2": "fb678202978212aae932e50a56a0d3dc7f0f794ce908988bf605ce9a6e04bc51", - "15/17/2": "2b27be7d0d105d84922201a60cda828d37723ca9e131d69dc71c55bd43d2e750", + "15/17/2": "6067cb7414bfbd09108bbf1e9c4825f8e1a81296bd24933193a01d91cc76fd82", } func TestGraphHashFollowsVersion(t *testing.T) { From 7c7ed2bd3646151f75123c1cc096b5b3c865d2c9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 14:05:14 +0000 Subject: [PATCH 10/11] Add the namespace package case and document the ambiguous rule Two distributions that share a namespace package, as google-auth and google-cloud-storage under google, had no case. The case shows that an import of a module under the namespace resolves into the one unit that holds it, and that no record names a unit that no import loads. An ambiguous site keeps no method name, so it concerns every target. The docs now say why: the site gets no edge, so the search cannot follow what the candidates call. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- docs/projectgraph.md | 6 ++ .../expect.yaml | 79 +++++++++++++++++++ .../project/Procfile | 1 + .../project/main.py | 5 ++ .../site-packages/google/auth/__init__.py | 0 .../venv/site-packages/google/auth/crypt.py | 5 ++ .../google/cloud/storage/__init__.py | 5 ++ .../google_auth-2.0.0.dist-info/METADATA | 2 + .../google_auth-2.0.0.dist-info/RECORD | 3 + .../METADATA | 2 + .../RECORD | 2 + .../summaries/pypi_google-auth_2.0.0.json | 20 +++++ .../pypi_google-cloud-storage_2.1.0.json | 20 +++++ test/projectgraph/version_test.go | 6 +- 14 files changed, 153 insertions(+), 3 deletions(-) create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/expect.yaml create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/Procfile create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/main.py create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/__init__.py create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/crypt.py create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/cloud/storage/__init__.py create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/METADATA create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/RECORD create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/METADATA create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/RECORD create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-auth_2.0.0.json create mode 100644 test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-cloud-storage_2.1.0.json diff --git a/docs/projectgraph.md b/docs/projectgraph.md index 3d0412ed..f64dd624 100644 --- a/docs/projectgraph.md +++ b/docs/projectgraph.md @@ -168,6 +168,12 @@ reason: `computed-name`, `dynamic-import`, `unknown-receiver`, `trait-receiver` or `no-dependency-source`. The site keeps the method name of the call, so the reach query can tell which targets it concerns. +A site with more than 20 candidates of its method name, as `render` in a +Django project, is `ambiguous` and keeps no method name. The site gets no +edge, so the search cannot follow what the candidates call, and the record +concerns every target. A record with a method name concerns the targets +that a definition of the name reaches. + A call on the result of another call, as `html.NewTokenizer(r).Next()`, needs the type of that result. When the link step does not know it, the site is a record with the method name, so a target of that method is diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/expect.yaml b/test/projectgraph/cases/python/dependency-namespace-two-dists/expect.yaml new file mode 100644 index 00000000..236776c1 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/expect.yaml @@ -0,0 +1,79 @@ +source: safedep:hand +description: >- + google-auth and google-cloud-storage share the namespace package google in one site-packages + directory. Each unit is the package directory that its RECORD gives under the namespace. An import of + google.auth.crypt resolves into google-auth alone, and one of google.cloud.storage into + google-cloud-storage alone, so neither unit misses a module of the other, and no record names a unit + that no import loads. +units: + venv/site-packages/google/auth: pypi:google-auth@2.0.0 + venv/site-packages/google/cloud/storage: pypi:google-cloud-storage@2.1.0 +import_dirs: + - venv/site-packages +roots: + - key: main.py + kind: manifest + rule: procfile +edges: + - from: main.py + to: google//auth//crypt//sign + kind: direct + confidence: high + site: main.py:4 + - from: google//auth//crypt//sign + to: pypi:google-auth@2.0.0//google/auth/crypt.py//sign + kind: package + confidence: high + site: pypi:google-auth@2.0.0//google/auth/crypt.py:4 + - from: main.py + to: google//cloud//storage//upload + kind: direct + confidence: high + site: main.py:5 + - from: google//cloud//storage//upload + to: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py//upload + kind: package + confidence: high + site: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py:4 +unresolved: [] +results: + - target: hashlib//sha256 + status: found + path: + - key: main.py + - key: google//auth//crypt//sign + site: main.py:4 + kind: direct + confidence: high + - key: pypi:google-auth@2.0.0//google/auth/crypt.py//sign + site: pypi:google-auth@2.0.0//google/auth/crypt.py:4 + kind: package + confidence: high + - key: hashlib//sha256 + site: pypi:google-auth@2.0.0//google/auth/crypt.py:5 + kind: direct + confidence: high + call_sites: + - caller: pypi:google-auth@2.0.0//google/auth/crypt.py//sign + site: pypi:google-auth@2.0.0//google/auth/crypt.py:5 + reached: true + - target: socket//create_connection + status: found + path: + - key: main.py + - key: google//cloud//storage//upload + site: main.py:5 + kind: direct + confidence: high + - key: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py//upload + site: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py:4 + kind: package + confidence: high + - key: socket//create_connection + site: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py:5 + kind: direct + confidence: high + call_sites: + - caller: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py//upload + site: pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py:5 + reached: true diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/Procfile b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/Procfile new file mode 100644 index 00000000..629b83ad --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/Procfile @@ -0,0 +1 @@ +web: python main.py diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/main.py b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/main.py new file mode 100644 index 00000000..3d763828 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/main.py @@ -0,0 +1,5 @@ +from google.auth import crypt +from google.cloud import storage + +crypt.sign(b"x") +storage.upload("f") diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/__init__.py b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/crypt.py b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/crypt.py new file mode 100644 index 00000000..b8215033 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/auth/crypt.py @@ -0,0 +1,5 @@ +import hashlib + + +def sign(data): + return hashlib.sha256(data) diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/cloud/storage/__init__.py b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/cloud/storage/__init__.py new file mode 100644 index 00000000..c9c7be19 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google/cloud/storage/__init__.py @@ -0,0 +1,5 @@ +import socket + + +def upload(name): + socket.create_connection(("h", 1)) diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/METADATA b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/METADATA new file mode 100644 index 00000000..2f9fa652 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/METADATA @@ -0,0 +1,2 @@ +Name: google-auth +Version: 2.0.0 diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/RECORD b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/RECORD new file mode 100644 index 00000000..26bfac75 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_auth-2.0.0.dist-info/RECORD @@ -0,0 +1,3 @@ +google/auth/__init__.py,, +google/auth/crypt.py,, +google_auth-2.0.0.dist-info/METADATA,, diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/METADATA b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/METADATA new file mode 100644 index 00000000..18eadc49 --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/METADATA @@ -0,0 +1,2 @@ +Name: google-cloud-storage +Version: 2.1.0 diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/RECORD b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/RECORD new file mode 100644 index 00000000..223381cf --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/project/venv/site-packages/google_cloud_storage-2.1.0.dist-info/RECORD @@ -0,0 +1,2 @@ +google/cloud/storage/__init__.py,, +google_cloud_storage-2.1.0.dist-info/METADATA,, diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-auth_2.0.0.json b/test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-auth_2.0.0.json new file mode 100644 index 00000000..1e7a400f --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-auth_2.0.0.json @@ -0,0 +1,20 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "pypi:google-auth@2.0.0", + "symbols": [ + { + "key": "google//auth//crypt//sign", + "definitions": [ + "pypi:google-auth@2.0.0//google/auth/crypt.py//sign" + ], + "reaches": [ + "hashlib//sha256" + ] + } + ], + "targets": [ + "hashlib//sha256" + ] +} diff --git a/test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-cloud-storage_2.1.0.json b/test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-cloud-storage_2.1.0.json new file mode 100644 index 00000000..6586eb5d --- /dev/null +++ b/test/projectgraph/cases/python/dependency-namespace-two-dists/summaries/pypi_google-cloud-storage_2.1.0.json @@ -0,0 +1,20 @@ +{ + "version": "2", + "index_version": "15", + "link_version": "17", + "unit": "pypi:google-cloud-storage@2.1.0", + "symbols": [ + { + "key": "google//cloud//storage//upload", + "definitions": [ + "pypi:google-cloud-storage@2.1.0//google/cloud/storage/__init__.py//upload" + ], + "reaches": [ + "socket//create_connection" + ] + } + ], + "targets": [ + "socket//create_connection" + ] +} diff --git a/test/projectgraph/version_test.go b/test/projectgraph/version_test.go index 7b02b001..a07e8df3 100644 --- a/test/projectgraph/version_test.go +++ b/test/projectgraph/version_test.go @@ -31,7 +31,7 @@ var graphHashes = map[string]string{ "14/14": "0f3c9c38b39f4de1835f34e8a58bcea85461f37e7aa2d5a2dd484569e670ff48", "14/15": "bbc708aba767ccf554c75a7e06816abda3fc1804ad6d0876ad8ce9575f4c3a88", "14/16": "823b1f82c4aed8ba27d19275505c8d1dee755086be48fcda64d323f55b4c62a5", - "15/17": "7535543fc7ace2a4833b71de9ffac5a1dff1f349d508c2fe9bd9144ee2a2ae96", + "15/17": "d823ce8ba221c534ab062f7e48abfe91d1c90d50b2a2fdb00d41183dc0e9da09", } // resultHashes records the hash of the reach results of every case for @@ -47,7 +47,7 @@ var resultHashes = map[string]string{ "14/14/2": "a832ea709bf3df852814375a56e6329c35e2ff4c49ae34ee49afe042e9502d17", "14/15/3": "77dfd0ea8ab71869ef612d41b8c2e46852d4fb67a49458cb645321692af77ea0", "14/16/3": "13b6eecb686ddec3c2ba470fc9752ea66cff445d5e18e0772a71b15ae7fe1c9c", - "15/17/4": "5b005738c979fc81b1e4617a0d922f5ab4c6e149c5f739271052180bdbbfd4ae", + "15/17/4": "1b32d7c1dbb0a4637853eb7a00b0db91a0a4e1a31f221b37ed0d0654a66ed13e", } // summaryHashes records the hash of the golden summaries of the cases for @@ -64,7 +64,7 @@ var summaryHashes = map[string]string{ "14/14/2": "27424c4e2525450479faba78421959738748811d70b9a93b81e2bbdf1733bf0d", "14/15/2": "5969f3cda15b666cbd902bbc2db953cbdef8c0ce48d5e77446cf243cdd0bb0e7", "14/16/2": "fb678202978212aae932e50a56a0d3dc7f0f794ce908988bf605ce9a6e04bc51", - "15/17/2": "6067cb7414bfbd09108bbf1e9c4825f8e1a81296bd24933193a01d91cc76fd82", + "15/17/2": "00a6bd5871f70459a617079f431939ef7c92e4b3f360013c1dedd017fe1146be", } func TestGraphHashFollowsVersion(t *testing.T) { From 618ad613fe975ad5151f142e160067b3f50e9699 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 14:05:45 +0000 Subject: [PATCH 11/11] Report the distribution graph and the standard library unit The report had no section for steps 5 and 6. This adds a run of the distribution graph on httpie of the corpus and what the standard library case shows. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA --- test/projectgraph/report.md | 39 +++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/test/projectgraph/report.md b/test/projectgraph/report.md index b9cc671c..f2d08968 100644 --- a/test/projectgraph/report.md +++ b/test/projectgraph/report.md @@ -1321,3 +1321,42 @@ TypeScript types. `docs/projectgraph.md` holds the rules. - `PROJECTGRAPH_METHOD_DETAIL` now writes one line per target with its outcome and the sites of its path or of its uncertainties, so the next step can read the cause of each number. + +#### After the distribution graph and the standard library unit + +The commits after step 3 add steps 5 and 6 of the plan. Neither moves the +method measure: its corpus names no standard library unit, and the +distribution graph is a graph of its own. + +`distgraph` builds the distribution graph of a Python project, as +`modgraph` builds the module graph of a Go module. `docs/distgraph.md` +holds the rules. A run of `examples/distgraph` on httpie 3.2.1 of the +corpus, with requests 2.28.2, urllib3 1.26.14, idna 3.6 and PyYAML 5.3.1 +unpacked into one site-packages directory, reads 200 files and gives these +nodes. + +| Node | State | Reason | +| --- | --- | --- | +| httpie | open | `httpie/cli/argparser.py:576` imports `rich.text`, which the directory does not hold | +| requests | open | `requests/certs.py:14` imports `certifi`, which the directory does not hold | +| urllib3 | open | `urllib3/contrib/ntlmpool.py:11` imports `ntlm` outside a guard | +| PyYAML | open | the extension module `_yaml` can import any module | +| idna | imports | | + +The first two reasons are true of the directory: the environment of the run +lacks rich and certifi, and a real install has them. The third is the +rule at work on a `contrib` module whose optional import has no `try` +guard. The fourth is the rule on an extension module, which +`TrustCompiled` turns off for a caller that reads the graph for a path and +not for the absence of one. The edges that the run reads are the ones a +reader expects: httpie imports requests and urllib3, requests imports idna +and urllib3, and urllib3 imports idna in `contrib/pyopenssl.py`. + +The unit `python:std@` of the `Lib` directory of CPython gives +the standard library to the project graph. The case +`python/dependency-stdlib` shows `Thread.start` of the unit reaching the +`run` of a project class with a `cha` edge, through the external key of +`Thread` that the class names as its base, and a path into `subprocess` +that goes on to `os.execvp`. The reach result still reports the shorter +path through the reference edge of the override rule, because the search +returns a shortest path.