Repository navigation
feat(projectgraph): link Python dependency code, read its type hints, and add the distribution graph - #55
Open
abhisek wants to merge 12 commits into
Open
feat(projectgraph): link Python dependency code, read its type hints, and add the distribution graph#55abhisek wants to merge 12 commits into
abhisek wants to merge 12 commits into
Conversation
The Python plan starts with a number. The link step reads no type hint of Python, and no Python project is in the method corpus, so nobody could say how often a Python method target ends uncertain. receivers_test.go has 18 Python rows against a pypi unit under testdata/receivers/python. Two rows give their answer today. The other 16 name the type-hint step of the plan and their answer before it. A call on a variable that holds the result of an external call gets the false key dep//new_thing//do and no record, so the method of the real type is no-such-target. The method corpus has 10 Python projects with 14 pinned dependencies. OSV lists no symbols for a PyPI advisory, so scripts/pypi_advisories.py holds the symbols of 72 advisories by hand, as the advisory text names them, and writes them with the ranges from OSV. fetch_method_corpus.sh unpacks a dependency from its wheel, so the unit has the layout of site-packages, and a project from its sdist. methods_test.go reads the pypi ecosystem and takes PROJECTGRAPH_METHOD_LANGUAGES to index one language. report.md has the baseline: 5 of 107 targets found, none not-found and none no-such-target. Every search meets an uncertainty that concerns every target. The causes are the bundled JavaScript that the index truncates, the decorator-wrap flag on every framework decorator, a truncated test file, and the unknown receiver of 27 of 63 methods. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
…s that the program runs The Python baseline of the method measure had every target uncertain. Three causes hid the receiver forms that the type-hint step must fix, and none of them was a call that the graph could not see. A Python project ships the JavaScript of its web pages, and the index step truncates a minified file of it. Every search counted the file as a budget. The index step now reports the language of each file, the link step keeps it on the module node, and Stats.Languages holds the languages of the files with a root other than a test root. Graph.Runs reads them, with the languages that the ecosystem of a unit installs. The reach query and the summary leave out the budget of a file that the program does not run. A truncated test file counts only for a search that includes the tests. The index flagged every decorator that was not a bare name as decorator-wrap, so each click or Flask project made every target uncertain. A decorator that is a name, a dotted name or a call of one is a call of the scope that defines the function, which the link step reads. The flag stays for a decorator that the graph cannot name, as @handlers[name]. A call of the value of a call, as click.option(...)(f), is no computed-name record of its own, unless the inner call reflects. An absolute Python import was first party when a file of the name was anywhere in the project, so import types in cryptography and import collections in Django were records that concerned every target. The first segment must now be a module under an import root or in the directory of the importer, as the import system reads it. Five cases cover the rules. flask-route now reads a dead call as not-found. pycg-micro keeps 0.954 precision and 0.708 recall, with the unresolved rate down from 0.340 to 0.312. The Python budget of the measure falls from 47 to 5 targets, and the run of every language gives the same numbers as the run of Python alone. report.md has the table. fileindex.Version 15, projectgraph.Version 17, reach.Version 4. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
…rules commit The rules commit went out with three tests red: the module-getattr line of the Python flag fixture moved with the new decorator, the index goldens had no hash row for fileindex.Version 15, and the golden graph of a project with test roots alone still listed python as a program language. go test ./... passes with exit 0 now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
…efinitions external keys A pypi unit knew only the files that the index step read. The link step named its import name from the package name, so PyYAML gave no yaml, an extension module was an external edge that hid its code, every test file of the package was in the graph, and a target that only the code of the package called matched nothing. modules.PythonDistributions reads the .dist-info directories at the root of a unit, as a site-packages directory or an unpacked wheel holds them: the name and the version of METADATA, the import names of top_level.txt or RECORD, the files of RECORD and its extension modules. The link step matches the one of the unit by the normalized name and gives the Python resolver its import names and its extension modules. An import of an extension module is a record with the reason no-dependency-source. The Python profile gets the symbol rule that Go and Rust have, so each definition of a unit carries the key that an importer of its module writes. The index step leaves out the test files and the wheel metadata of a pypi unit, as it leaves out the test files of a Go unit. examples/projectgraph takes -site-packages dir and adds each distribution of the directory as a unit, by the roots that its RECORD names, so a namespace package that two distributions share and a single-file module both get their units. The measure and the receiver rows say found/low for a path with a hop of low confidence. The external keys move 54 of 107 Python targets from uncertain to found/low, and unknown-receiver falls from 27 to 4 of 63 methods. report.md has the table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
SafeDep Report Summary
No dependency changes detected. Nothing to scan. This report is generated by SafeDep GitHub App |
|
Important Review skippedToo many files! This PR contains 249 files, which is 149 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9494392f0b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #55 +/- ##
==========================================
- Coverage 77.23% 77.13% -0.10%
==========================================
Files 174 180 +6
Lines 17129 18135 +1006
==========================================
+ Hits 13230 13989 +759
- Misses 3115 3301 +186
- Partials 784 845 +61 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…elper
The record of a call of the value of a call went away when the inner
callee was no reflection. operator.methodcaller("system", cmd)(os) calls
a method by a name in a string, as getattr(os, name)() does, and no edge
names that method. So os.system read as no-such-target. methodcaller and
attrgetter of the operator module count as reflective callees now, and
the outer call keeps its computed-name record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
The link step did not know the class of a Python receiver that a type hint names. A call on a typed parameter, on a local that holds the result of a function with a return hint, or on a field of a class had a record of an unknown receiver and low name edges. The receiver rows of Python marked 16 forms as gaps. The index step records the hints: the class of a parameter, with Optional, Union and `| None` unwrapped, the return type and the tuple positions of a function, an annotated local, a name that holds a call result, a typed name of the top level of a module, and the fields of a class from its annotations and from the attributes that its methods set. A call on a typed name gets the key of the name in its scope as its target, as the Go index does, so the link step reads the type. A call that the call graph bound to a method of a class of the file keeps that target. A result of a reflective call, as import_module, has no type. The link step trusts the hints as it trusts TypeScript types. A hint of a dependency class gives the external key of the method, the class hierarchy analysis adds the overrides in the project and in the unit, and an abstract method with no override is a record with the reason interface-receiver. The walk now sets the class of an instance of an external class, so the analysis runs for it. All 16 Python receiver rows give their wanted answer. Six cases cover the forms. The external-base-attribute case gets a member edge in place of a name edge. The method measure adds a detail file per target. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
The method report had no section for step 3. This adds the numbers of the run after the hints, the two targets that move, and why the rest do not: the paths of the corpus cross dependency code with no hints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
A caller of the project graph had no way to tell which installed Python distribution imports which, as modgraph tells it for the modules of a Go build. Without the edges, a caller cannot trust that no code imports a distribution, and a vulnerable distribution hides behind a false unused. distgraph builds the graph from the code on the disk. It reads the .dist-info directories of each site-packages directory and the import statements of the Python files of the project and of each distribution that an import reaches. It runs no Python. The graph never misses an edge that a run has: a distribution with an extension module, a computed import, an import that nothing holds, a file that does not parse, or a file past the budget is open, and a path that meets it reaches each distribution. An import in a try statement that catches an ImportError opens nothing. A distribution with no Python file gets its Requires-Dist lines as evidence of edges. The distribution reader gains the Requires-Dist names and the test-path rule, which the index step shares. examples/distgraph prints the graph, and docs/distgraph.md holds the rules. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
A call of the standard library on self, as self.run in Thread.start, reached the override of the project only through the reference edge of the override rule, because the graph had no code of the library. A path into a module of the library ended at its external key. The Lib directory of CPython is now the unit python:std@<version>, as GOROOT/src is go:std. The index step leaves out its test suite, the IDLE, the turtle demos and lib2to3. The link step knows that the python ecosystem installs Python, so a truncated file of the unit counts only for a Python program. A class of a unit gives its external key to the class hierarchy analysis, because a class of the project names that key as its base, so Thread.run reaches Worker.run with a cha edge. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
Two distributions that share a namespace package, as google-auth and google-cloud-storage under google, had no case. The case shows that an import of a module under the namespace resolves into the one unit that holds it, and that no record names a unit that no import loads. An ambiguous site keeps no method name, so it concerns every target. The docs now say why: the site gets no edge, so the search cannot follow what the candidates call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
The report had no section for steps 5 and 6. This adds a run of the distribution graph on httpie of the corpus and what the standard library case shows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149TuYSppnJAa4JVsXXqHDA
abhisek
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The whole-project call graph links Python dependency code, but Python had none of the second layer that #51 to #53 gave Go and Rust: a measure of how often a method target of a dependency ends uncertain, external keys for the definitions of a unit, source rules for the files of a unit, declared types for the receivers, and a graph of which dependency imports which. This pull request is the whole Python plan on the code side: the measure, the pypi unit, the type hints, the distribution graph and the standard library unit.
Step 1: the measure
receivers_test.gohas 18 Python rows against a pypi unit undertestdata/receivers/python.scripts/pypi_advisories.pyholds the symbols of 72 advisories by hand and pulls the ranges from OSV.fetch_method_corpus.shunpacks a dependency from its wheel and a project from its sdist.PROJECTGRAPH_METHOD_DETAILwrites one line per target with its outcome and sites.Step 2: four rules that the baseline forced, and the pypi unit
Graph.Runsdecides, fromNode.LanguageandStats.Languages.@handlers[name]. A call of the value of a call, asclick.option("-v")(f), is nocomputed-namerecord, unless the inner call reflects.modules.PythonDistributionsreads the.dist-infoat the root of a unit: the import names, the files ofRECORD, its extension modules and itsRequires-Distlines. An import of an extension module is a record with the reasonno-dependency-source.symbolrule, so a definition of a unit carries the key that an importer writes, asrequests//api//get.examples/projectgraphtakes-site-packages <dir>.Step 3: the type hints
Optional,Unionand| Noneunwrapped, the return type and the tuple positions of a function, an annotated local, a name that holds a call result, a typed name of the top level of a module, and the fields of a class from its annotations and from the attributes that its methods set.import_module, has no type.interface-receiver.Step 5: the distribution graph
distgraphbuilds the graph of which installed distribution imports which, from the.dist-infodirectories of each site-packages directory and the import statements of the Python files, with no Python. It follows the rule ofmodgraph: it never misses an edge that a run has. A distribution with an extension module, a computed import, an import that nothing holds, a file that does not parse, or a file past the budget is open. An import in atrystatement that catches anImportErroropens nothing.examples/distgraphprints the graph, anddocs/distgraph.mdholds the rules.Step 6: the standard library unit
The
Libdirectory of CPython is the unitpython:std@<version>, asGOROOT/srcisgo:std. The index step leaves out its test suite, the IDLE, the turtle demos andlib2to3. A class of a unit gives its external key to the class hierarchy analysis, soThread.runin the unit reaches therunof a project class with achaedge.Numbers
The corpus moves little in step 3 because its paths cross requests, Django 4.2 and urllib3 1.26, which have no type hints on the receivers that the paths cross. The receiver rows and the cases prove the mechanism. One false
found/lowpath through anameedge in Django becomesambiguous. pycg-micro keeps 0.954 precision and 0.708 recall. govulncheck stays at 9 of 9 stacks.report.mdhas the tables and the causes of each step.Versions:
fileindex.Version15,projectgraph.Version17,reach.Version4.Verification
go test ./...,go vet ./...,go vet -tags projectgraph ./test/projectgraph/,golangci-lint runandscripts/check_projectgraph_gates.shpass.TestBuildHoldsEachEdgeOfGoListofmodgraphfails in the development container onmaintoo, because the module cache there lacks the source of two modules.Generated by Claude Code