Skip to content

Commit d908720

Browse files
PhysShellPhysShell
authored
Merge pull request #19 from PhysShell/claude/zen-pasteur-76hfs1
Corpus miner: run own-check over public C# repos + aggregate a report
2 parents 0facc5a + d66b5ff commit d908720

7 files changed

Lines changed: 464 additions & 3 deletions

File tree

‎.github/workflows/mine.yml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: mine (corpus)
2+
3+
# On-demand corpus mining: clone one public C# repo and run the Own.NET leak
4+
# check over it, uploading a structured report. Evaluation tooling for the
5+
# analyser — see docs/notes/mining.md. One repo per run (be a good citizen).
6+
#
7+
# Trigger from the Actions tab ("Run workflow") or the API. Inputs are passed to
8+
# the miner via env (never interpolated into the shell) to avoid script injection.
9+
10+
on:
11+
workflow_dispatch:
12+
inputs:
13+
repo:
14+
description: "Target: owner/repo (e.g. DapperLib/Dapper) or a git URL"
15+
required: true
16+
ref:
17+
description: "Branch / tag / sha to mine (optional, default: repo HEAD)"
18+
required: false
19+
default: ""
20+
paths:
21+
description: "Subdir of the target to scan (optional, default: whole repo)"
22+
required: false
23+
default: ""
24+
25+
permissions:
26+
contents: read
27+
28+
jobs:
29+
mine:
30+
name: mine ${{ inputs.repo }}
31+
runs-on: ubuntu-latest
32+
steps:
33+
- uses: actions/checkout@v4
34+
- uses: actions/setup-python@v5
35+
with:
36+
python-version: "3.13"
37+
- uses: actions/setup-dotnet@v4
38+
with:
39+
dotnet-version: "8.0.x"
40+
- name: Mine the target
41+
env:
42+
REPO: ${{ inputs.repo }}
43+
REF: ${{ inputs.ref }}
44+
PATHS: ${{ inputs.paths }}
45+
run: |
46+
args=()
47+
[[ -n "$REF" ]] && args+=(--ref "$REF")
48+
[[ -n "$PATHS" ]] && args+=(--paths "$PATHS")
49+
scripts/mine.sh "${args[@]}" "$REPO"
50+
- name: Publish the report to the run summary
51+
if: always()
52+
run: |
53+
report=$(find corpus/mined -name report.md -type f 2>/dev/null | head -1 || true)
54+
if [[ -n "$report" ]]; then
55+
cat "$report" >> "$GITHUB_STEP_SUMMARY"
56+
else
57+
echo "no report produced (see the Mine step log)" >> "$GITHUB_STEP_SUMMARY"
58+
fi
59+
- name: Upload the report
60+
if: always()
61+
uses: actions/upload-artifact@v4
62+
with:
63+
name: mine-report
64+
path: |
65+
corpus/mined/*/report.md
66+
corpus/mined/*/report.json
67+
corpus/mined/*/findings.txt
68+
corpus/mined/*/extract.log
69+
if-no-files-found: warn

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,8 @@ __pycache__/
77
# .NET build output (golden_arraypool demo)
88
bin/
99
obj/
10+
11+
# Corpus mining: cloned third-party source + raw reports (scripts/mine.sh).
12+
# Never commit other projects' code; promote interesting findings into
13+
# corpus/real-world/ as minimal reduced cases instead.
14+
corpus/mined/

‎corpus/targets.txt‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Seed targets for the miner — scripts/mine.sh / .github/workflows/mine.yml.
2+
#
3+
# Well-known, permissively-licensed C# repos with IDisposable-dense code
4+
# (ADO.NET / streams / readers), good for stress-testing the leak detector on
5+
# unseen shapes. Mine ONE at a time — shallow, read-only; this is a spot-check,
6+
# not a crawler.
7+
#
8+
# One `owner/repo` per line; everything after `#` is a note. Verify the license
9+
# before doing anything beyond local analysis (e.g. reporting bugs upstream).
10+
DapperLib/Dapper # micro-ORM, ADO.NET (SqlConnection/IDataReader) — closest to GTM's domain
11+
JoshClose/CsvHelper # TextReader/TextWriter dense
12+
JamesNK/Newtonsoft.Json # JsonReader/Writer (IDisposable), streams
13+
restsharp/RestSharp # HttpClient / streams

‎docs/notes/mining.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Corpus mining — stress-testing the analyser on real repos
2+
3+
A spot-check harness: take a public C# repo, run the Own.NET leak check over it,
4+
and aggregate the result into a structured report. The goal is **evaluating the
5+
analyser**, not crawling GitHub — one repo at a time, shallow and read-only.
6+
7+
## Why it's cheap
8+
9+
The Roslyn extractor (P-014 Tier A) builds a best-effort `SemanticModel` from the
10+
runtime's trusted-platform assemblies and is error-tolerant: it reads symbols
11+
without a `dotnet restore`/build of the target, and external (NuGet) types it
12+
can't resolve become an honest `OWN050` "unchecked" marker rather than a guess.
13+
So mining needs **no per-repo build setup** — just point it at the `.cs`.
14+
15+
## Run it
16+
17+
In CI (no local .NET needed) — Actions tab → **mine (corpus)** → *Run workflow*,
18+
or via the API; the report lands in the run summary and as an artifact:
19+
20+
```text
21+
inputs: repo = DapperLib/Dapper ref = (optional) paths = (optional subdir)
22+
```
23+
24+
Locally (needs `dotnet`, `git`, Python 3.11+):
25+
26+
```sh
27+
scripts/mine.sh DapperLib/Dapper # whole repo
28+
scripts/mine.sh --paths src JoshClose/CsvHelper # focus a subdir
29+
```
30+
31+
Output → `corpus/mined/<slug>/` (gitignored): `findings.txt`, `extract.log`,
32+
`report.md`, `report.json`. Seed targets live in `corpus/targets.txt`.
33+
34+
## What the report says — and how to read it
35+
36+
`scripts/mine_report.py` aggregates the findings into: counts by OWN code, the
37+
error/advisory split, resource kinds, the noisiest files, and a triage list of
38+
the error-severity findings.
39+
40+
- **A clean run is a signal, not a dud.** On well-disciplined code (lots of
41+
`using`) zero findings is the *precision* result we want to see.
42+
- **A pile of `OWN001`s** is either real leaks (reduce one to a minimal `.cs` and
43+
add it to `corpus/real-world/` as a regression) **or** a false-positive pattern
44+
— the cue to harden the extractor (a new exemption, better escape analysis, a
45+
new lowering).
46+
- **A high `OWN050` count** is a *coverage* gap: the declaring types are
47+
unresolved external references. Not wrong, just not analysed.
48+
49+
## The loop
50+
51+
`mine → triage → (a) regressions in corpus/, (b) fixes in the extractor/exemptions`
52+
→ repeat. The methodology matches the GTM triage (real leaks kept, dispose-optional
53+
FPs exempted → 100% precision); mining stresses that on shapes we haven't seen.
54+
55+
## Honest gaps (v1)
56+
57+
- **No coverage/skip rate yet.** A method the extractor can't model (a `for`/`do`
58+
loop, `try`, …) is silently absent from the facts, so the report can't say
59+
"analysed N of M methods". Adding a `--stats` summary to the extractor is the
60+
planned next step.
61+
- **One target, by hand.** Auto-discovery (GitHub code search for IDisposable
62+
patterns) is deliberately out of scope — keep it a deliberate spot-check.
63+
- **Reporting upstream is a separate, manual step.** If a finding is a real bug
64+
worth disclosing, do it deliberately (and check the license); the miner never
65+
contacts the target project.

‎scripts/mine.sh‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
#!/usr/bin/env bash
2+
#
3+
# mine.sh — clone a public C# repo (shallow) and run own-check over it, writing a
4+
# structured Markdown report. Evaluation tooling for the analyser itself: see
5+
# docs/notes/mining.md. Mine ONE repo at a time — shallow, read-only; this is a
6+
# spot-check, not a crawler. Be a good citizen.
7+
#
8+
# Usage:
9+
# scripts/mine.sh [--ref <branch|tag|sha>] [--paths <subdir>] [--format human]
10+
# [--out <dir>] [--keep-src] <owner/repo | git-url>
11+
#
12+
# Output goes to corpus/mined/<slug>/ (gitignored): findings.txt, extract.log,
13+
# report.md, report.json (and src/ with --keep-src).
14+
#
15+
# Requires: git, a .NET SDK (dotnet), Python 3.11+.
16+
17+
set -euo pipefail
18+
19+
ref=""
20+
subpaths=""
21+
format="human"
22+
outdir=""
23+
keep_src=0
24+
target=""
25+
26+
while [[ $# -gt 0 ]]; do
27+
case "$1" in
28+
--ref) [[ $# -ge 2 ]] || { echo "mine: --ref needs a value" >&2; exit 2; }; ref="$2"; shift 2 ;;
29+
--paths) [[ $# -ge 2 ]] || { echo "mine: --paths needs a value" >&2; exit 2; }; subpaths="$2"; shift 2 ;;
30+
--format) [[ $# -ge 2 ]] || { echo "mine: --format needs a value" >&2; exit 2; }; format="$2"; shift 2 ;;
31+
--out) [[ $# -ge 2 ]] || { echo "mine: --out needs a value" >&2; exit 2; }; outdir="$2"; shift 2 ;;
32+
--keep-src) keep_src=1; shift ;;
33+
-h|--help) sed -n '2,19p' "$0"; exit 0 ;;
34+
--) shift; [[ $# -gt 0 ]] && { target="$1"; shift; } ;;
35+
*) target="$1"; shift ;;
36+
esac
37+
done
38+
39+
[[ -n "$target" ]] || { echo "mine: a target (owner/repo or git URL) is required" >&2; exit 2; }
40+
command -v git >/dev/null || { echo "mine: git not found" >&2; exit 2; }
41+
command -v python >/dev/null || { echo "mine: python not found" >&2; exit 2; }
42+
if ! command -v dotnet >/dev/null; then
43+
echo "mine: a .NET SDK (dotnet) is required to run the extractor." >&2
44+
echo "mine: run this in CI via .github/workflows/mine.yml, or install the SDK." >&2
45+
exit 2
46+
fi
47+
48+
root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
49+
50+
# owner/repo -> https URL; pass full git URLs through untouched.
51+
case "$target" in
52+
http://*|https://*|git@*) url="$target" ;;
53+
*) url="https://github.com/${target}.git" ;;
54+
esac
55+
slug="$(printf '%s' "$target" | sed -E 's#^https?://[^/]+/##; s#^git@[^:]+:##; s#\.git$##; s#[^A-Za-z0-9._-]#_#g')"
56+
[[ -n "$outdir" ]] || outdir="$root/corpus/mined/$slug"
57+
58+
mkdir -p "$outdir"
59+
src="$outdir/src"
60+
rm -rf "$src"
61+
62+
echo "mine: $target -> $outdir" >&2
63+
clone_args=(--quiet --depth 1)
64+
[[ -n "$ref" ]] && clone_args+=(--branch "$ref")
65+
git clone "${clone_args[@]}" "$url" "$src"
66+
commit="$(git -C "$src" rev-parse HEAD)"
67+
68+
scan="$src"
69+
[[ -n "$subpaths" ]] && scan="$src/$subpaths"
70+
[[ -e "$scan" ]] || { echo "mine: scan path '$scan' does not exist in the repo" >&2; exit 2; }
71+
72+
echo "mine: scanning $scan (commit $commit)" >&2
73+
# own-check sends host-parseable findings to stdout and dotnet/build chatter to
74+
# stderr; keep them apart. Without --fail-on-finding it exits 0 even with leaks;
75+
# rc>=2 is a hard error (bad facts) — note it but still report what we captured.
76+
set +e
77+
"$root/scripts/own-check.sh" --root "$root" --format "$format" -- "$scan" \
78+
>"$outdir/findings.txt" 2>"$outdir/extract.log"
79+
rc=$?
80+
set -e
81+
[[ "$rc" -ge 2 ]] && echo "mine: own-check hard error (rc=$rc); see $outdir/extract.log" >&2
82+
83+
python "$root/scripts/mine_report.py" "$outdir/findings.txt" \
84+
--repo "$target" --commit "$commit" --json "$outdir/report.json" \
85+
>"$outdir/report.md"
86+
87+
[[ "$keep_src" -eq 1 ]] || rm -rf "$src"
88+
89+
echo "mine: done -> $outdir/report.md" >&2
90+
grep -E '^- findings:' "$outdir/report.md" || true

0 commit comments

Comments
 (0)