Skip to content

Commit 79fe2aa

Browse files
feat(inline-scripts): add cache layout, meta.json sidecar, env-usability guard (PEP 723 PR 2/3)
Helpers that resolve the on-disk cache layout for PEP 723 inline-script envs (under <globalStorageUri>/script-envs-v1/<hash>/), read/write the .meta.json sidecar that lives at the root of every cached env, and verify on cache hit that the base interpreter the env was built against still exists on disk. Second foundation PR; pure utility, no behavior change on its own. What is in: - src/common/inlineScriptCacheLayout.ts - Path helpers: getScriptEnvCacheRoot, getScriptEnvDir, getMetaJsonPath. All take globalStorageUri: Uri rather than ExtensionContext so they are easy to test. - InlineScriptEnvMeta interface with schemaVersion: 1. Minimal shape: only fields with a concrete consumer in a planned PR (scriptFsPath for cache cleanup, lastUsedAt for TTL, optional requiresPython for cache-hit re-verify). Extras dropped on read per the v1 evolution policy. - readMetaJson(envDir) -- never throws. Returns undefined (with a traceWarn that includes err.code) for: missing file, non-regular file, oversize file (>1 MiB defensive cap), malformed JSON, invalid shape, unknown schemaVersion, non-canonical ISO 8601 timestamps. Strict ISO check uses round-trip equality. - writeMetaJson(envDir, meta) -- atomic temp-file + native fs.rename (NOT fs-extra move(), which does remove+rename and has both a no-sidecar window and a crash-loses-data failure mode). - CacheEntrySummary + selectStaleEntries(entries, now, ttlMs) -- pure selector for Q7s TTL eviction path. Zero I/O. Entries with undefined lastUsedAt are never TTLed (deliberate: do not delete what we do not understand). - verifyEnvUsable(envDir): Promise<boolean> -- cache-hit guard that confirms the base interpreter the cached env was built against still exists on disk. The cache-key path hash detects "user switched to a different Python" because the path changes; it does NOT detect "user uninstalled the Python at the cached path" (the cache directory and .meta.json are unchanged, only the file at the launcher target is gone). Common triggers: pyenv/uv python uninstall, brew/apt removal, Add/Remove Programs. * POSIX: stat <envDir>/bin/python. fs.stat follows symlinks so a dead symlink to a removed base throws ENOENT. * Windows: <envDir>/Scripts/python.exe is a COPY of the base (not a symlink), so checking it does not help. Read pyvenv.cfg, parse `home = <dir>`, stat <home>/python.exe. * Never throws. Returns false + traceWarn(message) on any failure mode (missing launcher, missing/malformed pyvenv.cfg, non-regular file at the launcher path, EACCES etc.). Mirrors readMetaJson's failure policy. Internal helpers `statRegularFile` (POSIX/Windows shared stat+isFile+error-tagging) and `parsePyvenvHome` (CRLF + extra whitespace tolerated). - META_SCHEMA_VERSION evolution policy documented inline. - src/test/common/inlineScriptCacheLayout.unit.test.ts -- 44 unit tests (real-fs, mirroring readMetaJson conventions): * path-helper shape * round-trip writeMetaJson/readMetaJson, concurrent writers, and all rejection paths (ENOENT-specific warn, malformed JSON, unknown schemaVersion, non-canonical ISO, size cap, etc.) * selectStaleEntries boundaries (TTL=0, exact-equal age, undefined lastUsedAt, future timestamps) * verifyEnvUsable POSIX branch (isWindows stubbed to false): bin/python exists / missing / is-a-directory; symlink-alive and dead-symlink cases gated on process.platform !== 'win32' * verifyEnvUsable Windows branch (isWindows stubbed to true): home points to existing / removed python.exe, missing pyvenv.cfg, no home= line, empty home value, whitespace and CRLF tolerance. Design context Implements Q2 (disk location), Q3 (env folder contents + .meta.json shape), Q4 step 4 (cache-hit interpreter-existence guard), and the pure selector half of Q7 (TTL eviction) from pep723_design_questions.md. The disk walk and deletion calls live in a later PR. The verifyEnvUsable guard was added in response to reviewer feedback on the design doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent b16150a commit 79fe2aa

2 files changed

Lines changed: 740 additions & 0 deletions

File tree

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT License.
3+
4+
import * as crypto from 'crypto';
5+
import * as fsapi from 'fs-extra';
6+
import * as path from 'path';
7+
import { Uri } from 'vscode';
8+
import { traceWarn } from './logging';
9+
import { isWindows } from './utils/platformUtils';
10+
11+
/**
12+
* Versioned name of the cache root under the extension's `globalStorageUri`.
13+
*
14+
* Bump the `-v1` suffix together with {@link META_SCHEMA_VERSION} on any
15+
* incompatible on-disk change, so old envs sit unread and TTL out naturally
16+
* instead of being migrated in place.
17+
*/
18+
export const INLINE_SCRIPT_CACHE_DIR_NAME = 'script-envs-v1';
19+
20+
export const META_JSON_FILENAME = '.meta.json';
21+
22+
/**
23+
* Schema version embedded in every {@link InlineScriptEnvMeta}.
24+
*/
25+
export const META_SCHEMA_VERSION = 1 as const;
26+
export interface InlineScriptEnvMeta {
27+
readonly schemaVersion: typeof META_SCHEMA_VERSION;
28+
readonly scriptFsPath: string;
29+
readonly lastUsedAt: string;
30+
readonly requiresPython?: string;
31+
}
32+
33+
export function getScriptEnvCacheRoot(globalStorageUri: Uri): Uri {
34+
return Uri.joinPath(globalStorageUri, INLINE_SCRIPT_CACHE_DIR_NAME);
35+
}
36+
37+
export function getScriptEnvDir(globalStorageUri: Uri, cacheKey: string): Uri {
38+
return Uri.joinPath(getScriptEnvCacheRoot(globalStorageUri), cacheKey);
39+
}
40+
41+
export function getMetaJsonPath(envDir: Uri): Uri {
42+
return Uri.joinPath(envDir, META_JSON_FILENAME);
43+
}
44+
45+
const MAX_META_JSON_BYTES = 1024 * 1024;
46+
47+
export async function readMetaJson(envDir: Uri): Promise<InlineScriptEnvMeta | undefined> {
48+
const metaPath = getMetaJsonPath(envDir).fsPath;
49+
50+
try {
51+
const stat = await fsapi.stat(metaPath);
52+
if (!stat.isFile()) {
53+
traceWarn(`inline-script meta: not a regular file at ${metaPath}`);
54+
return undefined;
55+
}
56+
if (stat.size > MAX_META_JSON_BYTES) {
57+
traceWarn(`inline-script meta: refusing to read ${metaPath} (${stat.size} bytes > cap)`);
58+
return undefined;
59+
}
60+
} catch (err) {
61+
if (isFileNotFoundError(err)) {
62+
traceWarn(`inline-script meta: not found at ${metaPath}`);
63+
} else {
64+
const code = (err as NodeJS.ErrnoException | undefined)?.code ?? 'unknown';
65+
traceWarn(`inline-script meta: failed to stat ${metaPath} (code=${code}):`, err);
66+
}
67+
return undefined;
68+
}
69+
70+
let raw: string;
71+
try {
72+
raw = await fsapi.readFile(metaPath, 'utf8');
73+
} catch (err) {
74+
const code = (err as NodeJS.ErrnoException | undefined)?.code ?? 'unknown';
75+
traceWarn(`inline-script meta: failed to read ${metaPath} (code=${code}):`, err);
76+
return undefined;
77+
}
78+
79+
let parsed: unknown;
80+
try {
81+
parsed = JSON.parse(raw);
82+
} catch (err) {
83+
traceWarn(`inline-script meta: malformed JSON in ${metaPath}:`, err);
84+
return undefined;
85+
}
86+
87+
const validated = validateMeta(parsed);
88+
if (!validated) {
89+
traceWarn(`inline-script meta: invalid shape in ${metaPath}`);
90+
return undefined;
91+
}
92+
return validated;
93+
}
94+
95+
/**
96+
* Atomically write the `.meta.json` sidecar via temp-file + rename.
97+
*/
98+
export async function writeMetaJson(envDir: Uri, meta: InlineScriptEnvMeta): Promise<void> {
99+
await fsapi.ensureDir(envDir.fsPath);
100+
const finalPath = getMetaJsonPath(envDir).fsPath;
101+
const tmpSuffix = crypto.randomBytes(6).toString('hex');
102+
const tmpPath = `${finalPath}.tmp-${tmpSuffix}`;
103+
const payload = JSON.stringify(meta, undefined, 2);
104+
try {
105+
await fsapi.writeFile(tmpPath, payload, 'utf8');
106+
await fsapi.rename(tmpPath, finalPath);
107+
} catch (err) {
108+
await fsapi.remove(tmpPath).catch(() => undefined);
109+
throw err;
110+
}
111+
}
112+
113+
/**
114+
* Snapshot of one cached entry, populated by the (separate, I/O-doing) disk
115+
* walk.
116+
*/
117+
export interface CacheEntrySummary {
118+
readonly envDirPath: string;
119+
readonly lastUsedAt: Date | undefined;
120+
}
121+
122+
/**
123+
* Pure selector: returns the env-dir paths whose age exceeds `ttlMs`.
124+
*/
125+
export function selectStaleEntries(entries: ReadonlyArray<CacheEntrySummary>, now: Date, ttlMs: number): string[] {
126+
const stale: string[] = [];
127+
const nowMs = now.getTime();
128+
for (const entry of entries) {
129+
if (entry.lastUsedAt === undefined) {
130+
continue;
131+
}
132+
const ageMs = nowMs - entry.lastUsedAt.getTime();
133+
if (ageMs > ttlMs) {
134+
stale.push(entry.envDirPath);
135+
}
136+
}
137+
return stale;
138+
}
139+
140+
/**
141+
* Verify that a cached env's base interpreter still exists on disk.
142+
*/
143+
export async function verifyEnvUsable(envDir: Uri): Promise<boolean> {
144+
if (isWindows()) {
145+
return verifyWindowsBaseInterpreter(envDir);
146+
}
147+
return verifyPosixBaseInterpreter(envDir);
148+
}
149+
150+
async function verifyPosixBaseInterpreter(envDir: Uri): Promise<boolean> {
151+
const launcherPath = Uri.joinPath(envDir, 'bin', 'python').fsPath;
152+
return statRegularFile(launcherPath, 'base interpreter');
153+
}
154+
155+
async function verifyWindowsBaseInterpreter(envDir: Uri): Promise<boolean> {
156+
const pyvenvPath = Uri.joinPath(envDir, 'pyvenv.cfg').fsPath;
157+
let raw: string;
158+
try {
159+
raw = await fsapi.readFile(pyvenvPath, 'utf8');
160+
} catch (err) {
161+
if (isFileNotFoundError(err)) {
162+
traceWarn(`inline-script env: missing pyvenv.cfg at ${pyvenvPath}`);
163+
} else {
164+
const code = (err as NodeJS.ErrnoException | undefined)?.code ?? 'unknown';
165+
traceWarn(`inline-script env: failed to read ${pyvenvPath} (code=${code}):`, err);
166+
}
167+
return false;
168+
}
169+
const home = parsePyvenvHome(raw);
170+
if (home === undefined) {
171+
traceWarn(`inline-script env: no 'home =' line in ${pyvenvPath}`);
172+
return false;
173+
}
174+
const launcherPath = path.join(home, 'python.exe');
175+
return statRegularFile(launcherPath, 'base interpreter');
176+
}
177+
178+
async function statRegularFile(filePath: string, label: string): Promise<boolean> {
179+
try {
180+
const stat = await fsapi.stat(filePath);
181+
if (!stat.isFile()) {
182+
traceWarn(`inline-script env: ${label} is not a regular file at ${filePath}`);
183+
return false;
184+
}
185+
return true;
186+
} catch (err) {
187+
if (isFileNotFoundError(err)) {
188+
traceWarn(`inline-script env: ${label} missing at ${filePath}`);
189+
} else {
190+
const code = (err as NodeJS.ErrnoException | undefined)?.code ?? 'unknown';
191+
traceWarn(`inline-script env: failed to stat ${filePath} (code=${code}):`, err);
192+
}
193+
return false;
194+
}
195+
}
196+
197+
function parsePyvenvHome(raw: string): string | undefined {
198+
for (const line of raw.split(/\r?\n/)) {
199+
const m = line.match(/^\s*home\s*=\s*(.+?)\s*$/);
200+
if (m) {
201+
return m[1];
202+
}
203+
}
204+
return undefined;
205+
}
206+
207+
function isFileNotFoundError(err: unknown): boolean {
208+
return typeof err === 'object' && err !== null && 'code' in err && (err as NodeJS.ErrnoException).code === 'ENOENT';
209+
}
210+
211+
function validateMeta(value: unknown): InlineScriptEnvMeta | undefined {
212+
if (typeof value !== 'object' || value === null || Array.isArray(value)) {
213+
return undefined;
214+
}
215+
const obj = value as Record<string, unknown>;
216+
if (obj.schemaVersion !== META_SCHEMA_VERSION) {
217+
return undefined;
218+
}
219+
if (typeof obj.scriptFsPath !== 'string' || obj.scriptFsPath.length === 0) {
220+
return undefined;
221+
}
222+
if (!isCanonicalIsoTimestamp(obj.lastUsedAt)) {
223+
return undefined;
224+
}
225+
if (obj.requiresPython !== undefined && typeof obj.requiresPython !== 'string') {
226+
return undefined;
227+
}
228+
229+
return {
230+
schemaVersion: META_SCHEMA_VERSION,
231+
scriptFsPath: obj.scriptFsPath,
232+
lastUsedAt: obj.lastUsedAt,
233+
requiresPython: obj.requiresPython,
234+
};
235+
}
236+
237+
function isCanonicalIsoTimestamp(value: unknown): value is string {
238+
if (typeof value !== 'string') {
239+
return false;
240+
}
241+
const ms = Date.parse(value);
242+
if (Number.isNaN(ms)) {
243+
return false;
244+
}
245+
return new Date(ms).toISOString() === value;
246+
}

0 commit comments

Comments
 (0)