Skip to content

Add Llama to Puzzletron v2 - #2454

Draft
grzegorz-k-karch wants to merge 1 commit into
feature/puzzletron_v2from
gkarch/puzzletron_v2_llama
Draft

grzegorz-k-karch wants to merge 1 commit into
feature/puzzletron_v2from
gkarch/puzzletron_v2_llama

Conversation

@grzegorz-k-karch

Copy link
Copy Markdown
Contributor

What does this PR do?

Type of change: Bug fix

Enables Llama 3.1 8B Instruct pruning campaigns in Puzzletron v2 with minimal implementation changes:

  • Uses the descriptor-based generic decoder converter to produce the expected Llama checkpoint schema.
  • Fixes width-importance completion checks for absolute, repository-relative, and campaign-relative output paths.
  • Adds Llama family/model configs, a memory-constrained FFN pruning campaign, compression sweeps, and a reduced-cost smoke configuration.
  • Configures the joint sweep to include attention/FFN no-op combinations and whole-block candidates, with block-level replacement and MIP scoring.
  • Sweeps retained-memory limits of 50%, 60%, 70%, 80%, and 90%, preserving the legacy 7-billion-parameter cap across all targets.
  • Adds regression coverage for conversion, path handling, and resolved campaign settings.

Usage

With a prepared dataset and site-specific runner/execution configurations:

export PUZZLETRON_DATASET_PATH=/path/to/dataset                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                             
python examples/puzzletron/orchestrate.py \                                                                                                                                                                                                                                  
  --experiment examples/puzzletron/configs/families/llama/llama3p1_8b/runs/pruneffn_memory_joint_sweep.yaml \                                                                                                                                                                
  --runner /path/to/runner.yaml \                                                                                                                                                                                                                                            
  --execution /path/to/execution.yaml \                                                                                                                                                                                                                                      
  --override puzzle_dir=/path/to/new/run \                                                                                                                                                                                                                                   
  --stage full --dry-run                                                                                                                                                                                                                                                     

Remove --dry-run to launch. Use pruneffn_memory_joint_sweep_smoke.yaml for a reduced-cost pipeline check; smoke results are not accuracy-comparison results.

Testing

Validated the isolated staged tree, excluding unrelated working-tree changes:

  • 38 tests passed across test_generic_decoder_converter.py, test_orchestration_lightweight.py, and test_llama3p1_8b_pruneffn_memory_config.py.
  • All applicable pre-commit checks passed, including formatting, lint, mypy, YAML, license, and security checks.

Before your PR is "Ready for review"

Follow the Contributor guidelines, including signed commits (git commit -s -S), and [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-pract\
ices-for-contributors).

  • Is this change backward compatible?: ✅ — preserves the legacy converter entry point; previously incompatible Llama conversion artifacts should be regenerated.
  • If you copied code from other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md?: N/A — no new runtime dependencies or external code.
  • Did you write any new necessary tests?: ✅
  • Did you update Changelog?: N/A — scoped bug fixes and example configurations.
  • Did you get Claude approval on this PR?: ❌ — not run.

Additional Information

Scope: commit 759dd8b51 (11 files). Uses existing sweep infrastructure; no new sweep engine is introduced. Setup-wizard changes, depth-pool changes, and broader orchestration hardening are excluded.

Signed-off-by: Grzegorz Karch <gkarch@nvidia.com>
@grzegorz-k-karch grzegorz-k-karch self-assigned this Sep 17, 2026
@grzegorz-k-karch grzegorz-k-karch added the puzzletron_v2 Related to feature/puzzletron_v2 branch label Sep 17, 2026
@copy-pr-bot

copy-pr-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://NVIDIA.github.io/Model-Optimizer/pr-preview/pr-2454/

Built to branch gh-pages at 2026-09-17 00:22 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 29.41176% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.62%. Comparing base (60df4db) to head (759dd8b).

Files with missing lines Patch % Lines
.../puzzletron/orchestration/adapters/stage_compat.py 0.00% 12 Missing ⚠️
Additional details and impacted files
@@                    Coverage Diff                    @@
##           feature/puzzletron_v2    #2454      +/-   ##
=========================================================
- Coverage                  50.63%   50.62%   -0.01%     
=========================================================
  Files                        713      713              
  Lines                      94313    94320       +7     
=========================================================
- Hits                       47756    47752       -4     
- Misses                     46557    46568      +11     
Flag Coverage Δ
puzzletron 29.55% <29.41%> (-0.01%) ⬇️
unit 28.65% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

puzzletron_v2 Related to feature/puzzletron_v2 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant