prep

prep is an agent-friendly CLI for converting LLM/ML datasets into common standard training and evaluation schemas backed by Hugging Face datasets.

Installation

uv tool install prep-cli
# pip install prep-cli # or pip
prep --help

Alternatively, you can install from source:

# as tool
uv tool install 'prep-cli @ https://github.com/Moenupa/prep.git'
prep --help

# as folder
git clone https://github.com/Moenupa/prep.git
uv sync --dev
uv run prep --help

Usage

# export UI=1 # if you want interactive access
prep TARGET_FORMAT PIPELINE_ID [SPLIT] [SOURCE[@SUBSET][:SPLIT]] [OPTIONS]

# `auto`: generic conversion pipeline, requires SPLIT and SOURCE:
prep sft auto train path/to/data.jsonl --no-save

# a registered dataset-specific formatter pipeline:
prep verl geo3k test --save

# list registered pipelines and local output status:
ppls

Documentation

The full usage reference lives in skills/prep, maintained for both humans and coding agents:

• SKILL.md — shared usage: CLI syntax, source loading, auto conversion controls, save/upload flow, and ppls.
• sft.md / verl.md / eval.md / cls.md — per-format options and output schemas.
• show.md — dataset diagnostics.
• dev.md — extending prep with new formatter pipelines.