datasets

The full lifecycle of a dataset from the command line — create, watch, preview, approve or iterate, download.

The datasets group is the heart of the CLI. Eight subcommands cover the whole lifecycle: you describe what you want, follow progress, inspect the three-sample preview, either approve or send feedback, then download the delivered dataset.

A typical session looks like this:

datagen datasets create --brief "Restructuring analysis tasks for my IB research agent"
datagen datasets watch <id>         # follow waypoints until preview is ready
datagen datasets preview <id>       # inspect the three samples
datagen datasets approve <id>       # or: feedback <id> "the rubric should weight sourcing higher"
datagen datasets watch <id>         # follow again until complete
datagen datasets download <id> -o ./my-dataset/

See the authoring flow for what's happening between those commands, and Reviewing your preview for what to look for at the preview step.

datasets create

Start a new dataset from a plain-language brief.

datagen datasets create --brief "your brief text" [--feedback "refinement note"]

Flags

Flag	Type	Default	Description
`--brief`	string	(required)	Plain-language description of the dataset you want.
`--feedback`	string	`None`	Optional refinement note applied to a fresh attempt when you want to nudge the shape without starting over conceptually.
`--format`	`text`\|`json`	`text`	Output format.

Example

$ datagen datasets create --brief "Complex restructuring analysis tasks for my IB research agent"
dataset_id: ds_abc123def
display_name: Restructuring analysis tasks
status: Understanding your ask
brief: Complex restructuring analysis tasks for my IB research agent
preview_available: False
download_available: False

Start datagen datasets watch ds_abc123def to follow progress.

datasets list

List datasets you've created, most recently updated first.

datagen datasets list [--limit N] [--cursor <token>]

Flags

Flag	Type	Default	Description
`--limit`	integer	`20`	Maximum number of datasets to return.
`--cursor`	string	`None`	Pagination cursor, printed by the previous page.
`--format`	`text`\|`json`	`text`	Output format.

When more pages exist, the next cursor is printed to stderr as Next page: --cursor <token>.

Example

$ datagen datasets list --limit 5
                        Datasets
┃ ID           ┃ Name                       ┃ Status         ┃ Updated ┃
│ ds_abc123def │ Restructuring analysis     │ Preview ready  │ 2m ago  │
│ ds_9f2k4qpw  │ Support triage tasks       │ Complete       │ 3h ago  │
│ ds_7h1m3zxc  │ SQL debugging tasks        │ Your review    │ 1d ago  │

datasets get

Fetch the current state of a single dataset.

datagen datasets get <dataset_id>

Arguments

Argument	Description
`dataset_id`	The dataset ID returned by `create`.

Flags

Flag	Type	Default	Description
`--format`	`text`\|`json`	`text`	Output format.

Example

$ datagen datasets get ds_abc123def
dataset_id: ds_abc123def
display_name: Restructuring analysis tasks
status: Preview ready
brief: Complex restructuring analysis tasks for my IB research agent
preview_available: True
download_available: False
message: Three samples are ready for review.

datasets preview

Render the three-sample preview for a dataset.

datagen datasets preview <dataset_id>

Arguments

Argument	Description
`dataset_id`	Dataset ID. Preview must be ready — check `get` first if unsure.

Flags

Flag	Type	Default	Description
`--format`	`text`\|`json`	`text`	Output format. `text` renders three panels; `json` emits the raw payload.

Each sample is rendered as a self-contained panel showing the task instruction, verifier, and metadata. See Reviewing your preview for what a good preview looks like and how to spot issues.

datasets approve

Approve the preview and kick off full generation.

datagen datasets approve <dataset_id> [--yes]

Arguments

Argument	Description
`dataset_id`	Dataset ID. Must have a preview ready.

Flags

Flag	Type	Default	Description
`--yes`, `-y`	flag	off	Skip the confirmation prompt.
`--format`	`text`\|`json`	`text`	Output format.

Without --yes, the CLI asks for confirmation before approving. Scripts and coding agents should pass -y.

Example

$ datagen datasets approve ds_abc123def -y
dataset_id: ds_abc123def
status: Generating full dataset

datasets feedback

Send a feedback note on the preview. The dataset respins with your note applied, then produces a new preview.

datagen datasets feedback <dataset_id> "<what should change>"

Arguments

Argument	Description
`dataset_id`	Dataset ID.
`text`	Your feedback, as a single positional string. Quote it.

Flags

Flag	Type	Default	Description
`--format`	`text`\|`json`	`text`	Output format.

Feedback works best when it's specific and actionable.

Example

$ datagen datasets feedback ds_abc123def "The rubric should weight sourcing criteria higher; a task isn't successful if claims aren't backed by the provided documents."
dataset_id: ds_abc123def
status: Designing your dataset

datasets watch

Stream progress waypoints over a long-lived connection. The command exits when the dataset reaches a terminal state: preview_ready, your_review, complete, or still_refining.

datagen datasets watch <dataset_id>

Arguments

Argument	Description
`dataset_id`	Dataset ID.

Flags

Flag	Type	Default	Description
`--format`	`text`\|`json`	`text`	Output format. JSON emits one event per line.

Example

$ datagen datasets watch ds_abc123def
Understanding your ask
Designing your dataset: drafting task shapes
Designing your dataset: grounding rubrics
Preview ready: three samples ready for review

If the connection drops mid-stream, the command exits with a stream_error and a hint to rerun.

The waypoint vocabulary is deliberately warm — "understanding your ask," "designing your dataset," "preview ready" — because each one represents a real slice of the work. See the authoring flow for what each one means.

datasets download

Download the delivered dataset to disk.

datagen datasets download <dataset_id> [-o path] [--file-format FORMAT]

Arguments

Argument	Description
`dataset_id`	Dataset ID. The dataset must be complete.

Flags

Flag	Type	Default	Description
`--output`, `-o`	path	`./<dataset_id>.<ext>`	Where to write the downloaded file.
`--file-format`	string	`None`	Preferred file format when multiple are available (`jsonl`, `parquet`). Filters the available artifacts.
`--format`	`text`\|`json`	`text`	Output format for the CLI's own report, not the downloaded file.

Example

$ datagen datasets download ds_abc123def -o ./my-dataset.tar.gz
Downloaded 18403222 bytes to ./my-dataset.tar.gz

The downloaded artifact is a Harbor-format taskset — one subfolder per task. See Dataset formats for the layout, and task anatomy for a field-by-field walkthrough.

On this page