Question 1

Why not just load datasets from the Hub directly?

Accepted Answer

Interchangeability is the point. Every task is normalized to a MultipleChoice, Classification, or TokenClassification template with canonical fields so aligned datasets mix in one loop, and each preprocessing is a small readable function. Inputs stay raw by default; `prompted=True` appends the annotation's question when the inputs alone do not say what to predict.

Question 2

How do I actually get the Jev decision data?

Accepted Answer

Call `load_task(id, recast="jev")` and render a row with `render_typed_decision`, or use the published tasksource-jev-typed-decisions set, whose runbook covers smoke tests, resumable builds, validation, and publication. Conversion is deterministic and criteria are not paraphrased, with one documented exception where a trailing all/none-of-the-above becomes all/none of the other options.

Question 3

What is soft labeling, and when should I use it?

Accepted Answer

Some annotations are distributions rather than single labels, such as annotator votes or survey counts. `load_task(id, soft=True)` returns those as probabilities over options, and each records whether it summarizes votes or mean ratings and how many annotators judged the item. Vote shares from three annotators are coarse, so `min_annotators=5` leaves them out entirely.

Question 4

Can I rely on the license filter before shipping something commercial?

Accepted Answer

Treat it as a starting point. `license_use` keeps tasks whose sources permit commercial or non-commercial use and takes the most restrictive license found, reading Hub cards plus Data Provenance Initiative annotations cached in the package, and `task_licenses(fresh=True)` re-reads the current cards. The README calls it a best-effort filter and not legal advice.

Question 5

How do I install tasksource and load tasks?

Accepted Answer

pip install tasksource, then from tasksource import list_tasks, load_task. list_tasks returns a frame you can filter by task_type, and load_task(id) gives a standardized dataset. All yielded datasets share the same columns and can be used interchangeably.

Question 6

How much disk space do large multi-task runs need?

Accepted Answer

Ensure more than 100GB is available for large multi-task runs. Datasets are downloaded to $HF_DATASETS_CACHE, the same location any Hugging Face dataset uses.

Question 7

Can I load vision or instruction variants of these tasks?

Accepted Answer

Visual annotations live in vision_tasks.py: discover them with list_tasks(vision=True) and load with load_task(id, vision=True). Pass recast="jev" or recast="instruct" to keep images alongside the rendered decisions or prompts.

Tasksource

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community