TopolearnTrainer and most examples expect datasets.Dataset objects; map and splits are the daily tools.
The companion datasets library (pip install datasets) loads data from the Hub or local files into Arrow-backed tables that are memory-mapped from disk, so even large datasets do not need to fit in RAM.
load_dataset("stanfordnlp/imdb")returns aDatasetDictof splits (train,test).load_dataset("csv", data_files="reviews.csv")(or"json","parquet") loads local files.Dataset.from_dict({...})orDataset.from_pandas(df)builds one in memory.
Each row is a dict; columns have typed features (e.g. ClassLabel with names). Inspect with ds[0], ds.features, ds.column_names, ds.num_rows.
The core transformation is ds.map(fn, batched=True): fn receives a dict of column lists and returns new or changed columns. Results are cached on disk, so re-running a script skips finished work.
Other everyday methods:
filter(fn),select(range(1000))for a quick subset,shuffle(seed=42).train_test_split(test_size=0.1, seed=42)to create a validation split.remove_columns,rename_column,cast_column.set_format("torch")orwith_format("torch")to return tensors.
For data too large to download, load_dataset(..., streaming=True) returns an IterableDataset that reads examples on the fly.
Example
from datasets import Dataset
ds = Dataset.from_dict({
"text": ["great film", "awful plot", "loved it", "meh", "superb cast", "boring"],
"label": [1, 0, 1, 0, 1, 0],
})
print(ds) # Dataset({features: ['text', 'label'], num_rows: 6})
print(ds[0]) # {'text': 'great film', 'label': 1}
def add_length(batch): # batched: gets lists, returns new columns
return {"n_words": [len(t.split()) for t in batch["text"]]}
ds = ds.map(add_length, batched=True)
short = ds.filter(lambda ex: ex["n_words"] == 1)
print(list(short["text"])) # ['meh', 'boring']
splits = ds.train_test_split(test_size=0.33, seed=42)
print(splits["train"].num_rows, splits["test"].num_rows) # 4 2
# from datasets import load_dataset; imdb = load_dataset("stanfordnlp/imdb")Remember
load_datasetreturns Arrow-backed, memory-mapped splitsmap(fn, batched=True)is the main, cached transformation- Use
train_test_split,selectandfilterto shape data
Watch out for
- Using batched=True but writing fn as if it received a single example, so it operates on lists by mistake.