TopolearnA model you cannot reload or share reproducibly is not finished.
Every model, tokenizer and config has save_pretrained(dir) and from_pretrained(dir). A saved directory contains config.json, model.safetensors (sharded with an index above a size threshold), tokenizer files and the generation config. Always save the tokenizer with the model; a model directory without its tokenizer is a common deployment bug.
With Trainer:
trainer.save_model("final")writes the model plus theprocessing_class.- Intermediate checkpoints (
checkpoint-500/) also include optimizer, scheduler and RNG state for resuming;save_total_limit=2keeps disk usage bounded.
Sharing on the Hub:
- Log in with
hf auth login(or setHF_TOKEN). model.push_to_hub("you/my-model")andtok.push_to_hub("you/my-model")upload to a repo, creating it if needed;private=Truekeeps it private.TrainingArguments(push_to_hub=True, hub_model_id=...)pushes checkpoints during training, andtrainer.push_to_hub()uploads the final model with an auto-generated model card listing hyperparameters and metrics.
Edit the model card (README.md) to describe the training data, intended use and limitations. Set id2label in the config before saving so pipelines show readable labels.
To reload, point any from_pretrained or pipeline(model=...) at the local path or repo id.
Example
from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline
name = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(name)
tok = AutoTokenizer.from_pretrained(name)
model.save_pretrained("my-sentiment")
tok.save_pretrained("my-sentiment") # keep tokenizer next to weights
# my-sentiment/: config.json, model.safetensors, tokenizer.json, ...
clf = pipeline("text-classification", model="my-sentiment")
print(clf("Reloaded fine.")[0]["label"]) # POSITIVE
# after `hf auth login`:
# model.push_to_hub("your-name/my-sentiment", private=True)
# tok.push_to_hub("your-name/my-sentiment", private=True)Remember
save_pretrained/from_pretrainedround-trip models, tokenizers and configs- Checkpoints include optimizer state for resuming; final saves need only the model and tokenizer
push_to_hubpublishes to a repo; document it in the model card
Watch out for
- Saving only the model and later loading a different tokenizer, which silently produces wrong token IDs.