TopolearnDebugging odd model behaviour, adding a custom head, or shipping a new architecture requires knowing these building blocks.
Every architecture in Transformers is built from three pieces:
- A config class (subclass of
PreTrainedConfig) holding hyperparameters such as hidden size, layers, heads and vocabulary size; it is whatconfig.jsonserializes. - A base model (
LlamaModel,BertModel) subclassingPreTrainedModel, which suppliesfrom_pretrained,save_pretrained, weight initialisation, embedding resizing, gradient checkpointing and Hub integration on top oftorch.nn.Module. - Heads (
LlamaForCausalLM,BertForSequenceClassification) wrapping the base model and returningModelOutputdataclasses withlosswhen labels are provided.
The library deliberately repeats code per model ("single model, single file") so each modeling file is readable on its own; in the source repo, modular files let new models inherit from existing ones and are expanded into standalone files.
To build your own:
1. Subclass PreTrainedConfig with a unique model_type. 2. Subclass PreTrainedModel, set config_class, and implement forward. 3. Register: AutoConfig.register("my-model", MyConfig) and AutoModel.register(MyConfig, MyModel).
Now save_pretrained and from_pretrained work, and with register_for_auto_class() plus push_to_hub others can load it via trust_remote_code=True.
Reading a model's modeling file is the fastest way to answer questions like "where is the RoPE scaling applied" or "which layer names should LoRA target".
Example
import torch
from torch import nn
from transformers import AutoConfig, AutoModel, PreTrainedConfig, PreTrainedModel
class TinyConfig(PreTrainedConfig):
model_type = "tiny-mlp"
def __init__(self, in_dim=4, hidden=8, num_labels=2, **kwargs):
self.in_dim, self.hidden = in_dim, hidden
super().__init__(num_labels=num_labels, **kwargs)
class TinyModel(PreTrainedModel):
config_class = TinyConfig
def __init__(self, config):
super().__init__(config)
self.net = nn.Sequential(nn.Linear(config.in_dim, config.hidden), nn.ReLU(),
nn.Linear(config.hidden, config.num_labels))
self.post_init()
def forward(self, x):
return {"logits": self.net(x)}
AutoConfig.register("tiny-mlp", TinyConfig)
AutoModel.register(TinyConfig, TinyModel)
TinyModel(TinyConfig()).save_pretrained("tiny")
model = AutoModel.from_pretrained("tiny")
print(type(model).__name__, model(torch.ones(1, 4))["logits"].shape) # TinyModel torch.Size([1, 2])Remember
- Config + PreTrainedModel base + task heads make up every architecture
- Register custom configs and models with the Auto classes to get save/load for free
- Modeling files are intentionally self-contained; read them to debug behaviour
Watch out for
- Forgetting to call post_init() in a custom PreTrainedModel, which skips weight initialisation and setup hooks.