The .ipynb format and nbformat

Tooling such as linters, report generators, graders and migration scripts all work by manipulating notebook JSON.

An .ipynb file is JSON following the nbformat schema (major version 4, minor 5 today). The top level has:

cells: a list of cells, each with cell_type ("code", "markdown" or "raw"), source, metadata and, since 4.5, a unique id;
code cells also have execution_count and outputs, a list of output objects of type stream (stdout/stderr text), execute_result and display_data (MIME bundles), or error (with ename, evalue and traceback);
metadata: kernelspec, language info, and anything tools add (Jupytext, papermill);
nbformat and nbformat_minor.

source may be stored as a list of lines or a single string; tools should accept both. Images are base64 strings inside output bundles, which is why plots inflate file size.

The nbformat library reads and writes notebooks safely: nbformat.read(path, as_version=4), nbformat.v4.new_notebook(), new_code_cell, new_markdown_cell, nbformat.validate(nb) and nbformat.write. It upgrades older files and normalizes source to strings.

nbclient executes a notebook object: NotebookClient(nb, timeout=600, kernel_name="python3").execute() fills in outputs, raising CellExecutionError on failure unless allow_errors=True. nbconvert and papermill are built on it.

Cell tags (metadata.tags) are the standard hook for tooling, such as parameters for papermill or remove-input for nbconvert's TagRemovePreprocessor.

Example

import nbformat
from nbformat.v4 import new_notebook, new_code_cell, new_markdown_cell
from nbclient import NotebookClient

nb = new_notebook()
nb.cells = [
    new_markdown_cell("# Generated report"),
    new_code_cell("x = 21\nx * 2"),
    new_code_cell("print('done')", metadata={"tags": ["remove-input"]}),
]
nbformat.validate(nb)                       # raises if the schema is violated
NotebookClient(nb, timeout=60, kernel_name="python3").execute()

out = nb.cells[1].outputs[0]
print(out.output_type, out.data["text/plain"])   # execute_result 42
print(nb.cells[2].outputs[0].text.strip())       # done
print(nb.nbformat, nb.nbformat_minor >= 5, len(nb.cells[1].id) > 0)  # 4 True True
nbformat.write(nb, "report.ipynb")

Remember

A notebook is JSON: cells with source, metadata and outputs, plus notebook metadata
Outputs are stream, execute_result, display_data or error objects with MIME bundles
Use nbformat to read, build and validate notebooks and nbclient to execute them

Watch out for

Editing notebook JSON by hand with string operations and producing files that fail schema validation.

Check yourself

Where is a cell's chart image stored in an executed .ipynb file?

Official docs: nbformat.readthedocs.io/en/latest/format_description.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks