TopolearnA few summary calls reveal ranges, outliers, typos and missing data before you trust any analysis.
Before analysing a dataset, profile it. A handful of methods do most of the work.
`df.info()` shows each column's dtype, non-null count and total memory: the fastest way to spot missing values and columns read with the wrong type.
`df.describe()` summarises numeric columns: count, mean, std, min, quartiles and max. Look for impossible minimums (negative ages), suspicious maximums (outliers or unit errors) and counts below the row count (missing values). describe(include="all") adds text columns with their number of unique values, the most frequent one and its frequency.
`s.value_counts()` counts each distinct value, sorted by frequency. It is the quickest way to see categories, typos ("NY" vs "ny"), and imbalance. Useful options:
normalize=Truegives proportions instead of countsdropna=Falseincludes a count of missing values (they're excluded by default)bins=5groups a numeric column into ranges
df.value_counts(["a", "b"]) counts combinations of several columns.
Also handy: s.nunique() (number of distinct values), s.unique(), df.isna().sum() (missing per column), df.duplicated().sum(), s.quantile([0.05, 0.95]), and df.corr(numeric_only=True) for pairwise correlations.
Remember that statistics skip NaN by default, so mean is the mean of the values present. That's convenient but can hide how much is missing, which is why counting missing values belongs in the first look too.
Example
import pandas as pd
df = pd.DataFrame({
"city": ["NY", "ny", "LA", "NY", "SF", None],
"age": [34, 29, -1, 41, 38, 52],
"spend": [120.0, 80.5, None, 300.0, 95.0, 60.0],
})
df.info() # dtypes, non-null counts, memory
print(df.describe().loc[["min", "max"], "age"].tolist()) # [-1.0, 52.0] -1 is suspicious
print(df["city"].value_counts().to_dict()) # {'NY': 2, 'ny': 1, 'LA': 1, 'SF': 1}
print(len(df["city"].value_counts(dropna=False))) # 5 groups, one is missing
print(df["city"].str.upper().value_counts(normalize=True).round(2).to_dict())
# {'NY': 0.6, 'LA': 0.2, 'SF': 0.2}
print(df.isna().sum().to_dict()) # {'city': 1, 'age': 0, 'spend': 1}
print(df["city"].nunique()) # 4 (NaN not counted)Remember
- Start with
info()anddescribe()on any new dataset value_countsreveals categories, typos and imbalance;normalize=Truefor sharesdropna=Falseis needed to see missing values in countsdf.isna().sum()counts missing values per column
Watch out for
- Reading
describe()means without checkingcount: statistics silently ignore missing values.