Data pipelines break silently when upstream data changes. Tools like Great Expectations and Pandera let you declare what the data should look like. Build a small version.
Implement validate(df, rules), where rules maps column names to dicts that may contain:
"dtype": one of "int", "float", "str" or "datetime" (check with pd.api.types)"nullable" (default True): if False, no missing values allowed"unique": if True, no duplicate non-null values"min" / "max": bounds on non-null values (inclusive)"allowed": a set of permitted non-null values"regex": every non-null value must fully match (as a string)Return a list of failure messages f"{column}: {check} ({count} rows)" in column order of rules, with checks in the order listed above, where check is missing column, dtype, nulls, duplicates, min, max, allowed or regex. For the missing column and dtype checks, use a count of 0. Skip the remaining checks for a missing column.