Skip to content

Historically reproducible model assessment

A temporal split alone does not establish what was known when a model would have been trained. Use one row per loan, feature snapshots from the decision date or earlier, and dates when the target outcomes actually became available.

run_predictive_backtest keeps one result row for every requested fold, including skipped folds. Supply label_available_col to enforce label availability at the validation-period start. The comparison is inclusive: a label available exactly at the cutoff can be used. Date inputs are normalized to UTC.

If explicit availability dates are unavailable, target_horizon_months derives conservative availability as the split decision date plus that many calendar months. A stale feature snapshot does not move the target window earlier. This assumes immediate reporting at horizon end; it does not account for reporting delays. Supply actual availability dates for delayed outcomes. With both options, negative labels cannot become available before the declared horizon; positive events may be known earlier. A label cannot predate its decision date.

feature_as_of_col identifies the feature snapshot date. It must not be missing or later than the corresponding split decision date. Without it, the split date is the assumed feature date. The package cannot verify that a supplied feature value actually existed then: extract historical snapshots and do not substitute current corrected or subsequently enriched values.

A complete historical call

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from cranalytics import predictive

frame = pd.DataFrame({
    "loan_id": [f"L{i}" for i in range(48)],
    "decision_date": np.repeat(pd.date_range("2024-01-01", periods=6, freq="MS"), 8),
    "feature": np.tile([1., 2., 3., 4., 5., 6., 7., 8.], 6),
    "bad": np.tile([0., 1.], 24),
})
frame["feature_date"] = frame["decision_date"]
frame["label_date"] = frame["decision_date"] + pd.DateOffset(months=2)
frame.loc[0, "feature"] = np.nan
session = predictive.run(
    frame, ["feature"], "bad", "decision_date",
    model_family="logistic", n_splits=3,
    feature_as_of_col="feature_date", label_available_col="label_date",
    target_horizon_months=2,
    evaluation_as_of="2024-07-01", training_as_of="2024-04-01",
    preprocessor=make_pipeline(SimpleImputer(), StandardScaler()),
)
assert session.training_metadata["n_training_rows"] == 16
assert session.backtest["n_train"].tolist() == [16, 24, 32]
assert session.backtest["n_pending_validation"].tolist() == [0, 0, 8]
print(session.backtest[["training_cutoff", "n_train", "n_purged", "status", "reason"]])

The final model and the historical evaluation have distinct dates:

  • evaluation_as_of limits the outcomes used to score validation folds. Unavailable outcomes are counted in n_pending_validation. Future validation folds are retained with reason validation_after_evaluation_cutoff.
  • training_as_of limits the final session fit to decision rows and labels known by that date. Training metadata reports the date, fitted row count and excluded row count. If omitted, the final fit uses the supplied frame; historical fold cutoffs do not restrict it automatically.

Fold output includes training_cutoff, max_train_label_available, feature_date_basis, target_horizon_months, evaluation_as_of, label_availability_enforced and loan_identity_checked, alongside existing counts, discrimination metrics and skipped-fold reasons. A legacy numeric-period call remains supported, but label_availability_enforced=False explicitly means that only temporal ordering was checked. If loan IDs are absent, loan_identity_checked=False; uniqueness cannot be established.

Loan grain and preprocessing

This workflow supports one observation per loan. It checks loan_id when present, or a custom loan_id_col. Missing or repeated IDs raise, including repeats across folds. Repeated-observation / behavioral-panel modeling requires a separate design and is not enabled by dropping the ID column.

Pass an unfitted sklearn-compatible transformer or pipeline as preprocessor. The package clones it for each training fold and for the final fit. Imputation, scaling, supervised feature selection and compatible binning transformers fit only on that fold's training rows and labels, after availability filtering. Validation rows are transformed with that fitted object. The returned estimator retains preprocessing for later score_model calls. Pre-fitted objects are cloned without their fitted state.

Features currently pass through the existing numeric predictive contract. Transformers receive arrays in feature_cols order; use integer column selectors, not column-name selectors. Raw categorical encoders are not supported by this contract. Do not transform or select features on the full dataset before calling the backtest. Logistic folds with missing features and no preprocessor are reported as skipped (missing_features_without_preprocessor); invalid model configuration still raises. strict=True raises on skipped folds.

These checks address observation timing and leakage. AUC, Gini and KS remain measures of discrimination; they do not establish calibration or expected dollar loss. ML Modeling covers the general workflow.