Lesson 9 of 10

Error analysis

Confusion matrix, per-tier and per-metro scores: where the model still struggles.

Outputs will appear here after the full run. The code is final.

Left: what & whyRight: the code and its real output

9. Error analysis

Where does the chosen model go wrong? The predictions below are the cross-fitted out-of-fold ones (each fold decoded with cut-points/weights tuned on the other folds).

from sklearn.metrics import confusion_matrix, f1_score as sk_f1
best_r = ranked[0]
o_, _ = best_r["out"]
cf_pred = np.empty(n_tr, int)
for f in range(FOLDS):
    m_ = fold_id == f
    prm = best_r["fit"](o_[~m_], y[~m_], W_tr[~m_], MT_tr[~m_])
    cf_pred[m_] = best_r["apply"](o_[m_], prm, MT_tr[m_])
print("chosen:", best_r["scenario"], "|", best_r["decoder"])
print("per-tier F1:", dict(zip(["tier 1", "tier 2", "tier 3", "tier 4"], sk_f1(y, cf_pred, average=None).round(3))))
cm = confusion_matrix(y, cf_pred, normalize="true")
fig, ax = plt.subplots(figsize=(5, 4))
ax.imshow(cm, cmap="Blues")
for i in range(K):
    for j in range(K):
        ax.text(j, i, f"{cm[i, j]:.2f}", ha="center", va="center", color="white" if cm[i, j] > 0.5 else "black")
ax.set_xticks(range(K), [f"pred {k + 1}" for k in range(K)]); ax.set_yticks(range(K), [f"true {k + 1}" for k in range(K)])
ax.set_title("Confusion matrix (row-normalised)"); plt.tight_layout(); plt.show()
err = cf_pred != y
print(f"wrong: {err.mean():.1%} of listings | of the mistakes, {(np.abs(cf_pred - y)[err] == 1).mean():.1%} are off by just one tier")
per_metro = pd.DataFrame({"metro": train.metro.astype(str), "y": y, "p": cf_pred}).groupby("metro")[["y", "p"]].apply(
    lambda d: pd.Series({"listings": len(d), "macro F1": round(sk_f1(d.y, d.p, average="macro"), 3)}))
display(per_metro.sort_values("macro F1"))

Reading this. Look at where the mistakes sit in the confusion matrix: mostly next to the diagonal means the model ranks listings well and only the borders between tiers are blurry. Middle tiers usually score lowest because they border two tiers each. The per-metro table shows which markets are hardest to predict.