Lesson 4 of 10

Feature experiments

Two plausible feature ideas, measured honestly. Neither helped.

Outputs will appear here after the full run. The code is final.

Left: what & whyRight: the code and its real output

4. Feature-block experiments

Two extra feature blocks were tested with a fast model on 50% of operators (grouped CV, full data, measured on Kaggle):

feature set	features	grouped CV macro F1
v2 (base + geo + operator profile + target features)	392	0.7028
v2 + H1 demand relative to comparables (`rel_*`)	405	0.7016
v2 + H2 facility price effects (`facte_*`)	398	0.7029
v2 + both	411	0.7026

Neither block beats noise (±0.002), so the simpler v2 set is used (RUN_ABLATION = False skips re-running this). Lesson: a hypothesis that is plausible and even visible in the data can still add nothing once the model already has related features.

Fast settings on 50% of operators and 3 folds (XGBoost on GPU, otherwise LightGBM). Each result is checkpointed. Differences under ~0.002 are within noise, and then the smaller set is kept.

prog.start("ablation of feature blocks")
FEATURE_SETS = {
    "v2": [c for c in ALL_COLS if not c.startswith(("rel_", "facte_"))],
    "v2 + H1 relative": [c for c in ALL_COLS if not c.startswith("facte_")],
    "v2 + H2 facility": [c for c in ALL_COLS if not c.startswith("rel_")],
    "v2 + H1 + H2": ALL_COLS,
}
abl_rows = []
if RUN_ABLATION:
    sub = (host[:n_tr] * 2654435761 % 1000) < 500
    for name, cols in FEATURE_SETS.items():
        idx = [ALL_COLS.index(c) for c in cols]
        scores = []
        for f in range(3):
            ckn = f"{TAG}_abl_{name.replace(' ', '_').replace('+', 'p')}_{f}"
            ck = ck_load(ckn)
            if ck is None:
                tr = np.where(sub & (fold_id != f))[0]
                va = np.where(sub & (fold_id == f))[0]
                if GPU:
                    pa = dict(objective="multi:softprob", num_class=K, eta=0.1, max_depth=8, min_child_weight=5, subsample=0.8,
                              colsample_bytree=0.5, max_bin=128, tree_method="hist", device="cuda", eval_metric="mlogloss", seed=SEED)
                    dva = xgb.DMatrix(X_full[va][:, idx], label=y[va])
                    b = xgb.train(pa, xgb.DMatrix(X_full[tr][:, idx], label=y[tr]), 3000, evals=[(dva, "v")],
                                  early_stopping_rounds=50, verbose_eval=False)
                    p = b.predict(dva, iteration_range=(0, b.best_iteration + 1))
                else:
                    pa = dict(objective="multiclass", num_class=K, learning_rate=0.1, num_leaves=127, min_child_samples=40,
                              feature_fraction=0.5, bagging_fraction=0.8, bagging_freq=1, max_bin=63, verbose=-1, seed=SEED, num_threads=N_THREADS)
                    m = lgb.train(pa, lgb.Dataset(X_full[tr][:, idx], y[tr]), 3000, valid_sets=[lgb.Dataset(X_full[va][:, idx], y[va])],
                                  callbacks=[lgb.early_stopping(50, verbose=False)])
                    p = m.predict(X_full[va][:, idx], num_iteration=m.best_iteration)
                ck = {"f1": np.array([fast_f1(y[va], p.argmax(1))])}
                ck_save(ckn, **ck)
            scores.append(float(ck["f1"][0]))
        abl_rows.append({"feature set": name, "n features": len(cols), "grouped CV macro F1": round(np.mean(scores), 4)})
        print(abl_rows[-1], flush=True)
    abl = pd.DataFrame(abl_rows)
    best = abl["grouped CV macro F1"].max()
    # simplest set within 0.002 of the best (fewer features = faster + less overfitting)
    FINAL_SET = abl[abl["grouped CV macro F1"] >= best - 0.002].sort_values("n features").iloc[0]["feature set"]
    display(abl)
else:
    FINAL_SET = "v2"  # winner of the measured experiments above
print("chosen feature set:", FINAL_SET)
SEL = [ALL_COLS.index(c) for c in FEATURE_SETS[FINAL_SET]]
COLS = FEATURE_SETS[FINAL_SET]
X, Xt = X_full[:, SEL], Xt_full[:, SEL]
CAT_IDX = [COLS.index(c) for c in feat["cat_cols"] if c in COLS]
SET_TAG = FINAL_SET.replace(" ", "").replace("+", "p")
prog.finish()