The story: from 0.70 to the final model

Version 1: a strong model with a broken ruler

The first notebook (Lesson 0) was already sophisticated: about 350 features, out-of-fold target encodings, LightGBM multiclass + ordinal models, a blend, and class weights tuned for macro F1.

A 6-point gap means the validation wasn't measuring the real task.

The diagnosis: Kaggle split by operator

Listings are run by operators, and one operator's listings are usually in the same tier (about 3 in 4 of them).

With random folds, the model could learn "operator X is expensive" and get rewarded on X's other listings. On the real test set every operator is new, so that knowledge is useless. Several of v1's strongest features (target encodings keyed on operator_ref) were really "recognise the operator" features.

Proof. Same features, same model, on a 35% sample:

The grouped number matches the leaderboard. From then on, every decision is made on grouped CV.

Version 2: fix the ruler, then improve

• Rebuild an operator id from the operator columns (name + platform age + class + portfolio size).
• Use StratifiedGroupKFold with that id.
• Drop operator-name target encodings (on the test set they'd match a different operator who happens to share the name).
• Add label-free operator-profile features and "comparable listings" features.

Small, honest gains. The big win of v2 is that the CV now predicts the leaderboard.

Version 3 and the final notebook: experiments, more models, better decoding

• Two more feature hypotheses (demand vs. comparables, facility price effects) were measured. Neither beat noise, so they were dropped. That's a normal outcome.
• More model families (Logistic Regression baseline, XGBoost, XGBoost regression, CatBoost) run on GPU.
• A stacker smooths predictions across each operator's listings.
• Decoders tuned for macro F1 (class weights, cut-points on the expected tier, per-metro cut-points), all cross-fitted.
• A scenario table picks the best combination automatically.

Final result

From the executed final notebook:

submission.csv           ← stack | A class weights | CV (test-weighted) 0.7214
submission_rank2.csv     ← opt_blend + xgb_reg (a=0.7) | B cut-points | CV (test-weighted) 0.7199
submission_rank3.csv     ← opt_blend + xgb_reg (a=0.5) | B cut-points | CV (test-weighted) 0.7198

The lessons that transfer to any project

1. Make validation imitate the test set. Ask how the test set was built: random rows? Future dates? New users? New operators?
2. A big CV vs. leaderboard gap is a bug report, not bad luck.
3. Measure every idea. Plausible ideas often add nothing.
4. Use the structure of the problem. Ordered tiers → regression and cut-points. Operator consistency → smoothing within operators.