{ "cells": [ { "cell_type": "markdown", "id": "59f09141", "metadata": { "id": "59f09141" }, "source": [ "# Workspace price tier prediction: v3\n", "\n", "**v1 → v2 recap.** Kaggle split train/test **by operator** (only ~1% of test rows come from an operator seen in train).\n", "v1's random folds let the model see each operator's other listings, so its CV (0.758) was inflated. Operator-grouped CV gives 0.697,\n", "which matches the leaderboard (0.70). From v2 on, **every decision is made on the grouped CV**.\n", "\n", "**v3 = an experiment notebook.** Everything below is a hypothesis that gets measured; the notebook keeps only what wins.\n", "\n", "| # | hypothesis | why it might help |\n", "|---|---|---|\n", "| H1 | **demand relative to comparable listings** (booked less / more open days / fewer reviews than similar spaces nearby) | for the same kind of space in the same zone, the pricier ones get booked less. Data check: corr −0.085 *within* comparables. Trees can't compute \"vs. the group average\" on their own |\n", "| H2 | **facility price effects** (each facility's average tier, after removing the metro effect) | 150+ one-hot columns are hard to use; one \"how premium are these amenities\" score is easier |\n", "| H3 | **more model families** (LightGBM, XGBoost, CatBoost) + blending | different algorithms make different mistakes |\n", "| H4 | **ordinal decoding** (expected-tier score + tuned cut-points) | tiers are ordered; cut-points can be tuned directly for macro F1 |\n", "| H5 | **operator smoothing** (stacker using the operator's other listings' predictions) | an operator's listings are usually in the same tier, and all of a test operator's listings are in test |\n", "| H6 | **test-distribution weighting** (metro mix differs: e.g. MTR-15 is 4% of train, 10% of test) | macro F1 depends on the class mix; tune decisions for the test's metro mix |\n", "| H7 | **per-metro cut-points** | tiers mean very different things in different metros |\n", "\n", "Decoders are evaluated with **cross-fitting** (tuned on 4 folds' predictions, scored on the 5th), so the scenario table isn't fooled by tuning on the same rows.\n", "\n", "### How to run\n", "1. Settings → Accelerator → **GPU T4** (XGBoost and CatBoost are ~10× faster). It also works on CPU, but CatBoost is then skipped.\n", "2. Optional: set `QUICK_TEST = True` and *Run All* (~10 min) to check that everything works, then set it back to `False`.\n", "3. **Save Version → \"Save & Run All (Commit)\"**. This runs on Kaggle's servers: closing the browser or losing internet doesn't stop it.\n", "4. **If a run is ever cut off**: *Add Input → Notebook Output →* this notebook's last version, then run again.\n", " It finds the old `ckpt/` folder and continues where it stopped. Every finished fold is saved." ] }, { "cell_type": "code", "source": [ "from google.colab import files; files.download('submission.csv')" ], "metadata": { "id": "0QPuG_wrUzMz", "outputId": "f66c8951-5612-4292-dae9-6ea174f637c7", "colab": { "base_uri": "https://localhost:8080/", "height": 17 } }, "id": "0QPuG_wrUzMz", "execution_count": 11, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "\n", " async function download(id, filename, size) {\n", " if (!google.colab.kernel.accessAllowed) {\n", " return;\n", " }\n", " const div = document.createElement('div');\n", " const label = document.createElement('label');\n", " label.textContent = `Downloading \"${filename}\": `;\n", " div.appendChild(label);\n", " const progress = document.createElement('progress');\n", " progress.max = size;\n", " div.appendChild(progress);\n", " document.body.appendChild(div);\n", "\n", " const buffers = [];\n", " let downloaded = 0;\n", "\n", " const channel = await google.colab.kernel.comms.open(id);\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", "\n", " for await (const message of channel.messages) {\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", " if (message.buffers) {\n", " for (const buffer of message.buffers) {\n", " buffers.push(buffer);\n", " downloaded += buffer.byteLength;\n", " progress.value = downloaded;\n", " }\n", " }\n", " }\n", " const blob = new Blob(buffers, {type: 'application/binary'});\n", " const a = document.createElement('a');\n", " a.href = window.URL.createObjectURL(blob);\n", " a.download = filename;\n", " div.appendChild(a);\n", " a.click();\n", " div.remove();\n", " }\n", " " ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "download(\"download_f95780c9-7c03-4a0f-ae36-cbf3eeef125f\", \"submission.csv\", 919746)" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "id": "633e51a2", "metadata": { "id": "633e51a2" }, "source": [ "## 0. Colab setup (skipped automatically on Kaggle)\n", "Runtime → Change runtime type → **T4 GPU** first. This cell installs CatBoost and downloads the competition data from the course Drive link." ] }, { "cell_type": "code", "execution_count": 1, "id": "ef58ce6f", "metadata": { "id": "ef58ce6f", "outputId": "700fd56c-cded-4573-d2ed-812c85bfe0c7", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "Downloading...\n", "From (original): https://drive.google.com/uc?id=1v6Ce6HLFvByXzDdcHQD1NMgA-XnJf5kf\n", "From (redirected): https://drive.google.com/uc?id=1v6Ce6HLFvByXzDdcHQD1NMgA-XnJf5kf&confirm=t&uuid=a761f33a-75fa-4c7c-8029-abe08af30049\n", "To: /content/data_download\n", "100%|██████████| 53.7M/53.7M [00:01<00:00, 33.5MB/s]\n" ] }, { "output_type": "stream", "name": "stdout", "text": [ "train.csv found at: ['/content/data/kaggle_ready_dataset/train.csv']\n" ] } ], "source": [ "import sys, subprocess, zipfile\n", "from pathlib import Path as _P\n", "if \"google.colab\" in sys.modules:\n", " subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"catboost\", \"gdown\"], check=True)\n", " if not any(_P(\"/content\").rglob(\"train.csv\")):\n", " import gdown\n", " gdown.download(id=\"1v6Ce6HLFvByXzDdcHQD1NMgA-XnJf5kf\", output=\"/content/data_download\", quiet=False)\n", " z = _P(\"/content/data_download\")\n", " if zipfile.is_zipfile(z):\n", " zipfile.ZipFile(z).extractall(\"/content/data\")\n", " for inner in list(_P(\"/content/data\").rglob(\"*.zip\")): # zip inside a zip\n", " zipfile.ZipFile(inner).extractall(inner.parent)\n", " print(\"train.csv found at:\", [str(p) for p in _P(\"/content\").rglob(\"train.csv\")])" ] }, { "cell_type": "code", "execution_count": 2, "id": "83bb0422", "metadata": { "id": "83bb0422", "outputId": "d7c39bb8-836e-4da8-929c-351233d872af", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "GPU: True | CPU threads: 2 | models: ['xgb', 'xgb_reg', 'cat'] | quick test: False\n", "checkpoint folders searched: ['ckpt']\n" ] } ], "source": [ "import json\n", "import os\n", "import shutil\n", "import subprocess\n", "import time\n", "import warnings\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import lightgbm as lgb\n", "import xgboost as xgb\n", "from catboost import CatBoostClassifier\n", "from scipy import sparse\n", "from scipy.optimize import minimize\n", "from sklearn.cluster import MiniBatchKMeans\n", "from sklearn.metrics import log_loss\n", "from sklearn.model_selection import StratifiedGroupKFold\n", "from sklearn.neighbors import NearestNeighbors\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "# ======================= SETTINGS =======================\n", "QUICK_TEST = bool(int(os.environ.get(\"PT_QUICK\", \"0\"))) # True = 10% of operators + few rounds (pipeline check only)\n", "USE_LIGHTGBM = False # LightGBM runs on CPU only (its GPU mode is slower on Kaggle): ~8 min/fold\n", "RUN_ABLATION = False # test H1/H2 (feature blocks) before the big training\n", "FOLDS, SEED, K = 5, 42, 4\n", "MAX_ROUNDS = 300 if QUICK_TEST else 6000\n", "LOG_EVERY = 100 if QUICK_TEST else 250 # print training progress every N rounds\n", "# =========================================================\n", "\n", "TAG = f\"v3b{'q' if QUICK_TEST else ''}_f{FOLDS}\" # checkpoint names (quick-test checkpoints never mix with full ones)\n", "WORK = Path(\"/kaggle/working\") if Path(\"/kaggle/working\").exists() else Path(\".\")\n", "CKPT = WORK / \"ckpt\"\n", "CKPT.mkdir(exist_ok=True)\n", "SEARCH = [CKPT] + (sorted(p for p in Path(\"/kaggle/input\").glob(\"**/ckpt\") if p.is_dir()) if Path(\"/kaggle/input\").exists() else [])\n", "\n", "\n", "def gpu_available():\n", " if shutil.which(\"nvidia-smi\") is None:\n", " return False\n", " try:\n", " return subprocess.run([\"nvidia-smi\"], capture_output=True, timeout=10).returncode == 0\n", " except Exception:\n", " return False\n", "\n", "\n", "GPU = gpu_available()\n", "if GPU: # all-GPU models; LightGBM only if asked for\n", " MODELS = [\"xgb\", \"xgb_reg\", \"cat\"] + ([\"lgb_multi\", \"lgb_reg\"] if USE_LIGHTGBM else [])\n", "else:\n", " MODELS = [\"lgb_multi\", \"lgb_reg\", \"xgb\", \"xgb_reg\"] + ([\"cat\"] if os.environ.get(\"PT_FORCE_CAT\") else [])\n", " print(\"No GPU found: running on CPU and skipping CatBoost. Turn on GPU T4 for a much faster run.\")\n", "N_THREADS = os.cpu_count()\n", "print(f\"GPU: {GPU} | CPU threads: {N_THREADS} | models: {MODELS} | quick test: {QUICK_TEST}\")\n", "print(\"checkpoint folders searched:\", [str(p) for p in SEARCH])" ] }, { "cell_type": "markdown", "id": "da2c5856", "metadata": { "id": "da2c5856" }, "source": [ "## Helpers: checkpoints, progress bar, fast macro F1\n", "Checkpoints are plain `.npz` files (NumPy arrays only, no pickle). They are written to a temp file and then renamed,\n", "so a crash during a save never leaves a broken checkpoint." ] }, { "cell_type": "code", "execution_count": 3, "id": "0784be45", "metadata": { "id": "0784be45" }, "outputs": [], "source": [ "def ck_load(name):\n", " for d in SEARCH:\n", " p = d / f\"{name}.npz\"\n", " if p.exists():\n", " with np.load(p, allow_pickle=False) as z:\n", " return {k: z[k] for k in z.files}\n", " return None\n", "\n", "\n", "def ck_save(name, **arrays):\n", " tmp = CKPT / f\"{name}.tmp.npz\"\n", " np.savez(tmp, **arrays)\n", " tmp.replace(CKPT / f\"{name}.npz\")\n", "\n", "\n", "class Progress:\n", " \"\"\"Overall progress bar over all big steps. ETA = remaining steps x the measured time of the same kind of step\n", " (e.g. each remaining XGBoost fold is estimated from finished XGBoost folds, not from the slower LightGBM ones).\"\"\"\n", "\n", " def __init__(self, plan):\n", " self.plan, self.total, self.done, self.t0, self.kind_times = plan, len(plan), 0, time.time(), {}\n", "\n", " def bar(self, width=30):\n", " full = int(width * self.done / self.total)\n", " return \"█\" * full + \"░\" * (width - full) + f\" {self.done}/{self.total} ({100 * self.done / self.total:.0f}%)\"\n", "\n", " def start(self, label):\n", " self.label, self.ts = label, time.time()\n", " print(f\"\\n▶ {label} [{self.bar()}]\", flush=True)\n", "\n", " def finish(self, from_checkpoint=False):\n", " self.done += 1\n", " took = time.time() - self.ts\n", " if not from_checkpoint:\n", " self.kind_times.setdefault(self.label.split(\":\")[0], []).append(took)\n", " left = [s.split(\":\")[0] for s in self.plan[self.done:]]\n", " timed = sum(np.mean(self.kind_times[k]) for k in left if k in self.kind_times)\n", " untimed = sum(k not in self.kind_times for k in left)\n", " eta = \"-\" if not left else f\"{timed / 60:.0f} min\" + (f\" + {untimed} steps not timed yet\" if untimed else \"\")\n", " how = \"loaded from checkpoint\" if from_checkpoint else f\"took {took / 60:.1f} min\"\n", " print(f\"✓ {self.label}: {how} | [{self.bar()}] | elapsed {(time.time() - self.t0) / 60:.1f} min | ETA ~{eta}\", flush=True)\n", "\n", "\n", "def conf(yt, yp, w=None):\n", " return np.bincount(yt * K + yp, weights=w, minlength=K * K).reshape(K, K)\n", "\n", "\n", "def f1_from_conf(cm):\n", " tp = np.diag(cm)\n", " return float(np.mean(2 * tp / np.maximum(cm.sum(0) + cm.sum(1), 1e-12)))\n", "\n", "\n", "def fast_f1(yt, yp, w=None):\n", " \"\"\"macro F1 (optionally sample-weighted), ~50× faster than sklearn; used inside the tuning loops\"\"\"\n", " return f1_from_conf(conf(yt, yp, w))\n", "\n", "\n", "PLAN = ([\"load data + grouped folds\", \"build features\", \"ablation of feature blocks\"]\n", " + [f\"{m}: fold {f + 1}/{FOLDS}\" for m in MODELS for f in range(FOLDS)]\n", " + [\"level-2 operator stacker\", \"scenario table\", \"write submissions\"])\n", "prog = Progress(PLAN)" ] }, { "cell_type": "markdown", "id": "7fca8d54", "metadata": { "id": "7fca8d54" }, "source": [ "## 1. Data, operator id, grouped folds" ] }, { "cell_type": "code", "execution_count": 4, "id": "c3dd74c8", "metadata": { "id": "c3dd74c8", "outputId": "14c05569-1013-49cb-eab6-eca417c2dba0", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "▶ load data + grouped folds [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0/21 (0%)]\n", "(279746, 60) (114538, 59) | test rows whose operator is in train: 0.0098\n", "✓ load data + grouped folds: took 1.0 min | [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1/21 (5%)] | elapsed 1.0 min | ETA ~0 min + 20 steps not timed yet\n" ] } ], "source": [ "prog.start(\"load data + grouped folds\")\n", "\n", "\n", "DATA_DIR = \"\" # optional: paste the folder that contains train.csv, e.g. \"/kaggle/input/workstation-rental-price-tier-prediction\"\n", "\n", "\n", "def find_data():\n", " if DATA_DIR:\n", " return Path(DATA_DIR)\n", " if \"PT_DATA\" in os.environ:\n", " return Path(os.environ[\"PT_DATA\"])\n", " for root in [\"/kaggle/input\", \"/content\", \".\"]:\n", " for dirpath, _, files in os.walk(root, followlinks=True): # followlinks: Kaggle mounts inputs as symlinks\n", " if \"train.csv\" in files and \"test.csv\" in files:\n", " return Path(dirpath)\n", " found = [str(q) for q in Path(\"/kaggle/input\").glob(\"*\")] if Path(\"/kaggle/input\").exists() else []\n", " raise FileNotFoundError(\n", " \"train.csv / test.csv not found. The competition data is not attached to this notebook. \"\n", " \"Fix: right panel -> 'Add Input' -> 'Competitions' -> search 'workstation-rental-price-tier-prediction' -> Add. \"\n", " f\"Currently in /kaggle/input: {found}\")\n", "\n", "\n", "DATA = find_data()\n", "train = pd.read_csv(DATA / \"train.csv\")\n", "test = pd.read_csv(DATA / \"test.csv\")\n", "\n", "# operator_ref alone isn't unique (different operators share names); these columns together identify one operator\n", "HOST = [\"operator_ref\", \"operator_platform_age_years\", \"operator_platform_age_months\", \"operator_active_years\",\n", " \"operator_active_months\", \"operator_class\", \"operator_portfolio_size\"]\n", "host = pd.concat([train[HOST], test[HOST]], ignore_index=True).groupby(HOST, dropna=False, sort=False).ngroup().to_numpy()\n", "if QUICK_TEST: # keep 10% of operators (whole operators, so the split logic stays the same)\n", " keep = (host * 2654435761 % 1000) < 100\n", " n0 = len(train)\n", " train, test = train[keep[:n0]].reset_index(drop=True), test[keep[n0:]].reset_index(drop=True)\n", " host = host[keep]\n", "\n", "n_tr = len(train)\n", "y = train[\"price_tier\"].to_numpy() - 1\n", "folds = list(StratifiedGroupKFold(FOLDS, shuffle=True, random_state=SEED).split(np.zeros(n_tr), y, host[:n_tr]))\n", "fold_id = np.zeros(n_tr, int)\n", "for i, (_, v) in enumerate(folds):\n", " fold_id[v] = i\n", "print(train.shape, test.shape, \"| test rows whose operator is in train:\", round(np.isin(host[n_tr:], host[:n_tr]).mean(), 4))\n", "prog.finish()" ] }, { "cell_type": "markdown", "id": "b8e4357c", "metadata": { "id": "b8e4357c" }, "source": [ "## 2. Features (checkpointed)\n", "Blocks (the ablation in section 3 decides which to keep):\n", "- **v2 set**: base columns, facility one-hots (≥ 0.5% of listings), geo transforms, demand/space ratios, local density, zone context,\n", " label-free operator profile, and out-of-fold target features on grouped folds (zone / comparables / geo neighbours / kNN comparables).\n", "- **H1 `rel_*`**: listing minus the average of its comparables (same zone + access type + seats), for bookings, open days, reviews, terms, facilities, scores. No labels used.\n", "- **H2 `facte_*`**: out-of-fold facility effects. For each facility, the average tier residual after removing the metro average; per listing, the mean / sum / max / min over its facilities." ] }, { "cell_type": "code", "execution_count": 5, "id": "2459aa8d", "metadata": { "id": "2459aa8d", "outputId": "990ab664-ed03-453e-db71-a652229f7037", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "▶ build features [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1/21 (5%)]\n", "✓ build features: took 3.1 min | [██░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2/21 (10%)] | elapsed 4.2 min | ETA ~0 min + 19 steps not timed yet\n", "all features: (279746, 411) | test: (114538, 411)\n" ] } ], "source": [ "prog.start(\"build features\")\n", "feat = ck_load(f\"{TAG}_features\")\n", "if feat is None:\n", " raw = pd.concat([train.drop(columns=\"price_tier\"), test], ignore_index=True)\n", " rawx = raw.assign(seats_b=raw.max_seats.clip(upper=12))\n", " df = raw.copy()\n", " for c in [\"operator_certified\", \"operator_has_avatar\", \"operator_id_verified\", \"bookable\"]:\n", " df[c] = df[c].map({\"t\": 1, \"f\": 0})\n", " for c in [\"snapshot_start_date\", \"snapshot_end_date\", \"first_review_date\", \"last_review_date\"]:\n", " df[c] = (pd.to_datetime(df[c]) - pd.Timestamp(\"2000-01-01\")).dt.days\n", " for c in [\"operator_ref\", \"zone\", \"borough\"]:\n", " df[c] = raw[c].map(raw[c].value_counts())\n", " nfac = raw[\"facilities\"].str.count(\"FCL-\").fillna(0)\n", " df[\"n_facilities\"] = nfac\n", " df[\"occupancy\"] = df.booked_load / 365\n", " df[\"review_speed_12m\"] = df.review_count_12m / (df.review_count + 1)\n", " df[\"review_trend\"] = (df.review_count_12m + 1) / (df.review_count_prev_yr + 1)\n", " df[\"days_since_last_review\"] = df.snapshot_end_date - df.last_review_date\n", " df[\"listing_age_days\"] = df.snapshot_end_date - df.first_review_date\n", " df[\"open_ratio_a_d\"] = df.open_slots_a / (df.open_slots_d + 1)\n", " df[\"open_ratio_e_d\"] = df.open_slots_e / (df.open_slots_d + 1)\n", " df[\"seats_per_ws\"] = df.max_seats / df.workstations.replace(0, np.nan)\n", " df[\"seats_per_zone\"] = df.max_seats / df.zones.replace(0, np.nan)\n", " df[\"seats_per_util\"] = df.max_seats / df.utility_rooms.replace(0, np.nan)\n", " df[\"term_span\"] = df.max_term_days - df.min_term_days\n", "\n", " codes = raw[\"facilities\"].fillna(\"[]\").map(json.loads).explode().dropna()\n", " fac = pd.crosstab(codes.index, codes).reindex(raw.index, fill_value=0).clip(upper=1).astype(np.uint8)\n", " fac = fac.loc[:, fac.mean() >= 0.005]\n", " fac.columns = [\"f_\" + c[4:] for c in fac.columns]\n", " code_id, code_names = pd.factorize(codes.values)\n", " F_all = sparse.csr_matrix((np.ones(len(codes), np.float32), (codes.index.to_numpy(), code_id)), shape=(len(raw), len(code_names)))\n", " F_all.data[:] = 1.0\n", " F_all = F_all[:, np.where(np.asarray(F_all.sum(0)).ravel() >= 30)[0]].tocsr()\n", "\n", " lat, lon = raw[\"geo_lat\"], raw[\"geo_lon\"]\n", " geo = pd.DataFrame({\"lat_p_lon\": lat + lon, \"lat_m_lon\": lat - lon})\n", " for a in (15, 30, 60):\n", " r = np.deg2rad(a)\n", " geo[f\"rot{a}_x\"] = lat * np.cos(r) - lon * np.sin(r)\n", " geo[f\"rot{a}_y\"] = lat * np.sin(r) + lon * np.cos(r)\n", " for k in (\"metro\", \"zone\"):\n", " center = raw.groupby(k)[[\"geo_lat\", \"geo_lon\"]].transform(\"median\")\n", " geo[f\"dist_{k}_center\"] = np.hypot(lat - center[\"geo_lat\"], lon - center[\"geo_lon\"])\n", " km = MiniBatchKMeans(100, random_state=0, n_init=3, batch_size=4096).fit(raw[[\"geo_lat\", \"geo_lon\"]])\n", " geo[\"geo_cluster\"] = km.labels_\n", " geo[\"geo_cluster_dist\"] = km.transform(raw[[\"geo_lat\", \"geo_lon\"]]).min(1)\n", " dist_all, _ = NearestNeighbors(n_neighbors=51).fit(raw[[\"geo_lat\", \"geo_lon\"]]).kneighbors(raw[[\"geo_lat\", \"geo_lon\"]])\n", " geo[\"density_d10\"], geo[\"density_d50\"] = dist_all[:, 10], dist_all[:, 50]\n", " geo[\"zone_mean_seats\"] = df.max_seats.groupby(raw.zone).transform(\"mean\")\n", " geo[\"zone_mean_nfac\"] = nfac.groupby(raw.zone).transform(\"mean\")\n", " geo[\"seats_vs_zone\"] = df.max_seats - geo.zone_mean_seats\n", " geo[\"nfac_vs_zone\"] = nfac - geo.zone_mean_nfac\n", "\n", " # label-free operator profile: what kind of operator is this, and how does this listing compare with its portfolio?\n", " d = pd.DataFrame({\"seats\": raw.max_seats, \"ws\": raw.workstations, \"zones\": raw.zones, \"util\": raw.utility_rooms,\n", " \"nfac\": nfac, \"score\": raw.score_overall, \"rev\": raw.review_count,\n", " \"minterm\": raw.min_term_days, \"open_d\": raw.open_slots_d, \"load\": raw.booked_load})\n", " for c in [\"access_type\", \"space_class\"]:\n", " for v in raw[c].value_counts().index[:4]:\n", " d[f\"{c}_{v}\"] = (raw[c] == v).astype(float)\n", " g = d.groupby(host)\n", " prof = pd.DataFrame(index=d.index)\n", " prof[\"h_rows\"] = g.seats.transform(\"size\")\n", " prof[\"h_zones_n\"] = raw.groupby(host).zone.transform(\"nunique\")\n", " prof[\"h_metros_n\"] = raw.groupby(host).metro.transform(\"nunique\")\n", " for c in d.columns:\n", " m = g[c].transform(\"mean\")\n", " prof[f\"h_mean_{c}\"] = m\n", " if c in (\"seats\", \"ws\", \"zones\", \"util\", \"nfac\", \"score\", \"minterm\"):\n", " prof[f\"h_std_{c}\"] = g[c].transform(\"std\")\n", " prof[f\"h_diff_{c}\"] = d[c] - m\n", " prof[\"h_rank_seats\"] = g.seats.rank(pct=True)\n", " prof[\"h_rank_nfac\"] = g.nfac.rank(pct=True)\n", " bld = raw.groupby([\"geo_lat\", \"geo_lon\"]).ngroup().to_numpy()\n", " gb = d.groupby(bld)\n", " prof[\"b_rows\"] = gb.seats.transform(\"size\")\n", " prof[\"b_hosts\"] = pd.Series(host).groupby(bld).transform(\"nunique\")\n", " prof[\"b_mean_seats\"] = gb.seats.transform(\"mean\")\n", " prof[\"b_mean_nfac\"] = gb.nfac.transform(\"mean\")\n", "\n", " # H1: demand and quality relative to comparable listings (same zone + access type + seats); no labels\n", " comp = rawx.groupby([\"zone\", \"access_type\", \"seats_b\"], dropna=False, sort=False).ngroup().to_numpy()\n", " value_gap = raw.score_value - raw[[\"score_accuracy\", \"score_condition\", \"score_onboarding\", \"score_support\", \"score_access\"]].mean(axis=1)\n", " rel = pd.DataFrame({\"rel_value_gap\": value_gap, \"rel_comp_size\": pd.Series(comp).map(pd.Series(comp).value_counts()).to_numpy()})\n", " for c, s in {\"load\": raw.booked_load, \"open_a\": raw.open_slots_a, \"open_d\": raw.open_slots_d, \"rpm\": raw.reviews_per_month,\n", " \"rev12\": raw.review_count_12m, \"minterm\": raw.min_term_days, \"nfac\": nfac, \"score\": raw.score_overall,\n", " \"value\": value_gap, \"ws\": raw.workstations}.items():\n", " rel[f\"rel_{c}_vs_comps\"] = s - s.groupby(comp).transform(\"mean\")\n", " metro_comp = rawx.groupby([\"metro\", \"access_type\", \"seats_b\"], dropna=False, sort=False).ngroup().to_numpy()\n", " rel[\"rel_load_vs_metro_comps\"] = raw.booked_load - raw.booked_load.groupby(metro_comp).transform(\"mean\")\n", "\n", " df = df.drop(columns=[\"facilities\", \"space_id\"])\n", " CAT_COLS = list(df.select_dtypes(exclude=\"number\").columns)\n", " for c in CAT_COLS:\n", " df[c] = df[c].astype(\"category\").cat.codes # -1 = missing (LightGBM treats negative categories as missing)\n", " base = pd.concat([df, fac, geo, prof, rel], axis=1)\n", "\n", " # ---- target features, out-of-fold on grouped folds ----\n", " TE_GROUPS = {\n", " \"zone\": [\"zone\"], \"boro\": [\"borough\"], \"sc\": [\"space_class\"], \"metro_sc\": [\"metro\", \"space_class\"],\n", " \"zone_acc\": [\"zone\", \"access_type\"], \"zone_seats\": [\"zone\", \"max_seats\"], \"zone_sc\": [\"zone\", \"space_class\"],\n", " \"geo\": [\"geo_lat\", \"geo_lon\"], \"metro_acc_seats\": [\"metro\", \"access_type\", \"max_seats\"],\n", " \"zone_acc_seats\": [\"zone\", \"access_type\", \"seats_b\"],\n", " \"zone_acc_seats_ur\": [\"zone\", \"access_type\", \"seats_b\", \"utility_rooms\"],\n", " \"boro_acc_seats\": [\"borough\", \"access_type\", \"seats_b\"], \"metro_sc_seats\": [\"metro\", \"space_class\", \"seats_b\"],\n", " \"bld_acc\": [\"geo_lat\", \"geo_lon\", \"access_type\"], \"metro_acc_rooms\": [\"metro\", \"access_type\", \"zones\", \"utility_rooms\"],\n", " }\n", " RES_GROUPS = [\"zone\", \"geo\", \"zone_sc\", \"sc\"]\n", " gid = {k: rawx[c].groupby(c, sort=False, dropna=False).ngroup().to_numpy() for k, c in TE_GROUPS.items()}\n", " base_ids = raw[[\"access_type\", \"max_seats\"]].groupby([\"access_type\", \"max_seats\"], dropna=False).ngroup().to_numpy()\n", " metro_id = pd.factorize(raw.metro)[0]\n", " coords = raw[[\"geo_lat\", \"geo_lon\"]].to_numpy()\n", " prior = np.bincount(y, minlength=K) / len(y)\n", " tiers = np.arange(1, K + 1)\n", " space = np.column_stack([(lat - lat.mean()) / lat.std(), (lon - lon.mean()) / lon.std(), np.log1p(raw.max_seats) * 0.3,\n", " raw.zones.fillna(1) * 0.05, raw.utility_rooms.fillna(1) * 0.05, nfac * 0.002]).astype(np.float32)\n", " acc = raw.access_type.astype(str).to_numpy()\n", "\n", " def target_features(fit, apply):\n", " cols = {}\n", " for name, g_ in gid.items(): # smoothed tier distribution of each group\n", " cnt = np.zeros((g_.max() + 1, K)); np.add.at(cnt, (g_[fit], y[fit]), 1)\n", " tot = cnt.sum(1)\n", " p = ((cnt + 20 * prior) / (tot[:, None] + 20))[g_[apply]]\n", " for j in range(K):\n", " cols[f\"te_{name}_p{j + 1}\"] = p[:, j]\n", " cols[f\"te_{name}_mean\"] = p @ tiers\n", " cols[f\"te_{name}_n\"] = np.log1p(tot[g_[apply]])\n", " yt = y[fit] + 1.0 # residuals vs. an (access type, seats) baseline\n", " nb = base_ids.max() + 1\n", " bm = (np.bincount(base_ids[fit], yt, nb) + 20 * yt.mean()) / (np.bincount(base_ids[fit], minlength=nb) + 20)\n", " res = yt - bm[base_ids[fit]]\n", " for name in RES_GROUPS:\n", " g_ = gid[name]; n = g_.max() + 1\n", " c_ = np.bincount(g_[fit], minlength=n)[g_[apply]] + 10\n", " mres = np.bincount(g_[fit], res, n)[g_[apply]] / c_\n", " cols[f\"res_{name}_mean\"] = mres\n", " cols[f\"res_{name}_pred\"] = bm[base_ids[apply]] + mres\n", " pts, inv = np.unique(coords[fit], axis=0, return_inverse=True) # tiers of nearest locations\n", " cnt = np.zeros((len(pts), K)); np.add.at(cnt, (inv.ravel(), y[fit]), 1)\n", " dist, idx = NearestNeighbors(n_neighbors=50).fit(pts).kneighbors(coords[apply])\n", " for k in (1, 5, 20, 50):\n", " c_ = cnt[idx[:, :k]].sum(1)\n", " p = (c_ + 5 * prior) / (c_.sum(1)[:, None] + 5)\n", " for j in range(K):\n", " cols[f\"geo{k}_p{j + 1}\"] = p[:, j]\n", " cols[f\"geo{k}_mean\"] = p @ tiers\n", " cols[f\"geo{k}_dist\"] = dist[:, k - 1]\n", " for k in (10, 50): # comparable listings: nearest in (location, size) with the same access type\n", " for s in (\"mean\", \"p1\", \"p4\", \"dist\"):\n", " cols[f\"knn{k}_{s}\"] = np.full(len(apply), np.nan)\n", " for a in np.unique(acc):\n", " fa, aa = fit[acc[fit] == a], np.where(acc[apply] == a)[0]\n", " if len(fa) < 50 or len(aa) == 0:\n", " continue\n", " dist, idx = NearestNeighbors(n_neighbors=50).fit(space[fa]).kneighbors(space[apply[aa]])\n", " lab = y[fa][idx]\n", " for k in (10, 50):\n", " oh = np.stack([(lab[:, :k] == j).mean(1) for j in range(K)], 1)\n", " cols[f\"knn{k}_mean\"][aa] = oh @ tiers\n", " cols[f\"knn{k}_p1\"][aa], cols[f\"knn{k}_p4\"][aa] = oh[:, 0], oh[:, 3]\n", " cols[f\"knn{k}_dist\"][aa] = dist[:, k - 1]\n", " # H2: facility effects after removing the metro average (shrunk towards 0 for rare facilities)\n", " nm = metro_id.max() + 1\n", " mm = np.bincount(metro_id[fit], yt, nm) / np.maximum(np.bincount(metro_id[fit], minlength=nm), 1)\n", " r_ = yt - mm[metro_id[fit]]\n", " Ff = F_all[fit]\n", " dev = (Ff.T @ r_) / (np.asarray(Ff.sum(0)).ravel() + 50)\n", " Fa = F_all[apply]\n", " nf = np.asarray(Fa.sum(1)).ravel()\n", " has = nf > 0\n", " cols[\"facte_sum\"] = Fa @ dev\n", " cols[\"facte_mean\"] = np.where(has, cols[\"facte_sum\"] / np.maximum(nf, 1), np.nan)\n", " cols[\"facte_max\"] = np.where(has, Fa.multiply(dev + 10).max(1).toarray().ravel() - 10, np.nan)\n", " cols[\"facte_min\"] = np.where(has, 10 - Fa.multiply(10 - dev).max(1).toarray().ravel(), np.nan)\n", " cols[\"facte_n_premium\"] = Fa @ (dev > 0.05).astype(np.float32)\n", " cols[\"facte_n_budget\"] = Fa @ (dev < -0.05).astype(np.float32)\n", " return pd.DataFrame(cols)\n", "\n", " T_tr = pd.concat([target_features(f, v).set_index(v) for f, v in folds]).sort_index()\n", " T_te = target_features(np.arange(n_tr), np.arange(n_tr, len(raw)))\n", " X_df = pd.concat([base.iloc[:n_tr].reset_index(drop=True), T_tr], axis=1)\n", " Xt_df = pd.concat([base.iloc[n_tr:].reset_index(drop=True), T_te], axis=1)\n", " feat = dict(X=X_df.to_numpy(np.float32), Xt=Xt_df.to_numpy(np.float32), cols=np.array(X_df.columns, dtype=str),\n", " cat_cols=np.array(CAT_COLS, dtype=str), bld=bld)\n", " ck_save(f\"{TAG}_features\", **feat)\n", " del raw, rawx, df, base, X_df, Xt_df, T_tr, T_te, F_all\n", " prog.finish()\n", "else:\n", " prog.finish(from_checkpoint=True)\n", "\n", "X_full, Xt_full = feat[\"X\"], feat[\"Xt\"]\n", "ALL_COLS = list(feat[\"cols\"])\n", "bld = feat[\"bld\"]\n", "MT_tr = np.maximum(X_full[:, ALL_COLS.index(\"metro\")], 0).astype(int)\n", "MT_te = np.maximum(Xt_full[:, ALL_COLS.index(\"metro\")], 0).astype(int)\n", "print(\"all features:\", X_full.shape, \"| test:\", Xt_full.shape)" ] }, { "cell_type": "markdown", "id": "d065d01d", "metadata": { "id": "d065d01d" }, "source": [ "## 3. Ablation: do H1 (`rel_*`) and H2 (`facte_*`) help on grouped CV?\n", "Fast settings on 50% of operators and 3 folds (XGBoost on GPU, otherwise LightGBM).\n", "Each result is checkpointed. Differences under ~0.002 are within noise, and then the smaller set is kept." ] }, { "cell_type": "code", "execution_count": 6, "id": "af3c251a", "metadata": { "id": "af3c251a", "outputId": "c5623232-86ae-41e7-c157-51251bfb120b", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "▶ ablation of feature blocks [██░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2/21 (10%)]\n", "chosen feature set: v2 + H1 + H2\n", "✓ ablation of feature blocks: took 0.0 min | [████░░░░░░░░░░░░░░░░░░░░░░░░░░ 3/21 (14%)] | elapsed 4.2 min | ETA ~0 min + 18 steps not timed yet\n" ] } ], "source": [ "prog.start(\"ablation of feature blocks\")\n", "FEATURE_SETS = {\n", " \"v2\": [c for c in ALL_COLS if not c.startswith((\"rel_\", \"facte_\"))],\n", " \"v2 + H1 relative\": [c for c in ALL_COLS if not c.startswith(\"facte_\")],\n", " \"v2 + H2 facility\": [c for c in ALL_COLS if not c.startswith(\"rel_\")],\n", " \"v2 + H1 + H2\": ALL_COLS,\n", "}\n", "abl_rows = []\n", "if RUN_ABLATION:\n", " sub = (host[:n_tr] * 2654435761 % 1000) < 500\n", " for name, cols in FEATURE_SETS.items():\n", " idx = [ALL_COLS.index(c) for c in cols]\n", " scores = []\n", " for f in range(3):\n", " ckn = f\"{TAG}_abl_{name.replace(' ', '_').replace('+', 'p')}_{f}\"\n", " ck = ck_load(ckn)\n", " if ck is None:\n", " tr = np.where(sub & (fold_id != f))[0]\n", " va = np.where(sub & (fold_id == f))[0]\n", " if GPU:\n", " pa = dict(objective=\"multi:softprob\", num_class=K, eta=0.1, max_depth=8, min_child_weight=5, subsample=0.8,\n", " colsample_bytree=0.5, max_bin=128, tree_method=\"hist\", device=\"cuda\", eval_metric=\"mlogloss\", seed=SEED)\n", " dva = xgb.DMatrix(X_full[va][:, idx], label=y[va])\n", " b = xgb.train(pa, xgb.DMatrix(X_full[tr][:, idx], label=y[tr]), 3000, evals=[(dva, \"v\")],\n", " early_stopping_rounds=50, verbose_eval=False)\n", " p = b.predict(dva, iteration_range=(0, b.best_iteration + 1))\n", " else:\n", " pa = dict(objective=\"multiclass\", num_class=K, learning_rate=0.1, num_leaves=127, min_child_samples=40,\n", " feature_fraction=0.5, bagging_fraction=0.8, bagging_freq=1, max_bin=63, verbose=-1, seed=SEED, num_threads=N_THREADS)\n", " m = lgb.train(pa, lgb.Dataset(X_full[tr][:, idx], y[tr]), 3000, valid_sets=[lgb.Dataset(X_full[va][:, idx], y[va])],\n", " callbacks=[lgb.early_stopping(50, verbose=False)])\n", " p = m.predict(X_full[va][:, idx], num_iteration=m.best_iteration)\n", " ck = {\"f1\": np.array([fast_f1(y[va], p.argmax(1))])}\n", " ck_save(ckn, **ck)\n", " scores.append(float(ck[\"f1\"][0]))\n", " abl_rows.append({\"feature set\": name, \"n features\": len(cols), \"grouped CV macro F1\": round(np.mean(scores), 4)})\n", " print(abl_rows[-1], flush=True)\n", " abl = pd.DataFrame(abl_rows)\n", " best = abl[\"grouped CV macro F1\"].max()\n", " # simplest set within 0.002 of the best (fewer features = faster + less overfitting)\n", " FINAL_SET = abl[abl[\"grouped CV macro F1\"] >= best - 0.002].sort_values(\"n features\").iloc[0][\"feature set\"]\n", " display(abl)\n", "else:\n", " FINAL_SET = \"v2 + H1 + H2\"\n", "print(\"chosen feature set:\", FINAL_SET)\n", "SEL = [ALL_COLS.index(c) for c in FEATURE_SETS[FINAL_SET]]\n", "COLS = FEATURE_SETS[FINAL_SET]\n", "X, Xt = X_full[:, SEL], Xt_full[:, SEL]\n", "CAT_IDX = [COLS.index(c) for c in feat[\"cat_cols\"] if c in COLS]\n", "SET_TAG = FINAL_SET.replace(\" \", \"\").replace(\"+\", \"p\")\n", "prog.finish()" ] }, { "cell_type": "markdown", "id": "68da80f7", "metadata": { "id": "68da80f7" }, "source": [ "## 4. Level-1 models (H3), each fold checkpointed\n", "| model | output | why it's here |\n", "|---|---|---|\n", "| `lgb_multi` LightGBM multiclass | 4 probabilities | strong default for tabular data |\n", "| `lgb_reg` LightGBM regression on tier 1–4 | one score | uses the **order** of the tiers (H4) |\n", "| `xgb` XGBoost multiclass | 4 probabilities | a different boosting implementation, so its mistakes differ |\n", "| `xgb_reg` XGBoost regression on tier 1–4 | one score | a second ordered model, on GPU |\n", "| `cat` CatBoost multiclass | 4 probabilities | symmetric trees, strong regularisation, a third opinion |\n", "| `lgb_ord` (optional) | 4 probabilities | v1's ordinal model: 3 binary models P(tier > k) |\n", "\n", "Every model uses early stopping on its validation fold; test predictions are averaged over the 5 fold models." ] }, { "cell_type": "code", "execution_count": 7, "id": "079e2501", "metadata": { "id": "079e2501", "outputId": "22596fe0-d22a-46cf-c41e-ccbdbc360aac", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "LightGBM device: cpu | XGBoost device: cuda | CatBoost: GPU\n", "\n", "▶ xgb: fold 1/5 [████░░░░░░░░░░░░░░░░░░░░░░░░░░ 3/21 (14%)]\n", "[0]\tvalid-mlogloss:1.34464\n", "[250]\tvalid-mlogloss:0.68007\n", "[500]\tvalid-mlogloss:0.66389\n", "[750]\tvalid-mlogloss:0.65959\n", "[1000]\tvalid-mlogloss:0.65905\n", "[1048]\tvalid-mlogloss:0.65932\n", " best iteration: 948\n", "✓ xgb: fold 1/5: took 1.8 min | [█████░░░░░░░░░░░░░░░░░░░░░░░░░ 4/21 (19%)] | elapsed 6.0 min | ETA ~7 min + 13 steps not timed yet\n", " fold macro F1: 0.7135\n", "\n", "▶ xgb: fold 2/5 [█████░░░░░░░░░░░░░░░░░░░░░░░░░ 4/21 (19%)]\n", "[0]\tvalid-mlogloss:1.34317\n", "[250]\tvalid-mlogloss:0.67086\n", "[500]\tvalid-mlogloss:0.65344\n", "[750]\tvalid-mlogloss:0.64839\n", "[992]\tvalid-mlogloss:0.64763\n", " best iteration: 892\n", "✓ xgb: fold 2/5: took 1.8 min | [███████░░░░░░░░░░░░░░░░░░░░░░░ 5/21 (24%)] | elapsed 7.8 min | ETA ~5 min + 13 steps not timed yet\n", " fold macro F1: 0.7140\n", "\n", "▶ xgb: fold 3/5 [███████░░░░░░░░░░░░░░░░░░░░░░░ 5/21 (24%)]\n", "[0]\tvalid-mlogloss:1.34376\n", "[250]\tvalid-mlogloss:0.67625\n", "[500]\tvalid-mlogloss:0.65819\n", "[750]\tvalid-mlogloss:0.65356\n", "[1000]\tvalid-mlogloss:0.65274\n", "[1015]\tvalid-mlogloss:0.65285\n", " best iteration: 915\n", "✓ xgb: fold 3/5: took 1.8 min | [████████░░░░░░░░░░░░░░░░░░░░░░ 6/21 (29%)] | elapsed 9.6 min | ETA ~4 min + 13 steps not timed yet\n", " fold macro F1: 0.7160\n", "\n", "▶ xgb: fold 4/5 [████████░░░░░░░░░░░░░░░░░░░░░░ 6/21 (29%)]\n", "[0]\tvalid-mlogloss:1.34425\n", "[250]\tvalid-mlogloss:0.66630\n", "[500]\tvalid-mlogloss:0.64786\n", "[750]\tvalid-mlogloss:0.64293\n", "[1000]\tvalid-mlogloss:0.64131\n", "[1118]\tvalid-mlogloss:0.64156\n", " best iteration: 1018\n", "✓ xgb: fold 4/5: took 2.0 min | [██████████░░░░░░░░░░░░░░░░░░░░ 7/21 (33%)] | elapsed 11.6 min | ETA ~2 min + 13 steps not timed yet\n", " fold macro F1: 0.7199\n", "\n", "▶ xgb: fold 5/5 [██████████░░░░░░░░░░░░░░░░░░░░ 7/21 (33%)]\n", "[0]\tvalid-mlogloss:1.34550\n", "[250]\tvalid-mlogloss:0.68670\n", "[500]\tvalid-mlogloss:0.66952\n", "[750]\tvalid-mlogloss:0.66484\n", "[1000]\tvalid-mlogloss:0.66373\n", "[1025]\tvalid-mlogloss:0.66365\n", " best iteration: 925\n", "✓ xgb: fold 5/5: took 1.9 min | [███████████░░░░░░░░░░░░░░░░░░░ 8/21 (38%)] | elapsed 13.5 min | ETA ~0 min + 13 steps not timed yet\n", " fold macro F1: 0.7106\n", "\n", "▶ xgb_reg: fold 1/5 [███████████░░░░░░░░░░░░░░░░░░░ 8/21 (38%)]\n", "[0]\tvalid-rmse:1.07717\n", "[250]\tvalid-rmse:0.53944\n", "[500]\tvalid-rmse:0.53165\n", "[750]\tvalid-rmse:0.52869\n", "[1000]\tvalid-rmse:0.52744\n", "[1250]\tvalid-rmse:0.52697\n", "[1474]\tvalid-rmse:0.52669\n", " best iteration: 1374\n", "✓ xgb_reg: fold 1/5: took 0.7 min | [████████████░░░░░░░░░░░░░░░░░░ 9/21 (43%)] | elapsed 14.2 min | ETA ~3 min + 8 steps not timed yet\n", "\n", "▶ xgb_reg: fold 2/5 [████████████░░░░░░░░░░░░░░░░░░ 9/21 (43%)]\n", "[0]\tvalid-rmse:1.07141\n", "[250]\tvalid-rmse:0.53312\n", "[500]\tvalid-rmse:0.52613\n", "[750]\tvalid-rmse:0.52336\n", "[1000]\tvalid-rmse:0.52185\n", "[1250]\tvalid-rmse:0.52118\n", "[1500]\tvalid-rmse:0.52079\n", "[1620]\tvalid-rmse:0.52077\n", " best iteration: 1520\n", "✓ xgb_reg: fold 2/5: took 0.8 min | [██████████████░░░░░░░░░░░░░░░░ 10/21 (48%)] | elapsed 15.0 min | ETA ~2 min + 8 steps not timed yet\n", "\n", "▶ xgb_reg: fold 3/5 [██████████████░░░░░░░░░░░░░░░░ 10/21 (48%)]\n", "[0]\tvalid-rmse:1.07403\n", "[250]\tvalid-rmse:0.53195\n", "[500]\tvalid-rmse:0.52512\n", "[750]\tvalid-rmse:0.52253\n", "[1000]\tvalid-rmse:0.52124\n", "[1250]\tvalid-rmse:0.52051\n", "[1500]\tvalid-rmse:0.52004\n", "[1750]\tvalid-rmse:0.51983\n", "[1791]\tvalid-rmse:0.51982\n", " best iteration: 1691\n", "✓ xgb_reg: fold 3/5: took 0.8 min | [███████████████░░░░░░░░░░░░░░░ 11/21 (52%)] | elapsed 15.8 min | ETA ~2 min + 8 steps not timed yet\n", "\n", "▶ xgb_reg: fold 4/5 [███████████████░░░░░░░░░░░░░░░ 11/21 (52%)]\n", "[0]\tvalid-rmse:1.08347\n", "[250]\tvalid-rmse:0.53225\n", "[500]\tvalid-rmse:0.52568\n", "[750]\tvalid-rmse:0.52324\n", "[1000]\tvalid-rmse:0.52239\n", "[1250]\tvalid-rmse:0.52193\n", "[1500]\tvalid-rmse:0.52150\n", "[1750]\tvalid-rmse:0.52143\n", "[1811]\tvalid-rmse:0.52156\n", " best iteration: 1711\n", "✓ xgb_reg: fold 4/5: took 0.8 min | [█████████████████░░░░░░░░░░░░░ 12/21 (57%)] | elapsed 16.7 min | ETA ~1 min + 8 steps not timed yet\n", "\n", "▶ xgb_reg: fold 5/5 [█████████████████░░░░░░░░░░░░░ 12/21 (57%)]\n", "[0]\tvalid-rmse:1.05853\n", "[250]\tvalid-rmse:0.54156\n", "[500]\tvalid-rmse:0.53495\n", "[750]\tvalid-rmse:0.53234\n", "[1000]\tvalid-rmse:0.53085\n", "[1250]\tvalid-rmse:0.53024\n", "[1500]\tvalid-rmse:0.53005\n", "[1750]\tvalid-rmse:0.52972\n", "[2000]\tvalid-rmse:0.52947\n", "[2144]\tvalid-rmse:0.52952\n", " best iteration: 2044\n", "✓ xgb_reg: fold 5/5: took 1.0 min | [██████████████████░░░░░░░░░░░░ 13/21 (62%)] | elapsed 17.6 min | ETA ~0 min + 8 steps not timed yet\n", "\n", "▶ cat: fold 1/5 [██████████████████░░░░░░░░░░░░ 13/21 (62%)]\n", "0:\tlearn: 1.3104611\ttest: 1.3108248\tbest: 1.3108248 (0)\ttotal: 342ms\tremaining: 34m 12s\n", "250:\tlearn: 0.6315131\ttest: 0.6943245\tbest: 0.6943245 (250)\ttotal: 10.5s\tremaining: 4m 1s\n", "500:\tlearn: 0.5764154\ttest: 0.6770451\tbest: 0.6770451 (500)\ttotal: 20.5s\tremaining: 3m 44s\n", "750:\tlearn: 0.5394861\ttest: 0.6696765\tbest: 0.6696765 (750)\ttotal: 30.2s\tremaining: 3m 30s\n", "1000:\tlearn: 0.5079702\ttest: 0.6657385\tbest: 0.6657385 (1000)\ttotal: 40s\tremaining: 3m 19s\n", "1250:\tlearn: 0.4808617\ttest: 0.6630347\tbest: 0.6630347 (1250)\ttotal: 49.8s\tremaining: 3m 8s\n", "1500:\tlearn: 0.4562570\ttest: 0.6617513\tbest: 0.6617412 (1496)\ttotal: 59.4s\tremaining: 2m 58s\n", "1750:\tlearn: 0.4343020\ttest: 0.6606891\tbest: 0.6606179 (1746)\ttotal: 1m 9s\tremaining: 2m 47s\n", "2000:\tlearn: 0.4141874\ttest: 0.6597028\tbest: 0.6596745 (1989)\ttotal: 1m 18s\tremaining: 2m 37s\n", "2250:\tlearn: 0.3948419\ttest: 0.6591688\tbest: 0.6591688 (2250)\ttotal: 1m 28s\tremaining: 2m 27s\n", "bestTest = 0.6588616286\n", "bestIteration = 2344\n", "Shrink model to first 2345 iterations.\n", "✓ cat: fold 1/5: took 1.8 min | [████████████████████░░░░░░░░░░ 14/21 (67%)] | elapsed 19.4 min | ETA ~7 min + 3 steps not timed yet\n", " fold macro F1: 0.7113\n", "\n", "▶ cat: fold 2/5 [████████████████████░░░░░░░░░░ 14/21 (67%)]\n", "0:\tlearn: 1.3106888\ttest: 1.3105116\tbest: 1.3105116 (0)\ttotal: 83ms\tremaining: 8m 18s\n", "250:\tlearn: 0.6327457\ttest: 0.6862685\tbest: 0.6862685 (250)\ttotal: 10.3s\tremaining: 3m 55s\n", "500:\tlearn: 0.5791844\ttest: 0.6697529\tbest: 0.6697529 (500)\ttotal: 20.1s\tremaining: 3m 41s\n", "750:\tlearn: 0.5413548\ttest: 0.6622135\tbest: 0.6622062 (749)\ttotal: 30s\tremaining: 3m 29s\n", "1000:\tlearn: 0.5105216\ttest: 0.6578358\tbest: 0.6578358 (1000)\ttotal: 39.8s\tremaining: 3m 18s\n", "1250:\tlearn: 0.4837006\ttest: 0.6553028\tbest: 0.6553028 (1250)\ttotal: 49.4s\tremaining: 3m 7s\n", "1500:\tlearn: 0.4582976\ttest: 0.6532074\tbest: 0.6532062 (1497)\ttotal: 59.1s\tremaining: 2m 57s\n", "1750:\tlearn: 0.4351029\ttest: 0.6515225\tbest: 0.6515130 (1741)\ttotal: 1m 8s\tremaining: 2m 47s\n", "2000:\tlearn: 0.4146589\ttest: 0.6507526\tbest: 0.6507141 (1997)\ttotal: 1m 18s\tremaining: 2m 36s\n", "2250:\tlearn: 0.3947096\ttest: 0.6502793\tbest: 0.6500409 (2203)\ttotal: 1m 28s\tremaining: 2m 26s\n", "bestTest = 0.6499261437\n", "bestIteration = 2317\n", "Shrink model to first 2318 iterations.\n", "✓ cat: fold 2/5: took 1.7 min | [█████████████████████░░░░░░░░░ 15/21 (71%)] | elapsed 21.2 min | ETA ~5 min + 3 steps not timed yet\n", " fold macro F1: 0.7113\n", "\n", "▶ cat: fold 3/5 [█████████████████████░░░░░░░░░ 15/21 (71%)]\n", "0:\tlearn: 1.3096588\ttest: 1.3087899\tbest: 1.3087899 (0)\ttotal: 89.6ms\tremaining: 8m 57s\n", "250:\tlearn: 0.6329501\ttest: 0.6868962\tbest: 0.6868962 (250)\ttotal: 10.3s\tremaining: 3m 55s\n", "500:\tlearn: 0.5793761\ttest: 0.6717701\tbest: 0.6717701 (500)\ttotal: 20.1s\tremaining: 3m 41s\n", "750:\tlearn: 0.5400870\ttest: 0.6645552\tbest: 0.6645552 (750)\ttotal: 30s\tremaining: 3m 29s\n", "1000:\tlearn: 0.5082666\ttest: 0.6605761\tbest: 0.6605537 (999)\ttotal: 39.9s\tremaining: 3m 19s\n", "1250:\tlearn: 0.4806219\ttest: 0.6581867\tbest: 0.6581867 (1250)\ttotal: 49.7s\tremaining: 3m 8s\n", "1500:\tlearn: 0.4560681\ttest: 0.6564406\tbest: 0.6564333 (1482)\ttotal: 59.3s\tremaining: 2m 57s\n", "1750:\tlearn: 0.4327583\ttest: 0.6551005\tbest: 0.6551005 (1750)\ttotal: 1m 8s\tremaining: 2m 47s\n", "2000:\tlearn: 0.4121772\ttest: 0.6543631\tbest: 0.6543631 (2000)\ttotal: 1m 18s\tremaining: 2m 37s\n", "bestTest = 0.6538370685\n", "bestIteration = 2126\n", "Shrink model to first 2127 iterations.\n", "✓ cat: fold 3/5: took 1.6 min | [██████████████████████░░░░░░░░ 16/21 (76%)] | elapsed 22.8 min | ETA ~3 min + 3 steps not timed yet\n", " fold macro F1: 0.7130\n", "\n", "▶ cat: fold 4/5 [██████████████████████░░░░░░░░ 16/21 (76%)]\n", "0:\tlearn: 1.3101796\ttest: 1.3103196\tbest: 1.3103196 (0)\ttotal: 81.3ms\tremaining: 8m 7s\n", "250:\tlearn: 0.6347616\ttest: 0.6812992\tbest: 0.6812992 (250)\ttotal: 10.3s\tremaining: 3m 55s\n", "500:\tlearn: 0.5802238\ttest: 0.6640989\tbest: 0.6640989 (500)\ttotal: 20.1s\tremaining: 3m 40s\n", "750:\tlearn: 0.5424698\ttest: 0.6569316\tbest: 0.6569316 (750)\ttotal: 29.9s\tremaining: 3m 28s\n", "1000:\tlearn: 0.5109335\ttest: 0.6526652\tbest: 0.6526652 (1000)\ttotal: 39.6s\tremaining: 3m 17s\n", "1250:\tlearn: 0.4825299\ttest: 0.6500226\tbest: 0.6500226 (1250)\ttotal: 49.5s\tremaining: 3m 7s\n", "1500:\tlearn: 0.4575551\ttest: 0.6482046\tbest: 0.6480990 (1474)\ttotal: 59.2s\tremaining: 2m 57s\n", "1750:\tlearn: 0.4350413\ttest: 0.6461287\tbest: 0.6461287 (1750)\ttotal: 1m 8s\tremaining: 2m 46s\n", "2000:\tlearn: 0.4146811\ttest: 0.6451048\tbest: 0.6450542 (1997)\ttotal: 1m 18s\tremaining: 2m 36s\n", "2250:\tlearn: 0.3949343\ttest: 0.6443657\tbest: 0.6443068 (2247)\ttotal: 1m 28s\tremaining: 2m 26s\n", "bestTest = 0.6443067982\n", "bestIteration = 2247\n", "Shrink model to first 2248 iterations.\n", "✓ cat: fold 4/5: took 1.7 min | [████████████████████████░░░░░░ 17/21 (81%)] | elapsed 24.5 min | ETA ~2 min + 3 steps not timed yet\n", " fold macro F1: 0.7160\n", "\n", "▶ cat: fold 5/5 [████████████████████████░░░░░░ 17/21 (81%)]\n", "0:\tlearn: 1.3093647\ttest: 1.3122034\tbest: 1.3122034 (0)\ttotal: 91.8ms\tremaining: 9m 10s\n", "250:\tlearn: 0.6286168\ttest: 0.7022837\tbest: 0.7022837 (250)\ttotal: 10.3s\tremaining: 3m 55s\n", "500:\tlearn: 0.5762297\ttest: 0.6866022\tbest: 0.6866022 (500)\ttotal: 20.1s\tremaining: 3m 40s\n", "750:\tlearn: 0.5384355\ttest: 0.6791796\tbest: 0.6791789 (745)\ttotal: 29.9s\tremaining: 3m 28s\n", "1000:\tlearn: 0.5079056\ttest: 0.6749869\tbest: 0.6749869 (1000)\ttotal: 39.6s\tremaining: 3m 17s\n", "1250:\tlearn: 0.4810091\ttest: 0.6723687\tbest: 0.6723687 (1250)\ttotal: 49.3s\tremaining: 3m 7s\n", "1500:\tlearn: 0.4565549\ttest: 0.6701565\tbest: 0.6701030 (1490)\ttotal: 58.9s\tremaining: 2m 56s\n", "1750:\tlearn: 0.4342202\ttest: 0.6684288\tbest: 0.6684288 (1750)\ttotal: 1m 8s\tremaining: 2m 46s\n", "2000:\tlearn: 0.4134261\ttest: 0.6676836\tbest: 0.6676835 (1999)\ttotal: 1m 18s\tremaining: 2m 36s\n", "2250:\tlearn: 0.3946886\ttest: 0.6668935\tbest: 0.6668866 (2249)\ttotal: 1m 27s\tremaining: 2m 26s\n", "2500:\tlearn: 0.3767622\ttest: 0.6665843\tbest: 0.6665651 (2494)\ttotal: 1m 37s\tremaining: 2m 16s\n", "2750:\tlearn: 0.3600809\ttest: 0.6664604\tbest: 0.6661767 (2697)\ttotal: 1m 47s\tremaining: 2m 6s\n", "bestTest = 0.6661766598\n", "bestIteration = 2697\n", "Shrink model to first 2698 iterations.\n", "✓ cat: fold 5/5: took 2.0 min | [█████████████████████████░░░░░ 18/21 (86%)] | elapsed 26.5 min | ETA ~0 min + 3 steps not timed yet\n", " fold macro F1: 0.7114\n", "xgb grouped CV macro F1 (plain argmax): 0.7148\n", "cat grouped CV macro F1 (plain argmax): 0.7127\n" ] } ], "source": [ "def pick_lgb_device():\n", " \"\"\"LightGBM's GPU mode (OpenCL) isn't always available or faster: time 30 rounds on 60k rows on each device, keep the faster.\"\"\"\n", " if not GPU or not any(m.startswith(\"lgb\") for m in MODELS):\n", " return \"cpu\"\n", " rows_ = np.random.RandomState(0).choice(n_tr, min(60000, n_tr), replace=False)\n", " times = {}\n", " for dev in (\"cpu\", \"gpu\"):\n", " try:\n", " pa = dict(objective=\"multiclass\", num_class=K, learning_rate=0.05, num_leaves=127, max_bin=63, feature_fraction=0.5,\n", " verbose=-1, num_threads=N_THREADS, device_type=dev)\n", " t0 = time.time()\n", " lgb.train(pa, lgb.Dataset(X[rows_], y[rows_], categorical_feature=CAT_IDX), 30)\n", " times[dev] = time.time() - t0\n", " except Exception as e:\n", " print(f\"LightGBM on {dev} not available: {str(e)[:150]}\")\n", " print(\"LightGBM speed test (30 rounds, 60k rows):\", {k: f\"{v:.1f}s\" for k, v in times.items()})\n", " return min(times, key=times.get) if times else \"cpu\"\n", "\n", "\n", "LGB_DEVICE = pick_lgb_device()\n", "print(\"LightGBM device:\", LGB_DEVICE, \"| XGBoost device:\", \"cuda\" if GPU else \"cpu\", \"| CatBoost:\", \"GPU\" if GPU else \"CPU\")\n", "LGB_BASE = dict(learning_rate=0.05, num_leaves=127, min_child_samples=40, feature_fraction=0.5, bagging_fraction=0.8,\n", " bagging_freq=1, lambda_l2=1.0, max_bin=63, verbose=-1, seed=SEED, num_threads=N_THREADS, device_type=LGB_DEVICE)\n", "XGB_PARAMS = dict(objective=\"multi:softprob\", num_class=K, eta=0.05, max_depth=8, min_child_weight=5, subsample=0.8,\n", " colsample_bytree=0.5, max_bin=128, tree_method=\"hist\", eval_metric=\"mlogloss\", seed=SEED,\n", " device=\"cuda\" if GPU else \"cpu\", nthread=N_THREADS)\n", "CAT_PARAMS = dict(loss_function=\"MultiClass\", iterations=MAX_ROUNDS, learning_rate=0.08, depth=8, border_count=128,\n", " random_seed=SEED, task_type=\"GPU\" if GPU else \"CPU\", thread_count=N_THREADS, od_type=\"Iter\", od_wait=100)\n", "\n", "\n", "def train_lgb(params, Xtr, ytr, Xva, yva, Xte):\n", " dtr = lgb.Dataset(Xtr, ytr, categorical_feature=CAT_IDX)\n", " dva = lgb.Dataset(Xva, yva, categorical_feature=CAT_IDX, reference=dtr)\n", " m = lgb.train(params, dtr, MAX_ROUNDS, valid_sets=[dva],\n", " callbacks=[lgb.early_stopping(100, verbose=False), lgb.log_evaluation(LOG_EVERY)])\n", " print(f\" best iteration: {m.best_iteration}\")\n", " return m.predict(Xva, num_iteration=m.best_iteration), m.predict(Xte, num_iteration=m.best_iteration)\n", "\n", "\n", "def cumulative_to_classes(p_above):\n", " p_above = np.minimum.accumulate(p_above, axis=1)\n", " edges = np.column_stack([np.ones(len(p_above)), p_above, np.zeros(len(p_above))])\n", " return np.clip(edges[:, :-1] - edges[:, 1:], 1e-6, 1)\n", "\n", "\n", "def fit_lgb_multi(Xtr, ytr, Xva, yva, Xte):\n", " return train_lgb({**LGB_BASE, \"objective\": \"multiclass\", \"num_class\": K}, Xtr, ytr, Xva, yva, Xte)\n", "\n", "\n", "def fit_lgb_reg(Xtr, ytr, Xva, yva, Xte):\n", " pv, pt = train_lgb({**LGB_BASE, \"objective\": \"regression\"}, Xtr, ytr + 1.0, Xva, yva + 1.0, Xte)\n", " return pv[:, None], pt[:, None]\n", "\n", "\n", "def fit_lgb_ord(Xtr, ytr, Xva, yva, Xte):\n", " av, at = [], []\n", " for k in range(K - 1):\n", " v, t = train_lgb({**LGB_BASE, \"objective\": \"binary\"}, Xtr, (ytr > k).astype(int), Xva, (yva > k).astype(int), Xte)\n", " av.append(v); at.append(t)\n", " return cumulative_to_classes(np.column_stack(av)), cumulative_to_classes(np.column_stack(at))\n", "\n", "\n", "def fit_xgb(Xtr, ytr, Xva, yva, Xte):\n", " dtr, dva = xgb.DMatrix(Xtr, label=ytr), xgb.DMatrix(Xva, label=yva)\n", " b = xgb.train(XGB_PARAMS, dtr, MAX_ROUNDS, evals=[(dva, \"valid\")], early_stopping_rounds=100, verbose_eval=LOG_EVERY)\n", " it = (0, b.best_iteration + 1)\n", " print(f\" best iteration: {b.best_iteration}\")\n", " return b.predict(dva, iteration_range=it), b.predict(xgb.DMatrix(Xte), iteration_range=it)\n", "\n", "\n", "XGB_REG = {**{k: v for k, v in XGB_PARAMS.items() if k not in (\"objective\", \"num_class\", \"eval_metric\")},\n", " \"objective\": \"reg:squarederror\", \"eval_metric\": \"rmse\"}\n", "\n", "\n", "def fit_xgb_reg(Xtr, ytr, Xva, yva, Xte):\n", " dtr, dva = xgb.DMatrix(Xtr, label=ytr + 1.0), xgb.DMatrix(Xva, label=yva + 1.0)\n", " b = xgb.train(XGB_REG, dtr, MAX_ROUNDS, evals=[(dva, \"valid\")], early_stopping_rounds=100, verbose_eval=LOG_EVERY)\n", " it = (0, b.best_iteration + 1)\n", " print(f\" best iteration: {b.best_iteration}\")\n", " return b.predict(dva, iteration_range=it)[:, None], b.predict(xgb.DMatrix(Xte), iteration_range=it)[:, None]\n", "\n", "\n", "def fit_cat(Xtr, ytr, Xva, yva, Xte):\n", " m = CatBoostClassifier(**CAT_PARAMS)\n", " m.fit(Xtr, ytr, eval_set=(Xva, yva), verbose=LOG_EVERY, use_best_model=True)\n", " return m.predict_proba(Xva), m.predict_proba(Xte)\n", "\n", "\n", "TRAINERS = {\"lgb_multi\": fit_lgb_multi, \"lgb_reg\": fit_lgb_reg, \"lgb_ord\": fit_lgb_ord, \"xgb\": fit_xgb, \"xgb_reg\": fit_xgb_reg,\n", " \"cat\": fit_cat}\n", "OOF, TEST = {}, {}\n", "for key in MODELS:\n", " for f, (tr, va) in enumerate(folds):\n", " name = f\"{TAG}_{SET_TAG}_{key}_fold{f}\"\n", " prog.start(f\"{key}: fold {f + 1}/{FOLDS}\")\n", " ck = ck_load(name)\n", " if ck is None:\n", " pv, pt = TRAINERS[key](X[tr], y[tr], X[va], y[va], Xt)\n", " ck = {\"oof\": np.asarray(pv, np.float32), \"test\": np.asarray(pt, np.float32)}\n", " ck_save(name, **ck)\n", " prog.finish()\n", " else:\n", " prog.finish(from_checkpoint=True)\n", " if key not in OOF:\n", " OOF[key] = np.zeros((n_tr, ck[\"oof\"].shape[1]), np.float32)\n", " TEST[key] = np.zeros((len(Xt), ck[\"test\"].shape[1]), np.float32)\n", " OOF[key][va] = ck[\"oof\"]\n", " TEST[key] += ck[\"test\"] / FOLDS\n", " if ck[\"oof\"].shape[1] == K:\n", " print(f\" fold macro F1: {fast_f1(y[va], ck['oof'].argmax(1)):.4f}\")\n", "for key in OOF:\n", " if OOF[key].shape[1] == K:\n", " print(f\"{key:10s} grouped CV macro F1 (plain argmax): {fast_f1(y, OOF[key].argmax(1)):.4f}\")" ] }, { "cell_type": "markdown", "id": "e927a7d2", "metadata": { "id": "e927a7d2" }, "source": [ "## 5. Level-2 stacker: operator smoothing (H5)\n", "For each listing, we average the level-1 predictions of the **same operator's other listings** (leave-one-out),\n", "and of other listings at the **same coordinates**. A small LightGBM learns how much to trust these, per metro and access type.\n", "With grouped folds every operator sits entirely inside one validation fold, so this is measured honestly." ] }, { "cell_type": "code", "execution_count": 8, "id": "98858464", "metadata": { "id": "98858464", "outputId": "9857fd1f-05a2-4982-afa5-3b35d6ae1de1", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "▶ level-2 operator stacker [█████████████████████████░░░░░ 18/21 (86%)]\n", "✓ level-2 operator stacker: took 2.7 min | [███████████████████████████░░░ 19/21 (90%)] | elapsed 29.1 min | ETA ~0 min + 2 steps not timed yet\n", "stack grouped CV macro F1 (plain argmax): 0.7204\n" ] } ], "source": [ "prog.start(\"level-2 operator stacker\")\n", "PROB_MODELS = [k for k in OOF if OOF[k].shape[1] == K]\n", "REG_MODELS = [k for k in OOF if OOF[k].shape[1] == 1]\n", "\n", "\n", "def loo_mean(P, key):\n", " _, inv = np.unique(key, return_inverse=True)\n", " S = np.zeros((inv.max() + 1, P.shape[1])); np.add.at(S, inv, P)\n", " other = np.bincount(inv)[inv] - 1\n", " M = (S[inv] - P) / np.maximum(other, 1)[:, None]\n", " M[other == 0] = np.nan\n", " return M, other\n", "\n", "\n", "def stack_inputs(store, host_ids, bld_ids, Xsrc):\n", " parts = [store[k] for k in PROB_MODELS + REG_MODELS]\n", " avg = np.mean([store[k] for k in PROB_MODELS], axis=0)\n", " summary = np.column_stack([avg] + [store[k] for k in REG_MODELS])\n", " H, hn = loo_mean(summary, host_ids)\n", " B, bn = loo_mean(summary, bld_ids)\n", " ctx = Xsrc[:, [ALL_COLS.index(c) for c in (\"metro\", \"access_type\", \"max_seats\")]]\n", " return np.column_stack(parts + [H, hn, np.log1p(hn), B, bn, ctx]).astype(np.float32)\n", "\n", "\n", "Z = stack_inputs(OOF, host[:n_tr], bld[:n_tr] * FOLDS + fold_id, X_full) # buildings: only rows predicted together\n", "Zt = stack_inputs(TEST, host[n_tr:], bld[n_tr:], Xt_full)\n", "stack_name = f\"{TAG}_{SET_TAG}_stack_{'-'.join(MODELS)}\"\n", "ck = ck_load(stack_name)\n", "if ck is None:\n", " P2 = dict(objective=\"multiclass\", num_class=K, learning_rate=0.05, num_leaves=31, min_child_samples=200,\n", " feature_fraction=0.9, bagging_fraction=0.8, bagging_freq=1, verbose=-1, seed=SEED, num_threads=N_THREADS)\n", " oof2, test2 = np.zeros((n_tr, K)), np.zeros((len(Zt), K))\n", " for f, v in folds:\n", " m = lgb.train(P2, lgb.Dataset(Z[f], y[f]), 3000, valid_sets=[lgb.Dataset(Z[v], y[v])],\n", " callbacks=[lgb.early_stopping(100, verbose=False)])\n", " oof2[v] = m.predict(Z[v], num_iteration=m.best_iteration)\n", " test2 += m.predict(Zt, num_iteration=m.best_iteration) / FOLDS\n", " ck = {\"oof\": oof2, \"test\": test2}\n", " ck_save(stack_name, **ck)\n", " prog.finish()\n", "else:\n", " prog.finish(from_checkpoint=True)\n", "OOF[\"stack\"], TEST[\"stack\"] = ck[\"oof\"], ck[\"test\"]\n", "print(f\"stack grouped CV macro F1 (plain argmax): {fast_f1(y, OOF['stack'].argmax(1)):.4f}\")" ] }, { "cell_type": "markdown", "id": "274dff62", "metadata": { "id": "274dff62" }, "source": [ "## 6. Scenarios: every combination, honestly scored\n", "**Outputs** (H3, H5): single models, a plain average, a weight-optimised average (weights minimise log-loss), probabilities mixed with the regression score, and the stacker.\n", "\n", "**Decoders** (how an output becomes a tier):\n", "- **A. class weights**: multiply the 4 probabilities by tuned weights and take the largest (v1's method),\n", "- **B. cut-points** (H4): expected tier Σ p·tier (or the regression score), cut at 3 tuned thresholds,\n", "- **C. per-metro cut-points** (H7): one set of thresholds per metro (tried on the 4 best B scenarios).\n", "\n", "**Scoring** (H6): every decoder is tuned with sample weights that make the train metro mix look like the test metro mix.\n", "It is then **cross-fitted**: tuned on 4 folds' out-of-fold predictions and scored on the 5th, so no scenario wins by tuning on its own answers.\n", "The main column, \"test-weighted\", estimates the score on the test set's metro mix. \"plain\" is the unweighted version." ] }, { "cell_type": "code", "execution_count": 9, "id": "2b4fdde3", "metadata": { "id": "2b4fdde3", "outputId": "22cac947-3ad6-42a6-d7c1-9fef78f7b3a5", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "▶ scenario table [███████████████████████████░░░ 19/21 (90%)]\n", "metro weights (test share / train share): [1.09 0.85 1.16 0.97 0.95 1.14 0.69 1.09 1.04 0.69 0.91 0.85 1. 0.87\n", " 2.24 1.12]\n", "optimised blend weights: {'xgb': np.float64(0.551), 'cat': np.float64(0.449)}\n", " scored: xgb\n", " scored: cat\n", " scored: stack\n", " scored: xgb_reg\n", " scored: avg(xgb+cat)\n", " scored: opt_blend\n", " scored: opt_blend + xgb_reg (a=0.3)\n", " scored: opt_blend + xgb_reg (a=0.5)\n", " scored: opt_blend + xgb_reg (a=0.7)\n", " scored: stack + xgb_reg (a=0.3)\n", " scored: stack + xgb_reg (a=0.5)\n", " scored: stack + xgb_reg (a=0.7)\n", " scored per-metro: opt_blend + xgb_reg (a=0.7)\n", " scored per-metro: opt_blend + xgb_reg (a=0.5)\n", " scored per-metro: stack\n", " scored per-metro: stack + xgb_reg (a=0.7)\n", "✓ scenario table: took 6.7 min | [████████████████████████████░░ 20/21 (95%)] | elapsed 35.9 min | ETA ~0 min + 1 steps not timed yet\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ " scenario decoder \\\n", "0 stack A class weights \n", "1 opt_blend + xgb_reg (a=0.7) B cut-points \n", "2 opt_blend + xgb_reg (a=0.5) B cut-points \n", "3 stack B cut-points \n", "4 opt_blend + xgb_reg (a=0.5) C per-metro cut-points \n", "5 stack + xgb_reg (a=0.7) B cut-points \n", "6 opt_blend + xgb_reg (a=0.7) C per-metro cut-points \n", "7 stack + xgb_reg (a=0.5) B cut-points \n", "8 opt_blend + xgb_reg (a=0.3) B cut-points \n", "9 stack + xgb_reg (a=0.7) C per-metro cut-points \n", "10 stack C per-metro cut-points \n", "11 opt_blend B cut-points \n", "12 avg(xgb+cat) A class weights \n", "13 stack + xgb_reg (a=0.3) B cut-points \n", "14 opt_blend A class weights \n", "15 avg(xgb+cat) B cut-points \n", "16 xgb B cut-points \n", "17 xgb A class weights \n", "18 cat A class weights \n", "19 cat B cut-points \n", "20 xgb_reg B cut-points \n", "\n", " CV macro F1 (test-weighted) CV macro F1 (plain) \n", "0 0.7214 0.7221 \n", "1 0.7199 0.7205 \n", "2 0.7198 0.7205 \n", "3 0.7194 0.7200 \n", "4 0.7193 0.7199 \n", "5 0.7190 0.7197 \n", "6 0.7188 0.7193 \n", "7 0.7186 0.7193 \n", "8 0.7185 0.7193 \n", "9 0.7185 0.7193 \n", "10 0.7183 0.7190 \n", "11 0.7178 0.7185 \n", "12 0.7177 0.7184 \n", "13 0.7177 0.7183 \n", "14 0.7175 0.7182 \n", "15 0.7174 0.7181 \n", "16 0.7169 0.7176 \n", "17 0.7158 0.7168 \n", "18 0.7144 0.7152 \n", "19 0.7139 0.7146 \n", "20 0.7138 0.7146 " ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
scenariodecoderCV macro F1 (test-weighted)CV macro F1 (plain)
0stackA class weights0.72140.7221
1opt_blend + xgb_reg (a=0.7)B cut-points0.71990.7205
2opt_blend + xgb_reg (a=0.5)B cut-points0.71980.7205
3stackB cut-points0.71940.7200
4opt_blend + xgb_reg (a=0.5)C per-metro cut-points0.71930.7199
5stack + xgb_reg (a=0.7)B cut-points0.71900.7197
6opt_blend + xgb_reg (a=0.7)C per-metro cut-points0.71880.7193
7stack + xgb_reg (a=0.5)B cut-points0.71860.7193
8opt_blend + xgb_reg (a=0.3)B cut-points0.71850.7193
9stack + xgb_reg (a=0.7)C per-metro cut-points0.71850.7193
10stackC per-metro cut-points0.71830.7190
11opt_blendB cut-points0.71780.7185
12avg(xgb+cat)A class weights0.71770.7184
13stack + xgb_reg (a=0.3)B cut-points0.71770.7183
14opt_blendA class weights0.71750.7182
15avg(xgb+cat)B cut-points0.71740.7181
16xgbB cut-points0.71690.7176
17xgbA class weights0.71580.7168
18catA class weights0.71440.7152
19catB cut-points0.71390.7146
20xgb_regB cut-points0.71380.7146
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "table", "summary": "{\n \"name\": \"table\",\n \"rows\": 21,\n \"fields\": [\n {\n \"column\": \"scenario\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 12,\n \"samples\": [\n \"cat\",\n \"xgb\",\n \"stack\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"decoder\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"A class weights\",\n \"B cut-points\",\n \"C per-metro cut-points\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"CV macro F1 (test-weighted)\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.0019789246719223263,\n \"min\": 0.7138,\n \"max\": 0.7214,\n \"num_unique_values\": 19,\n \"samples\": [\n 0.7214,\n 0.719,\n 0.7177\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"CV macro F1 (plain)\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.001932701046032042,\n \"min\": 0.7146,\n \"max\": 0.7221,\n \"num_unique_values\": 16,\n \"samples\": [\n 0.7221,\n 0.7205,\n 0.7193\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {} } ], "source": [ "prog.start(\"scenario table\")\n", "TIERS = np.arange(1, K + 1)\n", "share_tr = np.bincount(MT_tr, minlength=MT_tr.max() + 1) / n_tr\n", "share_te = np.bincount(MT_te, minlength=MT_tr.max() + 1) / len(MT_te)\n", "W_tr = (share_te / np.maximum(share_tr, 1e-12))[MT_tr]\n", "print(\"metro weights (test share / train share):\", np.round(share_te / np.maximum(share_tr, 1e-12), 2))\n", "\n", "\n", "def fit_cw(P, yy, w, mt):\n", " W = np.ones(K)\n", " for _ in range(3):\n", " for k in range(1, K):\n", " grid = W[k] * np.exp(np.linspace(-0.4, 0.4, 41))\n", " sc = [fast_f1(yy, (P * np.r_[W[:k], g_, W[k + 1:]]).argmax(1), w) for g_ in grid]\n", " W[k] = grid[int(np.argmax(sc))]\n", " return W\n", "\n", "\n", "def apply_cw(P, W, mt):\n", " return (P * W).argmax(1)\n", "\n", "\n", "def fit_cuts(s, yy, w, mt=None):\n", " cuts = np.quantile(s, np.cumsum(np.bincount(yy, minlength=K) / len(yy))[:-1])\n", " for _ in range(3):\n", " for i in range(K - 1):\n", " lo = cuts[i - 1] + 1e-6 if i else s.min()\n", " hi = cuts[i + 1] - 1e-6 if i < K - 2 else s.max()\n", " grid = np.linspace(max(lo, cuts[i] - 0.4), min(hi, cuts[i] + 0.4), 61)\n", " sc = [fast_f1(yy, np.digitize(s, np.r_[cuts[:i], g_, cuts[i + 1:]]), w) for g_ in grid]\n", " cuts[i] = grid[int(np.argmax(sc))]\n", " return cuts\n", "\n", "\n", "def apply_cuts(s, cuts, mt):\n", " return np.digitize(s, cuts)\n", "\n", "\n", "def fit_metro_cuts(s, yy, w, mt):\n", " g0 = fit_cuts(s, yy, w)\n", " C = np.tile(g0, (MT_tr.max() + 1, 1))\n", " pred = np.digitize(s, g0)\n", " cm = conf(yy, pred, w)\n", " for _ in range(2):\n", " for m_ in np.unique(mt):\n", " ix = mt == m_\n", " for i in range(K - 1):\n", " rest = cm - conf(yy[ix], pred[ix], w[ix])\n", " lo = C[m_, i - 1] + 1e-6 if i else s.min()\n", " hi = C[m_, i + 1] - 1e-6 if i < K - 2 else s.max()\n", " grid = np.linspace(max(lo, C[m_, i] - 0.5), min(hi, C[m_, i] + 0.5), 41)\n", " best_g, best_s = C[m_, i], -1\n", " for g_ in grid:\n", " p_ = np.digitize(s[ix], np.r_[C[m_, :i], g_, C[m_, i + 1:]])\n", " sc = f1_from_conf(rest + conf(yy[ix], p_, w[ix]))\n", " if sc > best_s:\n", " best_g, best_s = g_, sc\n", " C[m_, i] = best_g\n", " pred[ix] = np.digitize(s[ix], C[m_])\n", " cm = rest + conf(yy[ix], pred[ix], w[ix])\n", " return C\n", "\n", "\n", "def apply_metro_cuts(s, C, mt):\n", " out = np.empty(len(s), int)\n", " for m_ in np.unique(mt):\n", " ix = mt == m_\n", " out[ix] = np.digitize(s[ix], C[m_])\n", " return out\n", "\n", "\n", "def crossfit(o, fit, apply):\n", " pred = np.empty(n_tr, int)\n", " for f in range(FOLDS):\n", " m_ = fold_id == f\n", " prm = fit(o[~m_], y[~m_], W_tr[~m_], MT_tr[~m_])\n", " pred[m_] = apply(o[m_], prm, MT_tr[m_])\n", " return fast_f1(y, pred, W_tr), fast_f1(y, pred)\n", "\n", "\n", "def opt_weights(models):\n", " Ps = [OOF[m] for m in models]\n", " def loss(z):\n", " w = np.exp(z) / np.exp(z).sum()\n", " return log_loss(y, np.clip(sum(wi * p for wi, p in zip(w, Ps)), 1e-7, 1))\n", " z = minimize(loss, np.zeros(len(Ps)), method=\"Nelder-Mead\", options={\"maxiter\": 300}).x\n", " return np.exp(z) / np.exp(z).sum()\n", "\n", "\n", "candidates = {} # name -> (oof output, test output, kind)\n", "for m_ in PROB_MODELS + [\"stack\"]:\n", " candidates[m_] = (OOF[m_], TEST[m_], \"prob\")\n", "for m_ in REG_MODELS:\n", " candidates[m_] = (OOF[m_][:, 0], TEST[m_][:, 0], \"score\")\n", "if len(PROB_MODELS) > 1:\n", " candidates[\"avg(\" + \"+\".join(PROB_MODELS) + \")\"] = (np.mean([OOF[m_] for m_ in PROB_MODELS], 0),\n", " np.mean([TEST[m_] for m_ in PROB_MODELS], 0), \"prob\")\n", " wb = opt_weights(PROB_MODELS)\n", " print(\"optimised blend weights:\", dict(zip(PROB_MODELS, wb.round(3))))\n", " candidates[\"opt_blend\"] = (sum(wi * OOF[m_] for wi, m_ in zip(wb, PROB_MODELS)),\n", " sum(wi * TEST[m_] for wi, m_ in zip(wb, PROB_MODELS)), \"prob\")\n", "if REG_MODELS:\n", " reg_o = np.mean([OOF[m_][:, 0] for m_ in REG_MODELS], 0)\n", " reg_t = np.mean([TEST[m_][:, 0] for m_ in REG_MODELS], 0)\n", " if len(REG_MODELS) > 1:\n", " candidates[\"avg(\" + \"+\".join(REG_MODELS) + \")\"] = (reg_o, reg_t, \"score\")\n", " reg_opts = {m_: (OOF[m_][:, 0], TEST[m_][:, 0]) for m_ in REG_MODELS}\n", " if len(REG_MODELS) > 1:\n", " reg_opts[\"avg_reg\"] = (reg_o, reg_t)\n", " for src in [c for c in (\"opt_blend\", \"stack\") if c in candidates]:\n", " for rname, (ro, rt) in reg_opts.items():\n", " for a in (0.3, 0.5, 0.7):\n", " candidates[f\"{src} + {rname} (a={a})\"] = (a * (candidates[src][0] @ TIERS) + (1 - a) * ro,\n", " a * (candidates[src][1] @ TIERS) + (1 - a) * rt, \"score\")\n", "\n", "rows = []\n", "for name, (o, t, kind) in candidates.items():\n", " s_o, s_t = (o @ TIERS, t @ TIERS) if kind == \"prob\" else (o, t)\n", " if kind == \"prob\":\n", " rows.append(dict(scenario=name, decoder=\"A class weights\", out=(o, t), fit=fit_cw, apply=apply_cw,\n", " score=crossfit(o, fit_cw, apply_cw)))\n", " rows.append(dict(scenario=name, decoder=\"B cut-points\", out=(s_o, s_t), fit=fit_cuts, apply=apply_cuts,\n", " score=crossfit(s_o, fit_cuts, apply_cuts)))\n", " print(f\" scored: {name}\", flush=True)\n", "for r in sorted([r for r in rows if r[\"decoder\"].startswith(\"B\")], key=lambda r: -r[\"score\"][0])[:4]:\n", " rows.append(dict(scenario=r[\"scenario\"], decoder=\"C per-metro cut-points\", out=r[\"out\"], fit=fit_metro_cuts,\n", " apply=apply_metro_cuts, score=crossfit(r[\"out\"][0], fit_metro_cuts, apply_metro_cuts)))\n", " print(f\" scored per-metro: {r['scenario']}\", flush=True)\n", "\n", "table = pd.DataFrame([{\"scenario\": r[\"scenario\"], \"decoder\": r[\"decoder\"],\n", " \"CV macro F1 (test-weighted)\": round(r[\"score\"][0], 4), \"CV macro F1 (plain)\": round(r[\"score\"][1], 4)}\n", " for r in rows]).sort_values(\"CV macro F1 (test-weighted)\", ascending=False).reset_index(drop=True)\n", "prog.finish()\n", "display(table)" ] }, { "cell_type": "markdown", "id": "bc827048", "metadata": { "id": "bc827048" }, "source": [ "## 7. Submission\n", "`submission.csv` = the best scenario. The decoder is re-tuned on **all** out-of-fold predictions, then applied to the test predictions.\n", "The 2nd and 3rd best are saved too (`submission_rank2.csv`, `submission_rank3.csv`).\n", "**Pick by CV, not by the public leaderboard**: the public part is only a slice of the test set." ] }, { "cell_type": "code", "execution_count": 10, "id": "36ec1c4d", "metadata": { "id": "36ec1c4d", "outputId": "3fbb979a-45c8-42e4-8ed0-1f12553bfdae", "colab": { "base_uri": "https://localhost:8080/", "height": 365 } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "▶ write submissions [████████████████████████████░░ 20/21 (95%)]\n", "submission.csv ← stack | A class weights | CV (test-weighted) 0.7214\n", "submission_rank2.csv ← opt_blend + xgb_reg (a=0.7) | B cut-points | CV (test-weighted) 0.7199\n", "submission_rank3.csv ← opt_blend + xgb_reg (a=0.5) | B cut-points | CV (test-weighted) 0.7198\n", "✓ write submissions: took 0.2 min | [██████████████████████████████ 21/21 (100%)] | elapsed 36.0 min | ETA ~-\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "price_tier\n", "1 25533\n", "2 37412\n", "3 26906\n", "4 24687\n", "Name: count, dtype: int64" ], "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
count
price_tier
125533
237412
326906
424687
\n", "

" ] }, "metadata": {}, "execution_count": 10 } ], "source": [ "prog.start(\"write submissions\")\n", "ranked = sorted(rows, key=lambda r: -r[\"score\"][0])\n", "for i, r in enumerate(ranked[:3]):\n", " o, t = r[\"out\"]\n", " prm = r[\"fit\"](o, y, W_tr, MT_tr)\n", " pred = r[\"apply\"](t, prm, MT_te)\n", " fname = \"submission.csv\" if i == 0 else f\"submission_rank{i + 1}.csv\"\n", " pd.DataFrame({\"ID\": test[\"space_id\"], \"price_tier\": pred.astype(int) + 1}).to_csv(WORK / fname, index=False)\n", " print(f\"{fname:24s} ← {r['scenario']} | {r['decoder']} | CV (test-weighted) {r['score'][0]:.4f}\")\n", "prog.finish()\n", "pd.read_csv(WORK / \"submission.csv\").price_tier.value_counts().sort_index()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "name": "python3" }, "accelerator": "GPU", "colab": { "gpuType": "T4", "provenance": [] } }, "nbformat": 4, "nbformat_minor": 5 }