Feature Engineering: Encoding, Scaling, Selection

TL;DR: Encode categoricals by cardinality (one-hot for low, target or hashing for high, learned embeddings for very high and when a neural net is downstream), scale numerics when the model uses distances or gradients (kNN, SVM, linear, neural nets) and skip it for trees, and select features with filter, wrapper, or embedded methods to cut noise and serving cost. Done carefully it moves metrics more than swapping the model, and done carelessly it leaks the target.

Encoding: let cardinality decide

A model sees numbers, so every category becomes one, and the right transform is set mostly by how many distinct values the column has.

One-hot turns each level into its own 0/1 column. The safe default for low cardinality (under ~15 levels) and required for linear and logistic models, where an integer code imposes a meaningless order. The cost is dimensionality: a 10,000-value column becomes 10,000 sparse columns that bloat memory and starve tree splits.

Target (mean) encoding replaces each level with the mean of the target for that level. One column, and it works well for high-cardinality fields like zip code or merchant id. The danger is leakage: computing the mean on rows you then train on lets the model memorize the target. Compute it out-of-fold and smooth rare levels toward the global mean, or it overfits hard.

Hashing maps levels into a fixed number of buckets via a hash function. It bounds dimensionality and handles unseen categories at serve time for free, at the price of collisions. It shines for huge, churning vocabularies (URLs, user agents) you cannot enumerate ahead of time.

Learned embeddings map each level to a dense low-dimensional vector trained with the model. The move for very high cardinality with a neural net downstream (user and item ids in recommenders): the embedding captures similarity between levels that one-hot throws away.

Scaling: depends entirely on the model

Scaling is not universally needed; it depends on whether the model cares about the magnitude or geometry of features. It matters for any method that measures distances or rides gradients: kNN and SVM compute distances so an unscaled large-range feature dominates, gradient descent converges far faster when features share a scale, and PCA picks directions of variance so it is meaningless on unscaled data. Standardization (subtract mean, divide by std) is the usual choice; min-max to [0,1] suits bounded inputs and image pixels.

It does not matter for tree models (decision trees, random forests, gradient boosting). A tree splits on thresholds, and any monotonic rescaling gives the same splits, so scaling XGBoost features is wasted effort. Knowing this cold is a quick credibility check. The rule: fit the scaler on training data only and apply it to validation and test, or test statistics leak into training.

Selection: filter, wrapper, embedded

More features are not free: they add variance, slow training and serving, and invite overfitting, so cutting dead weight is part of the job.

• Filter methods score features independently of any model: correlation, mutual information, chi-squared, variance threshold. Cheap, but blind to interactions and redundancy.
• Wrapper methods train a model on feature subsets and search (recursive feature elimination, forward selection). Accurate but expensive, and at risk of overfitting the selection to your validation split.
• Embedded methods get selection from the model itself: L1 (lasso) drives coefficients to zero, tree models expose importances or, better, permutation importance. Usually the best signal per unit of compute.

A concrete pass

Take a churn model with columns country (200 levels), monthly_spend (dollars), plan_tier (3 levels), and last_url (millions). A reasonable pass: target-encode country out-of-fold, one-hot plan_tier, hash last_url into 1,024 buckets, standardize monthly_spend only if you feed a logistic or neural model (skip it for XGBoost). Then run permutation importance and drop features that do not move validation AUC. Swapping logistic for gradient boosting might add a point of AUC; a leak-free country encoding plus a spend-per-tenure ratio often adds more.

Why interviewers probe this

It separates people who have shipped from people who have only trained on clean Kaggle data. The interviewer wants cardinality to drive the encoding choice, model family to drive the scaling decision, and an explicit mention of leakage with the out-of-fold fix. The strong move is to volunteer "I would compute target encoding out-of-fold to avoid leaking the label" before being asked. The held-back follow-up is train-serve skew: how do you guarantee the same transform at training and inference? A feature store or serialized pipeline, fit on training data, versioned, applied identically in both paths.

Common misconceptions

• "Always one-hot encode categoricals." One-hot explodes on high cardinality and slows tree splits; use target, hashing, or embeddings there.
• "Always scale your features." Trees are invariant to monotonic scaling; scaling matters only for distance and gradient models.
• "Target encoding is a free win." Without out-of-fold computation it leaks the label and silently inflates validation scores.
• "Feature selection is about accuracy only." It also cuts serving latency, memory, and drift surface, which often matters more than a fractional metric gain.

Key takeaways

• Encode by cardinality: one-hot (low), target out-of-fold (high), hashing (huge or streaming), embeddings (very high with a neural net).
• Scale for distance and gradient models (kNN, SVM, linear, neural nets, PCA); skip it for tree ensembles.
• Fit every transform on training data only, then apply to validation, test, and serving to prevent leakage and train-serve skew.
• Selection is filter, wrapper, or embedded; embedded (L1, tree importances) usually gives the best signal per unit of compute.