Feature Engineering · Notebook 4 companion articleFeature Engineering for Categorical Variables

Start hereIntroduction#

Notebook 3 showed that every numeric transformation makes an assumption, and that the wrong one can quietly wreck a model. Categories need the same care, because a model can't use them at all until you turn them into numbers.

A lot of code treats encoding as a formatting step: strings go in, numbers come out, the errors stop. But every encoding choice makes a claim about how the categories relate to each other, and scikit-learn will build whatever claim you give it, true or not.

If you say a two-year contract is twice as far from month-to-month as a one-year contract, and that's roughly true, the model gets useful information for free. Say something similar about payment methods, where no such distance exists, and the model doesn't push back. It just learns something false and uses it confidently.

This notebook covers six approaches to the same problem, each with a measured cost and benefit:

• label and ordinal encoding
• one-hot encoding
• target encoding
• frequency encoding
• the hashing trick
• unseen and rare categories, which only cause trouble after a model ships

None of these is always right or always wrong. Each one works best in specific situations, and this notebook shows what those are instead of giving you one default.

The leakage rule from the last three notebooks comes back too, as notebook 3 promised. Encoders have a fit method like scalers and imputers. And in section 4 you'll meet the one encoder that learns from the target itself, where breaking the rule costs the most.

The dataset, and one new column#

Telco is a good fit because almost every column is a category. But none of its columns has many different values. One-hot encoding is the right answer for columns with just a few categories. The interesting trade-offs only show up when a column has too many categories to one-hot without thinking.

So this notebook builds one. Each customer has some combination of nine services, and that combination describes their account in a way no single column does. Joining those nine columns into one string gives a real column with many categories. It stands in for the kind of many-valued column you meet in real work, like city, product or store.

One honest note: you could also give the model the nine service columns directly. We use bundle because it's a real column with many categories, not because it's the only way to use this information.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction import FeatureHasher
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, TargetEncoder

plt.style.use('seaborn-v0_8-whitegrid')
pd.set_option('display.max_columns', 30)
pd.set_option('display.width', 170)

RANDOM_STATE = 42

telco = pd.read_csv('datasets/telco_churn.csv')
telco['TotalCharges'] = pd.to_numeric(telco['TotalCharges'], errors='coerce')
telco = telco[telco['tenure'] > 0].reset_index(drop=True)

churned = (telco['Churn'] == 'Yes').astype(int)
NUMERIC = ['tenure', 'MonthlyCharges', 'TotalCharges']

# Build one high-cardinality column out of the nine service columns
SERVICES = ['PhoneService', 'MultipleLines', 'InternetService', 'OnlineSecurity',
            'OnlineBackup', 'DeviceProtection', 'TechSupport', 'StreamingTV', 'StreamingMovies']
telco['bundle'] = telco[SERVICES].agg('|'.join, axis=1)

print(f'Rows          : {len(telco):,}')
print(f'Churn rate    : {churned.mean():.1%}')
print(f'Unique bundles: {telco["bundle"].nunique()}')
print()
print('Two example bundles:')
for b in telco['bundle'].value_counts().head(2).index:
    print(f'  {b}')

Rows          : 7,032
Churn rate    : 26.6%
Unique bundles: 322

Two example bundles:
  Yes|No|No|No internet service|No internet service|No internet service|No internet service|No internet service|No internet service
  Yes|Yes|No|No internet service|No internet service|No internet service|No internet service|No internet service|No internet service

That gives 322 different combinations of nine service columns across 7,032 customers. This column comes up throughout the notebook. It's useful, as the next section shows, and far too big to one-hot encode without problems.

Section 1Reading a categorical column before encoding it#

Two questions decide the right method. Answering them takes longer than writing the code

Does the order mean anything? An ordinal column has a real ranking. Contract goes month-to-month, one year, two year, which shows rising commitment. A nominal column has no ranking. PaymentMethod has four values, and none is "higher" than another.

Pandas can't answer this for you. Both kinds of column are just strings to the computer. Only a person who knows the subject can say which is which. That's why notebook 1 made Contract an ordered Categorical as soon as we knew the order, and promised this notebook would put that order to use.

How many different values are there? With few categories, one-hot encoding is easy: a few new columns, plenty of rows behind each one. With many categories, it gets hard, and every alternative has a real trade-off.

categorical_cols = [c for c in telco.columns
                    if c not in NUMERIC + ['customerID', 'Churn', 'bundle']]

inventory = pd.DataFrame({
    'column': categorical_cols,
    'n_categories': [telco[c].nunique() for c in categorical_cols],
    'most_common': [telco[c].value_counts().index[0] for c in categorical_cols],
    'rarest_count': [telco[c].value_counts().iloc[-1] for c in categorical_cols],
})
inventory.loc[len(inventory)] = ['bundle (engineered)', telco['bundle'].nunique(),
                                 telco['bundle'].value_counts().index[0][:28] + '...',
                                 telco['bundle'].value_counts().iloc[-1]]

print(inventory.to_string(index=False))

             column  n_categories                     most_common  rarest_count
             gender             2                            Male          3483
      SeniorCitizen             2                               0          1142
            Partner             2                              No          3393
         Dependents             2                              No          2099
       PhoneService             2                             Yes           680
      MultipleLines             3                              No           680
    InternetService             3                     Fiber optic          1520
     OnlineSecurity             3                              No          1520
       OnlineBackup             3                              No          1520
   DeviceProtection             3                              No          1520
        TechSupport             3                              No          1520
        StreamingTV             3                              No          1520
    StreamingMovies             3                              No          1520
           Contract             3                  Month-to-month          1472
   PaperlessBilling             2                             Yes          2864
      PaymentMethod             4                Electronic check          1521
bundle (engineered)           322 Yes|No|No|No internet servic...             1

The sixteen original columns each have two to four categories. The new bundle column has 322. That gap is really what this notebook is about. Each of the sixteen original columns can be one-hot encoded easily, making at most four new columns. Doing the same to bundle makes 322 new columns, and section 3 shows how badly that goes.

Before deciding how to encode bundle, check that it's worth the effort. A column with no link to the target doesn't need careful handling, however many categories it has.

bundle_stats = (pd.DataFrame({'bundle': telco['bundle'], 'churned': churned})
                .groupby('bundle')['churned'].agg(['mean', 'size'])
                .rename(columns={'mean': 'churn_rate', 'size': 'customers'}))

common = bundle_stats[bundle_stats['customers'] >= 20].sort_values('churn_rate')

print(f'Bundles with at least 20 customers: {len(common)}')
print(f'Their churn rates run from {common["churn_rate"].min():.1%} to {common["churn_rate"].max():.1%}')
print(f'(the dataset average is {churned.mean():.1%})')
print()
print('Lowest risk bundles:')
print(common.head(3).round(3).to_string())
print()
print('Highest risk bundles:')
print(common.tail(3).round(3).to_string())

Bundles with at least 20 customers: 76
Their churn rates run from 0.0% to 73.9%
(the dataset average is 26.6%)

Lowest risk bundles:
                                                 churn_rate  customers
bundle                                                                
Yes|Yes|DSL|Yes|Yes|Yes|Yes|No|No                       0.0         21
Yes|Yes|DSL|No|Yes|Yes|Yes|Yes|Yes                      0.0         25
No|No phone service|DSL|Yes|Yes|Yes|Yes|Yes|Yes         0.0         25

Highest risk bundles:
                                         churn_rate  customers
bundle                                                        
Yes|Yes|Fiber optic|No|No|No|No|Yes|Yes       0.664        110
Yes|Yes|Fiber optic|No|No|No|No|No|Yes        0.727         77
Yes|No|Fiber optic|No|No|No|No|Yes|No         0.739         69

Among the 76 bundles with at least 20 customers, churn ranges from 0% to 73.9%, against a 26.6% average overall. So this column matters. The table also shows why the row count matters alongside the rate. The most common bundle has 1,180 customers, and the rarest have just one. A churn rate from one or two customers is much weaker evidence than one from a thousand. Section 4 comes back to this.

The 0.8087 baseline below is the same number notebooks 1 and 3 got from these three numeric columns, so every gain in this notebook comes from the categories alone.

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)

def logreg_auc(X):
    # Scaled logistic regression: sensitive to how categories are encoded
    return cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000)),
                           X, churned, cv=cv, scoring='roc_auc').mean()

def forest_auc(X):
    return cross_val_score(RandomForestClassifier(n_estimators=150, random_state=RANDOM_STATE, n_jobs=1),
                           X, churned, cv=cv, scoring='roc_auc').mean()


numeric_only = telco[NUMERIC]
BASELINE = logreg_auc(numeric_only)
print(f'Baseline, the three numeric columns and nothing else: AUC {BASELINE:.4f}')
print('Every encoding below gets compared against this number.')

Baseline, the three numeric columns and nothing else: AUC 0.8087
Every encoding below gets compared against this number.

Section 2Label and ordinal encoding#

The simplest technique, and a default you should know about before it makes a choice for you

The simplest way to turn a category into a number is to give each one an integer: Bank transfer becomes 0, Credit card becomes 1, and so on. When the column has a real order, this is the right approach, called ordinal encoding. When the column has no order, the same operation is usually called label encoding, and it quietly adds a ranking that doesn't exist.

One default to know: if you don't give an order, scikit-learn sorts the categories alphabetically.

payment_categories = sorted(telco['PaymentMethod'].unique())

print('The order sklearn assigns by default, purely alphabetical:')
for i, cat in enumerate(payment_categories):
    print(f'  {i} = {cat}')
print()
print('That numbering claims Mailed check (3) is three units "more" than Bank transfer (0),')
print('and that Credit card (1) sits between them. None of that means anything.')

The order sklearn assigns by default, purely alphabetical:
  0 = Bank transfer (automatic)
  1 = Credit card (automatic)
  2 = Electronic check
  3 = Mailed check

That numbering claims Mailed check (3) is three units "more" than Bank transfer (0),
and that Credit card (1) sits between them. None of that means anything.

That numbering says Mailed check is three steps above Bank transfer, and Credit card is exactly in between. None of that is real. It just reflects which words start with which letters.

label_encoded = numeric_only.assign(
    payment=OrdinalEncoder().fit_transform(telco[['PaymentMethod']]).ravel())

one_hot = pd.concat([numeric_only,
                     pd.get_dummies(telco['PaymentMethod'], prefix='pay').astype(int)], axis=1)

print(f'{"encoding":16s} {"columns":>8s} {"LogReg AUC":>12s} {"RandomForest AUC":>18s}')
print('-' * 58)
print(f'{"label (0-3)":16s} {label_encoded.shape[1]:>8d} {logreg_auc(label_encoded):>12.4f} {forest_auc(label_encoded):>18.4f}')
print(f'{"one-hot":16s} {one_hot.shape[1]:>8d} {logreg_auc(one_hot):>12.4f} {forest_auc(one_hot):>18.4f}')

encoding          columns   LogReg AUC   RandomForest AUC
----------------------------------------------------------

label (0-3)             4       0.8088             0.7827

one-hot                 7       0.8175             0.7795

Label encoding lowers the logistic regression's AUC by 0.0087, from one-hot's 0.8175 to 0.8088. The random forest barely changes, and even does very slightly better with one label column than with four one-hot columns. (The forest scores lower than logistic regression overall on these few columns. What matters here is how each model reacts to the change in encoding.) There's a clear reason for the difference between the two models.

A linear model gives each column one coefficient. With payment method coded 0 to 3, it can only learn one effect that steadily rises or falls as the number goes up. The order is arbitrary, so there's no such steady effect, and the model is fitting a line through noise.

A tree works differently. It splits again and again, so it can isolate the value 2 with two cuts and rebuild the original categories despite the meaningless numbers. That's why trees handle label encoding much better, and why people using gradient boosting often use it without much harm.

Now the same mistake on a much bigger scale.

label_bundle = numeric_only.assign(
    bundle=OrdinalEncoder().fit_transform(telco[['bundle']]).ravel())

print(f'Bundle label-encoded as integers 0 to {telco["bundle"].nunique() - 1}:')
print(f'  LogReg AUC       {logreg_auc(label_bundle):.4f}')
print(f'  Baseline (no bundle at all) {BASELINE:.4f}')
print()
print(f'  RandomForest AUC {forest_auc(label_bundle):.4f}')

Bundle label-encoded as integers 0 to 321:
  LogReg AUC       0.8087
  Baseline (no bundle at all) 0.8087

  RandomForest AUC 0.7940

The logistic regression scores 0.8087, the same as the baseline that ignores bundle. A column with churn rates from 0% to 73.9% added nothing, because the model got it as an alphabetical number from 0 to 321. The information was in the data. The encoding destroyed it.

The random forest reaches 0.7940. There's no forest score without bundle to compare against, so it's hard to say how much it gained. A tree can at least split the numbers into chunks of neighboring bundles, but alphabetical neighbors aren't similar customers, so it's still a poor use of such a useful column.

Now a case where the order is real, so ordinal encoding should work well. Notebook 1 found churn of 42.7% for month-to-month, 11.3% for one year and 2.8% for two year, a clean drop as commitment goes up.

correct_order = ['Month-to-month', 'One year', 'Two year']
wrong_order = ['One year', 'Two year', 'Month-to-month']    # alphabetical-ish, and meaningless

def encode_with(order):
    return numeric_only.assign(contract=telco['Contract'].map({c: i for i, c in enumerate(order)}))


ordinal_right = encode_with(correct_order)
ordinal_wrong = encode_with(wrong_order)
contract_onehot = pd.concat(
    [numeric_only, pd.get_dummies(telco['Contract'], prefix='contract').astype(int)], axis=1)

print(f'{"encoding":34s} {"columns":>8s} {"LogReg AUC":>12s}')
print('-' * 56)
print(f'{"ordinal, correct order":34s} {ordinal_right.shape[1]:>8d} {logreg_auc(ordinal_right):>12.4f}')
print(f'{"ordinal, meaningless order":34s} {ordinal_wrong.shape[1]:>8d} {logreg_auc(ordinal_wrong):>12.4f}')
print(f'{"one-hot":34s} {contract_onehot.shape[1]:>8d} {logreg_auc(contract_onehot):>12.4f}')

encoding                            columns   LogReg AUC
--------------------------------------------------------
ordinal, correct order                    4       0.8278
ordinal, meaningless order                4       0.8197
one-hot                                   6       0.8276

Three results to look at.

Correct ordinal encoding scores 0.8278 with one column. One-hot scores 0.8276 with three. They're basically tied, and the ordinal version uses a third of the columns, because the order we gave it was real information the model would otherwise have to work out itself. Notebook 1 also checked that the two steps are roughly even on the log-odds scale (about 1.8 and 1.5), which is why equal steps of 0, 1, 2 fit so well.

The wrong order costs 0.0081 AUC. Same technique, same number of columns. The only change is which number went to which category. Ordinal encoding is only as good as the order you give it. A wrong order is worse than claiming no order at all (one-hot), though here it still beats leaving the column out.

A small surprise: for Contract, alphabetical order (Month, One, Two) happens to be the right order, so the default would have worked by luck. Don't rely on that for any other column.

One-hot didn't need to be told anything. It makes no claim about order, so it can't get the order wrong, and it found about the same pattern from the data. That safety is what the two extra columns buy you.

Section 3One-hot encoding, and where it stops working#

The standard choice for unordered columns, and a technique people keep using long after it stops helping

One-hot encoding creates one 0/1 column per category. A customer on a two-year contract gets a 1 in contract_Two year and 0 in the other contract columns. There's no order implied, and each category gets its own coefficient. It's the right default for unordered columns.

You'll sometimes see drop='first', which removes one column per feature. The dropped category becomes the "all zeros" case. That matters for a plain linear regression with no penalty, because a full set of one-hot columns always adds up to 1 and duplicates the intercept. Regularized models, which is scikit-learn's default for logistic regression, don't need it.

But there's an easy-to-miss problem worth seeing first.

sample = telco[['PaymentMethod']].head(4)

encoder = OneHotEncoder(sparse_output=False, dtype=int)
encoded = encoder.fit_transform(sample)

print('Input, the first four customers:')
print(sample.to_string(index=False))
print()
print('Output:')
print(pd.DataFrame(encoded, columns=encoder.get_feature_names_out()).to_string(index=False))
print()
print(f'Columns created : {encoded.shape[1]}')
print(f'Categories in the full dataset: {telco["PaymentMethod"].nunique()}')

Input, the first four customers:
            PaymentMethod
         Electronic check
             Mailed check
             Mailed check
Bank transfer (automatic)

Output:
 PaymentMethod_Bank transfer (automatic)  PaymentMethod_Electronic check  PaymentMethod_Mailed check
                                       0                               1                           0
                                       0                               0                           1
                                       0                               0                           1
                                       1                               0                           0

Columns created : 3
Categories in the full dataset: 4

The encoder only knows the categories it saw during fit. PaymentMethod has four categories in the full data. Fitting on these four sample rows, which only include three of them, gave three columns. Credit card (automatic) has no column at all, because the encoder never saw it.

That's fine here, because it's a small demo sample. The lesson is that the list of categories is learned, and where it's learned from matters.

This is also the main reason to use OneHotEncoder instead of the handier-looking pd.get_dummies. pd.get_dummies creates columns only for the categories in the data you give it. Train on data with four payment methods and you get four columns. Run the same line next month on data with only three, and you get three columns, with every column after the missing one shifted over by one place.

The model then gets the wrong values in the wrong places, with no error.

OneHotEncoder avoids this. It learns the full list of categories once, during fit, and always uses that same list, even for categories missing from the current data. That's why it belongs in a pipeline, like every other transformer in this series. get_dummies is fine for a quick look at the data. OneHotEncoder is what you use in anything that runs more than once.

Now a problem that good code can't fix, because the code isn't the problem.

bundle_onehot = pd.concat(
    [numeric_only, pd.get_dummies(telco['bundle'], prefix='b').astype(int)], axis=1)

print(f'{"approach":34s} {"columns":>8s} {"LogReg AUC":>12s}')
print('-' * 56)
print(f'{"numeric only, bundle ignored":34s} {numeric_only.shape[1]:>8d} {BASELINE:>12.4f}')
print(f'{"bundle one-hot encoded":34s} {bundle_onehot.shape[1]:>8d} {logreg_auc(bundle_onehot):>12.4f}')

approach                            columns   LogReg AUC
--------------------------------------------------------
numeric only, bundle ignored              3       0.8087

bundle one-hot encoded                  325       0.8007

One-hot encoding bundle makes the model worse than ignoring the column, dropping AUC from 0.8087 to 0.8007. Section 1 showed the column is useful, so the encoding is the problem, not the data. The picture below shows why.

Three problems add up. Almost every column is mostly zeros. With 322 categories across 7,032 rows, the average column has a 1 in only about 22 rows. That's very little evidence to estimate a coefficient from. Rare categories get coefficients that are just noise. A bundle that appears twice gets its own column and coefficient, learned from two customers. The model's default penalty shrinks these coefficients a little, but not enough to stop them from fitting noise.

325 columns for 7,032 rows gives the model more freedom than the data can support, so it fits patterns in the training folds that don't hold up in the validation fold.

One more thing adds to the problem here: the pipeline scales every column. A 0/1 column that's 1 for only two customers has a tiny standard deviation, so after scaling, those two 1s become values around 59. Rare categories end up with extreme values. Many pipelines leave one-hot columns unscaled for this reason.

This is where you need something other than one-hot. Sections 4 and 5 cover the alternatives, and section 6 shows that one-hot can still work if you deal with rare categories first.

Section 4Target encoding, and the leak that comes with it#

The strongest technique here for columns with many categories, and the one most likely to give you a great but false score

Target encoding replaces each category with the average target value for that category. Each of the 322 bundles becomes the churn rate of its customers. So 322 categories become one number column with a clear meaning: a bundle encoded as 0.65 is one whose customers churn 65% of the time.

It's the strongest technique in this notebook for columns with many categories. It's also the most likely to give you a great validation score for a model that fails in production. So we'll do it the wrong way first and measure the cost.

# THE WRONG WAY: compute the mapping from the whole dataset, then cross-validate
leaky_map = churned.groupby(telco['bundle']).mean()
leaky = numeric_only.assign(bundle_te=telco['bundle'].map(leaky_map))

print('Each bundle replaced by its churn rate, computed over all 7,032 rows:')
print(leaky[['bundle_te']].head(3).round(4).to_string(index=False))
print()
print(f'Cross-validated AUC: {logreg_auc(leaky):.4f}')
print(f'Baseline           : {BASELINE:.4f}')
print()
print('That looks like a large improvement. It is not real.')

Each bundle replaced by its churn rate, computed over all 7,032 rows:
 bundle_te
    0.3871
    0.1111
    0.2051

Cross-validated AUC: 0.8450
Baseline           : 0.8087

That looks like a large improvement. It is not real.

That looks like a big jump, from a baseline of 0.8087 to 0.8450. It isn't real, and it's easy to miss because nothing in the code looks broken.

cross_val_score does split the data into five folds and holds one out each time. But the churn-rate mapping was built before the split, using every row, including the ones in the held-out fold. So when the model is tested on fold 3, each customer's encoded value was partly calculated from that customer's own churn result.

It's worst where there's the least data. A bundle with only two customers gets the average of those two outcomes. If both churned, the value is 1.0, and the model sees that 1.0 for those same two customers during testing. This isn't a small bias. It's like giving the model part of the answer key.

There are two separate problems, so there are two fixes. The first is the usual leakage rule from this series: fit the mapping inside a pipeline so it only sees training rows. The second is specific to target encoding and easy to miss: even within the training rows, a row shouldn't help build the value used to encode that same row. That needs internal cross-fitting instead of one pass. sklearn.preprocessing.TargetEncoder handles both for you.

One quirk to know: the cross-fitting only happens in fit_transform. If you call fit and then transform on the same training rows, it's skipped and the second leak comes back. Inside a pipeline, this is handled for you.

# THE RIGHT WAY: TargetEncoder inside the pipeline, cross-fitted internally
preprocess = ColumnTransformer([
    ('numeric', 'passthrough', NUMERIC),
    ('target_encode', TargetEncoder(target_type='binary', random_state=RANDOM_STATE), ['bundle']),
])

proper = make_pipeline(preprocess, StandardScaler(), LogisticRegression(max_iter=3000))
proper_auc = cross_val_score(proper, telco[NUMERIC + ['bundle']], churned,
                             cv=cv, scoring='roc_auc').mean()

print(f'{"approach":38s} {"columns":>8s} {"AUC":>9s}')
print('-' * 58)
print(f'{"numeric only":38s} {3:>8d} {BASELINE:>9.4f}')
print(f'{"bundle one-hot":38s} {325:>8d} {logreg_auc(bundle_onehot):>9.4f}')
print(f'{"bundle target-encoded, LEAKED":38s} {4:>8d} {logreg_auc(leaky):>9.4f}')
print(f'{"bundle target-encoded, PROPER":38s} {4:>8d} {proper_auc:>9.4f}')

approach                                columns       AUC
----------------------------------------------------------
numeric only                                  3    0.8087
bundle one-hot                              325    0.8007

bundle target-encoded, LEAKED                 4    0.8450
bundle target-encoded, PROPER                 4    0.8250

The leak raised the score by 0.0200 AUC: a real 0.8250 reported as 0.8450. Think about what that means on a real project. You'd report 0.845 to your team, make decisions based on it, and then watch the live model perform like the 0.825 model it always was.

The small size is what makes it dangerous. A leak that pushes a score to 0.99 gets caught, because nobody believes it. A leak that adds two points looks like a good day of feature engineering and gets through review.

The real result is still very good. Proper target encoding reaches 0.8250 with one column, ahead of one-hot's 0.8007 with 322 columns and the 0.8087 baseline. That's a gain of 0.0163 from a single column. To make the leak concrete, the next cell shows it on a real held-out test set, not just in cross-validation.

train, test, y_train, y_test = train_test_split(
    telco, churned, test_size=0.25, random_state=RANDOM_STATE, stratify=churned)

# Version A: mapping built from ALL rows, including the test set
map_all = churned.groupby(telco['bundle']).mean()
model_a = make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000)).fit(
    train[NUMERIC].assign(te=train['bundle'].map(map_all)), y_train)
auc_a = roc_auc_score(y_test, model_a.predict_proba(
    test[NUMERIC].assign(te=test['bundle'].map(map_all)))[:, 1])

# Version B: mapping built from TRAINING rows only, unseen bundles fall back to the prior
map_train = y_train.groupby(train['bundle']).mean()
prior = y_train.mean()
model_b = make_pipeline(StandardScaler(), LogisticRegression(max_iter=3000)).fit(
    train[NUMERIC].assign(te=train['bundle'].map(map_train).fillna(prior)), y_train)
auc_b = roc_auc_score(y_test, model_b.predict_proba(
    test[NUMERIC].assign(te=test['bundle'].map(map_train).fillna(prior)))[:, 1])

print(f'Mapping built from all rows,      test AUC = {auc_a:.4f}   <- flattering and false')
print(f'Mapping built from training only, test AUC = {auc_b:.4f}   <- what you would actually get')

Mapping built from all rows,      test AUC = 0.8418   <- flattering and false
Mapping built from training only, test AUC = 0.8153   <- what you would actually get

0.8418 against 0.8153. The first number isn't a better model. It's the same model taking a test it had already seen the answers to.

Version B only fixes the first problem. Its training rows were still encoded using their own outcomes, so the model learned to trust the encoded column a bit too much. That's likely part of why 0.8153 is lower than the 0.8250 from cross-validation (the test split is also different). TargetEncoder's cross-fitting fixes the second problem too.

Smoothing, for categories with too little data#

One problem is left even after fixing the leak. A bundle seen only three times, with no churners, has a rate of exactly 0.0. Taken literally, that tells the model these customers never leave. Three customers can't support a claim that strong.

Smoothing fixes this by pulling each category's average toward the overall average, based on how much data the category has:

encoded = (n × category_mean + m × overall_mean) / (n + m)

Here n is the number of rows in the category, and m is a number you choose that controls how strongly the overall average pulls on small categories. Categories with lots of rows barely move. Categories with very few rows get pulled most of the way to the overall average.

overall = churned.mean()
counts = telco['bundle'].value_counts()

examples = [counts.index[0], counts.index[1], counts[counts <= 3].index[0]]

print(f'Overall churn rate (the prior) = {overall:.3f}')
print()
print(f'{"customers":>10s} {"raw mean":>10s} {"m=0":>8s} {"m=10":>8s} {"m=50":>8s}')
print('-' * 48)
for bundle in examples:
    rows = telco['bundle'] == bundle
    n = rows.sum()
    raw = churned[rows].mean()
    smoothed = [(n * raw + m * overall) / (n + m) for m in (0, 10, 50)]
    print(f'{n:>10d} {raw:>10.3f} ' + ' '.join(f'{s:>8.3f}' for s in smoothed))

Overall churn rate (the prior) = 0.266

 customers   raw mean      m=0     m=10     m=50
------------------------------------------------
      1180      0.087    0.087    0.089    0.095
       340      0.029    0.029    0.036    0.060
         3      0.000    0.000    0.204    0.251

The bundle with 1,180 customers barely moves, from 0.087 to 0.095, even with the strongest smoothing. It has plenty of data. The bundle with 3 customers moves from 0.000 to 0.251, close to the overall rate of 0.266. With only three customers, the honest answer is "we don't know much about this group yet," and heavy smoothing says exactly that.

TargetEncoder smooths automatically, using smooth='auto' by default, so we didn't have to do it by hand above. It's still worth understanding: if a target-encoded column's values all sit close to the overall average, smoothing on small categories is usually why. It's not a bug.

Section 5Frequency encoding and the hashing trick#

Two more options for columns with many categories. Neither uses the target, so neither can leak

Frequency encoding replaces each category with how often it appears. It creates one column, doesn't use the target, and assumes that how common something is might help predict the outcome. One side effect: two categories with the same count get the same number, so the model can't tell them apart.

frequencies = telco['bundle'].map(telco['bundle'].value_counts() / len(telco))
freq_encoded = numeric_only.assign(bundle_freq=frequencies)

print(f'Frequency encoded : AUC {logreg_auc(freq_encoded):.4f}')
print(f'Baseline          : AUC {BASELINE:.4f}')
print(f'Target encoded    : AUC {proper_auc:.4f}')

Frequency encoded : AUC 0.8094
Baseline          : AUC 0.8087
Target encoded    : AUC 0.8250

0.8094 against a 0.8087 baseline, which is basically nothing. It's important to report that plainly, because a null result is still a result. Frequency encoding helps when being rare actually means something, like in fraud detection, where an unusual merchant type is itself suspicious.

Here, how common a service bundle is has almost nothing to do with churn, so the encoding captures something that doesn't matter much for this question. It's cheap and safe, so it's worth a quick try. But it's not a replacement for target encoding when you can use target encoding.

The hashing trick works differently. It runs each category through a hash function and uses the result to pick a column. That way any number of categories fits into a fixed number of columns you choose ahead of time. Different categories will sometimes land in the same column, and the bet is that these collisions are rare and random enough not to matter much. (Scikit-learn's FeatureHasher also gives each category a random +1 or −1, so collisions tend to cancel out instead of piling up.)

print(f'{"columns":>9s} {"categories per bucket":>24s} {"LogReg AUC":>12s}')
print('-' * 48)
for n_features in [8, 16, 32, 64]:
    hasher = FeatureHasher(n_features=n_features, input_type='string')
    hashed = pd.DataFrame(hasher.transform([[b] for b in telco['bundle']]).toarray(),
                          index=telco.index).add_prefix('hash_')
    score = logreg_auc(pd.concat([numeric_only, hashed], axis=1))
    print(f'{n_features:>9d} {telco["bundle"].nunique() / n_features:>24.1f} {score:>12.4f}')

print('-' * 48)
print(f'{325:>9d} {1.0:>24.1f} {logreg_auc(bundle_onehot):>12.4f}   <- one-hot, for comparison')

  columns    categories per bucket   LogReg AUC
------------------------------------------------

        8                     40.2       0.8087
       16                     20.1       0.8107
       32                     10.1       0.8142
       64                      5.0       0.8156
------------------------------------------------

      325                      1.0       0.8007   <- one-hot, for comparison

Hashing 322 categories into 64 columns scores 0.8156, beating one-hot's 0.8007 with a fifth of the columns. That's surprising, since hashing throws away information by letting categories collide.

The reason is that one-hot's problem was never too little information. It was too many coefficients, each learned from too few rows. Hashing makes several categories share a column, so each column has many more rows behind it and the coefficients can be learned properly. The collisions cost less than the original sparseness did.

Hashing is most useful at a scale where one-hot and target encoding struggle. With 500,000 different user IDs, one-hot isn't possible, and target encoding needs a 500,000-row lookup table stored with the model. A hasher doesn't store any list, since the hash function is fixed. It handles new categories without any special code, though a new category simply inherits whatever its column learned from others. Its memory use stays the same however many categories show up.

The cost is that you can't interpret the columns. Column hash_37 is a random mix of unrelated categories and means nothing to a person, so you can't explain feature importance on hashed columns. Use hashing when the number of categories is truly huge, and use target encoding when it's just large.

Section 6Unseen and rare categories#

Two problems that only appear after deployment, and stay hidden while you work with one fixed dataset

Unseen categories are values that show up at prediction time but never appeared in training. A new payment method launches, a new product is added, a new city gets a warehouse. The encoder has never seen that value but still has to handle it.

# How many unseen bundles appear depends entirely on which split you happen to draw
print(f'{"test size":>10s} {"seed":>6s} {"unseen bundles":>16s} {"rows affected":>15s}')
print('-' * 52)
for test_size in [0.25, 0.40]:
    for seed in range(4):
        tr, te = train_test_split(telco, test_size=test_size, random_state=seed, stratify=churned)
        unseen = set(te['bundle']) - set(tr['bundle'])
        print(f'{test_size:>10.2f} {seed:>6d} {len(unseen):>16d} {te["bundle"].isin(unseen).sum():>15d}')

 test size   seed   unseen bundles   rows affected
----------------------------------------------------
      0.25      0                2               4

      0.25      1                0               0
      0.25      2                2               5
      0.25      3                2               5
      0.40      0                8              24
      0.40      1                3               6
      0.40      2                8              22
      0.40      3                7              16

Between 0 and 8 unseen categories, depending only on the random seed and the test size. That variation is the real lesson. With seed 1 and a 25% test size, no unseen categories appear, so you'd think the encoding is fine and ship it. With seed 0 on the same data, the problem shows up right away. Whether you catch this bug in development is mostly luck, so you need to handle it by design from the start.

Since we can't rely on a lucky split, the next cell forces the problem with a category that definitely isn't in the data.

encoder_input = telco[['bundle']]
brand_new = pd.DataFrame({'bundle': ['Yes|Yes|Satellite|Yes|Yes|Yes|Yes|Yes|Yes']})   # a service we never sold

print('Fitting on all 322 known bundles, then asking about a bundle that has never existed.')
print()

try:
    OneHotEncoder(handle_unknown='error').fit(encoder_input).transform(brand_new)
except ValueError as err:
    print(f"handle_unknown='error'   -> ValueError: {str(err)[:70]}...")

ignored = OneHotEncoder(handle_unknown='ignore', sparse_output=False).fit(encoder_input).transform(brand_new)
print(f"handle_unknown='ignore'  -> all {ignored.shape[1]} columns are zero: {bool((ignored == 0).all())}")

infreq = OneHotEncoder(min_frequency=20, handle_unknown='infrequent_if_exist',
                       sparse_output=False).fit(encoder_input)
routed = infreq.transform(brand_new)
print(f"handle_unknown='infrequent_if_exist' -> routed into the rare bucket, "
      f"column {int(np.argmax(routed))} of {routed.shape[1]}")

te_encoder = TargetEncoder(target_type='binary', random_state=RANDOM_STATE).fit(encoder_input, churned)
print(f"TargetEncoder            -> unseen category gets the overall mean: "
      f"{te_encoder.transform(brand_new)[0][0]:.4f}  (dataset rate {churned.mean():.4f})")

Fitting on all 322 known bundles, then asking about a bundle that has never existed.

handle_unknown='error'   -> ValueError: Found unknown categories ['Yes|Yes|Satellite|Yes|Yes|Yes|Yes|Yes|Yes']...
handle_unknown='ignore'  -> all 322 columns are zero: True
handle_unknown='infrequent_if_exist' -> routed into the rare bucket, column 76 of 77
TargetEncoder            -> unseen category gets the overall mean: 0.2658  (dataset rate 0.2658)

Four different answers to the same question, and none is always right.

• handle_unknown='error' fails loudly and right away. Use it when an unknown category means something broke upstream, and you want to know before making a prediction.
• handle_unknown='ignore' sets all of that column's one-hot columns to zero. The model still predicts, using the other features. This is usually right for a service that must stay up. Keep in mind that "all zeros" is also a pattern the model never saw in training.
• handle_unknown='infrequent_if_exist' puts an unknown value into the same bucket used for rare training categories. That's often better than all zeros, because that bucket was trained on real customers. The catch is that a brand-new category may not behave like the old rare ones.
• TargetEncoder returns the overall mean, 0.2658, which is the dataset's churn rate. When you know nothing about a category, predicting the average is the right default.

Rare categories are the related problem on the training side. They appear in training, but with so few rows that anything learned about them is noise. Section 3 showed one-hot encoding the full bundle column scored worse than dropping it, and rare categories were a big part of why.

counts = telco['bundle'].value_counts()

print(f'{"min rows to keep":>17s} {"categories":>12s} {"columns":>9s} {"LogReg AUC":>12s}')
print('-' * 54)
for threshold in [1, 5, 20, 50, 100]:
    keep = counts[counts >= threshold].index
    grouped = telco['bundle'].where(telco['bundle'].isin(keep), 'Other')
    X = pd.concat([numeric_only, pd.get_dummies(grouped, prefix='b').astype(int)], axis=1)
    print(f'{threshold:>17d} {grouped.nunique():>12d} {X.shape[1]:>9d} {logreg_auc(X):>12.4f}')

 min rows to keep   categories   columns   LogReg AUC
------------------------------------------------------
                1          322       325       0.8007

                5          266       269       0.8045
               20           77        80       0.8238
               50           23        26       0.8235
              100           10        13       0.8168

This is the most important result in this section. One-hot encoding all 322 bundles scored 0.8007, worse than ignoring the column. Grouping every bundle with fewer than 20 rows into one Other bucket leaves 77 categories, and the score jumps to 0.8238. We merged 246 rare categories into one on purpose, and the model got much better because of it.

There's a clear sweet spot. Grouping too little, with a cutoff of 5, leaves 266 categories and barely helps, reaching only 0.8045. Grouping too much, with a cutoff of 100, drops back to 0.8168, because really different bundles get merged just for having few rows. The cutoff is a setting to tune, and 20 to 50 works well on this column.

One caution: picking the best of five cutoffs using the same folds flatters the winner a little. In real work, tune it on a separate validation split.

for min_freq in [20, 50]:
    encoder = OneHotEncoder(min_frequency=min_freq, handle_unknown='infrequent_if_exist',
                            sparse_output=False)
    encoded = pd.DataFrame(encoder.fit_transform(telco[['bundle']]),
                           index=telco.index).add_prefix('b_')
    score = logreg_auc(pd.concat([numeric_only, encoded], axis=1))
    print(f'OneHotEncoder(min_frequency={min_freq:3d}) -> {encoded.shape[1]:3d} columns, AUC {score:.4f}')

print()
print('Same result as the manual grouping, and it survives into production,')
print('because unseen categories are routed to the infrequent bucket automatically.')

OneHotEncoder(min_frequency= 20) ->  77 columns, AUC 0.8238
OneHotEncoder(min_frequency= 50) ->  23 columns, AUC 0.8235

Same result as the manual grouping, and it survives into production,
because unseen categories are routed to the infrequent bucket automatically.

OneHotEncoder's min_frequency setting gives exactly the same result as the manual grouping, and keeps it inside a pipeline. It also handles production better than the manual version, because unseen categories are sent to the same rare-category bucket automatically, as described above.

Section 7Putting the comparison together#

Every encoding from this notebook, on the same column, scored the same way

summary = pd.DataFrame([
    ('ignore the column',        3,   BASELINE),
    ('label encoding',           4,   logreg_auc(label_bundle)),
    ('one-hot, all 322',         325, logreg_auc(bundle_onehot)),
    ('frequency encoding',       4,   logreg_auc(freq_encoded)),
    ('hashing, 64 columns',      67,  logreg_auc(pd.concat([numeric_only, pd.DataFrame(
        FeatureHasher(n_features=64, input_type='string').transform(
            [[b] for b in telco['bundle']]).toarray(),
        index=telco.index).add_prefix('hash_')], axis=1))),
    ('one-hot, rare grouped',    80,  logreg_auc(pd.concat([numeric_only, pd.get_dummies(
        telco['bundle'].where(telco['bundle'].isin(counts[counts >= 20].index), 'Other'),
        prefix='b').astype(int)], axis=1))),
    ('target encoding, proper',  4,   proper_auc),
    ('target encoding, LEAKED',  4,   logreg_auc(leaky)),
], columns=['encoding', 'columns', 'AUC']).sort_values('AUC').reset_index(drop=True)

print(summary.round(4).to_string(index=False))

               encoding  columns    AUC
       one-hot, all 322      325 0.8007
      ignore the column        3 0.8087
         label encoding        4 0.8087
     frequency encoding        4 0.8094
    hashing, 64 columns       67 0.8156
  one-hot, rare grouped       80 0.8238
target encoding, proper        4 0.8250
target encoding, LEAKED        4 0.8450

Read the table from the bottom up. The leaked target encoding is at the bottom, so it's the first one you read, and it's the only entry that isn't a real result. Setting it aside, proper target encoding wins at 0.8250 using four columns in total, with rare-grouped one-hot close behind at 0.8238 using eighty.

Those two are close enough that the choice is about engineering, not score. Target encoding is much more compact but needs careful cross-fitting to stay safe. Grouped one-hot is easier to understand, can't leak the way target encoding can, and every column still has a clear meaning. On a production system, you'd want a strong reason before choosing the method that can fail silently over the one that can't.

At the top of the table, label encoding and plain one-hot both score at or below the baseline that ignores the column. Both took a useful column and made it useless, from opposite directions. One hid the differences between categories behind a meaningless alphabetical order. The other kept all 322 and gave the model too little data to learn any of them.