Recall exactly where the checkpoint article left off. Six models trained on the churn dataset, results sitting in a table because the article's author happened to write them down. In a real project, that table doesn't exist unless someone builds it by hand, every time, and remembers to keep it updated. Nothing in Git, DVC, Docker, or FastAPI was ever meant to solve that problem, because none of them are about experiments. They're about code, data, environments, and APIs.
MLflow is what closes that gap. Specifically, this article is about MLflow Tracking, one of MLflow's four components, the one that logs parameters, metrics, and artifacts automatically, every time you train something, so the question "which run had max_depth=6 and what was its recall" has an actual answer instead of a guess.
By the end of this article, the churn project's six experiments will be logged properly, comparable side by side in a real UI, and you'll understand exactly what MLflow captured, what it didn't, and why that distinction matters.
What this article covers
| # | Topic |
|---|---|
| 1 | The problem, generalized beyond one project |
| 2 | What MLflow is, and what Tracking specifically does |
| 3 | The mental model: experiments, runs, and where they're stored |
| 4 | Installing MLflow and your first logged run |
| 5 | Tracking the churn experiments properly |
| 6 | Comparing runs in the MLflow UI |
| 7 | Autologging: the one-liner, and its real limits |
| 8 | Logging artifacts beyond metrics |
| 9 | When AIC and BIC actually apply, and when they don't |
Part 1 — The problem, generalized
Article 7.5's chaos wasn't specific to that project. It's what happens on every real project, every time, unless something intervenes.
A data scientist trains a model. It's decent. They try a different algorithm. Better, maybe. They tune a hyperparameter. Better again, or worse, hard to tell from memory. By the fifth or sixth attempt, three things have quietly become true: nobody can say with confidence which exact configuration produced which exact number, the code that produced the best result may have already been overwritten by the next attempt, and the only durable record is whatever got written down by hand, if anything did.
This isn't a discipline problem. Careful people hit this too. It's a tooling gap: training a model produces information (parameters, metrics, the model itself) that has nowhere permanent to live unless you build that place yourself, from scratch, every single project.
Part 2 — What MLflow is, and what Tracking does
MLflow was built at Databricks and open-sourced in June 2018, created by Matei Zaharia, the same person behind Apache Spark. It was contributed to the Linux Foundation in 2020 and is now used by an enormous number of teams: MLflow reports over 30 million monthly downloads.1
MLflow ships four components, and it's worth knowing the names even though this article only covers one of them:
- MLflow Tracking — logs parameters, metrics, and artifacts from training runs. This article.
- MLflow Projects — a standard format for packaging reusable ML code so it runs the same way anywhere. Not covered here.
- MLflow Models — a standard format for packaging trained models so they can be deployed through multiple serving tools without rewriting anything. Touched on briefly when we log models below.
- MLflow Model Registry — a centralized store for managing which model version is live. Article 9.
Tracking specifically answers one question, reliably: for any model you've ever trained through it, what exactly produced it, and how did it perform?
Part 3 — The mental model: experiments, runs, and where they live
Four ideas, and the rest of MLflow Tracking is just applying them.
A run is one execution of your training code, one specific attempt. In Article 7.5's terms, each of the six experiments is a run.
An experiment is a named group of related runs. All six churn attempts belong to one experiment, something like churn-prediction. Different projects, or genuinely different problems within one project, get different experiments.
Each run logs three kinds of things: parameters (the settings you chose, like n_estimators=200), metrics (the numbers that came out, like accuracy=0.8048), and artifacts (files, the trained model itself, a confusion matrix plot, anything you want attached to that specific run).
All of this needs somewhere to actually live, and here's a detail worth getting right before writing a line of code: as of MLflow 3.7, a fresh install defaults to a small SQLite database (mlflow.db) for storing run metadata, not the classic ./mlruns folder full of files that older MLflow versions used and that most existing MLflow tutorials still show.2 If you have an existing project with ./mlruns data already in it, MLflow keeps using that automatically. If you're starting fresh, which this project is, you'll get the database by default.
This split matters for a concrete reason coming up in Article 9: the Model Registry requires a database-backed store. The classic pure-file ./mlruns setup can't support it at all. The fact that MLflow now defaults to a database isn't just a performance choice, it's what makes next article's content possible without an extra setup step.
Part 4 — Installing MLflow and your first logged run
pip install mlflowThe smallest possible tracked run:
import mlflow
with mlflow.start_run():
mlflow.log_param("n_estimators", 100)
mlflow.log_metric("accuracy", 0.87)Three lines, and something real just happened: MLflow created an experiment (the default one, since none was named), started a run inside it, and recorded one parameter and one metric to mlflow.db. with mlflow.start_run(): is the pattern for everything that follows, it opens a run, and closes it cleanly when the block ends, even if your training code raises an exception partway through.
Look at it:
mlflow ui --port 5000Open http://localhost:5000. One run, one parameter, one metric, already there. This is the same interface you'll use to compare all six churn experiments in a moment.
A practical note on timing: MLflow's UI server takes noticeably longer to start than Flask or FastAPI's dev servers, sometimes 15 to 20 seconds before it's actually answering requests, since it spins up multiple worker processes underneath. If you hit connection refused right after starting it, that's normal. Give it a few more seconds.
Part 5 — Tracking the churn experiments properly
Here's Article 7.5's explore_models.py, rewritten with real tracking instead of print statements. Everything else about the pipeline, the ColumnTransformer, the train/test split, is untouched.
"""track_experiments.py — the same experiments from Article 7.5, now tracked."""
import pandas as pd
import mlflow
import mlflow.sklearn
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
df = pd.read_csv("data/telco_churn.csv")
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce").fillna(0)
df = df.drop(columns=["customerID"])
df["Churn"] = df["Churn"].map({"Yes": 1, "No": 0})
X = df.drop(columns=["Churn"])
y = df["Churn"]
numeric_features = ["SeniorCitizen", "tenure", "MonthlyCharges", "TotalCharges"]
categorical_features = [c for c in X.columns if c not in numeric_features]
preprocessor = ColumnTransformer(transformers=[
("num", StandardScaler(), numeric_features),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features),
])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
mlflow.set_experiment("churn-prediction")
def run_experiment(run_name, clf, params):
with mlflow.start_run(run_name=run_name):
mlflow.log_params(params)
pipe = Pipeline([("preprocessor", preprocessor), ("classifier", clf)])
pipe.fit(X_train, y_train)
pred = pipe.predict(X_test)
acc = accuracy_score(y_test, pred)
prec = precision_score(y_test, pred)
rec = recall_score(y_test, pred)
f1 = f1_score(y_test, pred)
mlflow.log_metric("accuracy", acc)
mlflow.log_metric("precision", prec)
mlflow.log_metric("recall", rec)
mlflow.log_metric("f1", f1)
mlflow.sklearn.log_model(pipe, name="model")
print(f"{run_name}: acc={acc:.4f} f1={f1:.4f}")
run_experiment("logreg_default", LogisticRegression(max_iter=1000, random_state=42),
{"model_type": "LogisticRegression", "max_iter": 1000})
run_experiment("rf_100", RandomForestClassifier(n_estimators=100, random_state=42),
{"model_type": "RandomForest", "n_estimators": 100, "max_depth": None})
run_experiment("rf_200_depth10", RandomForestClassifier(n_estimators=200, max_depth=10, random_state=42),
{"model_type": "RandomForest", "n_estimators": 200, "max_depth": 10})
run_experiment("rf_300_depth6", RandomForestClassifier(n_estimators=300, max_depth=6, random_state=42),
{"model_type": "RandomForest", "n_estimators": 300, "max_depth": 6})
run_experiment("gb_100_lr01", GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, random_state=42),
{"model_type": "GradientBoosting", "n_estimators": 100, "learning_rate": 0.1})
run_experiment("gb_200_lr005", GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, random_state=42),
{"model_type": "GradientBoosting", "n_estimators": 200, "learning_rate": 0.05})Every piece worth noticing: mlflow.set_experiment("churn-prediction") creates or reuses one named experiment so all six runs land in the same place, instead of scattering across MLflow's default unnamed one. mlflow.log_params(params) logs a whole dictionary at once, the exact configuration that produced this run, in one call. mlflow.sklearn.log_model(pipe, name="model") saves the entire fitted pipeline, preprocessing included, as an artifact attached to this specific run, the same "save the whole pipeline" habit from Article 7.5, just attached to a run instead of a loose file.
python track_experiments.pylogreg_default: acc=0.8055 f1=0.6040
rf_100: acc=0.7864 f1=0.5474
rf_200_depth10: acc=0.7999 f1=0.5766
rf_300_depth6: acc=0.8048 f1=0.5710
gb_100_lr01: acc=0.8027 f1=0.5813
gb_200_lr005: acc=0.8048 f1=0.5877Same numbers as Article 7.5, confirmed by actually rerunning the exact code. Nothing about the models changed. What changed is that this time, none of it is gone the moment the terminal scrolls past it.
Part 6 — Comparing runs in the UI
mlflow ui --port 5000Open the UI and go to the churn-prediction experiment. Every run is listed with its parameters and metrics as actual columns, sortable. Click the accuracy column to sort descending, and this is what comes back, pulled directly from MLflow's own run data:
| Run | Accuracy | F1 | Parameters |
|---|---|---|---|
| logreg_default | 0.8055 | 0.6040 | model_type=LogisticRegression, max_iter=1000 |
| gb_200_lr005 | 0.8048 | 0.5877 | model_type=GradientBoosting, n_estimators=200, learning_rate=0.05 |
| rf_300_depth6 | 0.8048 | 0.5710 | model_type=RandomForest, n_estimators=300, max_depth=6 |
| gb_100_lr01 | 0.8027 | 0.5813 | model_type=GradientBoosting, n_estimators=100, learning_rate=0.1 |
| rf_200_depth10 | 0.7999 | 0.5766 | model_type=RandomForest, n_estimators=200, max_depth=10 |
| rf_100 | 0.7864 | 0.5474 | model_type=RandomForest, n_estimators=100, max_depth=None |
That's the entire Article 7.5 dilemma, resolved in one click. Which run had max_depth=6? Third row, accuracy 0.8048. Which model actually won? Depends whether you're sorting by accuracy or recall, and now you can just sort by each and see, instead of scrolling through terminal history hoping you didn't lose it.
Click into any individual run and MLflow shows its full detail page: every logged parameter, every logged metric, the exact model artifact you can download or load directly, and, since MLflow captures it automatically, the exact Git commit your code was at when that run happened, if the project is a Git repository. That last part is worth sitting with. A run doesn't just remember its own metrics anymore. It remembers which version of your code produced them.
Part 7 — Autologging: the one-liner, and its real limits
Writing log_params and log_metric by hand, as above, gives you full control over exactly what gets recorded. MLflow also offers a shortcut:
mlflow.sklearn.autolog()Add that one line before training, and MLflow automatically captures nearly everything about a scikit-learn model without another line of logging code. Tested directly against the churn pipeline, here's what showed up with zero manual logging: every hyperparameter of every step in the pipeline, including the nested ColumnTransformer and its OneHotEncoder/StandardScaler settings, dozens of parameters in total, and a full set of automatic metrics, precision, recall, F1, accuracy, log loss, ROC AUC, plus the trained model itself as an artifact.
That's a lot of value for one line. There's a real catch worth knowing before you rely on it, confirmed by actually inspecting what got logged: autolog's automatic metrics are computed on the training set, not a held-out test set. In the test above, training_accuracy_score came back as 0.998, an almost perfect score, while the same model's actual test-set accuracy, logged separately, was 0.7864. The first number reflects how well the model memorized data it already saw. The second reflects what actually matters, how it performs on data it hasn't.
This isn't a bug, autolog is explicit that its metrics are training metrics. But it means autolog alone isn't quite enough for genuine model comparison. The practical pattern: use autolog() to capture the full hyperparameter surface and the model artifact for free, and still compute and log your own test-set metrics manually, the way Part 5 did, for the numbers you actually intend to compare runs by.
mlflow.sklearn.autolog()
with mlflow.start_run(run_name="rf_100_autolog"):
pipe = Pipeline([("preprocessor", preprocessor), ("classifier", RandomForestClassifier(n_estimators=100, random_state=42))])
pipe.fit(X_train, y_train)
# autolog already captured every hyperparameter and a training-set metric suite —
# this line adds the one number that actually decides which run wins
test_accuracy = pipe.score(X_test, y_test)
mlflow.log_metric("test_accuracy", test_accuracy)Part 8 — Logging artifacts beyond metrics
Anything that's a file can be an artifact: a confusion matrix plot, a sample of the input data, a text file of notes. Confusion matrices are the most immediately useful for a classification project like this one.
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
with mlflow.start_run(run_name="logreg_with_plot"):
pipe.fit(X_train, y_train)
pred = pipe.predict(X_test)
mlflow.log_metric("accuracy", accuracy_score(y_test, pred))
fig, ax = plt.subplots(figsize=(5, 5))
ConfusionMatrixDisplay.from_predictions(y_test, pred, ax=ax)
mlflow.log_figure(fig, "confusion_matrix.png")
plt.close(fig)mlflow.log_figure saves the plot directly to the run's artifact folder, no manual file handling, no separate upload step. Open that run in the UI and the confusion matrix is right there next to its metrics, permanently attached to the exact run that produced it.
This is also where the DVC habit from Article 3 pays off a second time. Log the DVC data hash as a tag on the run:
mlflow.set_tag("data_version", "telco_churn.csv.dvc hash a1b2c3d4")Now a run doesn't just remember its code version and its hyperparameters. It remembers exactly which version of the data trained it too.
Part 9 — When AIC and BIC actually apply
AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) are model-selection metrics that balance how well a model fits the data against how complex the model is, penalizing extra parameters so a model can't win purely by memorizing more detail. Lower is better for both, and BIC penalizes complexity more heavily than AIC does.
Here's the honest part: neither one applies cleanly to the Random Forest and Gradient Boosting models used throughout this project. AIC and BIC are computed from a model's log-likelihood, which requires a model with an explicit, well-defined probability structure, the kind linear and logistic regression have. Tree ensembles don't have that in the same sense, which is why scikit-learn only exposes AIC/BIC support for one narrow case, LassoLarsIC, a linear regression variant, and nothing in the tree-ensemble family.3
Where AIC/BIC genuinely fit is a plain logistic regression, fit through statsmodels rather than scikit-learn, since statsmodels reports them natively. Using a handful of the same churn features, actually fit and verified, on the same train/test split as every other model in this project:
import statsmodels.api as sm
from sklearn.preprocessing import StandardScaler
df["is_month_to_month"] = (df["Contract"] == "Month-to-month").astype(int)
df["has_fiber"] = (df["InternetService"] == "Fiber optic").astype(int)
df["paperless"] = (df["PaperlessBilling"] == "Yes").astype(int)
features = ["tenure", "MonthlyCharges", "is_month_to_month", "has_fiber", "paperless", "SeniorCitizen"]
X_stats = df[features]
y_stats = df["Churn"]
# same split as every other model in this article — fit on train, not the full dataset
X_train_s, X_test_s, y_train_s, y_test_s = train_test_split(
X_stats, y_stats, test_size=0.2, random_state=42, stratify=y_stats
)
X_train_s = X_train_s.copy()
X_train_s[["tenure", "MonthlyCharges"]] = StandardScaler().fit_transform(X_train_s[["tenure", "MonthlyCharges"]])
X_train_s = sm.add_constant(X_train_s)
with mlflow.start_run(run_name="logreg_statsmodels_aic_bic"):
model = sm.Logit(y_train_s.values, X_train_s.astype(float)).fit(disp=0)
mlflow.log_metric("aic", model.aic)
mlflow.log_metric("bic", model.bic)
mlflow.log_metric("log_likelihood", model.llf)AIC: 4857.28
BIC: 4903.74Both numbers, and every coefficient in the fitted model, came back statistically significant. Logged into MLflow exactly like any other metric, this run sits right alongside the six from Part 5, directly comparable on accuracy and F1, and additionally carrying AIC/BIC where the other six simply can't. That's the honest way to use AIC/BIC in a project like this one: as an extra lens on the one model type built to support it, not a universal yardstick applied to every model regardless of whether the math behind it makes sense.
Summary
The problem Article 7.5 ended on is solved. Six real experiments, each with its exact parameters, metrics, and trained model, permanently recorded and sortable in seconds instead of reconstructed from memory. MLflow itself was built at Databricks in 2018 and has four components; this article covered Tracking, the one that logs runs. The mental model is small: experiments group runs, runs log parameters, metrics, and artifacts, and as of MLflow 3.7 all of that lives in a small SQLite database by default, which happens to be exactly what the next article needs. Manual logging gives precise control; autolog() captures far more with one line, but its automatic metrics are training-set numbers, so genuine model comparison still wants your own test-set metrics logged explicitly. Artifacts, plots, models, even a DVC data hash as a tag, attach permanently to the run that produced them. And AIC/BIC, while genuinely useful, belong to models with a real likelihood function, logistic regression through statsmodels here, not to the tree ensembles that did most of the work in this project.
One gap remains. Every run in this article logged a model as an artifact, but nothing yet decides which one is actually live. main.py would still need someone to know, by memory, which run's model to load. That's Article 9.