Why a data scientist should care whether two things move together.
Imagine you're a data scientist at Netflix. Your manager walks up and says: "We've noticed some users who watch more than 3 hours a day are churning faster. Can you look into it?"
Your first instinct is to check whether there's a relationship between hours watched and churn. Does watching more make people more likely to quit? Less likely? How strong is that link, if it's even there at all?
This is exactly where correlation comes in. It's the tool that tells you whether two variables are related, how strongly, and which direction they move in together.
Correlation is one of the first things you check on any new dataset. It helps you:
Correlation is a number that shows the strength and direction of the relationship between two numeric variables.
The key word is relationship. When two variables are correlated, it means changes in one tend to show up alongside changes in the other.
Correlation is expressed as a single number called the correlation coefficient, usually written as r. It always falls between -1 and +1, and that one number tells you two things at once: direction and strength.
Think of it as a ruler running from -1 on one end to +1 on the other, with 0 sitting in the middle.
When two variables move the same way, one goes up, the other goes up too, that's a positive correlation.
Everyday examples: taller people tend to weigh more. Hotter days mean more ice cream sold. More study hours tend to mean higher exam scores.
Big-tech example: at Amazon, the number of reviews a product has and its sales volume are positively correlated. More reviews build trust, and popular products naturally accumulate more reviews too, so the two feed each other. When Amazon's data team sees review counts drop in a category, it's often an early warning sign of a coming sales dip.
When two variables move opposite ways, one goes up while the other goes down, that's a negative correlation.
Everyday examples: more exercise tends to mean less body fat. As a product's price rises, demand tends to fall. More hours of TV generally means lower exam scores.
Big-tech example: at Uber, there's a well-known negative correlation between driver wait times and user retention. As estimated arrival time goes up, fewer users complete their booking. That single insight is a big part of why Uber invests so heavily in its driver-matching algorithms.
When there's no real pattern connecting two variables, r sits close to 0. Knowing one tells you nothing useful about the other.
Examples: someone's shoe size and their salary. How many films a person watches per year and their blood pressure. Daily coffee intake and IQ score.
The size of r, ignoring its sign, tells you how strong the correlation is. Here's a simple guide most data scientists lean on:
| Value of r | Strength of correlation |
|---|---|
| < 0.1 | No correlation |
| 0.1 – 0.3 | Little correlation |
| 0.3 – 0.5 | Medium correlation |
| 0.5 – 0.7 | High correlation |
| 0.7 – 1.0 | Very high correlation |
These are general guidelines, not hard rules. In social science, an r of 0.3 can be genuinely meaningful. In physics or engineering, you'd expect r above 0.95 before getting excited. Always read a correlation in the context of the field it came from.
Not all data is the same, so there isn't just one correlation coefficient. Depending on what your data looks like, you should reach for one of three main ones:
Let's go through each one properly.
Before we get to correlation, we need covariance, since correlation is built directly on top of it.
Covariance measures how two variables change together. If both tend to sit above their own average at the same time, covariance comes out positive. If one tends to be above average while the other is below, it comes out negative.
The formula for covariance is:
The covariance is 30. Is that big or small? There's no way to tell without context, exactly the "which ruler is this" problem Day 5 spent a whole article on. That's why we divide by the standard deviations to get correlation.
The problem with covariance: it depends entirely on the units you measured in. Measure height in centimetres instead of metres and you get a completely different covariance number, even though the actual relationship hasn't changed at all. That makes covariance impossible to compare fairly across different datasets.
This is exactly why correlation exists. It standardizes covariance by dividing it by the standard deviations of both variables, which gives you a number with no units that always sits between -1 and +1.
The Pearson correlation coefficient is the most widely used measure of correlation. It captures the strength and direction of the straight-line relationship between two continuous variables.
Dividing covariance by the product of the two standard deviations is what forces the result to always land between -1 and +1, no matter what units the original data used.
Say we're checking whether the hours a Netflix user watches per week is related to how satisfied they say they are in post-viewing surveys (rated 1 to 10).
Before you trust a Pearson r, check whether the relationship actually looks like a straight line. The only real way to check is to plot it and look.
In practice you rarely look at just two variables at a time. You look at many features at once using a correlation matrix: a table showing the correlation between every pair of variables. Companies like Google and Meta lean on exactly this during early data exploration.
Reading a heatmap is simple once you know the rule: red cells (near +1) mean a strong positive correlation, blue cells (near -1) mean a strong negative correlation, and white or light cells (near 0) mean barely any relationship at all.
In this example, cart_additions and purchase_value are highly correlated, which makes total sense. That's exactly the kind of finding that helps a product team decide what to prioritize in a checkout redesign.
Pearson's r is a parametric measure, which means it assumes certain things about your data. Break these assumptions and you can end up trusting a number that's actively misleading you.
Think about stress and job performance. Up to a point, more stress can actually push someone to perform better, a positive relationship. But past that point, high stress wrecks performance instead, a negative relationship. Pearson looks at the whole picture at once, sees something roughly linear on average, and completely misses that U-shaped curve underneath.
Here's a real, famous illustration of the same trap.
0.82 sounds like a strong linear relationship. It isn't one. This dataset is part of Anscombe's Quartet, four datasets built by statistician Francis Anscombe in 1973 specifically to prove this point: they all share the same mean, the same variance, and the same Pearson r, and they look nothing alike when you actually plot them.
Always visualize your data. Never trust a correlation number without also looking at a scatter plot.
One single point turned a strong positive relationship into a negative one. That's the entire relationship, flipped, by adding one data point out of nine. This is exactly why Spearman's correlation exists: it's built to shrug off exactly this kind of outlier.
Spearman's rank correlation is the go-to alternative whenever Pearson's assumptions don't hold up. Instead of working with the raw numbers, it first converts everything into ranks, then runs Pearson's formula on those ranks.
That one change makes it:
Pearson needs the relationship to be a straight line. Spearman only needs it to be monotonic.
A monotonic relationship just means: as X goes up, Y either always goes up or always goes down. It doesn't have to happen in a straight line, it just can't reverse direction.
Think of climbing a mountain on a winding road. You keep gaining altitude the whole way (monotonically increasing), even though the incline changes constantly. Spearman captures that perfectly. Pearson, expecting a straight line, struggles with it.
Monotonically increasing: as one variable increases, the other never decreases. Example: age and years of work experience.
Monotonically decreasing: as one variable increases, the other never increases. Example: time spent on a support call and the customer's satisfaction score.
Not monotonic: the other variable goes up and down as the first one increases. Example: caffeine and focus. A little caffeine helps focus, a moderate amount helps more, but too much wrecks it.
Given data for X and Y:
When there are no ties, there's a shortcut:
Here, di is the difference between the two ranks for each pair, and n is the number of observations.
Say you're at Google, looking at the Play Store. You want to know whether an app's engagement score (an internal metric, not normally distributed, with some extreme values) relates to its number of active installs.
Notice Pearson's number is actually higher here, not lower. That's worth sitting with for a second: Spearman being the "more appropriate" choice for skewed, outlier-prone data doesn't mean it always spits out a bigger or smaller number. It means it's measuring something more trustworthy: the consistency of the rank order, rather than a relationship that a couple of viral outlier apps could be quietly inflating.
Back in 2017, Netflix dropped its 5-star rating system for a simple thumbs up or thumbs down. Netflix's own stated reasons were about engagement, not statistics: thumbs got roughly 200% more ratings than stars did in testing, and a lot of users were confused about what a star average even represented. But there's a real statistical point sitting underneath that decision too. Star ratings are ordinal data, so the gap between 3 and 4 stars was never guaranteed to mean the same thing as the gap between 1 and 2. If you wanted to correlate star ratings with anything else, Spearman was always the more defensible tool, not Pearson.
In this particular toy example the two numbers land close together, both are very high, because the ranks and the raw values happen to agree almost perfectly here. That won't always happen. The real reason Spearman is the correct tool for star ratings has nothing to do with which number comes out bigger: it's that Pearson's formula silently assumes the gap between 3 and 4 stars equals the gap between 1 and 2 stars, and for a human rating scale, that assumption is usually just wrong.
Reach for Spearman when your data is ordinal (ratings, rankings, satisfaction levels), when it has real outliers, when it isn't normally distributed (income data is a classic example, since it's heavily right-skewed), or when the relationship looks curved but still monotonic on a scatter plot.
Kendall's tau is another rank-based correlation, similar in spirit to Spearman's. The difference is in exactly what it counts.
For every possible pair of observations, Kendall's tau asks a simple question: do X and Y move in the same direction for this pair? If yes, that's a concordant pair. If they move in opposite directions, it's a discordant pair.
Kendall's tau tends to be the better choice when your sample size is small (it stays more accurate with fewer points), when you have a lot of tied ranks, or when you want something easy to explain in plain language: a tau of 0.6 breaks down to a clean split, 80% of all pairs agreed on direction and 20% disagreed.
Out of all 10 possible pairs in this tiny dataset, 8 agreed on direction (concordant) and 2 didn't (discordant), which is exactly where the 0.6 comes from. With only 5 data points, the p-value of 0.23 is a reminder not to read too much into it: this sample is too small to call the result statistically significant, even though the tau itself looks decent.
Big-tech use case: at Microsoft, when engineers evaluate search result quality, they often rank pages by relevance and compare two different ranking algorithms' outputs using Kendall's tau. A high tau means the two algorithms broadly agree on what's relevant. A low one means they disagree a lot, and someone needs to dig into why.
Here's a simple way to decide:
Is your data continuous and roughly normally distributed?
Yes: use Pearson's r.
No, keep going.
Is your data ordinal, does it have outliers, or is the relationship non-linear but still monotonic?
Yes: use Spearman's ρ.
Is your sample very small (fewer than 30 points), or do you have a lot of tied ranks?
Yes: use Kendall's τ.
This might be the single most important idea in all of statistics. Two variables being correlated does not mean one of them causes the other.
There's a famous, silly-sounding example: the number of Nicolas Cage movie releases in a year correlates with the number of drowning deaths, since both tend to peak in summer. Obviously his movies aren't drowning anyone. Both are being driven by a third factor entirely: summer itself, when pools are busiest and movie releases cluster. That hidden third factor has a name: a confounding variable, sometimes called a lurking variable.
Big-tech example: early studies found that Facebook users who shared more posts also had higher engagement with ads. It would be tempting to conclude "let's get people to share more, and ad revenue will follow." But the causation actually runs the other way: people who are already more engaged with the platform in general tend to both share more and click more ads. Forcing people to share more without any real engagement behind it wouldn't move ad performance at all.
Whenever you see a correlation, ask: is there a third variable that could explain this? Could the cause and effect actually be reversed from what seems obvious? For a long list of entertaining examples of correlations that mean absolutely nothing, Tyler Vigen's Spurious Correlations is worth a look.
One of the most practical everyday uses of correlation in data science is feature selection: deciding which columns actually deserve a place in your model.
The idea is simple. If two input features are very highly correlated with each other (say, above 0.90), they're carrying almost identical information. Keeping both doesn't help your model. It can actually hurt it, by causing multicollinearity in linear models.
In this housing dataset, bedrooms, bathrooms, and area_sqft are all highly correlated with each other, since they're all really just measuring the size of the house from different angles. Before feeding this into a linear regression model, you'd want to drop one or two of them rather than keep all three.
Here's roughly how a working data scientist would approach a correlation analysis from start to finish, using that same housing dataset.
The function checked normality first (p = 0.74, comfortably normal, so it picked Pearson automatically), then ranked every feature by how strongly it relates to price. area_sqft, bedrooms, and bathrooms all come out strongly positive, and we already know from the heatmap above that those three are highly correlated with each other too. house_age is the one interesting result here: its correlation is weak enough (p = 0.22) that we can't even be confident it's real, not just noise in this particular sample.
Correlation is one of the most useful, and most misused, tools in data science. Used with the right method, backed up by an actual plot, and paired with a healthy dose of skepticism about causation, it can genuinely turn up real insight from a dataset. Skip any one of those three things and it can just as easily mislead you.
Four questions to test whether this stuck.
Not necessarily. Pearson only detects straight-line relationships. Two variables could have a strong curved relationship, like the stress and job-performance example, and still produce a Pearson r near zero. Always plot the data before ruling out a relationship.
Spearman's rho. Star ratings are ordinal, so the gap between 3 and 4 stars isn't guaranteed to mean the same thing as the gap between 1 and 2. Spearman only needs the relationship to move consistently in one direction, which fits ordinal data far better than Pearson's straight-line assumption.
No. Both tend to rise in summer, when more people are swimming and more people are buying ice cream. Summer is the confounding variable driving both, not any real connection between the two.
Consider dropping one of them. At that level of correlation they're carrying almost the same information, and keeping both can cause multicollinearity, which makes a linear model's coefficients unstable and much harder to interpret.