Pandas for Data Analysis: A Complete Beginner's Guide

This guide walks through the entire Pandas workflow from the ground up, using a real retail sales dataset as the running example. Throughout, you'll find visual diagrams that show exactly what's happening to your data at each step, because data manipulation is so much easier to understand when you can see it.

What Is Pandas?

Pandas is a Python library built on top of another library called NumPy, designed specifically for data analysis. You'll use it constantly for data manipulation, preparing data for visualization, and building machine learning models.

To use it, you import it with the near-universal nickname pd:

import pandas as pd
import numpy as np

Pandas has two core data structures, and everything else builds on them:

   SERIES                         DATAFRAME
   (one column)                   (a whole table)

   ┌─────────┐                    ┌──────┬────────┬───────┐
   │ Maths   │                    │ id   │ Profit │ Sales │
   │ ML      │                    ├──────┼────────┼───────┤
   │ Python  │                    │ 101  │   12   │  345  │
   │ Data    │                    │ 102  │   14   │  765  │
   │ Science │                    │ 103  │   21   │  321  │
   │ Java    │                    │ 104  │   65   │  876  │
   └─────────┘                    └──────┴────────┴───────┘

   1-dimensional                  2-dimensional

A Series is a single column of data. A DataFrame is a full table, and you can think of a DataFrame as simply a bunch of Series stacked side by side. The DataFrame is what you'll work with 95% of the time.

Part 1: The Pandas Series

A Series is like a one-dimensional array, but instead of being a row, it's represented as a single column. Critically, every value in a Series has an index (a label) attached to it.

Creating a Series

text_series = pd.Series(['Maths', 'ML', 'Python', 'Data', 'Science', 'Java'])

Here's what Pandas builds — the index on the left is added automatically:

 index   value
   0   │  Maths
   1   │  ML
   2   │  Python
   3   │  Data
   4   │  Science
   5   │  Java

Inspecting a Series

Pandas gives you a set of attributes to examine any Series. Most have no parentheses because they're properties that describe the object rather than actions it performs. The exception is memory_usage(), which is a method (it does a small calculation to include the index size).

One thing worth knowing on recent Pandas versions: .values doesn't always hand back a plain NumPy array. For a numeric Series it does, but text columns are now backed by Pandas's own StringArray by default, so .values on a text Series returns that instead. If you specifically need a NumPy array regardless of dtype, use .to_numpy().

Naming a Series

You can give a Series a name, which becomes its column header when it joins a DataFrame:

text_series_2 = pd.Series(['Maths', 'ML', 'Python', 'Data', 'Science', 'Java'],
                          name='text series')
text_series_2.name    # 'text series'

Accessing Series Data

You access elements much like a list:

text_series_2[0]      # 'Maths' — single element
text_series[0:4]      # the first four elements (a slice)

A Subtle but Important Point: Custom Indexes

You can assign your own index labels instead of the default 0, 1, 2...:

num_series = pd.Series([90, 23, 45, 21, 78, 98, 65, 43, 34],
                       index=[5, 6, 7, 8, 9, 10, 11, 12, 13])

 YOUR index    value
     5      │   90
     6      │   23
     7      │   45      ← num_series[7] returns 45
     8      │   21
     9      │   78
    10      │   98
    ...     │  ...

Now here's a quirk that trips up many beginners. When you have a custom index, indexing and slicing behave differently:

num_series[7]      # returns 45 → uses YOUR label (explicit index)
num_series[2:7]    # uses POSITION 2 to 6 (implicit index)

• Indexing with a single value uses your explicit (custom) index labels.
• Slicing with a range uses the implicit (default positional) index.

This inconsistency is exactly why Pandas gives us two dedicated tools, loc and iloc, to remove all ambiguity. We'll get to those shortly.

Part 2: The Pandas DataFrame

The DataFrame is the star of the show, a table with labeled rows (the index) and named columns. This is the structure you'll spend nearly all your time working with.

Creating a DataFrame from a Dictionary

The keys become column names, and the values (lists) become the columns:

df = pd.DataFrame({'Cust_id': [101, 102, 103, 104, 105],
                   'Profit':  [12, 14, 21, 65, 89],
                   'Sales':   [345, 765, 321, 876, np.nan]})

   ┌─────────┬────────┬───────┐
   │ Cust_id │ Profit │ Sales │
   ├─────────┼────────┼───────┤
 0 │   101   │   12   │  345  │
 1 │   102   │   14   │  765  │
 2 │   103   │   21   │  321  │
 3 │   104   │   65   │  876  │
 4 │   105   │   89   │  NaN  │  ← np.nan means "missing value"
   └─────────┴────────┴───────┘

That NaN ("Not a Number") is how Pandas represents missing data, an empty cell. You'll meet it constantly in real datasets, and handling it is a core part of data cleaning.

Think of it like a paper form where someone skipped a field. The form isn't invalid, one box is just blank. NaN is Pandas's way of marking that blank box instead of guessing a value or crashing.

Creating a DataFrame from a CSV File

In the real world, your data usually lives in files. The read_csv function loads a CSV straight into a DataFrame:

market = pd.read_csv('market_fact.csv')

This single line is the starting point of almost every data analysis project.

First Steps: Getting to Know Your Data

Whenever you load a new dataset, you run a handful of commands to understand what you're dealing with. These are your "first look" toolkit:

head() and info() are the two you'll run most. head() shows you what the data looks like. info() tells you what's in it and whether anything's missing.

Part 3: Indexing and Slicing a DataFrame

Selecting the specific rows and columns you need is the most fundamental skill in Pandas. Let's start with the basic approaches.

Selecting Rows by Position

market[100:501]         # rows 100 through 500
market[100:1001:10]     # rows 100 to 1000, taking every 10th row

Selecting Columns

market['Sales']                          # a single column → returns a Series
market[['Profit', 'Sales', 'Discount']]  # multiple columns → returns a DataFrame

Note the difference in brackets, and what comes back:

 market['Sales']              market[['Profit','Sales']]
 (single brackets)            (double brackets — a LIST of columns)

 → a SERIES                   → a DATAFRAME
 ┌───────┐                    ┌────────┬───────┐
 │ Sales │                    │ Profit │ Sales │
 ├───────┤                    ├────────┼───────┤
 │  345  │                    │   12   │  345  │
 │  765  │                    │   14   │  765  │
 └───────┘                    └────────┴───────┘

The rule: single brackets give a Series, double brackets give a DataFrame. When you want multiple columns, you pass a list of names, and a list itself needs square brackets, which is why you see two sets.

Part 4: The Right Way — loc and iloc

The plain indexing above works, but it has the ambiguity problems we saw with Series. Pandas solves this with two precise selection tools. Mastering these is what separates a beginner from someone who's comfortable in Pandas.

The mental model:

   iloc  →  selects by POSITION  (i = integer)   →  always 0, 1, 2, 3...
   loc   →  selects by LABEL     (l = label)     →  uses the actual index/column names

It's the difference between "give me the third person in line" (iloc, a seat number that never changes) and "give me the person named Priya" (loc, a name that stays attached to them no matter where they move in line).

iloc — Selection by Position

iloc uses integer positions only, exactly like coordinates on a grid. The format is iloc[rows, columns]:

market.iloc[2]              # the entire row at position 2
market.iloc[3, 7]           # single cell: row 3, column 7
market.iloc[2, 4:8]         # row 2, columns at positions 4 to 7
market.iloc[0:6, 3:8]       # rows 0-5, columns 3-7 (a rectangular block)
market.iloc[[1,3,7], [4,6,7,9]]   # specific rows AND specific columns

Visually, iloc[0:6, 3:8] grabs a rectangular block by position:

        col0 col1 col2 │col3 col4 col5 col6 col7│ col8
 row0    .    .    .   │ ███  ███  ███  ███  ███ │  .
 row1    .    .    .   │ ███  ███  ███  ███  ███ │  .
 row2    .    .    .   │ ███  ███  ███  ███  ███ │  .
 row3    .    .    .   │ ███  ███  ███  ███  ███ │  .
 row4    .    .    .   │ ███  ███  ███  ███  ███ │  .
 row5    .    .    .   │ ███  ███  ███  ███  ███ │  .
 ─────                 └────────────────────────┘
 row6    .    .    .     .    .    .    .    .      .
                         (the shaded block is what you get)

loc — Selection by Label

loc uses the actual labels. The index values and column names:

market.loc[2, ['Profit', 'Sales']]           # row labeled 2, named columns
market.loc[10:21, ['Sales', 'Profit']]        # row labels 10-21, specific columns by name

One Key Difference: Slicing Behavior

There's an important distinction in how the two handle ranges:

   iloc[0:6]  →  stops BEFORE 6  →  gives positions 0,1,2,3,4,5   (6 rows)
   loc[10:21] →  INCLUDES 21     →  gives labels 10 through 21    (inclusive)

iloc excludes the endpoint (like normal Python slicing). loc includes it, because with labels it would be confusing to leave out the exact row you named.

Why This Matters: Changing the Index

Here's where loc becomes essential. Suppose we replace the default numeric index with a meaningful column:

market_copy = market.copy()                       # always work on a copy!
market_copy.set_index('Cust_id', inplace=True)    # make Cust_id the new index

Now the index is customer IDs like Cust_1818 instead of 0, 1, 2...:

  BEFORE set_index            AFTER set_index
  ┌───┬───────────┐           ┌───────────┬────────┐
  │ # │  Cust_id  │           │ (index)   │ Profit │
  ├───┼───────────┤           ├───────────┼────────┤
  │ 0 │ Cust_1818 │    →      │ Cust_1818 │   ...  │
  │ 1 │ Cust_1755 │           │ Cust_1755 │   ...  │
  └───┴───────────┘           └───────────┴────────┘

To select by this new index, you must use loc, because iloc only ever understands positions:

market_copy.loc['Cust_1818', ['Profit', 'Sales']]   # works perfectly

The .copy() method creates an independent duplicate, so changes to market_copy don't affect the original market. And inplace=True tells Pandas to modify the DataFrame directly rather than returning a new one.

Part 5: Filtering Data with Conditions

This is one of the most useful things Pandas does. Filtering means selecting only the rows that meet a condition, like "show me only the unprofitable sales."

How Filtering Works Under the Hood

A condition on a column produces a Series of True/False values, one for each row. This is called a boolean mask:

market['Profit'] < 0

 row    Profit      Profit < 0
  0  │    52    │     False
  1  │   -30    │     True
  2  │    18    │     False
  3  │   -45    │     True
  4  │    91    │     False

When you pass this mask back into the DataFrame, Pandas keeps only the rows where the mask is True:

market[market['Profit'] < 0]    # only the loss-making rows

   The mask        applied to data      gives only True rows
   ────────        ───────────────      ───────────────────
   False              [keep? no]
   True       →       [keep? YES]   →    ┌─ row 1 (Profit -30) ─┐
   False              [keep? no]         └─ row 3 (Profit -45) ─┘
   True               [keep? YES]
   False              [keep? no]

Combining Multiple Conditions

You can stack conditions together using three operators. Important: in Pandas you use the symbols &, |, ~ (not the words and, or, not), and each condition must be wrapped in its own parentheses.

Skip the parentheses and you'll hit a genuinely confusing error, because & binds tighter than comparison operators like > in Python, so the expression gets grouped in a way you didn't intend:

market[market['Sales'] > 0 & market['Profit'] < 0]   # missing parens
# ValueError: The truth value of a Series is ambiguous.
# Use a.empty, a.bool(), a.item(), a.any() or a.all().

Wrapping each side in its own parentheses forces Python to evaluate the two comparisons first, and only then combine them with &, which is exactly what you want:

# AND — sales positive AND making a loss (sold but lost money)
market[(market['Sales'] > 0) & (market['Profit'] < 0)]

# OR — either condition is enough
market[(market['Sales'] > 0) | (market['Profit'] < 0)]

# NOT — the opposite of "profit >= 0"
market[~(market['Profit'] >= 0)]

Filtering on Text and Combining with Column Selection

You can filter on text columns too, and combine a filter with loc to grab specific columns from the matching rows:

# Find one specific customer
market[market['Cust_id'] == 'Cust_1818']

# Filter rows AND pick only certain columns at once
market.loc[market['Cust_id'] == 'Cust_1818',
           ['Cust_id', 'Sales', 'Profit', 'Prod_id']]

This filter-then-select pattern is one of the most frequently used patterns in daily analysis.

Part 6: Sorting Data

Sorting reorders your rows based on the values in one or more columns, using sort_values:

market.sort_values('Profit')                   # ascending (smallest first) — the default
market.sort_values('Profit', ascending=False)  # descending (largest first)
market.sort_values(['Profit', 'Sales'])        # sort by Profit, break ties using Sales

When you sort by two columns, Pandas sorts by the first, and uses the second only to break ties (which only matters when the first column has duplicate values):

 sort_values(['Profit', 'Sales'])

 Profit │ Sales         1. Sort by Profit first (ascending).
   12   │  345          2. Where Profit values are EQUAL,
   21   │  321             sort by Sales to break the tie.
   21   │  876    ←──   ← same Profit=21, so Sales decides: 321 < 876
   65   │  765
   89   │  773

In this example, two rows share Profit = 21. Pandas uses the Sales column to decide their order, placing the row with Sales 321 above the row with Sales 876.

One useful parameter: ignore_index=True renumbers the index 0, 1, 2... after sorting, keeping it tidy instead of preserving the scrambled original order:

market.sort_values('Profit', ignore_index=True)

Part 7: Combining DataFrames — Merge vs Concatenate

Real data is usually spread across multiple files. Pandas gives you two ways to bring them together, and it's important to understand which to use when.

   MERGE                              CONCATENATE
   Joins on a shared KEY column       Stacks tables directly
   (like a VLOOKUP in Excel)          (top-bottom or side-by-side)

Merging — Joining on a Common Column

Merge combines tables by matching values in a shared "key" column. For example, our sales data has a Cust_id, and a separate customer table also has Cust_id — merge uses that to stitch each sale to its customer's details.

It's the same move as a VLOOKUP in Excel: you have a list of orders with a student ID on each row, and a separate roster with each student's name and class. Merge walks down the ID column and pulls in the matching name and class for every order, without you copy-pasting anything by hand.

market   = pd.read_csv('market_fact.csv')   # the sales transactions
customer = pd.read_csv('cust_dimen.csv')     # customer details
product  = pd.read_csv('prod_dimen.csv')     # product details

new_df = pd.merge(market, customer, on='Cust_id', how='inner')
final_df = pd.merge(new_df, product, on='Prod_id', how='inner')

Here's the idea visually — matching rows on the shared key:

   market table              customer table
   ┌─────────┬───────┐       ┌─────────┬──────────┐
   │ Cust_id │ Sales │       │ Cust_id │  Region  │
   ├─────────┼───────┤       ├─────────┼──────────┤
   │  C101   │  345  │       │  C101   │   West   │
   │  C102   │  765  │       │  C102   │   East   │
   └─────────┴───────┘       └─────────┴──────────┘
                  │                 │
                  └──── merge on ───┘
                       'Cust_id'
                          ▼
            ┌─────────┬───────┬──────────┐
            │ Cust_id │ Sales │  Region  │
            ├─────────┼───────┼──────────┤
            │  C101   │  345  │   West   │
            │  C102   │  765  │   East   │
            └─────────┴───────┴──────────┘

The how='inner' part controls which rows survive. An inner join keeps only rows that have a match in both tables. Other options exist (left, right, outer) but inner is the most common starting point.

Once you've built a combined table, you can save it to a new CSV file:

final_df.to_csv('merged_data.csv', index=False)   # index=False skips the index column

Concatenating — Stacking Tables Directly

Concatenation doesn't match on a key — it just glues DataFrames together, either vertically or horizontally. The axis parameter controls the direction.

axis=0 — stack vertically (top and bottom). Use this when two tables have the same columns and you simply want to combine their rows:

pd.concat([df2, df3], axis=0, ignore_index=True)

   df2                    df3                 result (axis=0)
   ┌─────┬────────┐       ┌─────┬────────┐    ┌─────┬────────┐
   │ id  │ Profit │       │ id  │ Profit │    │ id  │ Profit │
   ├─────┼────────┤   +   ├─────┼────────┤ =  ├─────┼────────┤
   │ 101 │   65   │       │ 106 │   99   │    │ 101 │   65   │
   │ 102 │   90   │       │ 107 │  105   │    │ 102 │   90   │
   └─────┴────────┘       └─────┴────────┘    │ 106 │   99   │  ← df3
                                              │ 107 │  105   │     stacked
                                              └─────┴────────┘     below

axis=1 — stack horizontally (side by side). Use this when tables describe the same rows but hold different columns:

pd.concat([df2, df4], axis=1)

   df2                    df4              result (axis=1)
   ┌─────┬────────┐       ┌────────┐       ┌─────┬────────┬────────┐
   │ id  │ Profit │       │ Gender │       │ id  │ Profit │ Gender │
   ├─────┼────────┤   +   ├────────┤  =    ├─────┼────────┼────────┤
   │ 101 │   65   │       │   M    │       │ 101 │   65   │   M    │
   │ 102 │   90   │       │   F    │       │ 102 │   90   │   F    │
   └─────┴────────┘       └────────┘       └─────┴────────┴────────┘
                                            └─ glued side by side ─┘

The quick decision guide:

Part 8: Grouping and Summarizing — The "Split-Apply-Combine" Pattern

This is one of the most powerful ideas in all of data analysis. Often you don't want a single grand total — you want a total per category. "What's the profit per product category?" "Average sales per region?" That's the job of groupby.

The Core Idea, Visualized

groupby follows a three-step pattern called split-apply-combine:

Picture sorting a mail bag into pigeonholes by zip code (split), weighing the mail in each pigeonhole (apply), then writing up a single sheet listing total weight per zip code (combine). You never manually track which letter belongs where — the pigeonholes do that for you.

   1. SPLIT                2. APPLY              3. COMBINE
   Break data into         Run a calculation    Stack the results
   groups by category      on each group        into one summary

   ┌──────────────┐
   │ ALL DATA     │        Office → sum()       ┌──────────┬────────┐
   │              │        = 12,300             │ Category │ Profit │
   │ Office       │   →                    →    ├──────────┼────────┤
   │ Tech         │        Tech   → sum()       │ Office   │ 12,300 │
   │ Furniture    │        = 45,800             │ Tech     │ 45,800 │
   │ (mixed up)   │                             │ Furniture│  8,100 │
   └──────────────┘        Furn   → sum()       └──────────┴────────┘
                           = 8,100

Doing It in Pandas

First, a simple grand total (no grouping):

merged_data['Profit'].sum()    # total profit across everything

Now, profit by category. We group on a categorical column (one with repeating text categories), then apply our calculation:

# How many unique categories are there?
merged_data['Product_Category'].unique()
# ['OFFICE SUPPLIES', 'TECHNOLOGY', 'FURNITURE']

# How many rows in each?
merged_data['Product_Category'].value_counts()

When you group, the data is split into one chunk per category:

   merged_data splits into 3 groups:
   ├── OFFICE SUPPLIES   → shape (4610, 16)
   ├── TECHNOLOGY        → shape (2065, 16)
   └── FURNITURE         → shape (1724, 16)

grp_by_cat = merged_data.groupby('Product_Category')

grp_by_cat.groups                       # shows the row indices in each group
grp_by_cat.get_group('TECHNOLOGY')      # pull out just one group as a DataFrame

Now apply the summary — Pandas grabs the Profit column from each group and sums it separately:

grp_by_cat['Profit'].sum()

To present it cleanly, wrap it in a DataFrame and sort it:

pd.DataFrame(grp_by_cat['Profit'].sum()).sort_values(by='Profit', ascending=False)

   Result:
   ┌──────────────────┬─────────┐
   │ Product_Category │  Profit │
   ├──────────────────┼─────────┤
   │ TECHNOLOGY       │  45,800 │  ← highest
   │ OFFICE SUPPLIES  │  12,300 │
   │ FURNITURE        │   8,100 │  ← lowest
   └──────────────────┴─────────┘

Grouping by Two Columns

You can group by multiple categories at once for a finer breakdown — for instance, category and sub-category:

grp_by_cat_subcat = merged_data.groupby(['Product_Category', 'Product_Sub_Category'])
pd.DataFrame(grp_by_cat_subcat['Profit'].sum())

This creates nested groups — each main category broken down into its sub-categories, each with its own profit total.

Part 9: Pivot Tables

If you've used Excel, pivot tables will feel familiar. They're an alternative to groupby that produces Excel-like aggregate tables, and they're especially good at summarizing across two dimensions at once.

The method signature (called on a DataFrame):

df.pivot_table(values=..., index=..., columns=..., aggfunc='mean')

• values — the column you want to aggregate (e.g. Profit)
• index — what goes down the rows
• columns — what goes across the columns
• aggfunc — the calculation to apply (sum, mean, count...)

A simple one-dimensional pivot (similar to a groupby):

merged_data.pivot_table(values='Profit', index='Customer_Segment', aggfunc='sum')

The real power comes with a two-dimensional pivot — categories down the side, segments across the top:

merged_data.pivot_table(values='Profit',
                        index='Product_Category',
                        columns='Customer_Segment',
                        aggfunc='sum')

                   ┌──────────── Customer_Segment ─────────────┐
                   │ Consumer │ Corporate │ Home Office │ Small │
   ┌───────────────┼──────────┼───────────┼─────────────┼───────┤
   │ FURNITURE     │   2,100  │   3,400   │    1,200    │ 1,400 │
   │ OFFICE SUPP.  │   4,500  │   5,100   │    1,800    │   900 │
   │ TECHNOLOGY    │  18,200  │  15,600   │    7,300    │ 4,700 │
   └───────────────┴──────────┴───────────┴─────────────┴───────┘
       ▲                          ▲
    index =                   columns =
   Product_Category          Customer_Segment

   Each cell = total Profit for that Category × Segment combination

This single table answers "which segment buys the most technology?" at a glance, the kind of summary that would take many separate groupby calls to reproduce.

Part 10: Applying Functions with apply and Lambda

Sometimes you need to transform a column using your own custom logic — creating a new column or altering an existing one. The apply method runs a function against every value in a column.

Think of apply as a rubber stamp: you press it on every single value in the column, one at a time, and it imprints whatever your function returns. A lambda is just a stamp you carve on the spot for a single job, instead of machining a permanent, reusable one with def.

Using a Named Function

def is_positive(num):
    return num > 0

merged_data['Profit'].apply(is_positive)   # True/False for each row

apply takes your function and runs it against every single value in the column, returning a new Series of results.

Using a Lambda (the Shortcut)

For simple operations, writing a whole named function is overkill. A lambda is a tiny, one-line, anonymous function, perfect for use inside apply:

# Exactly the same as the is_positive function above, in one line:
merged_data['Profit'].apply(lambda x: x > 0)

Lambdas can include conditional logic:

merged_data['Profit'].apply(lambda x: 'Profitable' if x > 0 else 'Not Profitable')

Creating a New Column

The most common real use: take the result of apply and store it as a brand-new column. Here we label every transaction as profitable or not:

merged_data['profitable_or_not'] = merged_data['Profit'].apply(
    lambda x: 'Profitable' if x > 0 else 'Not Profitable'
)

   BEFORE                          AFTER
   ┌────────┐                      ┌────────┬───────────────────┐
   │ Profit │                      │ Profit │ profitable_or_not │
   ├────────┤                      ├────────┼───────────────────┤
   │   52   │        apply         │   52   │ Profitable        │
   │  -30   │   ───────────────►   │  -30   │ Not Profitable    │
   │   18   │   (lambda creates    │   18   │ Profitable        │
   │  -45   │    the new column)   │  -45   │ Not Profitable    │
   └────────┘                      └────────┴───────────────────┘

This pattern, using apply with a lambda to engineer a new column, is one of the most frequent operations in real data preparation, especially when getting data ready for machine learning.

The Complete Pandas Workflow at a Glance

Here's how all the pieces fit into a typical analysis, start to finish:

Quick Reference Summary

Pandas is the bridge between raw, messy data and real insight. Once these operations feel natural: loading, exploring, selecting, filtering, combining, grouping, and transforming, you'll be able to take almost any dataset and bend it to answer the questions you care about. The best way to internalize it is to grab a real CSV (your bank statement, a sports dataset, anything) and walk through the eight-step workflow above. Every diagram in this guide describes something you can now do, so open a notebook and try them on your own data.