The Standard Library Toolkit: collections, itertools, functools and datetime

Analogy: think of the standard library as the toolbox that comes with a new house. Before you drive to the hardware store (installing an outside library with pip), it's worth opening the toolbox you already own. These four modules are four well-stocked drawers in that toolbox.

You may already have seen one of these tools without an explanation. A sales report might build its totals like this:

from collections import defaultdict

revenue_by_category = defaultdict(float)
orders_by_customer = defaultdict(int)

Both lines will make sense by the end of Part 1, and you'll see that the second one is really a job for a different tool.

Everything in this guide comes with Python. There's no pip install and no requirements.txt, so nothing can go wrong on someone else's machine.

Here is what we will cover:

1. collections: defaultdict, Counter, namedtuple, deque
2. itertools: chain, islice, groupby, product, combinations, count, cycle
3. functools: partial, and a second look at lru_cache
4. datetime: reading dates from text, printing them, and doing maths with timedelta
5. The traps in each one, and how to spot them

The Data We'll Use

One small dataset of online shop orders runs through the whole guide, so each tool solves a problem you can already see. Each order is a tuple of five values: an order ID, the customer, the product category, the order date and the amount paid.

ORDERS = [
    # order_id, customer, category,      date,         amount
    ("A001", "Alice", "Books",       "2024-03-01",  59.98),
    ("A002", "Bob",   "Electronics", "2024-03-01",  45.00),
    ("A003", "Alice", "Appliances",  "2024-03-02",  89.99),
    ("A004", "Carol", "Books",       "2024-03-04",  89.97),
    ("A005", "Bob",   "Electronics", "2024-03-05",  75.00),
    ("A006", "Dave",  "Appliances",  "2024-03-05",  89.99),
    ("A007", "Alice", "Books",       "2024-03-08",  24.99),
    ("A008", "Carol", "Books",       "2024-03-11", 120.00),
]

print(f"{len(ORDERS)} orders, ${sum(o[4] for o in ORDERS):,.2f} total")

8 orders, $594.92 total

Part 1: collections

The collections module gives you four extra container types, each doing a job that the built-in ones (lists, dicts, tuples) handle awkwardly.

1.1 defaultdict: The KeyError You Keep Working Around

Grouping (for example, "total revenue per category") is one of the most common things you'll do with a list of records, and a plain dictionary makes it clumsy. The problem is that the first time you touch a key, it doesn't exist yet:

revenue = {}

try:
    revenue["Books"] += 59.98
except KeyError as e:
    print("KeyError:", e)

KeyError: 'Books'

revenue["Books"] += 59.98 really means revenue["Books"] = revenue["Books"] + 59.98, and Python can't read revenue["Books"] because there's nothing there yet.

So every grouping loop needs an extra check first:

revenue = {}

for order_id, customer, category, date, amount in ORDERS:
    if category not in revenue:         # the extra check
        revenue[category] = 0.0
    revenue[category] += amount

print(revenue)

{'Books': 294.94, 'Electronics': 120.0, 'Appliances': 179.98}

dict.setdefault() shortens that to one line, and you'll see it in plenty of code. It means "give me the value for this key, and if there isn't one, set it to this default first":

revenue[category] = revenue.setdefault(category, 0.0) + amount

defaultdict goes further. You give it a function that makes the starting value, and it calls that function automatically whenever you touch a key that isn't there yet:

from collections import defaultdict

revenue = defaultdict(float)            # float() is 0.0

for order_id, customer, category, date, amount in ORDERS:
    revenue[category] += amount         # no check, no setdefault

for category, total in revenue.items():
    print(f"{category:<12} ${total:>7.2f}")

Books        $ 294.94
Electronics  $ 120.00
Appliances   $ 179.98

No check, no setdefault. The first time "Books" is touched, defaultdict calls float(), which returns 0.0, and then adds the amount.

Analogy: a filing cabinet that creates folders by itself. With a normal cabinet, asking for the "Books" folder when it doesn't exist gets you an error, so you have to create it first. With a defaultdict cabinet, asking for a missing folder makes a new empty one appear, ready to use.

   revenue["Books"] += 59.98

   plain dict:   "Books" missing? ──► KeyError ✗

   defaultdict(float):
                 "Books" missing? ──► call float() → 0.0 ──► store it ──► 0.0 + 59.98 ✓

Pass a Function, Not a Value

The argument is a function, not a value. defaultdict(float) calls float() to get 0.0, and defaultdict(list) calls list() to get a fresh empty list. You'll use that second one the most, because it turns grouping into two lines:

orders_by_customer = defaultdict(list)

for order in ORDERS:
    orders_by_customer[order[1]].append(order[0])

for customer, ids in orders_by_customer.items():
    print(f"{customer:<6} {ids}")

Alice  ['A001', 'A003', 'A007']
Bob    ['A002', 'A005']
Carol  ['A004', 'A008']
Dave   ['A006']

Each customer automatically got their own empty list the first time they appeared, and then their order IDs were appended to it.

Notice it's defaultdict(list), not defaultdict([]). The whole idea is that each new key gets its own fresh starting value. If you could pass one actual list, that same list object would be shared by every key, so an append to one customer would show up under all of them.

Analogy: it's the difference between giving each shopper their own new basket (list, a basket-making function) and making every shopper share one single basket ([]). With one shared basket, everyone's items get mixed up.

Python doesn't let you make that mistake. It refuses anything that isn't a function (anything that isn't "callable"):

try:
    broken = defaultdict([])            # deliberately wrong: a list, not a function
    broken["Books"].append("A001")
except TypeError as e:
    print("TypeError:", e)

TypeError: first argument must be callable or None

The Trap: Reading a Missing Key Creates It

This is the part of defaultdict that surprises people. Looking up a key isn't just a read. It can change the dictionary:

counts = defaultdict(int)
counts["Books"] += 1

print("Electronics count   :", counts["Electronics"])   # just looking
print("the dictionary now  :", dict(counts))
print("length              :", len(counts))

Electronics count   : 0
the dictionary now  : {'Books': 1, 'Electronics': 0}
length              : 2

We only looked at counts["Electronics"], but that lookup created the key with a value of 0. The dictionary grew from 1 key to 2. If a report then loops over counts.items(), you get a row of zeros for every category anyone ever asked about.

Analogy: imagine a librarian who, every time you ask about a book that isn't on the shelf, quietly puts an empty box labelled with that title on the shelf. After a busy day, the shelves are full of empty boxes.

Use .get() when you only want to look. .get(key, default) returns the default if the key is missing, without creating it:

counts = defaultdict(int)
counts["Books"] += 1

print("with .get():", counts.get("Electronics", 0))
print("still just:", dict(counts))

with .get(): 0
still just: {'Books': 1}

A defaultdict is a real dict underneath, so everything you know about dictionaries works on it. When you've finished building it, dict(d) turns it back into a plain dictionary:

print("is it a dict?", isinstance(counts, dict))
print("plain again  :", type(dict(counts)).__name__)

is it a dict? True
plain again  : dict

1.2 Counter: Counting Made Easy

You could count orders per customer with defaultdict(int). That works, but counting is so common that it has its own container. Give Counter anything you can loop over, and it counts what's in it:

from collections import Counter

by_customer = Counter(order[1] for order in ORDERS)

print(by_customer)
print("Alice's orders:", by_customer["Alice"])

Counter({'Alice': 3, 'Bob': 2, 'Carol': 2, 'Dave': 1})
Alice's orders: 3

Analogy: a Counter is like the tally sheet used to count votes in an election. Every time a name comes up, you add a mark next to it. At the end, you can instantly see who has the most.

The main reason to prefer it over defaultdict(int) is most_common(). It sorts the counts from biggest to smallest, which is usually the question you actually wanted answered. (total() adds up all the counts. It needs Python 3.10 or later.)

by_category = Counter(order[2] for order in ORDERS)

print("all categories  :", by_category.most_common())
print("top 2           :", by_category.most_common(2))
print("total items     :", by_category.total())

all categories  : [('Books', 4), ('Electronics', 2), ('Appliances', 2)]
top 2           : [('Books', 4), ('Electronics', 2)]
total items     : 8

Counter also counts the words in a sentence or the characters in a string. That's often the first thing you do with a text dataset, like product reviews:

review = "the book was good the delivery was fast the price was fair"

words = Counter(review.split())
print(words.most_common(3))

letters = Counter("mississippi")
print(sorted(letters.items()))

[('the', 3), ('was', 3), ('book', 1)]
[('i', 4), ('m', 1), ('p', 2), ('s', 4)]

• review.split() breaks the sentence into a list of words, and Counter counts each word.
• Counter("mississippi") counts each character, because looping over a string gives you its characters one by one.

Counter Doesn't Create Missing Keys

Compare this with defaultdict. In a Counter, a missing key simply reads as 0, and looking doesn't add anything:

letters = Counter("mississippi")

print("count of 'z':", letters["z"])
print("is 'z' in it?", "z" in letters)
print("length still:", len(letters))

count of 'z': 0
is 'z' in it? False
length still: 4

Adding and Subtracting Counters

Counters can be subtracted, which answers "what changed?" in one line. It's like comparing two months of stock counts:

february = Counter({"Books": 3, "Electronics": 5, "Appliances": 1})
march    = Counter({"Books": 4, "Electronics": 2, "Appliances": 2})

print("month on month:", march - february)     # only positive counts survive
print("combined      :", march + february)

month on month: Counter({'Books': 1, 'Appliances': 1})
combined      : Counter({'Books': 7, 'Electronics': 7, 'Appliances': 3})

march - february dropped Electronics completely, because subtracting Counters keeps only the positive counts. (Electronics went from 5 to 2, a change of -3, so it disappeared.) When you want the negative changes too, subtract the values yourself:

change = {k: march[k] - february[k] for k in set(march) | set(february)}
print(dict(sorted(change.items())))

{'Appliances': 1, 'Books': 1, 'Electronics': -3}

set(march) | set(february) gives every category that appears in either month (the | combines two sets). Then the dict comprehension works out march[k] - february[k] for each one, and since a Counter reads a missing key as 0, no category causes an error.

1.3 namedtuple: A Tuple Whose Fields Have Names

Our orders are tuples, and reading them means remembering which position is which:

order = ORDERS[0]

print(order[2], order[4])       # which fields are these again?

Books 59.98

What was position 2 again? And position 4? Six months from now, you won't remember.

Tuples are great for data that shouldn't change, like a payment record. The downside is that the position number is the only way to get at a field. namedtuple builds a new tuple type that has a name for each position.

Order(*row) uses * to spread each row out into the five arguments, one per field:

from collections import namedtuple

Order = namedtuple("Order", "order_id customer category date amount")

orders = [Order(*row) for row in ORDERS]
first = orders[0]

print(first)
print(first.category, first.amount)

Order(order_id='A001', customer='Alice', category='Books', date='2024-03-01', amount=59.98)
Books 59.98

Analogy: a plain tuple is like a row of unlabelled boxes: you have to remember that the third box holds the category. A namedtuple is the same row of boxes, but each box has a label on it.

Look at what print(first) shows. A plain tuple prints as a row of values you have to decode. A namedtuple prints its field names too, which makes debugging much quicker.

It's still a tuple, so everything you know about tuples still works. You can unpack it, index it by position, and you can't change it:

order_id, customer, category, date, amount = first     # unpacking still works
print(order_id, customer)

print("index access :", first[1])
print("is a tuple   :", isinstance(first, tuple))

try:
    first.amount = 0.0                  # not allowed: a tuple can't be changed
except AttributeError as e:
    print("AttributeError:", e)

A001 Alice
index access : Alice
is a tuple   : True
AttributeError: can't set attribute

_replace() and _asdict()

Two methods you'll use:

• Since you can't edit a namedtuple, _replace() gives you a new one with one field changed. The original stays the same.
• _asdict() turns it into a dictionary, which is what you want just before saving it as JSON (for example, with json.dump).

corrected = first._replace(amount=61.98)

print("original :", first.amount)
print("corrected:", corrected.amount)
print("as a dict:", corrected._asdict())

original : 59.98
corrected: 61.98
as a dict: {'order_id': 'A001', 'customer': 'Alice', 'category': 'Books', 'date': '2024-03-01', 'amount': 61.98}

The underscore at the start of these names makes them look private, but they're meant for you to use. The underscore is only there so they can't clash with a field you might name replace or count.

When a Namedtuple Isn't Enough: @dataclass

A namedtuple is for plain data with no behaviour. Once the record needs methods (its own functions), or needs to change after it's created, a class is the answer. Writing that class by hand means an __init__ that assigns every field, plus something like __str__ so it prints nicely.

The dataclasses module writes that code for you, using a decorator (the @ line above the class).

The : str and : int after each field name are type hints: labels that say what kind of value belongs there. Python doesn't actually check them at runtime, but @dataclass reads these lines to find out which fields the class has:

from dataclasses import dataclass

@dataclass
class Shipment:
    order_id: str
    carrier: str
    days_in_transit: int = 3            # a default, like any function argument

    def is_late(self):                  # methods, which a namedtuple cannot have
        return self.days_in_transit > 5


s = Shipment("A001", "BlueDart")
print(s)                                # the dataclass wrote the printing code for us
print("late?", s.is_late())

s.days_in_transit = 7                   # can be changed, unlike a namedtuple
print("late now?", s.is_late())

Shipment(order_id='A001', carrier='BlueDart', days_in_transit=3)
late? False
late now? True

• Shipment("A001", "BlueDart") didn't pass days_in_transit, so it used the default of 3.
• print(s) showed all the fields neatly, because @dataclass wrote the printing code.
• is_late() is a method, which a namedtuple can't have.
• s.days_in_transit = 7 changed a field, which a namedtuple wouldn't allow.

Analogy: writing a class by hand is like designing a form from a blank page. @dataclass is like a form builder: you list the fields you want, and it produces a neat, working form for you.

Use a namedtuple for rows of data, and a dataclass when the thing needs behaviour.

1.4 deque: A List That's Fast at Both Ends

list.append() and list.pop() work on the end of a list, and they're fast. list.pop(0) works on the front, and to do that Python has to shift every remaining item one place to the left. On a short list you won't notice. On a long one, it's why a script that "worked on the sample" takes minutes on the full file.

Analogy: a Python list is like a row of numbered seats in a cinema. If the person in seat 1 leaves and everyone must stay numbered from 1, every single person has to shuffle one seat to the left. A deque is like a queue at a ticket counter: when the first person leaves, nobody needs to move seats; the next person simply becomes the front.

   list.pop(0)                                    deque.popleft()

   [ A ][ B ][ C ][ D ][ E ]                       front ──► A  B  C  D  E ◄── back
     ▲    ◄─── everyone shifts left ───             ▲                     ▲
   remove                                        remove here         add here
                                                 (instantly)         (instantly)
   slow when the list is long                    fast at BOTH ends

deque (say "deck", short for "double-ended queue") is built to be quick at both ends. Let's time it instead of taking that on trust:

from collections import deque
import time

N = 50_000
as_list = list(range(N))
as_deque = deque(range(N))

start = time.perf_counter()
while as_list:
    as_list.pop(0)                      # shifts everything left, every time
list_time = time.perf_counter() - start

start = time.perf_counter()
while as_deque:
    as_deque.popleft()
deque_time = time.perf_counter() - start

print(f"removing {N:,} items from the front")
print("deque was faster        :", deque_time < list_time)
print("by more than 10 times   :", list_time > deque_time * 10)

removing 50,000 items from the front
deque was faster        : True
by more than 10 times   : True

The exact times depend on your computer, so the cell prints the comparison instead of the raw numbers. The gap is big, and it grows as the data gets bigger.

deque has append/pop for the right end and appendleft/popleft for the left end:

queue = deque(["job-1", "job-2", "job-3"])

queue.append("job-4")                   # arrives at the back
queue.appendleft("urgent")              # jumps the queue

print(queue)
print("next up:", queue.popleft())
print("remaining:", list(queue))

deque(['urgent', 'job-1', 'job-2', 'job-3', 'job-4'])
next up: urgent
remaining: ['job-1', 'job-2', 'job-3', 'job-4']

maxlen: A Rolling Window

What makes deque really useful for data work is maxlen. Give it one, and the container only keeps the most recent N items, dropping the oldest ones automatically. That's called a rolling window, and it's fiddly to write by hand:

last_three = deque(maxlen=3)

for order in ORDERS:
    last_three.append(order[0])

print("the three most recent orders:", list(last_three))

the three most recent orders: ['A006', 'A007', 'A008']

Analogy: the "Recently played" list in a music app. It only shows your last few songs. Every time you play a new one, the oldest drops off the end.

   deque(maxlen=3), appending A001 ... A008

   after A003:  [A001, A002, A003]
   after A004:  [A002, A003, A004]      ← A001 pushed out
   after A005:  [A003, A004, A005]      ← A002 pushed out
   ...
   after A008:  [A006, A007, A008]      ← always just the last three

Part 2: itertools

itertools is a toolbox of ready-made generators. Everything in it is lazy: nothing is worked out until you ask for it, and nothing large is held in memory.

Analogy: if writing your own generators is like building your own machines for a factory, itertools is a catalogue of ready-built machines that you just plug into your assembly line.

2.1 chain: Several Sequences, One Loop

Say you have three monthly export files and want one report. Without chain, you'd either write three loops or build one big combined list. Let's create the files first, so we're working with real files:

import csv
import os

os.makedirs("toolkit_demo", exist_ok=True)

HEADER = ["order_id", "customer", "category", "date", "amount"]
months = {"2024_01": ORDERS[:3], "2024_02": ORDERS[3:6], "2024_03": ORDERS[6:]}

for label, rows in months.items():
    with open(f"toolkit_demo/orders_{label}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(HEADER)
        writer.writerows(rows)

print("wrote", len(months), "files, each with a header row")

wrote 3 files, each with a header row

Now let's read all three files as if they were one long file:

from contextlib import ExitStack     # chapter 14
from itertools import chain

paths = [f"toolkit_demo/orders_{label}.csv" for label in months]

with ExitStack() as stack:
    files = [stack.enter_context(open(p, newline="", encoding="utf-8")) for p in paths]

    readers = []
    for f in files:
        reader = csv.reader(f)
        next(reader)                    # step over this file's header
        readers.append(reader)

    every_row = chain.from_iterable(readers)
    total = sum(float(row[4]) for row in every_row)

print(f"total across {len(paths)} files: ${total:,.2f}")

total across 3 files: $594.92

How it works:

• ExitStack opens all three files and guarantees they will all be closed at the end.
• For each file, csv.reader reads rows, and next(reader) skips that file's header row.
• chain.from_iterable(readers) joins the three readers into one long stream of rows.
• sum(...) adds up the amount column (row[4]) across all of them.

Analogy: chain is like coupling railway coaches. Three separate coaches become one train, and passengers can walk from the first coach to the last without stepping outside.

chain.from_iterable takes one collection of iterables (here, a list of readers). chain(a, b, c) takes them as separate arguments. Both give back an iterator, so the three files were read one row at a time and never held in memory together:

print(type(chain([1, 2], [3, 4])).__name__)
print(list(chain([1, 2], [3, 4], [5])))

chain
[1, 2, 3, 4, 5]

2.2 islice: Slicing Things You Can't Normally Slice

Normal slicing ([:5]) works on lists, tuples and strings. A generator isn't one of those, so this fails:

squares = (n * n for n in range(1_000_000))

try:
    print(squares[:5])                  # a generator can't be sliced like this
except TypeError as e:
    print("TypeError:", e)

TypeError: 'generator' object is not subscriptable

islice slices any iterator, and only takes as much as you ask for. The other 999,995 squares are never worked out:

from itertools import islice

squares = (n * n for n in range(1_000_000))
print("first five:", list(islice(squares, 5)))

squares = (n * n for n in range(1_000_000))
print("items 10 to 14:", list(islice(squares, 10, 15)))

first five: [0, 1, 4, 9, 16]
items 10 to 14: [100, 121, 144, 169, 196]

Notice that we created squares again before the second islice. That's because the first islice used up the first five values, and a generator can't go back.

Analogy: islice is like taking just the first five chips out of a giant bag. You don't need to pour the whole bag onto the table to do that.

The everyday use is skipping a header row. The chain example above did that with next(reader). islice does it in one expression: islice(f, 1, None) means "start at line 1 (skipping line 0), and go to the end":

with open("toolkit_demo/orders_2024_03.csv", encoding="utf-8") as f:
    for line in islice(f, 1, None):     # everything after the header
        print(line.strip())

A007,Alice,Books,2024-03-08,24.99
A008,Carol,Books,2024-03-11,120.0

One limit to know: islice can't take a negative index. To find the last five items, Python would have to read to the end first, and an iterator doesn't know where its end is until it gets there:

try:
    islice(ORDERS, -5)
except ValueError as e:
    print("ValueError:", e)

ValueError: Indices for islice() must be None or an integer: 0 <= x <= sys.maxsize.

For the last N items, use the deque(maxlen=N) trick from Part 1.4 instead.

2.3 groupby: Easy to Get Wrong

groupby goes through an iterable and groups items that share a key, but only items that sit next to each other. That detail is easy to miss, and missing it gives you a report that looks fine but is wrong.

Our orders are in date order, not category order. Let's group them by category as they are:

from itertools import groupby

for category, group in groupby(ORDERS, key=lambda o: o[2]):
    print(f"{category:<12} count={len(list(group))}")

Books        count=1
Electronics  count=1
Appliances   count=1
Books        count=1
Electronics  count=1
Appliances   count=1
Books        count=2

Seven groups from three categories, and Books shows up three separate times. Nothing raised an error and nothing warned us. A report built on that output would undercount every category.

Analogy: think of a supermarket cashier who puts items into bags as they come down the conveyor belt, and starts a new bag every time the type of item changes. If the items arrive mixed up (book, phone, toaster, book), you end up with lots of half-empty bags. If you first sort the items by type, every type ends up in exactly one bag.

   groupby on UNSORTED data (by category):

   Books │ Electronics │ Appliances │ Books │ Electronics │ Appliances │ Books Books
   ──────┴─────────────┴────────────┴───────┴─────────────┴────────────┴────────────
     1          1             1         1          1             1           2      ← 7 groups ✗

   groupby on SORTED data:

   Appliances Appliances │ Books Books Books Books │ Electronics Electronics
   ──────────────────────┴────────────────────────┴────────────────────────
             2                       4                        2                     ← 3 groups ✓

groupby needs the data sorted by the same key first:

by_category = sorted(ORDERS, key=lambda o: o[2])

for category, group in groupby(by_category, key=lambda o: o[2]):
    rows = list(group)
    revenue = sum(row[4] for row in rows)
    print(f"{category:<12} count={len(rows)}  ${revenue:>7.2f}")

Appliances   count=2  $ 179.98
Books        count=4  $ 294.94
Electronics  count=2  $ 120.00

Sort by the key, then group by the same key. If you write one without the other, that's the bug.

The Second Trap: Each Group Gets Used Up

Each group is an iterator that only works while you're on it. Move on to the next group, and the previous one is used up. If you collect the pairs first and loop over them later, the groups come out empty:

pairs = list(groupby(by_category, key=lambda o: o[2]))

for category, group in pairs:
    print(f"{category:<12} {list(group)}")

Appliances   []
Books        []
Electronics  []

Three categories, three empty lists, and no error. Use up each group inside the loop, like the working version above does with rows = list(group).

Analogy: like a museum tour guide who leads one group at a time. Once the guide has moved on to the next group, the previous group has already left the building. If you want to know who was in each group, write their names down while they're still in front of you.

Often the Simpler Tool: defaultdict(list)

Because of both traps, a defaultdict(list) is often the better tool. It doesn't need sorted data, and it gives you real lists that you can use as many times as you like:

grouped = defaultdict(list)
for order in ORDERS:
    grouped[order[2]].append(order)

for category, rows in sorted(grouped.items()):
    print(f"{category:<12} count={len(rows)}")

Appliances   count=2
Books        count=4
Electronics  count=2

Use groupby when the data already arrives sorted, which is common when it comes from a database or a sorted file. Use a defaultdict(list) the rest of the time.

2.4 product and combinations: Every Combination, Without Nested Loops

product gives you every combination of several sequences. If you've ever written nested loops to try every setting, this replaces them. In machine learning, trying every combination of settings like this is called a hyperparameter grid:

from itertools import product

learning_rates = [0.01, 0.1]
depths = [3, 5, 7]

grid = list(product(learning_rates, depths))

for learning_rate, depth in grid:
    print(f"  learning_rate={learning_rate:<5} max_depth={depth}")

print("combinations to try:", len(grid))

  learning_rate=0.01  max_depth=3
  learning_rate=0.01  max_depth=5
  learning_rate=0.01  max_depth=7
  learning_rate=0.1   max_depth=3
  learning_rate=0.1   max_depth=5
  learning_rate=0.1   max_depth=7
combinations to try: 6

Analogy: choosing an outfit. With 2 jackets and 3 pairs of trousers, there are 2 × 3 = 6 possible outfits. product lists all six for you.

                   max_depth=3     max_depth=5     max_depth=7
   lr=0.01        (0.01, 3)       (0.01, 5)       (0.01, 7)
   lr=0.1         (0.1, 3)        (0.1, 5)        (0.1, 7)

   2 learning rates × 3 depths = 6 combinations

combinations picks groups from one sequence, with no repeats and without caring about order. Here it gives every pair of page versions to compare in an A/B test (a test where different users see different versions of a page, to see which works better):

from itertools import combinations, permutations

variants = ["control", "blue-button", "short-form", "new-copy"]

print("pairs to compare:")
for a, b in combinations(variants, 2):
    print(f"  {a} vs {b}")

print("\ncombinations:", len(list(combinations(variants, 2))))
print("permutations:", len(list(permutations(variants, 2))))

pairs to compare:
  control vs blue-button
  control vs short-form
  control vs new-copy
  blue-button vs short-form
  blue-button vs new-copy
  short-form vs new-copy

combinations: 6
permutations: 12

Those last two numbers show the difference:

• combinations treats ("control", "blue-button") and ("blue-button", "control") as the same pair, so it returns 6.
• permutations counts different orders separately, so it returns 12.

Analogy: handshakes vs race medals. When Alice shakes hands with Bob, that's the same handshake as Bob shaking hands with Alice (a combination). But Alice winning gold and Bob silver is a different result from Bob gold and Alice silver (a permutation).

These numbers grow fast. Pairs from 20 items come to 190. Groups of 5 from 20 items come to 15,504. Check the length before you loop over one.

2.5 count and cycle: Iterators That Never End

count is like range with no stop value, which makes it a neat ID generator:

from itertools import count, cycle

order_ids = count(1001)

print([next(order_ids) for _ in range(3)])
print("and it carries on:", next(order_ids))

[1001, 1002, 1003]
and it carries on: 1004

Analogy: the token machine at a bank: it hands out 1001, then 1002, then 1003, for as long as people keep arriving.

cycle repeats a sequence forever. That gives you round-robin assignment (taking turns in order) in one line:

servers = cycle(["eu-west-1", "us-east-1", "ap-south-1"])

for job in ["job-1", "job-2", "job-3", "job-4", "job-5"]:
    print(f"  {job} -> {next(servers)}")

  job-1 -> eu-west-1
  job-2 -> us-east-1
  job-3 -> ap-south-1
  job-4 -> eu-west-1
  job-5 -> us-east-1

Analogy: a rota for washing up: Monday is Asha, Tuesday is Ben, Wednesday is Chen, then back to Asha on Thursday, and so on forever.

Both go on forever, so list(count()) never finishes, and neither does a for loop over one without a break. (We won't demonstrate that, because the cell would never stop running.)

Pair them with islice when you want a fixed number:

print(list(islice(count(0, 5), 6)))          # start at 0, step 5, take 6
print(list(islice(cycle("AB"), 5)))

[0, 5, 10, 15, 20, 25]
['A', 'B', 'A', 'B', 'A']

count(0, 5) starts at 0 and steps by 5. cycle("AB") repeats the characters A and B.

Part 3: functools

functools is a module of tools for working with functions. You may already know reduce, wraps and lru_cache from it. Two more things here are worth knowing well.

3.1 partial: Filling In Arguments Ahead of Time

When you keep calling the same function with the same first argument, partial makes a new function with that argument already filled in:

from functools import partial

def tag(level, message):
    return f"[{level}] {message}"


warn = partial(tag, "WARNING")
error = partial(tag, "ERROR")

print(warn("disk almost full"))
print(error("connection refused"))

[WARNING] disk almost full
[ERROR] connection refused

warn = partial(tag, "WARNING") creates a new function. Calling warn("disk almost full") is the same as calling tag("WARNING", "disk almost full").

Analogy: a pre-filled form. If you write lots of letters to the same company, you might print envelopes with their address already on them. You only have to add the part that changes each time.

You could also write this as a closure (a function that remembers values), and you'd get exactly the same result. partial is just the short way to write it:

def make_tagger(level):                 # the chapter 13 version
    def tagger(message):
        return tag(level, message)
    return tagger


warn_closure = make_tagger("WARNING")

print(warn_closure("disk almost full"))
print("same result:", warn_closure("disk almost full") == warn("disk almost full"))

[WARNING] disk almost full
same result: True

partial with Built-In Functions

It works on built-in functions too, which is where it saves the most typing. int(text, base=16) reads a number written in base 16 (hexadecimal, the number system used in colour codes like #ff0000), so "ff" is 255. lambda text: int(text, base=16) would do the same job, but partial reads more clearly:

from_hex = partial(int, base=16)

print(from_hex("ff"), from_hex("1a"))

to_2dp = partial(round, ndigits=2)
print([to_2dp(o[4] * 1.18) for o in ORDERS[:4]])     # amounts with 18% tax

255 26
[70.78, 53.1, 106.19, 106.16]

to_2dp = partial(round, ndigits=2) makes a "round to 2 decimal places" function. Here it rounds each order amount after adding 18% tax (multiplying by 1.18).

3.2 lru_cache in Practice

@lru_cache is a decorator that remembers results, so calling the function again with the same arguments returns the saved answer instantly instead of working it out again. There are two things to watch out for, and here's what both look like with real data.

Arguments Have to Be Hashable

The cache stores answers in a dictionary, using the arguments as keys. So every argument has to be something a dictionary accepts as a key (Python calls this hashable). A list isn't, because lists can change. A tuple is, so convert a list to a tuple before you pass it in:

from functools import lru_cache

@lru_cache
def total(amounts):
    print("  (computing)")
    return sum(amounts)


print(total((59.98, 45.00, 89.99)))     # a tuple is fine
print(total((59.98, 45.00, 89.99)))     # second call is free

try:
    total([59.98, 45.00])               # a list is not
except TypeError as e:
    print("TypeError:", e)

  (computing)
194.97
194.97
TypeError: unhashable type: 'list'

The first call printed (computing); the second didn't, because the answer came straight from the cache.

A Cache Goes Stale

lru_cache assumes the function returns the same answer for the same arguments forever. If the function reads something that can change, it keeps handing back the old answer:

prices = {"Books": 10.0}

@lru_cache
def price_of(category):
    return prices[category]


print("first call        :", price_of("Books"))

prices["Books"] = 12.0                  # the price changes
print("after the change  :", price_of("Books"))
print("cache_info        :", price_of.cache_info())

first call        : 10.0
after the change  : 10.0
cache_info        : CacheInfo(hits=1, misses=1, maxsize=128, currsize=1)

The function returned a price that doesn't exist anywhere in the program anymore. cache_info() shows why: one miss (the first call did the work), then one hit (the second call reused the old saved answer).

Analogy: an old price sticker on a product. The shop changed the price in its system, but the sticker on the shelf still shows the old one, and that's what you read.

cache_clear() throws the saved answers away:

price_of.cache_clear()

print("after cache_clear :", price_of("Books"))
print("cache_info        :", price_of.cache_info())

after cache_clear : 12.0
cache_info        : CacheInfo(hits=0, misses=1, maxsize=128, currsize=1)

Only cache pure functions: same input, same output, and nothing read from anywhere that can change. A Fibonacci calculator is a perfect fit. A function that reads a file, a database, or a variable outside itself is a bad fit.

Part 4: datetime

Our orders store their dates as strings like "2024-03-01". But a date inside a string isn't really a date to Python, just a sequence of characters. You can't subtract one from another, sort them reliably, or ask what day of the week it was. The datetime module gives you real date and time objects.

4.1 The Four Types

Analogy: a date is a page on a calendar, a time is a reading on a clock, a datetime is the calendar page with the clock time written on it, and a timedelta is a distance between two points in time ("7 days"), not a point itself.

datetime is the one you'll use most:

from datetime import date, time, datetime, timedelta

launch = datetime(2024, 3, 1, 14, 30)

print(launch)
print("year :", launch.year)
print("day  :", launch.day)
print("date part:", launch.date())
print("time part:", launch.time())

2024-03-01 14:30:00
year : 2024
day  : 1
date part: 2024-03-01
time part: 14:30:00

datetime.now() gives you the current moment. The cell below doesn't print it, because the output would change every time it runs, but here's the type:

right_now = datetime.now()

print(type(right_now).__name__)
print("has a year:", right_now.year > 2000)

datetime
has a year: True

4.2 Parsing: String to datetime

strptime (think "string parse time") reads a date out of a string, using a format you give it. The format is made of % codes that describe what the string looks like. %Y is the 4-digit year, %m the month and %d the day (the full list is in Part 4.3):

order_date = datetime.strptime("2024-03-01", "%Y-%m-%d")

print(order_date)
print("weekday:", order_date.strftime("%A"))

2024-03-01 00:00:00
weekday: Friday

Analogy: strptime is like reading a handwritten date on a form. You need to know whether the writer puts the day or the month first; the format string tells Python exactly that.

Once it's a real datetime, you can ask questions a string could never answer, like which weekday it was (%A gives the weekday name).

Why This Matters: Strings Sort in the Wrong Order

A date written day first sorts alphabetically, which has nothing to do with time:

dates = ["09/03/2024", "15/02/2024", "01/12/2023"]

print("sorted as strings:", sorted(dates))

parsed = sorted(datetime.strptime(d, "%d/%m/%Y") for d in dates)
print("sorted as dates  :", [d.strftime("%d/%m/%Y") for d in parsed])

sorted as strings: ['01/12/2023', '09/03/2024', '15/02/2024']
sorted as dates  : ['01/12/2023', '15/02/2024', '09/03/2024']

Sorting the strings put December 2023 first, then March 2024, then February 2024. That's because string sorting compares character by character: "09..." comes before "15..." no matter what month or year follows. Only the parsed dates come out in real time order.

ISO 8601: The Format That Sorts Correctly

The exception is ISO 8601, the YYYY-MM-DD format our orders use. The parts go from biggest (year) to smallest (day), so alphabetical order happens to match date order. That's a good reason to store dates this way in every file you write.

Analogy: it's like writing an address from biggest to smallest: country, city, street, house. Sorting by the first part groups things correctly, then the next part, and so on.

Python has a parser just for this format, fromisoformat, so there's no format string to get wrong:

print(sorted(o[3] for o in ORDERS)[:3])        # strings, and correctly ordered

parsed = datetime.fromisoformat("2024-03-01")
print(parsed, "|", parsed.isoformat())

['2024-03-01', '2024-03-01', '2024-03-02']
2024-03-01 00:00:00 | 2024-03-01T00:00:00

When the format doesn't match the string, strptime raises a ValueError, which you can catch with try/except:

try:
    datetime.strptime("01/03/2024", "%Y-%m-%d")
except ValueError as e:
    print("ValueError:", e)

ValueError: time data '01/03/2024' does not match format '%Y-%m-%d'

4.3 Formatting: datetime to String

strftime (think "string format time") is strptime in reverse: it takes a datetime and a format, and gives you a string.

moment = datetime(2024, 3, 1, 14, 30)

print(moment.strftime("%Y-%m-%d"))
print(moment.strftime("%d %B %Y"))
print(moment.strftime("%A, %d %b %Y at %H:%M"))

2024-03-01
01 March 2024
Friday, 01 Mar 2024 at 14:30

The same moment, written three different ways, just by changing the format string.

4.4 timedelta: Date Arithmetic

Two simple rules cover most date maths:

• Subtract two datetimes and you get a timedelta (a length of time).
• Add a timedelta to a datetime and you get another datetime.

So you never need to write a loop to count how many days have passed.

   2024-03-01                                   2024-03-11                 2024-04-10
   first order ─────────── span = 10 days ────► last order ─── + 30 days ──► payment due
       │                                            │
       └──────── last - first = timedelta(days=10) ─┘

first = datetime.fromisoformat(ORDERS[0][3])
last = datetime.fromisoformat(ORDERS[-1][3])

span = last - first

print("first order:", first.date())
print("last order :", last.date())
print("span       :", span)
print("in days    :", span.days)
print("in hours   :", span.total_seconds() / 3600)

first order: 2024-03-01
last order : 2024-03-11
span       : 10 days, 0:00:00
in days    : 10
in hours   : 240.0

deadline = last + timedelta(days=30)
print("payment due:", deadline.date())

reminder = deadline - timedelta(days=7, hours=12)
print("reminder at:", reminder)

payment due: 2024-04-10
reminder at: 2024-04-02 12:00:00

timedelta(days=7, hours=12) is seven and a half days, so the reminder lands at noon, 7.5 days before the deadline.

Use .days for whole days and .total_seconds() for everything else. There's no .hours or .months, because months aren't all the same length (28 to 31 days). For "one month later, same day number" you'd need dateutil.relativedelta, which is a separate package you'd have to install, so it's outside this guide.

4.5 Naive and Aware Datetimes

A datetime with no timezone attached is called naive. One that has a timezone is called aware. Mixing the two raises an error, and it's better to meet that error here than in the middle of a real project:

from datetime import timezone

naive = datetime(2024, 3, 1, 9, 0)
aware = datetime(2024, 3, 1, 9, 0, tzinfo=timezone.utc)

print("naive:", naive)
print("aware:", aware)

try:
    print(aware - naive)
except TypeError as e:
    print("TypeError:", e)

naive: 2024-03-01 09:00:00
aware: 2024-03-01 09:00:00+00:00
TypeError: can't subtract offset-naive and offset-aware datetimes

Analogy: a naive datetime is like a message saying "let's meet at 9 o'clock" without saying which city. 9 o'clock in Mumbai and 9 o'clock in London are different moments. An aware datetime says "9 o'clock, London time", so there's no confusion. Python refuses to compare the two, because it can't know what the vague one meant.

UTC (Coordinated Universal Time) is the world's reference clock, the "neutral" timezone that every other timezone is measured from. The +00:00 at the end of the aware datetime means "0 hours away from UTC".

Pick one style and stick with it. For anything that moves between computers (like data from an API), store times in UTC as aware datetimes:

stamped = datetime(2024, 3, 1, 9, 0, tzinfo=timezone.utc)

print("stored as:", stamped.isoformat())
print("read back:", datetime.fromisoformat("2024-03-01T09:00:00+00:00"))

stored as: 2024-03-01T09:00:00+00:00
read back: 2024-03-01 09:00:00+00:00

Part 5: Putting It Together

Here's a small report that uses three of the four modules on the same data: namedtuple for readable rows, datetime for the dates, defaultdict and Counter for grouping and counting, and groupby on data we sorted ourselves. sorted(..., reverse=True) sorts from biggest to smallest.

from collections import Counter, defaultdict, namedtuple
from datetime import datetime
from itertools import groupby

Order = namedtuple("Order", "order_id customer category date amount")

orders = [Order(oid, cust, cat, datetime.fromisoformat(d), amt)
          for oid, cust, cat, d, amt in ORDERS]

print(f"{len(orders)} orders from {orders[0].date.date()} to {orders[-1].date.date()}")
print(f"total revenue: ${sum(o.amount for o in orders):,.2f}\n")

revenue = defaultdict(float)
for order in orders:
    revenue[order.category] += order.amount

print("revenue by category")
for category, total in sorted(revenue.items(), key=lambda pair: pair[1], reverse=True):
    print(f"  {category:<12} ${total:>7.2f}")

print("\nbusiest weekdays")
for day, n in Counter(o.date.strftime("%A") for o in orders).most_common(3):
    print(f"  {day:<10} {n} orders")

print("\nspend per customer")
for customer, group in groupby(sorted(orders, key=lambda o: o.customer),
                               key=lambda o: o.customer):
    rows = list(group)
    print(f"  {customer:<6} orders: {len(rows)}   ${sum(r.amount for r in rows):>7.2f}")

8 orders from 2024-03-01 to 2024-03-11
total revenue: $594.92

revenue by category
  Books        $ 294.94
  Appliances   $ 179.98
  Electronics  $ 120.00

busiest weekdays
  Friday     3 orders
  Monday     2 orders
  Tuesday    2 orders

spend per customer
  Alice  orders: 3   $ 174.96
  Bob    orders: 2   $ 120.00
  Carol  orders: 2   $ 209.97
  Dave   orders: 1   $  89.99

Let's walk through what each part of that report used:

• The header uses namedtuple (so we can write orders[0].date instead of orders[0][3]) and datetime.fromisoformat to turn the date strings into real dates.
• Revenue by category uses defaultdict(float) to total the amounts, then sorts by the total, biggest first.
• Busiest weekdays uses strftime("%A") to turn each date into a weekday name, and Counter(...).most_common(3) to rank them.
• Spend per customer uses groupby, and it works correctly because we sorted by customer first.

Every line of that is standard library. It runs on any computer with nothing but Python installed.

Part 6: Common Mistakes

Notice how many of these say "no error". Those are the dangerous ones: the code runs, the output looks reasonable, and the answer is quietly wrong. Knowing these traps in advance is the best protection.

Quick Reference Summary

collections

from collections import defaultdict, Counter, namedtuple, deque

totals = defaultdict(float)         # missing key becomes float() == 0.0
groups = defaultdict(list)          # missing key becomes []
counts = Counter(items)             # counts anything iterable
counts.most_common(3)               # the top three, already sorted
Row = namedtuple("Row", "a b c")    # a tuple with field names
window = deque(maxlen=100)          # keeps only the last 100 items

itertools (everything is lazy)

functools

from functools import partial, lru_cache, reduce, wraps

warn = partial(tag, "WARNING")      # fill in the first argument ahead of time
@lru_cache                          # remember results (hashable args, pure function)
def fib(n): ...
fib.cache_info()                    # hits, misses, size
fib.cache_clear()                   # throw the saved answers away

datetime

from datetime import date, datetime, timedelta, timezone

datetime.strptime("2024-03-01", "%Y-%m-%d")   # string  -> datetime
datetime.fromisoformat("2024-03-01")          # the same, for ISO 8601
moment.strftime("%d %B %Y")                   # datetime -> string
moment.isoformat()                            # the format to store
later = moment + timedelta(days=30)           # arithmetic
gap = later - moment                          # a timedelta
gap.days, gap.total_seconds()                 # reading it
datetime(2024, 3, 1, tzinfo=timezone.utc)     # aware, not naive

Which container