Working with APIs: Getting Data from Other Computers

This guide is long on purpose. It starts from zero (what an API actually is, what problem it solves, and the vocabulary people use), then works through every piece of code in the chapter's notebook, and finishes with the ways APIs show up in data science and machine learning work: collecting datasets, turning JSON into pandas DataFrames, calling paid AI models, testing API code, and serving your own model as an API.

Here is what we will cover:

1. What an API is, what problem it solves, and the words you need to know
2. How a web request actually travels: URLs, methods, headers, bodies and JSON
3. Installing requests with the project's requirements.txt
4. Making a request and reading the response
5. Status codes, and the errors that aren't exceptions
6. timeout and raise_for_status, the two lines people forget
7. Query parameters and headers
8. API keys, and why they never go in your code
9. Retrying, with an upgraded version of chapter 13's decorator
10. Rate limits, and how to stay inside them
11. Pagination, using chapter 12's generators
12. A small, complete API client
13. APIs in data science and machine learning
14. Being a good client, common mistakes, and a cheat sheet

About the output in this guide. Most examples read a saved copy of a real response from a sample_data/ folder, so they give the same answer every time and work without the internet. A handful make a real request to GitHub's API, and those are marked. Their output is from the day the notebook was built, so your numbers will be different.

Part 1: What Is an API?

API stands for Application Programming Interface. That's a mouthful, so let's take it apart:

• Application: a program.
• Programming: used by code, not by a person clicking around.
• Interface: the agreed way two things connect, like a plug and a socket.

Put together: an API is an agreed way for one program to ask another program for something.

The Restaurant Analogy

The most common way to picture an API is a restaurant:

• You are the customer. In our case, "you" is your Python code. This side is called the client.
• The kitchen has the food and does the work. In our case, it's another company's computer with the data. This side is called the server.
• The waiter is the API. You don't walk into the kitchen and start opening fridges. You give your order to the waiter, in the format the restaurant expects (an item from the menu), and the waiter brings back what you asked for.
• The menu is the API's documentation: the list of things you're allowed to ask for, and how to ask.

     YOU (client)                   WAITER (API)                  KITCHEN (server)
   ┌─────────────┐   your order   ┌─────────────┐   passes it   ┌──────────────────┐
   │ Python code │ ─────────────► │  agreed     │ ────────────► │ databases, models│
   │             │                │  rules      │               │ business logic   │
   │             │ ◄───────────── │             │ ◄──────────── │                  │
   └─────────────┘   your food    └─────────────┘   the result  └──────────────────┘
                     (JSON data)

Notice what the waiter protects. You never see how the kitchen is organised, which chef is working, or where the ingredients are stored. The kitchen can rebuild itself completely, and as long as the menu stays the same, your order still works. That separation is the whole point of an API.

What Problem Does an API Solve?

Imagine you want today's weather in Mumbai inside your Python program, and there's no API. What are your options?

1. Copy it by hand from a website. Fine once. Hopeless for 500 cities every hour.
2. Download the web page and dig the number out of the HTML. This is called web scraping (chapter 18). It works until the website changes its design, and then your code silently breaks. Web pages are made for human eyes, not for programs.
3. Ask for direct access to the weather company's database. No company will hand strangers the keys to its database: it's a security risk, and you could slow it down for everyone.

An API is the fourth option, and the one designed for exactly this job:

• It's made for programs. The answer comes back as clean, structured data (usually JSON), not a page full of layout and adverts.
• It's stable. The company promises the format won't change without warning, often by giving each version a number like /v1/ or /v2/.
• It's safe for both sides. You can only do what the API allows, and the company can control who calls it, how often, and what they see.
• It hides complexity. One line of your code can trigger a huge amount of work on their side (searching millions of records, running a machine learning model) without you needing to know how it's done.

Analogy: a bank's ATM is an API for your bank account. You don't get to walk into the vault. You get a small, fixed set of things you can do (check balance, withdraw, deposit), a card and PIN to prove who you are, and a limit on how much you can withdraw per day. That's almost exactly how a web API works: fixed operations, a key to prove who you are, and a limit on usage.

Not All APIs Are Web APIs

The word "API" is used more widely than you might expect:

• Library APIs. The functions and classes a Python library gives you are also called its API. When people say "pandas has a clean API", they mean df.groupby(...).mean() is pleasant to use.
• Operating system APIs. os.environ and open() talk to your operating system through its API.
• Web APIs. A program on another computer that you talk to over the internet, using HTTP. This guide is about web APIs, and when people in data science just say "an API", this is usually what they mean.

The Words You Need to Know

You'll meet these terms in every API's documentation. Don't try to memorise them all now; each one gets explained properly later in the guide, and you can come back to this table any time.

REST, and a Few Other Styles

Most APIs you'll use are REST APIs (or "RESTful"). The idea is simple:

• every thing (resource) has its own URL: /repos/python/cpython, /users/guido,
• you act on it with HTTP methods: GET it to read it, POST to create a new one, DELETE to remove it,
• each request is independent and carries everything the server needs (this is called being stateless: the server doesn't remember your previous request).

You'll also hear about a few other styles. You don't need them yet, but it helps to recognise the names:

• GraphQL: one single endpoint, and you send a query describing exactly which fields you want back. GitHub offers one alongside its REST API.
• gRPC: a fast, binary style used mainly between services inside a company.
• Streaming: the server sends the answer a piece at a time instead of all at once. AI chat APIs often use this, which is why answers appear word by word.

Part 2: How a Web Request Travels

Before writing code, it's worth seeing what actually happens when your program "calls an API". Every call is a request going out and a response coming back.

   YOUR PROGRAM                                                       THE API SERVER
       │                                                                  │
       │  REQUEST                                                         │
       │  ┌────────────────────────────────────────────────────────┐      │
       │  │ GET /repos/python/cpython            ← method + path   │      │
       │  │ Host: api.github.com                 ← headers         │ ───► │
       │  │ Accept: application/vnd.github+json                    │      │
       │  │ User-Agent: my-app/1.0                                 │      │
       │  └────────────────────────────────────────────────────────┘      │
       │                                                                  │
       │                                                        RESPONSE  │
       │       ┌────────────────────────────────────────────────────────┐ │
       │       │ 200 OK                               ← status code     │ │
       │ ◄───  │ Content-Type: application/json       ← headers         │ │
       │       │                                                        │ │
       │       │ {"full_name": "python/cpython", ...} ← body (JSON)     │ │
       │       └────────────────────────────────────────────────────────┘ │

Analogy: it's like sending a letter and getting a reply. The address on the envelope is the URL. The notes written on the envelope ("urgent", "reply in English") are the headers. The letter inside is the body. And the reply comes with a stamp at the top saying how it went ("delivered", "address unknown"), which is the status code.

The Parts of a URL

A URL isn't just an address; it's made of parts, and each one has a job:

   https://api.github.com/search/repositories?q=language%3Apython&sort=stars&per_page=3
   └─┬─┘   └─────┬──────┘└─────────┬────────┘ └──────────────────┬────────────────────┘
   scheme       host              path                     query string
   (how)       (which         (which thing           (options: name=value pairs,
              computer)        on it)                  joined with &, after the ?)

   base URL = https://api.github.com           endpoint = /search/repositories

Python's standard library can split a URL into these parts for you, which is handy for checking what a URL actually contains:

from urllib.parse import urlparse, parse_qs

url = "https://api.github.com/search/repositories?q=language%3Apython&sort=stars&per_page=3"
parts = urlparse(url)

print("scheme:", parts.scheme)
print("host  :", parts.netloc)
print("path  :", parts.path)
print("query :", parts.query)
print("params:", parse_qs(parts.query))

scheme: https
host  : api.github.com
path  : /search/repositories
query : q=language%3Apython&sort=stars&per_page=3
params: {'q': ['language:python'], 'sort': ['stars'], 'per_page': ['3']}

Notice that parse_qs also decoded the query: language%3Apython came back as language:python. We'll see why that %3A was there in Part 7.

The Body Is (Almost Always) JSON

The data in a response body is nearly always JSON (JavaScript Object Notation), the same format you read and wrote in chapter 10. It maps neatly onto Python:

So once the JSON arrives, you work with it using the dictionaries and lists you already know. The new part in this chapter is getting it.

Part 3: Installing requests

To make HTTP requests, we'll use a library called requests. Unlike everything in chapters 1 to 16, it isn't part of the standard library, so it has to be installed. This is the moment chapter 11's virtual environments were preparing you for.

Analogy: the standard library is the toolbox that came with the house. requests is the first tool we have to buy from the hardware store, and a virtual environment is the separate toolbox we keep for this project, so its tools don't get mixed up with other projects.

The course repository already has a requirements.txt: a short, hand-written list of the packages the course needs, with requests in it. Open a terminal in the course folder and run:

python3 -m venv .venv                          # create the environment, once
source .venv/bin/activate                      # Windows: .venv\Scripts\activate
python3 -m pip install -r requirements.txt     # install what the course needs
python3 -m pip install jupyterlab              # so the notebook can run in it
jupyter lab                                    # reopen the course from here

Line by line:

1. python3 -m venv .venv creates the virtual environment in a folder called .venv. You only do this once.
2. source .venv/bin/activate switches your terminal to use it. (Windows uses the command in the comment.)
3. python3 -m pip install -r requirements.txt installs every package listed in the file. -r means "read the list from this file".
4. python3 -m pip install jupyterlab installs Jupyter into the same environment, so the notebook runs with the same Python.
5. jupyter lab opens the course from inside the environment.

If the import below fails with ModuleNotFoundError, the notebook is running on a different Python from the one you installed into. Chapter 11's "Your notebook and your environment" section shows how to check which Python it's using, with sys.executable. This is also the first time in the course that running a notebook needs a step besides opening it.

import requests

print("requests version:", requests.__version__)

requests version: 2.32.3

Your version number may be different. That's fine, as long as it's at least the 2.31 that requirements.txt asks for.

Python's standard library does have a module called urllib.request, and it can do everything in this chapter. But it takes noticeably more code for the same job, so most people use requests, and that's what you'll meet in real projects. (Two other names you may see later: httpx, which works like requests and also supports async code, and aiohttp, which is async only.)

Part 4: Your First Request

requests.get(url) sends an HTTP GET request. GET is the request type that means "give me this thing". It returns a response object that holds everything the server sent back.

The next example talks to GitHub's public API, which lets you make a small number of requests without an account. (This one makes a real request.)

response = requests.get("https://api.github.com/repos/python/cpython", timeout=10)

print("status code :", response.status_code)
print("ok?         :", response.ok)
print("content type:", response.headers["Content-Type"])
print("url         :", response.url)

status code : 200
ok?         : True
content type: application/json; charset=utf-8
url         : https://api.github.com/repos/python/cpython

The timeout=10 says "give up if there's no answer within 10 seconds". Part 6 explains why you should never leave it out.

Four things on the response worth knowing:

Content-Type: application/json is the server telling us, "the body I'm sending is JSON". That's our cue to read it as JSON.

Reading the Body with .json()

The body is the part you came for, and .json() turns it into Python data. It's chapter 10's json.loads, with the download built in. (This one uses the real response from above.)

data = response.json()

print(type(data).__name__)
print("keys:", sorted(data)[:8], "...")
print()
print(data["full_name"], "|", data["description"])
print("stars:", data["stargazers_count"])

dict
keys: ['allow_forking', 'archive_url', 'archived', 'assignees_url', 'blobs_url', 'branches_url', 'clone_url', 'collaborators_url'] ...

python/cpython | The Python programming language
stars: 75972

• type(data).__name__ shows the JSON object became a Python dict.
• sorted(data)[:8] lists the first eight keys alphabetically. The real response has dozens of them; APIs usually send far more than you need.
• Then we pick out the fields we care about, exactly like any dictionary.

That output is from the day the notebook was built, and the star count has changed since.

Working from a Saved Copy

From here on, we'll mostly use a saved copy of a real response, so the numbers stay the same every time:

import json

with open("sample_data/github_repo.json", encoding="utf-8") as f:
    repo = json.load(f)

print(repo["full_name"], "|", repo["language"])
print("stars       :", f"{repo['stargazers_count']:,}")
print("open issues :", f"{repo['open_issues_count']:,}")
print("owner       :", repo["owner"]["login"], f"({repo['owner']['type']})")

python/cpython | Python
stars       : 75,971
open issues : 9,622
owner       : python (Organization)

Notice repo["owner"]["login"]: the owner is a dictionary inside a dictionary. Nested data like this is very common in API responses, and Part 13 shows how pandas can flatten it into columns.

sample_data/github_repo.json is a real response, trimmed down to the fields this chapter uses and saved in the repository. Saving a response while you work on the code that handles it is a good habit anyway:

• it's faster (reading a file takes microseconds; a network request takes much longer),
• it works without internet,
• and it doesn't use up your request allowance by sending the same request forty times while you fix a typo.

Analogy: it's like photocopying a library book you need for a week of study, instead of walking to the library every time you want to check one page.

The Other Request Types

GET asks for something. HTTP has other request types (often called verbs or methods) for changing things, and requests has a function for each:

Analogy: think of a shared notice board. GET is reading a notice. POST is pinning up a new one. PUT is replacing a notice with a completely new sheet. PATCH is correcting one line on it with a pen. DELETE is taking it down.

These verbs differ in an important way: is it safe to repeat?

• GET only reads, so it's always safe to repeat. Reading a notice twice changes nothing.
• POST isn't: sending the same order twice can create two orders. That's why a browser warns you before resending a form.
• PUT and DELETE give the same end result if you repeat them (replacing a notice with the same sheet twice, or taking down a notice that's already gone, leaves the board the same). This property is called being idempotent. They still change data, though, so don't send them casually.

This matters later, when we automatically retry failed requests: retrying a GET is harmless, retrying a POST might do the thing twice.

This chapter only reads data, so it only uses GET. But you can see exactly what a POST would send without actually sending it, by building the request and stopping there:

order = requests.Request("POST", "https://api.example.com/orders",
                         json={"item": "Python Crash Course", "quantity": 2},
                         headers={"Authorization": "Bearer not-a-real-key"})

prepared = order.prepare()                  # built, but never sent

print("method      :", prepared.method)
print("url         :", prepared.url)
print("content type:", prepared.headers["Content-Type"])
print("body        :", prepared.body)

method      : POST
url         : https://api.example.com/orders
content type: application/json
body        : b'{"item": "Python Crash Course", "quantity": 2}'

The json= argument did two things:

1. It turned the dictionary into a JSON string, like chapter 10's json.dumps.
2. It set the Content-Type header to application/json, so the server knows how to read it.

The b in front of the body means it's bytes: raw data, ready to go over the network. Passing data= instead of json= would send the values as form fields, which is what an HTML form on a web page does.

requests.Request(...).prepare() is a useful trick to remember. When an API rejects your request and you can't see why, preparing it first lets you look at exactly what would be sent.

Part 5: Status Codes

Every response comes with a three-digit status code. The first digit is the part to remember:

Analogy: think of status codes as the responses you get at a shop counter:

• 2xx: "Here you go."
• 3xx: "We've moved; the shop is next door now." (requests walks you there automatically.)
• 4xx: "You asked for something wrong": a product that doesn't exist, no membership card, or too many questions too fast.
• 5xx: "Sorry, our till has crashed." Not your fault, and trying again later may work.

The 4xx group is the one that wastes people's time, because each code needs a different fix:

401 is about authentication ("I don't know who you are"), and 403 is about authorization ("I know who you are, and you're not allowed"). The names are confusing for historical reasons, but the difference is worth knowing when you're debugging.

Here's a real 404 (this one makes a real request):

missing = requests.get("https://api.github.com/repos/python/not-a-real-repo", timeout=10)

print("status:", missing.status_code)
print("ok?   :", missing.ok)
print("body  :", missing.json()["message"])

status: 404
ok?   : False
body  : Not Found

A Failed Request Doesn't Raise an Exception

That example ran to the end without a try anywhere. requests counts "the server answered" as success, and a 404 is an answer.

Analogy: when you post a letter to a wrong address and it comes back marked "address unknown", the postal service didn't fail. It delivered a reply. It's up to you to read the stamp.

That leads to a very common bug in API code. The 404 body is valid JSON, so .json() works fine, and the program carries on until it looks for a key that isn't there. Here's GitHub's 404 body from above, treated as if it were the repository:

data = {"message": "Not Found"}            # what the 404 above sent back

try:
    print(data["full_name"])
except KeyError as e:
    print("KeyError:", e)

KeyError: 'full_name'

In a real program, that KeyError could show up thirty lines away from the request that actually failed, which makes it hard to track down. Part 6 shows how to stop at the real problem.

A 200 That Isn't JSON

.json() assumes the body is JSON. Sometimes something else answers instead: a login page, an error page from a network in between (like hotel or office Wi-Fi asking you to sign in), or a "down for maintenance" notice. Then you get HTML with a perfectly normal 200 status, and the trouble shows up here:

html = "<!DOCTYPE html><html><body>Sign in to continue</body></html>"

try:
    json.loads(html)                    # this is what response.json() calls
except json.JSONDecodeError as e:
    print("JSONDecodeError:", e)

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

response.json() raises that same exception. "Expecting value: line 1 column 1 (char 0)" means the very first character already wasn't JSON (here, the < of the HTML).

When you see it, a good next step is print(response.text[:200]). response.text is the body as plain text, and the page the server sent instead usually says what went wrong.

Part 6: The Two Lines People Forget

raise_for_status()

raise_for_status() turns a bad status into an exception, so the program stops right where the failure happened. It's the bridge between HTTP and chapter 9's exception handling. (This one makes a real request.)

try:
    bad = requests.get("https://api.github.com/repos/python/not-a-real-repo", timeout=10)
    bad.raise_for_status()
    print("this line is not reached")
except requests.exceptions.HTTPError as e:
    print("HTTPError:", e)

HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repos/python/not-a-real-repo

On a 2xx it does nothing at all, so you can call it after every request without thinking about it. It raises HTTPError for any 4xx or 5xx.

Analogy: it's like a smoke alarm for your request. Most of the time it's silent. When something's wrong, it makes sure you hear about it immediately, instead of discovering the fire thirty minutes later.

timeout

requests has no timeout by default. Leave it out, and if a server accepts your connection and then never answers, your program waits forever. Not thirty seconds: forever. When a script hangs overnight with no error message, a missing timeout is one of the first things to check.

Analogy: calling a helpline with no timeout is like being put on hold and promising yourself you'll never hang up. If they forget about you, you'll still be there tomorrow.

A tiny timeout of one thousandth of a second shows the exception you'd get (this one makes a real request):

try:
    requests.get("https://api.github.com/repos/python/cpython", timeout=0.001)
except requests.exceptions.Timeout as e:
    print("caught:", type(e).__name__)

caught: ConnectTimeout

There are two kinds of timeout, and which one you see depends on how far the request got before the time ran out:

• ConnectTimeout: the connection never even opened in time.
• ReadTimeout: the connection opened, and then the server went quiet for too long.

With a limit as tiny as 0.001 seconds, you might see either one, depending on your network. Both are children of Timeout, so catching Timeout covers both.

You can also give the two limits separately as a tuple, like timeout=(3, 30): up to 3 seconds to connect, and up to 30 seconds between pieces of data arriving. That's useful for slow endpoints, such as an AI model that takes a while to answer.

The Family of requests Exceptions

Timeout is in turn a child of requests.exceptions.RequestException, the parent of everything requests raises. So that's the one to catch when you want to treat "the network didn't work" as a single case:

   RequestException                    ← catch this to handle "anything went wrong"
   ├── ConnectionError
   ├── Timeout
   │   ├── ConnectTimeout
   │   └── ReadTimeout
   └── HTTPError

(ConnectTimeout is actually a child of both ConnectionError and Timeout, since it's both kinds of problem, but the tree above is the picture to keep in mind.)

Watch out for one name clash. requests.exceptions.ConnectionError is not the same as Python's built-in ConnectionError, so a plain except ConnectionError: won't catch it:

print("is requests' ConnectionError the built-in one?",
      issubclass(requests.exceptions.ConnectionError, ConnectionError))

is requests' ConnectionError the built-in one? False

Always write the full requests.exceptions... name when you mean the requests one.

Put together, every request you write should look like this:

try:
    response = requests.get(url, timeout=10)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    ...

Three parts, every time: a timeout so it can't hang, raise_for_status() so a bad status stops right there, and a try/except so a failure is handled instead of crashing the whole program.

Part 7: Query Parameters and Headers

Query Parameters

Anything after a ? in a URL is a query parameter: an option you pass to the endpoint, like a search term, a sort order or a page number.

You could build that part of the URL yourself, but you shouldn't. Values can contain spaces, &, = and non-English characters, and all of those have to be encoded in a special way. Here's what goes wrong when you don't:

from urllib.parse import urlencode

search = {"q": "R&D reports", "lang": "en"}

print("by hand:", "q=" + search["q"] + "&lang=" + search["lang"])
print("encoded:", urlencode(search))
print()
print("what the server reads from the hand-built one:")
print(" ", parse_qs("q=R&D reports&lang=en"))

by hand: q=R&D reports&lang=en
encoded: q=R%26D+reports&lang=en

what the server reads from the hand-built one:
  {'q': ['R'], 'lang': ['en']}

The search term R&D reports contains an &, and in a URL & means "the next parameter starts here". So the hand-built URL is read as q=R, followed by a broken piece, D reports, which has no = and is simply thrown away. The search silently became a search for "R". The encoded version turned & into %26 and the space into +, so the value arrives intact.

Analogy: encoding is like putting fragile items in bubble wrap before posting them. Characters like &, = and spaces have special jobs in a URL, so they're wrapped up (as %26, %3D, +) to travel safely, and unwrapped at the other end.

Pass a dictionary as params and requests does the encoding for you (this one makes a real request):

response = requests.get("https://api.github.com/search/repositories",
                        params={"q": "language:python topic:machine-learning",
                                "sort": "stars",
                                "per_page": 3},
                        timeout=10)

print("the URL that was actually sent:")
print(" ", response.url)
print()
for item in response.json()["items"]:
    print(f"  {item['full_name']:<28} {item['stargazers_count']:>7,} stars")

the URL that was actually sent:
  https://api.github.com/search/repositories?q=language%3Apython+topic%3Amachine-learning&sort=stars&per_page=3

  huggingface/transformers     164,744 stars
  pytorch/pytorch              102,740 stars
  d2l-ai/d2l-zh                 80,250 stars

Look at response.url:

• the spaces became +,
• the colons became %3A,
• and the three parameters were joined with &.

That's what you'd have had to get right by hand. With params=, you never have to think about it.

Path parameters are different. They're part of the path itself, like python/cpython in /repos/python/cpython, so you build them into the URL with an f-string, as the client in Part 12 does with f"/repos/{full_name}".

Headers

Headers are extra information a request sends about itself, written as name and value pairs. Two matter early on:

• User-Agent says who's calling. Some APIs reject requests without one, and a name plus a contact address is the polite way to fill it in.
• Accept says what format you want back.

They go in as a dictionary:

headers = {"User-Agent": "sales-report/1.0 (you@example.com)",
           "Accept": "application/vnd.github+json"}

requests.get(url, headers=headers, timeout=10)

application/vnd.github+json is GitHub's own name for "GitHub-style JSON, please". Many APIs document a specific Accept value like this.

Responses have headers too, and they're worth reading. GitHub uses them to tell you about your rate limit, which Part 10 covers. Other common response headers you'll meet are Content-Type (the format of the body), Retry-After (how long to wait before trying again) and Link (where the next page is, in Part 11).

Part 8: API Keys, and Where They Don't Go

Most APIs beyond a free tier need a key, which is like a password for your program. It tells the API who is calling, so it can apply your limits, charge your account, and block you if you misbehave. It usually travels in a header:

headers = {"Authorization": f"Bearer {api_key}"}

Bearer is a standard word meaning "whoever bears (carries) this token is allowed in". You'll see a few other ways of sending credentials in API documentation:

Whatever the style, the question is the same: where does api_key come from? Not from your code:

api_key = "sk-live-8f4c2a91d7e3"      # never do this

A key written into a file is a key you've published:

• It goes into git history, and deleting the line later doesn't remove it, because git keeps every version.
• It ends up in the notebook's saved output if you ever print it.
• It gets shared whenever you send someone the file.
• Bots scan public repositories for exactly this, around the clock, and a leaked cloud key can run up a big bill in hours.

Analogy: writing your API key into your code is like writing your bank PIN on the back of your card. It's convenient right up until the moment someone else picks up the card.

Environment Variables

Keep the key in an environment variable instead: a named value held by your operating system, outside your project. You set it in the terminal:

export WEATHER_API_KEY="your-key-here"        # macOS and Linux
setx WEATHER_API_KEY "your-key-here"          # Windows

(setx on Windows only takes effect in terminal windows you open afterwards. On macOS and Linux, export only lasts for the current terminal session; to keep it, people add the line to a startup file like ~/.bashrc or ~/.zshrc, or use a .env file as described below.)

Analogy: an environment variable is like keeping your key in a locker outside the classroom. Your code walks to the locker and opens it when it needs the key, but the key itself is never written in your notebook, so sharing the notebook shares nothing secret.

os.environ reads environment variables. You've used os since chapters 9 and 10:

import os

# Standing in for a key you would have exported in your shell.
os.environ["DEMO_API_KEY"] = "sk-demo-not-a-real-key"

print("present    :", os.environ.get("DEMO_API_KEY"))
print("not present:", os.environ.get("MISSING_API_KEY"))

present    : sk-demo-not-a-real-key
not present: None

The first line sets a fake key, standing in for one you'd normally export in your terminal. Then .get() reads it. For a variable that doesn't exist, .get() returns None.

Fail Loudly When the Key Is Missing

os.environ.get() returning None is the wrong kind of failure. The program carries on and sends the text Bearer None to the server, and the error you eventually see is a 401 that says nothing about the real cause.

Fail loudly instead, using chapter 9's raise:

def get_api_key(name):
    key = os.environ.get(name)
    if not key:
        raise RuntimeError(
            f"{name} is not set. Export it before running this script:\n"
            f'    export {name}="your-key-here"')
    return key


print("found:", get_api_key("DEMO_API_KEY")[:8] + "...")

try:
    get_api_key("MISSING_API_KEY")
except RuntimeError as e:
    print("\nRuntimeError:")
    print(e)

found: sk-demo-...

RuntimeError:
MISSING_API_KEY is not set. Export it before running this script:
    export MISSING_API_KEY="your-key-here"

Now a missing key stops the program immediately, with a message that says exactly what's wrong and how to fix it. Compare that with spending twenty minutes wondering why the server says "Unauthorized".

Two More Habits

• Many projects keep their keys in a file called .env, which a tool or your own code loads into environment variables. (A popular package for this is python-dotenv: load_dotenv() reads the file, and os.environ sees the values.) Add .env, and any other file holding secrets, to .gitignore on day one, so git never picks it up. This repository's .gitignore already has a Secrets section with .env in it, even though no secret is ever written here.
• Never print a key. The example above prints only the first eight characters so you can see it was found, and that's as far as it should ever go.

If a key ever does leak, don't just delete it from the file. Revoke it (cancel it) in the API provider's dashboard and create a new one, because the old one is already out there.

Part 9: Retrying, with an Upgraded Chapter 13 Decorator

Networks fail in ways that disks don't. A Wi-Fi blip, a server restarting, a busy moment at the provider: a request that failed a second ago might work now. So the right response to a connection error or a 503 is often to wait and try again.

Chapter 13 built a retry decorator factory for exactly this. Here's an upgraded version with three changes:

• delay waits between attempts, and the wait doubles each time. This is called exponential backoff. Three quick retries against a struggling server just give it three more problems.
• on says which exceptions are worth retrying, so other errors pass straight through.
• When it runs out of attempts, it re-raises the last error with a bare raise (chapter 9), instead of replacing it with a RuntimeError. That way the caller sees what actually went wrong.

import time
import functools


def retry(times=3, delay=0.5, on=Exception):
    def decorator(func):
        @functools.wraps(func)                  # chapter 13
        def wrapper(*args, **kwargs):
            wait = delay
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except on as e:
                    if attempt == times:
                        raise                   # out of tries, let it through
                    print(f"  attempt {attempt} failed ({type(e).__name__}), "
                          f"waiting {wait}s")
                    time.sleep(wait)
                    wait *= 2                   # back off
        return wrapper
    return decorator

Let's walk through the layers, since decorator factories have a lot of them:

• retry(times=3, delay=0.5, on=Exception) is the factory. You call it with your settings, and it returns decorator.
• decorator(func) receives the function being decorated and returns wrapper.
• wrapper(*args, **kwargs) is what actually runs when you call the decorated function. It loops through the attempts:
• return func(*args, **kwargs): if the call works, return its result straight away, which also ends the loop.
• except on as e: if it raises one of the "worth retrying" exceptions, check whether this was the last attempt. If it was, a bare raise re-raises the same error. If not, print a note, time.sleep(wait), and double the wait.
• @functools.wraps(func) (chapter 13) copies the original function's name and docstring onto wrapper, so the decorated function still looks like itself.

Analogy: exponential backoff is how a sensible person redials a busy phone line: try again after a moment, then wait a bit longer, then longer still. Redialling ten times a second doesn't get you through any faster; it just jams the line for everyone.

   attempt 1 ✗ ── wait 0.5s ── attempt 2 ✗ ── wait 1s ── attempt 3 ✗ ── give up, raise the error
                                                                       (or ✓ at any point: return)

In real systems, people often add a small random amount to each wait (called jitter), so that a thousand clients that failed at the same moment don't all retry at exactly the same moment too.

Seeing It Work

To show it working we need a function that fails in a predictable way. This one fails twice and then succeeds. The demos raise Python's built-in ConnectionError to stand in for a real network failure:

calls = {"count": 0}


@retry(times=4, delay=0.1, on=ConnectionError)
def flaky_fetch():
    calls["count"] += 1
    if calls["count"] < 3:
        raise ConnectionError(f"call {calls['count']} refused")
    return {"status": "ok", "on_call": calls["count"]}


print("result:", flaky_fetch())

  attempt 1 failed (ConnectionError), waiting 0.1s
  attempt 2 failed (ConnectionError), waiting 0.2s
result: {'status': 'ok', 'on_call': 3}

calls is a dictionary so that the function can change the count inside it (changing a value inside a dictionary doesn't need global or nonlocal). The first two calls raised, the decorator waited 0.1s and then 0.2s, and the third call returned a result.

And when it never succeeds, the error comes through instead of being swallowed. That's the same rule chapter 14 gave for context managers:

@retry(times=3, delay=0.1, on=ConnectionError)
def always_broken():
    raise ConnectionError("host unreachable")


try:
    always_broken()
except ConnectionError as e:
    print("gave up, and said so:", e)

  attempt 1 failed (ConnectionError), waiting 0.1s
  attempt 2 failed (ConnectionError), waiting 0.2s
gave up, and said so: host unreachable

What Is Worth Retrying?

One rule about what to retry:

• Worth retrying: a 503, a 502, a timeout, a dropped connection. Your request was fine; the server or the network wasn't, and that can change in a second.
• Not worth retrying: a 404 or a 401. You'll get the same answer every time, and retrying with a bad key just gets you blocked faster.
• Careful with POST: retrying a request that creates something might create it twice. Some APIs let you send an idempotency key (a unique ID for the operation) so the server can tell a retry from a genuinely new request.

Part 10: Rate Limits

If an API let everyone call it without limit, someone would take it down by accident (one loop with a bug in it is enough). So most APIs give you a budget: a number of requests per minute, hour or day. And the response tells you how much you have left. (This one makes a real request.)

limits = requests.get("https://api.github.com/rate_limit", timeout=10)
core = limits.json()["resources"]["core"]

print("allowed per hour:", core["limit"])
print("remaining       :", core["remaining"])
print("resets at       :", core["reset"], "(a Unix timestamp)")

allowed per hour: 60
remaining       : 57
resets at       : 1788461615 (a Unix timestamp)

GitHub allows 60 requests per hour without an account, and far more (thousands) with a token. Its search endpoints have their own, smaller per-minute limit. Every API sets its own numbers, which is why reading the documentation comes first.

Analogy: a rate limit is like a mobile data plan. You get a fixed allowance per period, the app tells you how much is left, it resets at a known time, and if you go over, you're slowed down or cut off until the reset.

Unix Timestamps

That reset value is a Unix timestamp: the number of seconds since 1 January 1970 (UTC), which is how many systems store times. It's just one big number, so it's easy to store and compare. datetime.fromtimestamp() turns one into a datetime, and tz=timezone.utc makes it an aware UTC datetime, like the ones in chapter 15:

from datetime import datetime, timezone

reset_at = datetime.fromtimestamp(1735689600, tz=timezone.utc)
print("a reset timestamp of 1735689600 is:", reset_at.isoformat())

a reset timestamp of 1735689600 is: 2025-01-01T00:00:00+00:00

When You Go Over: 429

When you go over the budget you get 429 Too Many Requests, usually with a Retry-After header saying how many seconds to wait. Treat that as an instruction, not a suggestion:

if response.status_code == 429:
    wait = int(response.headers.get("Retry-After", 60))
    time.sleep(wait)

.get("Retry-After", 60) means "use the header if it's there, otherwise assume 60 seconds".

One detail the official HTTP rules allow: Retry-After can be either a number of seconds ("30") or a date ("Wed, 21 Oct 2015 07:28:00 GMT"). The short version above handles the first form, which is what most APIs send. If an API you use sends dates, int() would fail on it, so a slightly fuller helper handles both:

from email.utils import parsedate_to_datetime


def seconds_to_wait(retry_after, default=60):
    if retry_after is None:                       # header missing: use a safe default
        return default
    if retry_after.isdigit():                     # "30" means 30 seconds
        return int(retry_after)
    when = parsedate_to_datetime(retry_after)     # otherwise it is a date
    return max(0, (when - datetime.now(timezone.utc)).total_seconds())


print(seconds_to_wait("30"))
print(seconds_to_wait(None))
print(seconds_to_wait("Wed, 21 Oct 2015 07:28:00 GMT"))      # already in the past

30
60
0

• "30" is all digits, so it's 30 seconds.
• No header at all falls back to the default.
• A date is turned into a datetime, and we wait until then. That date is long past, so max(0, ...) makes the wait zero instead of negative.

Three Habits That Keep You Inside Any Limit

1. Save what you fetch. Fetching the same thing again and again is an easy way to burn through a budget. Chapter 10's json.dump is all you need.
2. Ask for more per request. One call that returns 100 rows beats 100 calls that return one row each. Look for parameters like per_page or limit.
3. Pause between calls in a loop. Even time.sleep(0.1) turns a burst of requests into a steady trickle.

The first one only takes a few lines. This is a tiny cache: a place where results are kept so they don't have to be fetched again:

from contextlib import suppress

os.makedirs("api_demo", exist_ok=True)
CACHE = "api_demo/repo.json"

with suppress(FileNotFoundError):           # chapter 14: start each run empty
    os.remove(CACHE)


def cached_json(path, fetch):
    if os.path.exists(path):
        with open(path, encoding="utf-8") as f:
            return json.load(f), "cache"
    data = fetch()
    with open(path, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2)
    return data, "network"


def pretend_fetch():
    print("  (fetching over the network)")
    return {"full_name": "python/cpython", "stars": 75971}


data, source = cached_json(CACHE, pretend_fetch)
print("first call :", source)

data, source = cached_json(CACHE, pretend_fetch)
print("second call:", source, "->", data["full_name"])

  (fetching over the network)
first call : network
second call: cache -> python/cpython

How cached_json works:

1. If the cache file already exists, read it and return its contents, tagged "cache".
2. Otherwise, call fetch() to get the data, save it to the file with json.dump, and return it, tagged "network".

The first call found no file, so it "fetched over the network" (here a pretend function that just returns a dictionary). The second call found the saved file and never called pretend_fetch at all.

cached_json takes the fetching function as an argument. That's chapter 7's higher-order function idea doing real work: the caching code doesn't need to know anything about what it's caching. The same function could cache weather, stock prices or GitHub data.

The os.remove at the top (wrapped in chapter 14's suppress, so a missing file isn't an error) is only there so this example tells the same story every time it runs. In real use, the whole point of a cache is that it sticks around.

A note for data science work: saved responses are also how you make an analysis reproducible. An API returns today's data; next month it will return something different. If your report was built on what you fetched on the 3rd of September, save that exact response (with the date in the file name), and anyone can rebuild your results later.

Part 11: Pagination

An API won't hand you ten thousand rows at once. It gives you one page and expects you to ask for the next.

Analogy: it's like a search engine's results. You see ten results on page 1, and you click "Next" to see more. Nobody sends you all two million results at once.

Written as a plain loop, the calling code has to know about pages. Written as a generator (chapter 12), the caller just gets rows, one at a time, and never has to think about pages at all:

def all_items(get_page):
    # Yield items from every page, one at a time, until a page comes back empty.
    page = 1
    while True:
        items = get_page(page)
        if not items:                       # an empty page is the end
            return
        print(f"  (fetched page {page}: {len(items)} items)")
        yield from items                    # chapter 12
        page += 1

How it works:

1. Start at page = 1.
2. Ask get_page(page) for that page's items.
3. If the page is empty, that's the end: return stops the generator.
4. Otherwise, yield from items hands the items out one at a time (chapter 12), then move to the next page and repeat.

get_page is a function the caller supplies. That keeps the paging logic separate from where the pages come from, which is what lets us demonstrate it offline. The print is there so you can watch the requests happen. A real one would write to a log, or stay quiet.

This get_page reads the saved sample pages (issues_page_1.json to issues_page_4.json, where the last one is an empty list):

def get_page_from_samples(page):
    path = f"sample_data/issues_page_{page}.json"
    if not os.path.exists(path):
        return []
    with open(path, encoding="utf-8") as f:
        return json.load(f)


for issue in all_items(get_page_from_samples):
    print(f"    #{issue['number']} {issue['title'][:48]:<48} {issue['user']}")

  (fetched page 1: 3 items)
    #156898 [3.15] gh-89735: Document requirements for the * miss-islington
    #156897 gh-155742: Optimize json float parsing           vstinner
    #156896 Importing idlelib.run deletes tkinter dialog sub serhiy-storchaka
  (fetched page 2: 3 items)
    #156895 Configure script output doesn't mention CXX      smontanaro
    #156894 Wrong error position for tokenizer errors report serhiy-storchaka
    #156892 gh-156891: Fix shlex source inclusion at EOF     lpyu001
  (fetched page 3: 3 items)
    #156891 `shlex` silently discards a sourced stream when  lpyu001
    #156890 gh-155742: Add tests on the PyMarshal C API      vstinner
    #156889 [3.13] Reword `atexit` docs (GH-156086)          encukou

Nine issues, and four calls to get_page. You can't see the fourth one: it returned an empty list, the generator stopped, and the print never ran because it comes after the check. An empty page is how this API says "there's no more".

   get_page(1) ──► [3 items] ──► yield, yield, yield
   get_page(2) ──► [3 items] ──► yield, yield, yield
   get_page(3) ──► [3 items] ──► yield, yield, yield
   get_page(4) ──► []        ──► stop

Stopping Early

Because it's a generator, only the current page is held in memory, and the caller can stop early. Here chapter 15's islice takes the first four rows, and the later pages are never fetched at all:

from itertools import islice

for issue in islice(all_items(get_page_from_samples), 4):
    print(f"    #{issue['number']} by {issue['user']}")

  (fetched page 1: 3 items)
    #156898 by miss-islington
    #156897 by vstinner
    #156896 by serhiy-storchaka
  (fetched page 2: 3 items)
    #156895 by smontanaro

Two pages fetched instead of four calls. Against a real API, that's half the requests and half the waiting. This is the generator's laziness paying off: the third page is only requested if someone actually asks for a seventh row.

The live version only changes get_page, which now makes a real request (this one makes a real request):

def get_page_from_github(page):
    response = requests.get("https://api.github.com/repos/python/cpython/issues",
                            params={"per_page": 3, "page": page},
                            headers={"Accept": "application/vnd.github+json"},
                            timeout=10)
    response.raise_for_status()
    return response.json()


for issue in islice(all_items(get_page_from_github), 4):
    print(f"    #{issue['number']} {issue['title'][:44]}")

  (fetched page 1: 3 items)
    #156907 [3.13] gh-94242: Clarify comments for _MAX_W
    #156906 [3.14] gh-94242: Clarify comments for _MAX_W
    #156905 [3.15] gh-94242: Clarify comments for _MAX_W
  (fetched page 2: 3 items)
    #156904 [3.14] GH-146096: Fix segfault in BaseExcept

Notice that everything from Part 6 and Part 7 is in there: params for the page number, an Accept header, a timeout, and raise_for_status().

Other Ways APIs Do Pagination

"Ask for page 1, then page 2, until a page is empty" is one style. You'll meet three others, and it's worth recognising them in the documentation:

Cursors are common in large, fast-changing datasets, because page numbers can skip or repeat rows when new rows are added while you're paging. A cursor is a bookmark that says "carry on from exactly here".

GitHub actually uses Link headers: each response includes the URL of the next and last page. requests parses that header for you as response.links, using the same function you can call directly:

link_header = ('<https://api.github.com/repositories/81598961/issues?per_page=3&page=2>; rel="next", '
               '<https://api.github.com/repositories/81598961/issues?per_page=3&page=3164>; rel="last"')

for link in requests.utils.parse_header_links(link_header):
    print(f"  {link['rel']:<5} -> {link['url']}")

  next  -> https://api.github.com/repositories/81598961/issues?per_page=3&page=2
  last  -> https://api.github.com/repositories/81598961/issues?per_page=3&page=3164

So instead of guessing page numbers, a client can follow response.links["next"]["url"] until there's no "next" left. (Page 3164 of three issues each means about 9,500 issues: exactly the kind of size you don't want in one response.) The generator pattern stays the same whichever style an API uses; only get_page changes.

Part 12: Putting It Together

Here's a small client with everything this chapter recommended: a session, a timeout, headers, raise_for_status, and retries. A client is a class that wraps an API, so the rest of your code can say gh.repo("python/cpython") instead of repeating the HTTP details everywhere.

requests.Session is the one new piece. It does two useful things:

• It reuses the same connection between calls, which is faster. Opening a new secure connection takes several back-and-forth messages before any data moves.
• It holds headers (and other settings), so you set them once instead of on every request.

Analogy: a session is like staying on the phone with a shop and asking several questions, instead of hanging up and redialling, re-introducing yourself and re-explaining who you are for every question.

RETRYABLE = (requests.exceptions.ConnectionError,
             requests.exceptions.Timeout)


class GitHubClient:
    BASE = "https://api.github.com"

    def __init__(self, token=None, timeout=10):
        self.timeout = timeout
        self.session = requests.Session()
        self.session.headers.update({
            "Accept": "application/vnd.github+json",
            "User-Agent": "python-course-notebook/1.0",
        })
        if token:
            self.session.headers["Authorization"] = f"Bearer {token}"

    @retry(times=3, delay=0.5, on=RETRYABLE)
    def get(self, path, **params):
        response = self.session.get(f"{self.BASE}{path}",
                                    params=params, timeout=self.timeout)
        response.raise_for_status()
        return response.json()

    def repo(self, full_name):
        return self.get(f"/repos/{full_name}")

    def issues(self, full_name, per_page=3):
        return all_items(lambda page: self.get(f"/repos/{full_name}/issues",
                                               per_page=per_page, page=page))

    def close(self):
        self.session.close()

    def __enter__(self):                        # chapter 14
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.close()

A tour of the class:

• RETRYABLE lists the exceptions worth retrying: connection problems and timeouts.
• __init__ creates the session, sets the Accept and User-Agent headers once, and adds an Authorization header only if a token was given.
• get is the one method that actually talks to the network. It's decorated with our retry, it uses the session, and it always passes the timeout and calls raise_for_status().
• repo and issues are small, friendly methods built on top of get. issues reuses the all_items generator from Part 11, with a lambda (chapter 7) as the get_page function.
• close, __enter__ and __exit__ make the client a context manager (chapter 14).

Every method gets the timeout, the headers, the status check and the retries, because they're all set up in one place. Adding a token later means changing one line where you create the client, and nothing else.

What RETRYABLE Leaves Out

It would have been shorter to write on=requests.exceptions.RequestException and catch everything. But that would also retry 404s, which Part 9 said not to do: asking the same wrong question three times of a server that already answered.

Retrying a 503 is worth doing, but it needs more than an exception type, because raise_for_status() raises the same HTTPError for a 404 and a 503. You'd have to check response.status_code >= 500 before deciding to retry.

There's also a ready-made way to do that, built into requests itself (through the urllib3 library it uses underneath). You describe a retry policy, including which status codes to retry, and attach it to a session:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry_policy = Retry(total=3,
                     backoff_factor=0.5,                         # waits: none, then 1s, 2s, 4s ...
                     status_forcelist=[429, 500, 502, 503, 504],  # statuses worth retrying
                     allowed_methods=["GET"],                     # never re-send a POST
                     respect_retry_after_header=True)

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry_policy))

print(session.get_adapter("https://api.github.com").max_retries)
print("retries these statuses:", sorted(retry_policy.status_forcelist))
session.close()

Retry(total=3, connect=None, read=None, redirect=None, status=None)
retries these statuses: [429, 500, 502, 503, 504]

• backoff_factor=0.5 gives exponential backoff, like our decorator. The first retry happens straight away, and after that the wait is the factor times a doubling number: 1 second, then 2, then 4, up to a maximum of two minutes. (Older versions of urllib3 counted slightly differently, so check the documentation for the version you have.)
• status_forcelist lists the status codes worth retrying: rate limits and server errors, but not 401 or 404.
• allowed_methods=["GET"] makes sure a POST is never re-sent once it may have reached the server. (A request that failed before it even connected can still be retried, since nothing was sent.)
• respect_retry_after_header=True obeys the server's Retry-After.
• session.mount("https://", ...) attaches the policy to every https:// request made through this session.

If every retry fails on one of those statuses, requests raises requests.exceptions.RetryError, which is also a child of RequestException. Our hand-written decorator is still worth understanding, because it's the same idea made visible, and it works for any function, not just HTTP calls.

Using the Client

A session holds an open connection, which is exactly the "take it, then give it back" shape from chapter 14. That's why the class has close(), and also __enter__ and __exit__: a caller who uses with can't forget to close it. (This one makes real requests.)

with GitHubClient() as gh:
    repo = gh.repo("python/cpython")
    print(repo["full_name"], "|", f"{repo['stargazers_count']:,} stars")

    for issue in islice(gh.issues("python/cpython"), 3):
        print(f"  #{issue['number']} {issue['title'][:44]}")

python/cpython | 75,972 stars
  (fetched page 1: 3 items)
  #156907 [3.13] gh-94242: Clarify comments for _MAX_W
  #156906 [3.14] gh-94242: Clarify comments for _MAX_W
  #156905 [3.15] gh-94242: Clarify comments for _MAX_W

Seven chapters are working together in that one block: a class from chapter 8, exceptions from chapter 9, a generator from chapter 12, a decorator from chapter 13, a context manager from chapter 14, islice from chapter 15, and a lambda from chapter 7.

Part 13: APIs in Data Science and Machine Learning

Everything so far is general. This part is about how APIs show up specifically in data science and machine learning work, which is a lot more often than beginners expect.

   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
   │ 1. COLLECT   │ ─► │ 2. ENRICH    │ ─► │ 3. MODEL     │ ─► │ 4. SERVE     │
   │ data from    │    │ add info via │    │ train it, or │    │ your model   │
   │ public APIs  │    │ other APIs   │    │ call an AI   │    │ AS an API    │
   │              │    │ (geocoding,  │    │ model's API  │    │ for others   │
   │              │    │ currency...) │    │              │    │ to call      │
   └──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘
         APIs show up at every step of a data project

13.1 Where Data Scientists Get Data from APIs

A few common kinds of API you'll meet. The specific services change over time, so always check the current documentation, prices and terms before relying on one:

Many of these come with an official SDK: a Python package from the provider that wraps the HTTP calls in friendly functions (kaggle, huggingface_hub, cloud SDKs like boto3 for Amazon Web Services). An SDK is still calling the API underneath, so everything in this guide (keys in environment variables, rate limits, timeouts, retries) still applies. It just saves you writing the HTTP part.

13.2 From JSON to a pandas DataFrame

API data arrives as JSON, but analysis happens in pandas. The bridge between them is short, and you'll write it in almost every data project.

Nested JSON (dictionaries inside dictionaries) is the first hurdle. pd.json_normalize flattens it, turning each nested field into its own column with a dotted name:

import pandas as pd

flat = pd.json_normalize(repo)                       # repo is the saved GitHub response

print("columns:", len(flat.columns))
print([c for c in flat.columns if c.startswith(("owner", "license"))])
print()
print(flat[["full_name", "owner.login", "license.spdx_id", "stargazers_count"]].to_string(index=False))

columns: 19
['license.key', 'license.name', 'license.spdx_id', 'owner.login', 'owner.id', 'owner.type', 'owner.html_url']

     full_name owner.login license.spdx_id  stargazers_count
python/cpython      python     NOASSERTION             75971

repo["owner"]["login"] became a column called owner.login, and the license fields became license.key, license.name and so on. One row per record, one column per field: exactly the shape pandas wants.

Paged results go straight in too. Because all_items is a generator that yields one dictionary per row, list(...) collects every row from every page, and pd.DataFrame turns the list of dictionaries into a table:

issues = pd.DataFrame(list(all_items(get_page_from_samples)))

issues["created_at"] = pd.to_datetime(issues["created_at"])     # text -> real datetimes

print(issues[["number", "user", "comments", "created_at"]].head(4).to_string(index=False))
print()
print("rows:", len(issues))
print("most active:", issues["user"].value_counts().head(2).to_dict())

  (fetched page 1: 3 items)
  (fetched page 2: 3 items)
  (fetched page 3: 3 items)
 number             user  comments                created_at
 156898   miss-islington         1 2026-09-03 15:48:50+00:00
 156897         vstinner         1 2026-09-03 14:28:20+00:00
 156896 serhiy-storchaka         1 2026-09-03 13:28:13+00:00
 156895       smontanaro         0 2026-09-03 13:26:02+00:00

rows: 9
most active: {'vstinner': 2, 'serhiy-storchaka': 2}

Two details that matter in real projects:

• Dates arrive as text. JSON has no date type, so created_at is a string like "2026-09-03T15:48:50Z". pd.to_datetime turns it into real datetimes you can sort, filter and group by (the Z at the end means UTC, which is why the result shows +00:00).
• Check the row count. 9 rows is what we expected from three pages of three. When an analysis built from an API looks odd, the first question is often "did I actually get all the pages?"

13.3 Fetching Faster: Several Requests at Once

Most of a request's time is spent waiting for the network, not computing. So when you need many independent pages, you can wait for several at once using threads from the standard library's concurrent.futures:

from concurrent.futures import ThreadPoolExecutor


def slow_page(page):
    time.sleep(0.3)                     # pretend each request takes 0.3 seconds
    return get_page_from_samples(page)


start = time.perf_counter()
one_by_one = [slow_page(p) for p in [1, 2, 3]]
print(f"one at a time: {time.perf_counter() - start:.1f}s")

start = time.perf_counter()
with ThreadPoolExecutor(max_workers=3) as pool:
    together = list(pool.map(slow_page, [1, 2, 3]))
print(f"three at once: {time.perf_counter() - start:.1f}s")

print("same results :", one_by_one == together)

one at a time: 0.9s
three at once: 0.3s
same results : True

Three requests that take 0.3 seconds each took 0.9 seconds one after another, and about 0.3 seconds together, with the same results.

Analogy: it's like ordering at three food stalls at once instead of queuing at one, waiting for your food, and only then joining the next queue.

Be careful with this power, though. Running requests in parallel uses up a rate limit three times as fast, and some APIs block clients that send bursts. Keep max_workers small, stay inside the published limits, and combine it with the retry and backoff ideas above. (For very large jobs, people also use async libraries like httpx or aiohttp, which do the same kind of waiting-in-parallel in a different style.)

13.4 Calling AI Model APIs

Large language models and other AI services are usually used through an API: you send text (or images) in a POST request, the provider runs a huge model on their computers, and you get the answer back as JSON. For data scientists, that makes it possible to classify, summarise or extract information from text without training a model yourself.

Every provider's exact format is a little different (always follow their documentation or official SDK), but the shape is very consistent. Here's a typical request, built but never sent, using the same prepare() trick from Part 4 and an imaginary provider address:

llm_request = requests.Request(
    "POST", "https://api.example-llm.com/v1/chat",
    headers={"Authorization": f"Bearer {get_api_key('DEMO_API_KEY')}"},
    json={"model": "example-model-small",
          "messages": [{"role": "user",
                        "content": "Sentiment of: 'Fast delivery, great book!' "
                                   "Answer positive, negative or neutral."}],
          "max_tokens": 5,
          "temperature": 0})

prepared = llm_request.prepare()                  # built, never sent

print(prepared.method, prepared.url)
print(json.dumps(json.loads(prepared.body), indent=2))

POST https://api.example-llm.com/v1/chat
{
  "model": "example-model-small",
  "messages": [
    {
      "role": "user",
      "content": "Sentiment of: 'Fast delivery, great book!' Answer positive, negative or neutral."
    }
  ],
  "max_tokens": 5,
  "temperature": 0
}

What each field does, in the usual vocabulary of these APIs:

• model: which model to run. Providers offer several sizes; smaller ones are cheaper and faster.
• messages: the conversation so far. Each message has a role (user for your text) and content.
• max_tokens: the most the model may write back. A token is a chunk of text, roughly three-quarters of an English word on average.
• temperature: how random the answer is. 0 means "as consistent as possible", which is what you want when labelling data.

And here is the usual shape of a reply, and how you'd read it:

reply = {                                          # the usual shape of a reply
    "id": "msg_001",
    "model": "example-model-small",
    "output": [{"type": "text", "text": "positive"}],
    "usage": {"input_tokens": 24, "output_tokens": 1},
}

label = reply["output"][0]["text"]
used = reply["usage"]

price_in, price_out = 0.50, 2.00                   # made-up prices, per million tokens
cost = (used["input_tokens"] * price_in + used["output_tokens"] * price_out) / 1_000_000

print("label        :", label)
print("tokens       :", used)
print(f"cost         : ${cost:.6f}")
print(f"for 50,000 reviews: about ${cost * 50_000:.2f}")

label        : positive
tokens       : {'input_tokens': 24, 'output_tokens': 1}
cost         : $0.000014
for 50,000 reviews: about $0.70

The usage section is important: AI APIs are usually billed per token, counting both the text you send in and the text you get back. The prices above are invented to show the arithmetic, but the habit is real: estimate the cost of a whole job before running it, by multiplying the cost of one request by the number of rows you plan to send.

Everything else in this guide applies to AI APIs with extra force:

• Keys must be in environment variables; a leaked AI key is spent by someone else very quickly.
• Timeouts need to be longer, because generating text takes time (this is where timeout=(3, 60) style values are useful).
• Rate limits are often counted in tokens per minute as well as requests per minute.
• Retries with backoff are essential, because these services are often busy.
• Caching saves real money: if you've already labelled a review, save the answer instead of paying for it again.
• Privacy: anything you send leaves your computer. Check what your data agreement allows before sending customer data to any outside API.

13.5 Serving Your Own Model as an API

So far, your code has always been the client. In machine learning, you'll often build the server too: once a model is trained, the usual way to let a website, an app or another team use it is to put it behind an API. Their code sends a POST with the input features, and your API replies with a prediction. This is a big part of what MLOps and model deployment are about.

A popular Python library for this is FastAPI. It isn't part of this course's requirements.txt, so treat this as a preview of where this chapter leads, not something to install right now. The "model" here is a stand-in formula, so the example stays small:

from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel

app = FastAPI()


class House(BaseModel):                # what a request must contain
    area_sqft: float
    bedrooms: int


@app.post("/predict")
def predict(house: House):
    price = 50_000 + 120 * house.area_sqft + 10_000 * house.bedrooms   # a stand-in model
    return {"predicted_price": round(price, 2)}


client = TestClient(app)               # calls the API inside Python, no server needed

good = client.post("/predict", json={"area_sqft": 1200, "bedrooms": 3})
bad = client.post("/predict", json={"area_sqft": "big", "bedrooms": 3})

print(good.status_code, good.json())
print(bad.status_code, bad.json()["detail"][0]["msg"])

200 {'predicted_price': 224000.0}
422 Input should be a valid number, unable to parse string as a number

What's happening:

• app = FastAPI() creates the API.
• class House(BaseModel) describes what a valid request must contain: an area_sqft number and a whole-number bedrooms. (BaseModel comes from pydantic, a library for checking data.)
• @app.post("/predict") turns the function below it into an endpoint: a POST to /predict runs it. (A decorator from chapter 13, doing real work again.)
• TestClient lets us call the API inside Python, without starting a real server, which is also how APIs are tested.

Look at the two responses. The good request got 200 and a prediction. The bad one ("big" instead of a number) got 422, a 4xx code meaning "I understood you, but the data is invalid", with a message explaining exactly what was wrong. FastAPI checked the input automatically, before our model code ever ran. Everything you learned about status codes as a client is exactly what you'd send back as a server.

In a real deployment, you'd load a trained model (for example a scikit-learn model saved to a file) once when the API starts, add a simple GET /health endpoint so monitoring tools can check it's alive, version the endpoint (/v1/predict), and run it with a server program such as uvicorn.

13.6 Testing Code That Calls APIs

Tests that make real network requests are slow, use up your rate limit, and fail when the internet is down or the data changes. So in tests, people replace the real requests.get with a fake one that returns a known answer. Python's standard library has unittest.mock for exactly this:

from unittest.mock import Mock, patch


def fetch_stars(full_name):
    response = requests.get(f"https://api.github.com/repos/{full_name}", timeout=10)
    response.raise_for_status()
    return response.json()["stargazers_count"]


fake_response = Mock()
fake_response.json.return_value = {"stargazers_count": 123}

with patch("requests.get", return_value=fake_response) as fake_get:
    print("stars      :", fetch_stars("someone/some-repo"))
    print("called with:", fake_get.call_args)

stars      : 123
called with: call('https://api.github.com/repos/someone/some-repo', timeout=10)

• Mock() creates a fake object that accepts any method call. We tell its .json() method what to return.
• patch("requests.get", ...) temporarily replaces the real requests.get with our fake, only inside the with block.
• fetch_stars ran exactly as normal, but no network request happened. It got 123 stars from the fake.
• fake_get.call_args shows exactly how the function called requests.get, so a test can check that the URL is right and that a timeout was passed.

Analogy: it's like a flight simulator. Pilots practise emergencies without risking a real plane; your tests practise "the API returned X" without touching the real API. Chapter 19 covers testing properly.

13.7 Things That Go Wrong in Real Data Pipelines

When API data feeds a dashboard or a model that runs every day, a few problems show up again and again:

• The response changes shape. A field gets renamed, a number starts arriving as text, or a new nested level appears. Check that the fields you need exist before using them, and fail loudly if they don't.
• The API version changes. Many APIs let you pin a version (in the URL like /v2/, or in a header). Pin it, so an update on their side doesn't silently change your data.
• Missing pages. A request in the middle of a long paging job fails, and the job carries on with a hole in the data. Retrying, logging every page, and checking row counts all help.
• Silent duplicates. Data that changes while you're paging can cause rows to repeat or be skipped. Cursor-based paging and de-duplicating on an ID (df.drop_duplicates("id")) help.
• Time zones. Timestamps from APIs are usually UTC. Convert them deliberately, as chapter 15 showed, instead of assuming local time.
• Data drift. The world changes, so data fetched today may look different from the data a model was trained on. Saving dated snapshots makes this visible.
• Legal and ethical limits. API terms often restrict storing, sharing or reselling data, and data about people may be covered by privacy laws. Read the terms before building a dataset.

Part 14: Being a Good Client

An API runs on someone else's server, and answering you costs them money.

• Read the documentation first. Rate limits, required headers and the shape of the response are all written down. Guessing takes longer than reading.
• Say who you are with a real User-Agent. If something goes wrong, that's how they can contact you instead of blocking you.
• Cache as much as you can. The fastest and politest request is the one you don't make.
• Back off when asked. Respect Retry-After, and treat a 429 as a message, not an obstacle.
• Check the terms before you share the data. Being able to fetch something isn't the same as being allowed to republish it.
• Use the official API rather than scraping the website. It's faster, more stable, and allowed. Chapter 18 covers what to do when there's no API, and starts by telling you to look for one anyway.
• Keep HTTPS checks on. You may see advice online to pass verify=False to "fix" a certificate error. That switches off the check that you're talking to the real server, so fix the underlying problem instead.

Part 15: Common Mistakes

Most of these don't crash with a helpful message. They hang, or they fail somewhere else, or they cost money quietly. Building the habits into every request from the start is far easier than tracking these down later.

Quick Reference Summary

A request, written properly

import requests

try:
    response = requests.get(url,
                            params={"page": 1},
                            headers={"User-Agent": "my-app/1.0"},
                            timeout=10)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.RequestException as e:
    print("request failed:", e)

The response object

Status codes

Exceptions (all in requests.exceptions)

Keys

export WEATHER_API_KEY="..."

key = os.environ.get("WEATHER_API_KEY")
if not key:
    raise RuntimeError("WEATHER_API_KEY is not set")
headers = {"Authorization": f"Bearer {key}"}

Sessions

session = requests.Session()
session.headers.update({"User-Agent": "my-app/1.0"})
session.get(url, timeout=10)      # reuses the connection
session.close()

Pagination

def all_items(get_page):
    page = 1
    while True:
        items = get_page(page)
        if not items:
            return
        yield from items
        page += 1

The four lines never to leave out

timeout=10                        # or it hangs forever
response.raise_for_status()       # or a 404 becomes a KeyError
os.environ.get("API_KEY")         # or you publish your key
time.sleep(...)                   # between calls, and when told to

For data work