Web Scraping: Reading Data from Pages Built for People

Scraping means reading a page that was written for humans and pulling the data out of it. It's really useful, and it's also the most fragile thing in this course. You're relying on the layout of someone else's HTML, and they can change it any day without telling you.

Analogy: an API is like a shop giving you a neat printed price list. Scraping is like walking around the shop copying prices off the shelf labels with a notepad. You can get the same information, but you have to find each label yourself, and if the shop rearranges its shelves overnight, your route no longer works.

That's why this guide starts with two questions to answer before writing any code, and ends with the question of whether you should be doing it at all.

Here is what we will cover:

1. Whether to scrape at all, and whether you're allowed to
2. How a web page is put together
3. Why chapter 16's regex can't do this job, shown on a real page
4. BeautifulSoup: find, find_all, attributes and text
5. CSS selectors, which are usually the shorter way
6. Pulling out a table
7. Writing a scraper that survives missing pieces
8. Following pagination with a chapter 12 generator
9. Saving the result to CSV, from chapter 10
10. One real request, and how to make it politely
11. Scraping responsibly

About the pages in this guide. Almost everything here reads HTML files saved in a sample_data/ folder, so the output is the same every time and works offline. Three examples near the end share one real request, to a website built specifically for people to practise on. They're marked.

Part 1: Before You Write Any Code

Question 1: Is There an API?

This is the whole point of chapter 17. An API gives you organised data, a clear agreement about what you'll get, and permission to use it. Scraping gives you none of those.

Look for an API before you write a scraper. Some ways to find one:

• Try example.com/api, search for "site name API", and read the site's developer page.
• Open your browser's developer tools (usually the F12 key, or right-click and choose Inspect) and watch the Network tab while you use the site. A page that loads more data as you scroll is usually calling a JSON API behind the scenes, and you could often call that directly with chapter 17's tools.

Analogy: before copying a phone book by hand, check whether the phone company already sells it as a spreadsheet.

Question 2: Are You Allowed To?

That's really three separate questions, and they can have different answers.

robots.txt

robots.txt is a small text file at the root of a website (for example https://example.shop/robots.txt) that says which paths automated programs should stay away from. It's a request, not a lock: nothing physically stops a program from ignoring it. But ignoring it makes it very clear that you knew the rules and didn't care.

Analogy: robots.txt is like a "Staff only" sign on a door. The door isn't locked, and you could walk through. But if you're caught on the other side, "I saw the sign and ignored it" is a bad position to be in.

Python can read it with a standard library module:

from urllib.robotparser import RobotFileParser

ROBOTS = """
User-agent: *
Crawl-delay: 2
Disallow: /cart/
Disallow: /checkout/
Disallow: /search?
Allow: /

User-agent: GreedyBot
Disallow: /
"""

rules = RobotFileParser()
rules.parse(ROBOTS.splitlines())            # parse text we already have

for path in ["/books/", "/books/page-2.html", "/cart/", "/checkout/pay"]:
    allowed = rules.can_fetch("*", "https://example.shop" + path)
    print(f"  {path:<22} {'allowed' if allowed else 'DISALLOWED'}")

  /books/                allowed
  /books/page-2.html     allowed
  /cart/                 DISALLOWED
  /checkout/pay          DISALLOWED

Let's read that file line by line:

• User-agent: * means "the rules below apply to every program". (* means "everyone".)
• Crawl-delay: 2 asks for a pause of 2 seconds between requests.
• Disallow: /cart/ and Disallow: /checkout/ say "please don't visit anything under these paths". Shopping baskets and payments are no place for a scraper.
• Disallow: /search? asks programs not to run the site's search, which is expensive for the site.
• Allow: / says "everything else is fine".
• The second block, User-agent: GreedyBot with Disallow: /, applies only to a program calling itself GreedyBot, and bans it from the whole site.

In the code, rules.parse(ROBOTS.splitlines()) reads rules from text we already have, and rules.can_fetch("*", url) answers the question "may a general program (*) fetch this URL?".

That file tells you two more things:

print("crawl delay asked of everyone:", rules.crawl_delay("*"), "seconds")
print("GreedyBot may fetch the home page:", rules.can_fetch("GreedyBot", "https://example.shop/"))

crawl delay asked of everyone: 2 seconds
GreedyBot may fetch the home page: False

Crawl-delay: 2 is the site asking for no more than one request every two seconds. GreedyBot has been named and banned completely, which is what happens to a scraper that behaves badly.

In a real script you'd fetch the live file instead of pasting it in:

rules = RobotFileParser()
rules.set_url("https://example.shop/robots.txt")
rules.read()

set_url tells the parser where the file lives, and read() downloads and parses it.

If a site has no robots.txt, the request gets a 404 and the parser treats everything as allowed. That only means no rules were written down. It doesn't mean anything goes. (One detail worth knowing: if the server answers 401 or 403, meaning the file exists but you're not allowed to read it, Python's parser plays safe and treats everything as disallowed. It does the same if the server itself is failing with a 5xx error, since it can't tell what the rules would have said.)

One more detail, about the order of rules. When an Allow and a Disallow line both match a path, the robots.txt standard says the most specific (longest) rule wins, and recent Python releases follow that. Older Python releases instead use the first matching line, reading from the top. Our example lists the Disallow lines before Allow: /, so both give the same answers. If you write test rules of your own, put the specific lines first and the broad Allow: / last, and every version will agree.

The Terms of Service

The terms of service (sometimes called "terms of use") are the legally important document, and they're separate from robots.txt. Many sites forbid automated collection outright. Read them. They're usually linked at the bottom of every page.

Personal Data

Personal data is a separate question again. Names, emails, photos and profiles are protected by laws like the EU's GDPR, and similar laws in many other countries (India's Digital Personal Data Protection Act, for example), even when the data is publicly visible. "It was on a public page" doesn't make it fine to collect, and this is the part most likely to turn a scraping project into a legal problem.

   ┌────────────────────┐      ┌────────────────────┐      ┌────────────────────┐
   │ robots.txt         │      │ terms of service   │      │ personal data?     │
   │ "may programs go   │  +   │ "what does the     │  +   │ "would collecting  │
   │  there?"           │      │  contract allow?"  │      │  this harm people?"│
   └────────────────────┘      └────────────────────┘      └────────────────────┘
            all three need a good answer before you write the scraper

The rest of this guide assumes you've checked all three and the answer was yes.

Part 2: How a Web Page Is Put Together

To scrape a page, you need to know what a page is made of. Every web page is written in HTML (HyperText Markup Language): plain text with tags that mark what each piece is.

HTML is a tree of elements. Each element has:

• a tag name (like p for a paragraph or a for a link),
• optional attributes (extra information, like where a link goes),
• and content, which can be text or more elements.

Here's one product from a shop page:

<article class="product" data-sku="9781617294433">
  <h3><a href="/book/9781617294433.html">Deep Learning with Python</a></h3>
  <p class="price">£51.77</p>
</article>

• article, h3, a and p are tags. Most come in pairs: <p> opens a paragraph and </p> closes it.
• class, data-sku and href are attributes, written inside the opening tag as name="value". You'll target class and id most often, because the site's own styling uses them.
• The <a> is inside the <h3>, which is inside the <article>. That nesting is the tree you'll move around in.

Analogy: HTML is like a set of labelled boxes inside boxes. The <article> is a big box labelled "product". Inside it are smaller boxes: a heading box (with a link box inside that), and a price box. Scraping is opening the right boxes and reading the labels and contents.

   article  (class="product", data-sku="9781617294433")
   ├── h3
   │   └── a  (href="/book/9781617294433.html")
   │       └── "Deep Learning with Python"
   └── p  (class="price")
       └── "£51.77"

Common Tags You'll Meet

Two Attributes Worth Knowing Well

• class groups similar elements so the site can style them together. Every product card has class="product", every price has class="price". That's exactly what makes it useful for scraping: it says what an element is.
• id names one unique element on the page, like id="catalogue".
• Attributes starting with data- (like data-sku) are custom values the site's developers added for their own code. They're often the cleanest data on the page.

Our Sample Pages

sample_data/ has three pages from a small bookshop, written for this chapter. Here's the start of the first one:

from pathlib import Path

page1 = Path("sample_data/shop_page_1.html").read_text(encoding="utf-8")

print(page1[:640])

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>The Bookshop - page 1</title>
</head>
<body>
  <header>
    <h1>The Bookshop</h1>
    <p class="tagline">Books for people who write code</p>
  </header>

  <!-- <article class="product" data-sku="0000000000000">
         <h3><a href="/book/retired.html">Retired Title</a></h3>
         <p class="price">&pound;99.99</p>
       </article> -->

  <main id="catalogue">
    <article class="product" data-sku="9781617294433">
      <h3><a href="/book/9781617294433.html">Deep Learning with Python</a></h3>
      <p class="author">François Chollet</p>
      <p class="pri

Path(...).read_text(...) (from the pathlib module) reads the whole file as one string, and page1[:640] prints the first 640 characters.

Notice the block between <!-- and -->. That's an HTML comment. A browser doesn't show it, and anything reading the page properly should skip it too. Here, someone left an old, retired product commented out. Keep it in mind: it's about to cause trouble.

Also notice £ inside the comment. That's an HTML entity, a way of writing special characters using plain letters. £ means £, and © means ©. A proper HTML reader turns these back into the real characters for you.

Part 3: Why Chapter 16's Regex Can't Do This

Chapter 16's section on "when a regex is the wrong tool" showed one failing on a tiny three-tag example. Here's the same problem on a realistic page. The pattern looks exactly right:

import re

print(re.findall(r'<article class="product" data-sku="(\d+)">', page1))

['0000000000000', '9781617294433', '9781492056355']

The pattern says: find <article class="product" data-sku=", then capture one or more digits, then ">. Three results, and the page has three products, so it looks like it worked.

It didn't. Compare it with what an HTML parser finds:

from bs4 import BeautifulSoup

soup = BeautifulSoup(page1, "html.parser")

print([article["data-sku"] for article in soup.find_all("article", class_="product")])

['9781617294433', '9781593279288', '9781492056355']

Two different lists of the same length, which is the worst kind of wrong, because nothing looks off.

• 0000000000000 is the retired product inside the HTML comment. It isn't on the page. The regex only sees text, and has no idea what a comment is.
• 9781593279288 is a real product the regex missed, because its data-sku is on the next line (the HTML has a line break between class="product" and data-sku) and the pattern expected a single space.

Both mistakes are silent, and a report built on that list would be wrong in two ways at once.

Analogy: using a regex on HTML is like counting the people at a party by counting coats on the coat rack. Usually it's close. But someone left an old coat behind last week (the comment), and someone else hung theirs on a different hook (the line break). The numbers can even come out the same while being wrong about who's there.

A parser reads the page the way a browser does: comments are comments, line breaks inside a tag are just spacing, and you get a tree you can ask questions about.

Meet BeautifulSoup

BeautifulSoup (installed as the package beautifulsoup4, imported as bs4) is the most popular HTML parser library for Python. It's in this course's requirements.txt from chapter 17, so it's already installed if you followed that chapter's setup.

BeautifulSoup(page1, "html.parser") reads the page into a soup object: the whole tree, ready to search. We'll call it soup from now on.

"html.parser" is the parser built into Python, and it's what this guide uses. "lxml" is faster but needs installing. For a page you're reading once, the difference doesn't matter.

Part 4: Finding Things

Two methods do most of the work:

• find(...) returns the first match, or None.
• find_all(...) returns a list of every match, which might be empty.

You give them a tag name, and optionally a class or other attributes to narrow it down.

first = soup.find("article", class_="product")

print(type(first).__name__)
print(first.h3.get_text(strip=True))

Tag
Deep Learning with Python

• soup.find("article", class_="product") finds the first <article> whose class includes product.
• What comes back is a Tag object: one element of the tree, which you can search inside in exactly the same way.
• first.h3.get_text(strip=True) reads the text of the heading inside it.

The argument is class_, with an underscore. class is a Python keyword (chapter 8, where it starts a class definition), so it can't be used as a parameter name. BeautifulSoup adds the underscore to get around that.

Analogy: find is asking a librarian for "a book about Python", and getting the first one they spot. find_all is asking for "every book about Python", and getting a pile (possibly an empty one).

Reading a Tag: Text, Attributes, and the Tags Inside It

A tag gives you three things to read: its text, its attributes, and the tags inside it.

print("text of the whole card:")
print(" ", first.get_text(" ", strip=True))
print()
print("just the title  :", first.h3.a.get_text(strip=True))
print("the link        :", first.h3.a["href"])
print("the sku         :", first["data-sku"])
print("the price       :", first.find("p", class_="price").text)

text of the whole card:
  Deep Learning with Python François Chollet £51.77 Rating In stock (22 available)

just the title  : Deep Learning with Python
the link        : /book/9781617294433.html
the sku         : 9781617294433
the price       : £51.77

• first.get_text(" ", strip=True) gets all the text inside the card, with a space between each piece and extra whitespace tidied away.
• first.h3.a.get_text(strip=True) goes into the heading, then into the link, and reads only the title.
• first.h3.a["href"] reads the link's href attribute, with square brackets like a dictionary.
• first["data-sku"] reads an attribute of the article itself.
• first.find("p", class_="price").text searches inside the card for the price paragraph.

first.h3.a is a shortcut. Writing .h3 after a tag gives you the first <h3> inside it, and .a then gives you the first <a> inside that. It's handy, but fragile. If a tag is missing you get None, and the next step in the chain raises AttributeError, which Part 7 deals with. Use it for exploring, not in code you rely on.

Moving Around the Tree

Besides looking inside a tag, you can also step outwards and sideways from it. This is handy when the thing you can find easily sits right next to the thing you actually want:

price_tag = first.select_one("p.price")

print("the tag itself  :", price_tag)
print("its parent      :", price_tag.parent.name, price_tag.parent["data-sku"])
print("p just before it:", price_tag.find_previous_sibling("p").get_text(strip=True))
print("p just after it :", price_tag.find_next_sibling("p")["class"])

the tag itself  : <p class="price">£51.77</p>
its parent      : article 9781617294433
p just before it: François Chollet
p just after it : ['rating', 'star-four']

• .parent steps out one level: the price's parent is the <article>.
• .find_previous_sibling("p") and .find_next_sibling("p") look sideways, at the <p> elements just before and after it inside the same parent.

Analogy: if the tree is a family tree, .parent is the parent, and siblings are brothers and sisters: elements with the same parent, sitting side by side.

Three Ways to Get Text, and the One That Surprises People

title_tag = first.h3

print("h3 .text  :", repr(title_tag.text))
print("h3 .string:", repr(title_tag.string))
print()
print("article .text  :", repr(first.text[:40]), "...")
print("article .string:", repr(first.string))

h3 .text  : 'Deep Learning with Python'
h3 .string: 'Deep Learning with Python'

article .text  : '\nDeep Learning with Python\nFrançois Chol' ...
article .string: None

repr(...) shows strings with their quotes and with \n for line breaks, so we can see exactly what came back.

• On the <h3>, .text and .string agree.
• On the <article>, .text includes all the line breaks from the HTML ('\nDeep Learning with Python\nFrançois Chol...'), which is why .get_text(strip=True) is usually nicer.
• .string on the <article> is None.

.string worked on the <h3> because it holds exactly one thing, the <a>, which in turn holds exactly one thing, the text. .string follows that chain down. On the <article> it's None, because there are several children and no single string to return.

That's not an error, so a script would carry on with the None, and the problem would show up somewhere else later. Just use .get_text() and forget .string exists.

Attributes Are a Dictionary, Except class

print("the article's attributes:", first.attrs)
print()

rating = first.find("p", class_="rating")
print("class      :", rating["class"], "<- a list")
print("data-sku   :", first["data-sku"], "<- a string")
print("one class  :", rating["class"][1])

the article's attributes: {'class': ['product'], 'data-sku': '9781617294433'}

class      : ['rating', 'star-four'] <- a list
data-sku   : 9781617294433 <- a string
one class  : star-four

.attrs gives you every attribute of a tag as a dictionary. Look closely at the values:

• data-sku is a plain string.
• class is a list: ['rating', 'star-four'].

class comes back as a list because an element can have several classes at once. Here, rating says "this is a rating" and star-four says which one. A few other attributes that can hold several space-separated values, such as rel on links, come back as lists too:

button = BeautifulSoup('<a href="/basket" rel="nofollow noopener" class="btn primary">Buy</a>',
                       "html.parser").a

print(button.attrs)

{'href': '/basket', 'rel': ['nofollow', 'noopener'], 'class': ['btn', 'primary']}

Everything else (href, id, data-... and most others) is a plain string.

So comparing class to a string never matches. Check whether the class is in the list instead:

print('rating["class"] == "rating":', rating["class"] == "rating")
print('"rating" in rating["class"]:', "rating" in rating["class"])

rating["class"] == "rating": False
"rating" in rating["class"]: True

Analogy: asking rating["class"] == "rating" is like asking "is this person's full name exactly 'Rating'?" The answer is no, because their full name is "Rating Star-Four". Asking "rating" in rating["class"] is like asking "is one of their names 'Rating'?", which is yes.

(find(..., class_="rating") and CSS selectors already do the "is it in the list" check for you, which is why they worked earlier without any fuss.)

Finding Many, and Narrowing Down

Put together, the usual pattern is: find_all the repeating cards, then search inside each card for its fields.

for article in soup.find_all("article", class_="product"):
    title = article.h3.get_text(strip=True)
    price = article.find("p", class_="price").text
    stock = article.find("p", class_="stock").get_text(strip=True)
    print(f"  {title:<28} {price:>8}   {stock}")

  Deep Learning with Python      £51.77   In stock (22 available)
  Python Crash Course            £23.99   In stock (8 available)
  Fluent Python                  £44.50   Out of stock

Searching inside each article (instead of the whole soup) is what keeps each title paired with its own price and stock. If you instead collected all the titles from the whole page in one list and all the prices in another, a single missing price would shift every price after it onto the wrong book.

The f-string formatting lines the results up: :<28 pads the title to 28 characters on the left, and :>8 right-aligns the price in 8 characters.

Part 5: CSS Selectors

select() and select_one() take a CSS selector: the same pattern language the site's own stylesheet uses to pick elements. (CSS, Cascading Style Sheets, is the language that controls how a page looks: colours, fonts, spacing.) For anything nested, selectors are much shorter than chaining find calls.

• select(...) returns a list of every match, like find_all.
• select_one(...) returns the first match or None, like find.

Two symbols carry most of the meaning: a dot (.) means "class", and a hash (#) means "id". A space between two parts means "somewhere inside", and > means "directly inside".

Analogy: a CSS selector is like a postal address read from big to small. #catalogue h3 a means "in the catalogue, find a heading, and inside it find a link", just as "India, Mumbai, Marine Drive, building 12" narrows down step by step.

print("prices        :", [p.text for p in soup.select("article.product p.price")])
print("in stock      :", len(soup.select("p.stock.in-stock")), "of",
                          len(soup.select("article.product")))
print("first title   :", soup.select_one("#catalogue h3 a").get_text(strip=True))
print("the next link :", soup.select_one("nav.pagination a.next")["href"])

prices        : ['£51.77', '£23.99', '£44.50']
in stock      : 2 of 3
first title   : Deep Learning with Python
the next link : shop_page_2.html

What each line found:

• "article.product p.price": every price paragraph inside a product card, so all three prices.
• "p.stock.in-stock": stock paragraphs that have both the stock and in-stock classes. Two of the three books are in stock. (Fluent Python has out-of-stock instead.)
• "#catalogue h3 a": inside the element with id="catalogue", a link inside a heading. select_one returns just the first.
• "nav.pagination a.next": the "Next" link inside the pagination bar, which we'll follow in Part 8.

Copying a Selector from Your Browser

A tip that saves a lot of guessing: in your browser, right-click the thing you want and choose Inspect. The developer tools open with that element highlighted in the HTML. Then right-click the highlighted element and choose Copy → Copy selector.

Trim it down to the part that says what the element is, not exactly where it sits. The copied version is usually something like body > main > article:nth-child(2) > p:nth-child(3), which is so specific that any change to the page breaks it. article.product p.price says the same thing in a way that survives a redesign.

One caution: the Inspect panel shows the page as the browser has built it, which isn't always the HTML your script downloads. JavaScript may have added elements, and browsers quietly insert a <tbody> into any table that doesn't have one. A copied selector like table > tbody > tr can then find nothing in the downloaded HTML. View page source (Part 10) shows what the server actually sent, and that's what BeautifulSoup sees.

Part 6: Pulling Out a Table

Tables are the easiest thing to scrape, because their structure already matches what you want:

• <tr> is a row ("table row"),
• <th> is a heading cell ("table header"),
• <td> is a data cell ("table data").

Headings usually sit inside a <thead> section, and the rows of data inside a <tbody>.

   <table class="specs">
     <thead> <tr> <th>Method</th>   <th>Days</th> <th>Cost</th>  </tr> </thead>   ← headers
     <tbody> <tr> <td>Standard</td> <td>5</td>    <td>£2.99</td> </tr>            ← row 1
             <tr> <td>Express</td>  <td>2</td>    <td>£6.49</td> </tr>            ← row 2
             ...                                                     </tbody>
   </table>

table = soup.find("table", class_="specs")

headers = [th.get_text(strip=True) for th in table.select("thead th")]
rows = [[td.get_text(strip=True) for td in tr.find_all("td")]
        for tr in table.select("tbody tr")]

print(headers)
for row in rows:
    print(" ", row)

['Method', 'Days', 'Cost']
  ['Standard', '5', '£2.99']
  ['Express', '2', '£6.49']
  ['Next day', '1', '£9.99']

• headers is a list comprehension: for every <th> inside <thead>, take its text.
• rows is a nested list comprehension: for every <tr> in the body, build a list of the text of each of its <td> cells.

That's a list of lists, already the shape chapter 10's csv.writer wants. Notice the £ entities in the HTML came out as real £ signs.

The One-Line Version with pandas

When you move on to pandas (see "Where to go next" in the course README), pandas.read_html does this whole job in one line. It finds <table> elements in the HTML and returns a list of DataFrames, one per table:

import io
import pandas as pd

shipping = pd.read_html(io.StringIO(page1), attrs={"class": "specs"})[0]

print(shipping)
print()
print(shipping.dtypes)

     Method  Days   Cost
0  Standard     5  £2.99
1   Express     2  £6.49
2  Next day     1  £9.99

Method      str
Days      int64
Cost        str
dtype: object

• io.StringIO(page1) wraps our string so pandas reads it like a file. (Recent pandas versions want this instead of a raw string.)
• attrs={"class": "specs"} picks the table with that class, and [0] takes the first (and only) match.
• Look at dtypes: pandas even turned the Days column into numbers (int64) by itself. The Cost column stays text because of the £ sign, so it would still need chapter 16's cleaning. (Older pandas versions print object instead of str for text columns; it means the same thing here.)

read_html needs an extra parsing library installed alongside pandas, usually lxml, so it's a tool for later rather than something this chapter's requirements.txt covers.

Part 7: Writing a Scraper That Doesn't Fall Over

Every example so far has called .text straight on the result of a find, with no check. That worked on page 1, because page 1 has every field. Real pages often don't.

A few lines of code are the difference between a scraper that reports "8 of 9 rows had a price" and one that crashes halfway through the night. Page 2 has a product with no price, just like real catalogues sometimes do:

page2 = Path("sample_data/shop_page_2.html").read_text(encoding="utf-8")
soup2 = BeautifulSoup(page2, "html.parser")

article = soup2.select("article.product")[1]
print("found the price tag:", article.find("p", class_="price"))

found the price tag: None

soup2.select("article.product")[1] takes the second product on page 2 (counting from 0), and find looked for its price. It returned None, which is fine until the next line:

try:
    print(article.find("p", class_="price").text)
except AttributeError as e:
    print("AttributeError:", e)

AttributeError: 'NoneType' object has no attribute 'text'

'NoneType' object has no attribute 'text' is an error you'll see a lot when scraping. It means "you asked for .text on something that wasn't found". It's the same problem as chapter 16's re.search returning None.

Analogy: it's like asking someone to read the label on the third jar in the cupboard, when there are only two jars. They can't read the label of nothing.

Missing Attributes

Attributes fail differently. A missing one raises KeyError with [...], and gives None with .get(), just like a dictionary in chapter 3:

link = article.find("a")

try:
    link["data-discount"]
except KeyError as e:
    print("KeyError:", e)

print('with .get():', link.get("data-discount"))
print('with a default:', link.get("data-discount", "none"))

KeyError: 'data-discount'
with .get(): None
with a default: none

So the rule is the same as for dictionaries: use tag["name"] when the attribute must be there (and an error would be right if it isn't), and tag.get("name") or tag.get("name", default) when it might be missing.

One Helper for Every Field

So a parsing function that will hold up on a real site looks like this: one helper, and every field goes through it. x if found else default gives x when found is truthy and default otherwise:

def text_of(parent, selector, default=None):
    # The first match for `selector` as tidy text, or `default` if it is not there.
    found = parent.select_one(selector)
    return found.get_text(strip=True) if found else default


def parse_product(article):
    return {
        "sku": article.get("data-sku"),
        "title": text_of(article, "h3 a"),
        "author": text_of(article, "p.author"),
        "price": text_of(article, "p.price", default=""),
        "stock": text_of(article, "p.stock", default="unknown"),
    }


for article in soup2.select("article.product"):
    print(parse_product(article))

{'sku': '9781593279929', 'title': 'Automate the Boring Stuff', 'author': 'Al Sweigart', 'price': '£27.95', 'stock': 'In stock (13 available)'}
{'sku': '9781098139482', 'title': 'Effective Pandas', 'author': 'Matt Harrison', 'price': '', 'stock': 'In stock (3 available)'}
{'sku': '9781492041139', 'title': 'Data Science from Scratch', 'author': 'Joel Grus', 'price': '£41.05', 'stock': 'In stock (11 available)'}

How it works:

• text_of(parent, selector, default) looks for the selector inside parent. If it finds something, it returns its tidy text. If not, it returns the default instead of crashing.
• parse_product(article) builds a dictionary for one product, sending every field through text_of (and using .get for the attribute).
• A missing price becomes "" (an empty string), and a missing stock line would become "unknown", so the gap is visible in the output.

No exception, and the gap shows up in the output instead of being hidden. That second product, Effective Pandas, has 'price': ''.

Analogy: text_of is like a careful assistant filling in a form from a messy document. When a field is missing, they write "not found" in the box instead of throwing the whole form away.

Turning Prices into Numbers

A price like "£27.95" is still text. Turning it into a number is chapter 16's job:

def to_price(text):
    digits = re.sub(r"[^\d.]", "", text or "")      # chapter 16
    return float(digits) if digits else None


for article in soup2.select("article.product"):
    product = parse_product(article)
    print(f"  {product['title']:<28} {to_price(product['price'])}")

  Automate the Boring Stuff    27.95
  Effective Pandas             None
  Data Science from Scratch    41.05

• re.sub(r"[^\d.]", "", text or "") deletes everything that isn't a digit or a dot, leaving "27.95".
• text or "" means "use text, or an empty string if text is None or empty", so the function never crashes on a missing price.
• If nothing is left, it returns None instead of a number, which is the honest answer for a missing price.

(Remember chapter 16's warning about this pattern: it's fine for prices like £27.95, but a value like Rs. 500 would wrongly become 0.5. Always check cleaned values on your real data.)

Part 8: Following the Pages

In chapter 17 we paginated by asking for page 1, 2, 3 until one came back empty. A website usually tells you instead: there's a "next" link, until there isn't.

   shop_page_1.html ──"Next"──► shop_page_2.html ──"Next"──► shop_page_3.html ──(no Next link)──► stop

That's a generator again, and this time it stops when there's no next link:

def crawl(start_page, load):
    # Yield the soup for each page, following the 'next' link until it runs out.
    page = start_page
    while page:
        html = load(page)
        soup = BeautifulSoup(html, "html.parser")
        print(f"  (parsed {page})")
        yield soup

        next_link = soup.select_one("nav.pagination a.next")
        page = next_link["href"] if next_link else None

How crawl works:

1. Start with page = start_page.
2. While there is a page: load its HTML with the load function, parse it, and yield the soup.
3. Look for the "Next" link with select_one("nav.pagination a.next").
4. If there is one, its href becomes the next page. If not, page becomes None, and while page: ends the loop.

Like chapter 17's all_items, it takes a load function from the caller, so the same crawler works on saved files today and live pages tomorrow. (Scraping lots of pages by following links is often called crawling, and the program doing it a crawler or spider.)

Here's a load function that reads our saved pages, and a loop that collects every product across the catalogue:

def load_from_samples(name):
    return Path(f"sample_data/{name}").read_text(encoding="utf-8")


products = []
for page_soup in crawl("shop_page_1.html", load_from_samples):
    for article in page_soup.select("article.product"):
        products.append(parse_product(article))

print(f"\n{len(products)} products across the whole catalogue")
for product in products[:4]:
    print(f"  {product['title']:<40} {product['price']}")

  (parsed shop_page_1.html)
  (parsed shop_page_2.html)
  (parsed shop_page_3.html)

9 products across the whole catalogue
  Deep Learning with Python                £51.77
  Python Crash Course                      £23.99
  Fluent Python                            £44.50
  Automate the Boring Stuff                £27.95

Three pages, nine products, and the crawl stopped on its own, because page 3 has no next link.

On a real site, the href is often a relative link like /catalogue/page-2.html instead of a full address. urllib.parse.urljoin(current_url, href) turns it into the full URL you can request.

Stopping Early

Because crawl is a generator, chapter 15's islice still works, and pages you don't reach are never loaded:

from itertools import islice

for page_soup in islice(crawl("shop_page_1.html", load_from_samples), 2):
    print("   ", page_soup.title.get_text(strip=True))

  (parsed shop_page_1.html)
    The Bookshop - page 1
  (parsed shop_page_2.html)
    The Bookshop - page 2

We asked for two pages, so page 3 was never loaded or parsed. page_soup.title is the shortcut for the <title> tag, the text in the browser tab. On a real site, stopping early like this saves requests, time, and the site's resources, which is especially handy while you're still developing and only need a page or two to test against.

Part 9: Saving the Result

The point of scraping is the file you end up with. Chapter 10's csv.DictWriter takes a list of dictionaries directly, which is exactly what parse_product produced:

import csv
import os

os.makedirs("scraper_demo", exist_ok=True)
out_path = "scraper_demo/books.csv"

fields = ["sku", "title", "author", "price", "stock"]

with open(out_path, "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fields)
    writer.writeheader()
    writer.writerows(products)

print(f"wrote {len(products)} rows to {out_path}\n")

lines = Path(out_path).read_text(encoding="utf-8").splitlines()
print("\n".join(lines[:5]))                 # the header and the first four rows

wrote 9 rows to scraper_demo/books.csv

sku,title,author,price,stock
9781617294433,Deep Learning with Python,François Chollet,£51.77,In stock (22 available)
9781593279288,Python Crash Course,Eric Matthes,£23.99,In stock (8 available)
9781492056355,Fluent Python,Luciano Ramalho,£44.50,Out of stock
9781593279929,Automate the Boring Stuff,Al Sweigart,£27.95,In stock (13 available)

• fieldnames=fields sets the column order, matching the dictionary keys.
• writeheader() writes the header row, and writerows(products) writes every product.
• newline="" stops extra blank lines appearing on Windows, and encoding="utf-8" makes sure characters like ç and £ are saved correctly.

Then the cell reads the file back and prints the first five lines, which is always worth doing: look at what you saved.

A Quick Check on What You Collected

Chapter 15's Counter can now tell you something about what you collected. It's a natural next step, and a good check that the scrape worked:

from collections import Counter

print(Counter(p["stock"].split(" (")[0] for p in products).most_common())

priced = [p for p in products if to_price(p["price"]) is not None]
print(f"{len(priced)} of {len(products)} products had a price")
print("cheapest:", min(priced, key=lambda p: to_price(p["price"]))["title"])

[('In stock', 7), ('Out of stock', 2)]
8 of 9 products had a price
cheapest: Python Crash Course

• p["stock"].split(" (")[0] turns "In stock (22 available)" into just "In stock", so the Counter groups them.
• priced keeps only the products whose price could be turned into a number. 8 of 9 had one, which matches the one missing price we saw on page 2.
• min(priced, key=...) finds the cheapest by comparing the numeric prices.

These quick counts catch scraping problems early. If the scrape had silently failed on a whole page, you'd see "6 products" instead of 9. If the price selector had broken, you'd see "0 of 9 had a price".

Part 10: One Real Request

Everything so far read files from disk. Here's the same code against a live website, using requests from chapter 17 to fetch the page and BeautifulSoup to read it.

The site is books.toscrape.com, which exists so people can practise scraping without bothering anyone. This is the only request in the whole guide.

Three things make it a polite request:

• a real User-Agent saying what this is,
• a timeout, as chapter 17 insisted,
• a sleep afterwards, so a next request wouldn't follow straight away.

(This one makes a real request. Its output is from the day the notebook was built.)

import time
import requests

response = requests.get("https://books.toscrape.com/",
                        headers={"User-Agent": "python-course-notebook/1.0 (learning example)"},
                        timeout=10)
response.raise_for_status()

# response.text, not response.json(), because this body is HTML
first_price = BeautifulSoup(response.text, "html.parser").select_one("p.price_color")
print("the first price reads:", first_price.get_text(strip=True))

the first price reads: £51.77

Notice response.text, not response.json(): this body is HTML, not JSON. Then select_one("p.price_color") finds the first price on the page.

And the price reads £51.77. That's not a currency symbol anyone recognises.

Where Did £ Come From?

Computers store text as bytes (numbers from 0 to 255). An encoding is the rulebook for turning characters into bytes and back again. The two that matter here:

• UTF-8, the encoding almost every website uses today. It can store every character in every language. In UTF-8, £ takes up two bytes.
• Latin-1 (also called ISO-8859-1), an older encoding where every byte is one character.

The page is written in UTF-8, so £ was sent as two bytes, and something read those two bytes as two separate characters: Â and £.

You can recreate it without the internet. Encode £ as UTF-8, then decode the bytes as Latin-1:

pound = "£".encode("utf-8")

print("£ in UTF-8 is two bytes:", pound)
print("read back as Latin-1   :", "£51.77".encode("utf-8").decode("latin-1"))

£ in UTF-8 is two bytes: b'\xc2\xa3'
read back as Latin-1   : £51.77

b'\xc2\xa3' is the two bytes (written in hexadecimal). Read back with the wrong rulebook, the first byte \xc2 became Â, and the second byte \xa3 happened to be £ in Latin-1.

Analogy: it's like reading a message written in a code where some letters use two symbols, using a codebook where every symbol is one letter. Most letters come out fine, but the two-symbol ones turn into gibberish pairs.

Who Got It Wrong?

The something was requests. When a server doesn't say which encoding it used for a text page, requests assumes Latin-1. That's what an older version of the HTTP rules said to do, and requests still follows it. This site's server only says text/html, with no encoding. The HTML itself does declare UTF-8, but in a <meta> tag that requests never looks at. (This uses the real response from above.)

print("what requests assumed:", response.encoding)
print("what the bytes say    :", response.apparent_encoding)

what requests assumed: ISO-8859-1
what the bytes say    : utf-8

• response.encoding is what requests assumed from the headers: ISO-8859-1, which is Latin-1.
• response.apparent_encoding is a guess from the bytes themselves, made by a character-detection library that requests uses. Here it guesses right: utf-8.

The Fix

Set response.encoding to the right value, and response.text is decoded again, correctly, without a second request (the bytes were already downloaded; only the reading changes). (This uses the real response from above.)

response.encoding = response.apparent_encoding     # believe the bytes

live = BeautifulSoup(response.text, "html.parser")

for pod in live.select("article.product_pod")[:5]:
    title = pod.h3.a["title"]
    price = pod.select_one("p.price_color").get_text(strip=True)
    stock = pod.select_one("p.instock").get_text(strip=True)
    print(f"  {title[:34]:<34} {price:>8}   {stock}")

time.sleep(2)                               # do not fetch the next page immediately

  A Light in the Attic                 £51.77   In stock
  Tipping the Velvet                   £53.74   In stock
  Soumission                           £50.10   In stock
  Sharp Objects                        £47.82   In stock
  Sapiens: A Brief History of Humank   £54.23   In stock

Now the prices read £51.77, and we can read the first five books.

Garbled text like that £ is called mojibake (a Japanese word meaning roughly "character transformation"). It's a common scraping bug, and an easy one to miss because nothing fails. The script happily writes a CSV full of £, and everything that uses the file inherits the mess.

Two ways to avoid it:

1. Set response.encoding = "utf-8" when you know the site uses UTF-8, which almost every site does now.
2. Or give BeautifulSoup response.content, the raw bytes, and let it read the <meta> tag itself to work out the encoding.

The Title Came from an Attribute

Look at pod.h3.a["title"]. The title comes from the link's title attribute, not the visible text, because the visible text on that site is cut short with "..." for long titles. It's a small example of a general rule: look at the actual HTML before deciding where the data is. The same information often appears in more than one place, and one of them is usually more complete.

Finally, time.sleep(2) pauses before anything else happens, matching the kind of Crawl-delay we saw in Part 1.

When the Data Isn't in the HTML at All

One more situation you'll meet. Some websites send an almost empty HTML page, and then JavaScript running in the browser fetches the data and builds the page. requests only downloads the HTML; it doesn't run JavaScript. So you get the empty shell, and your selectors find nothing.

Two ways forward:

• Use the developer tools' Network tab (Part 1) to find the JSON API the page's JavaScript is calling, and call it directly with chapter 17's tools. This is usually the better answer.
• Use a browser automation tool such as Playwright or Selenium, which drives a real browser that runs the JavaScript. It's much slower and heavier, so it's a last resort.

A quick check: right-click the page and choose View page source. That shows the HTML the server actually sent. If the data you want isn't in there, it's being loaded by JavaScript.

Part 11: Scraping Responsibly

The technical part of this chapter is short. This part is what decides whether your scraper is a useful tool or a problem for someone else.

Don't Hurt the Site

A loop with no pause can send many requests a second, and to the site that looks like an attack and gets treated like one. One request every second or two is plenty for anything that isn't urgent, and robots.txt may ask for a specific delay, as ours did.

for url in urls:
    scrape(url)
    time.sleep(2)          # or whatever Crawl-delay asked for

Analogy: a scraper with no pause is like a customer who asks a shopkeeper fifty questions a second. Even if every question is reasonable, the shopkeeper can't serve anyone else, and you'll be asked to leave.

Scrape once and save the pages. Re-reading a file on your disk costs the site nothing, and that's why this guide has sample_data/. While you're developing your parsing code, you'll run it dozens of times; run it against saved HTML, not the live site.

Say Who You Are

A real User-Agent with your project name and a contact address means that if something goes wrong, the site owner can email you instead of blocking your whole network. Pretending to be Chrome does the opposite. If you feel the need to disguise your scraper, ask yourself why.

Take Only What You Need

Fetching one page and pulling out four fields isn't the same as copying a whole database. Think about what you'd say if the site owner asked what you were doing.

The Legal Side, in Brief

This isn't legal advice, and the law is different from country to country. Some general points:

• robots.txt isn't a law, but ignoring it weakens any claim that you acted in good faith.
• Terms of service are a contract, and many of them forbid automated collection. Breaking them can get your account closed and has been used in court cases.
• Copyright still applies. Plain facts generally can't be copyrighted, but the text, photos and layout that present them generally can. Collecting prices is a different thing from republishing reviews.
• Personal data is protected by laws like the GDPR in the EU, and similar laws in many other countries. Being publicly visible isn't the same as consent.
• Getting around a login, a paywall or a CAPTCHA can break computer-misuse laws in many countries. This guide deliberately doesn't cover any of that.

The short version: read robots.txt, read the terms, go slowly, say who you are, take as little as you need, avoid personal data, and use the API if there is one. If a project needs you to ignore several of those, that's a sign to rethink it.

Part 12: Common Mistakes

Notice how many of these fail silently: wrong data, None, garbled characters, with no error message. That's why the quick checks in Part 9 (counting rows, counting missing prices) are part of every good scraper.

Quick Reference Summary

The whole shape

import time

import requests
from bs4 import BeautifulSoup

response = requests.get(url, headers={"User-Agent": "my-project/1.0 (me@example.com)"},
                        timeout=10)
response.raise_for_status()
response.encoding = "utf-8"          # or the £ signs come out as £
soup = BeautifulSoup(response.text, "html.parser")
...
time.sleep(2)

Finding

Reading

Selectors

Not falling over

def text_of(parent, selector, default=None):
    found = parent.select_one(selector)
    return found.get_text(strip=True) if found else default

Checking you're allowed

from urllib.robotparser import RobotFileParser

rules = RobotFileParser()
rules.set_url("https://example.com/robots.txt")
rules.read()
rules.can_fetch("*", url)      # True or False
rules.crawl_delay("*")         # seconds, or None

Before you start

1. Is there an API?
2. What does robots.txt say?
3. What do the terms of service say?
4. Is there personal data in this?
5. How slowly can I afford to go?