Regular Expressions: Finding Patterns in Text

Analogy: a normal text search is like asking a friend, "Find me the word cat in this book." A regex is like saying, "Find me every phone number in this book." You don't know the actual numbers in advance. You only know what a phone number looks like, and that description is enough to find all of them.

Regex comes with Python as the re module, so there's nothing to install. You'll meet it all over data work:

• cleaning messy columns (prices typed as "$59.98", "USD 45", "1,299.00 INR"),
• pulling fields out of log lines,
• checking that input looks right (is this a date? an order ID?),
• splitting text that a plain .split() can't handle.

Where We Left Off in Chapter 2

Back in chapter 2, we wrote two small text parsers using only plain string methods. One looked for a mention (like @guido) in an Instagram caption:

mention_start = caption.find("@")
mention = caption[mention_start:]

The other turned a Google search box entry into a URL slug, the readable part of a web address like how-to-learn-python:

url_slug = search_query.lower().replace(" ", "-")

Each one worked on the single example it was written for. In this guide, we'll first give them a harder example and watch them struggle, and then build the tool that handles both properly.

Here is what we will cover:

1. Where chapter 2's string methods run out
2. Raw strings, and why every pattern starts with r
3. The main re functions you'll actually use
4. Character classes, quantifiers and anchors: the building blocks of a pattern
5. Groups, and named groups that keep patterns readable
6. Greedy and lazy matching
7. re.compile, flags, and re.escape
8. Real examples: log files, messy currency values, checking emails, splitting sentences
9. When a regex is the wrong tool
10. Redoing chapter 2's parsers properly

Part 1: Where Plain String Methods Run Out

Here's chapter 2's caption parser again. This time we give it a more realistic caption, with several hashtags and several mentions:

caption = "Learning #Python and #pandas! Thanks @guido and @raymondh for the help. #100DaysOfCode"

has_python = "#Python" in caption          # chapter 2's hashtag check
mention_start = caption.find("@")          # chapter 2's mention finder
mention = caption[mention_start:]

print("Hashtag present:", has_python)
print("User mentioned :", mention)

Hashtag present: True
User mentioned : @guido and @raymondh for the help. #100DaysOfCode

There are two problems here:

1. The mention is wrong. We wanted @guido, but we got everything from the first @ to the very end of the caption. caption[mention_start:] means "from this position to the end", and nothing told it where the name stops.
2. The hashtag check is too narrow. "#Python" in caption only checks for one hashtag whose name we already knew. Instagram needs to find every hashtag in a caption, including ones nobody has thought of yet.

Doing It by Hand

Finding every hashtag with plain string methods means writing the search loop yourself, one character at a time:

def find_hashtags(text):
    tags = []
    word = ""

    for char in text + " ":                # the trailing space closes the last word
        if char == "#":
            word = "#"
        elif word and (char.isalnum() or char == "_"):
            word += char
        elif word:
            tags.append(word)
            word = ""

    return tags


print(find_hashtags(caption))

['#Python', '#pandas', '#100DaysOfCode']

It works, but look at what it took:

• a dozen lines of code,
• three if/elif branches,
• and a sneaky trick: adding an extra space to the end of the text (text + " ") so the last hashtag isn't lost.

Worse, it has a bug:

print("with a space   :", find_hashtags("#python #pandas"))
print("without a space:", find_hashtags("#python#pandas"))

with a space   : ['#python', '#pandas']
without a space: ['#pandas']

When two hashtags are written with no space between them, the first one silently disappears. Here's why: when the loop reaches the second #, the first branch (if char == "#") runs and restarts the word as "#", throwing away #python without ever saving it.

One hashtag missing, and nothing told us. That's the real cost of writing loops like this by hand. The problem isn't the length. It's the cases you didn't think of.

The Same Job with a Regex

import re

print(re.findall(r"#\w+", caption))
print(re.findall(r"@\w+", caption))
print(re.findall(r"#\w+", "#python#pandas"))    # the case the loop got wrong

['#Python', '#pandas', '#100DaysOfCode']
['@guido', '@raymondh']
['#python', '#pandas']

One line each, and the tricky case works too.

Let's read the pattern #\w+ piece by piece:

   #\w+
   │ │ └── + : "one or more of the thing before me"
   │ └──── \w : "a word character" (a letter, a digit or an underscore)
   └────── # : just a real # character

   In words: "a #, followed by one or more word characters"

re.findall then returns every piece of text that fits this description. Notice the pattern stops automatically at the first character that isn't a word character (a space, a !, another #), which is exactly where a hashtag ends.

Analogy: writing the loop by hand is like giving someone step-by-step directions: "walk forward, if you see a # start writing, if you see a letter keep writing, if you see a space stop...". A regex is like showing them a photo of what you're looking for. You describe the target, and the search is done for you.

The URL Slug Has the Same Kind of Problem

Chapter 2's slug code assumed the search text was clean. Real search text rarely is:

query = "How to learn Python in 2026?  (beginner guide)"

print(query.lower().replace(" ", "-"))

how-to-learn-python-in-2026?--(beginner-guide)

A question mark and a pair of brackets ended up in the URL, and the double space became a double hyphen. Fixing that with .replace() would mean one call for every punctuation mark you can think of, and you won't think of them all.

The regex version takes a different approach. Instead of listing the exceptions, it states the rule: "replace every run of characters that isn't a lowercase letter or a digit with a single hyphen". Then .strip("-") removes any hyphens left over at the two ends:

slug = re.sub(r"[^a-z0-9]+", "-", query.lower()).strip("-")

print(slug)

how-to-learn-python-in-2026-beginner-guide

Let's read [^a-z0-9]+:

• [a-z0-9] would mean "one lowercase letter or one digit".
• The ^ just inside the square brackets flips it: [^a-z0-9] means "any one character that is NOT a lowercase letter or digit".
• The + means "one or more of those in a row".

So "? (" (a question mark, two spaces and a bracket, all next to each other) counts as one run and becomes one hyphen. That's why there's no double hyphen anymore.

Part 2: Raw Strings, and Why Every Pattern Starts with r

Every pattern above was written as r"...", with a letter r before the opening quote. That r makes a raw string: a string where a backslash is just a backslash.

Why does that matter? Because two different programs both treat backslashes as special:

1. Python itself, when it reads a normal string. For example, "\n" becomes a newline character.
2. The regex engine, when it reads a pattern. For example, \d means "a digit".

Without the r, Python reads the string first and converts any backslash sequence it recognises (like \n, \t or \b) before the regex engine ever sees it:

print("normal string:", repr("\b"), "length", len("\b"))
print("raw string   :", repr(r"\b"), "length", len(r"\b"))

normal string: '\x08' length 1
raw string   : '\\b' length 2

• In a normal string, "\b" is turned into one character: a "backspace" character (shown as '\x08'), left over from old typewriter-style terminals.
• In a raw string, r"\b" stays as two characters: a backslash and a b. That's what the regex engine needs to see, because to regex \b means "word boundary" (Part 4 explains what that is).

Analogy: imagine passing a note through a translator to reach a second person. You write \b meaning "word boundary" for the second person, but the first translator sees \b, decides it means "backspace", and changes it before passing it on. The raw string r"..." is like writing "don't translate this, pass it on exactly as written" on the note.

   "\bPython"   ──► Python reads it ──► [backspace]Python ──► regex looks for a backspace ✗
   r"\bPython"  ──► Python leaves it ──► \bPython          ──► regex sees "word boundary" ✓

This isn't a matter of style. Leave the r off, and the pattern quietly stops working:

text = "I like Python"

print("without r:", re.findall("\bPython", text))
print("with r   :", re.findall(r"\bPython", text))

without r: []
with r   : ['Python']

No error, no warning, just an empty list. Python turned \b into a backspace character before re saw the pattern, and the sentence "I like Python" has no backspace characters in it.

What about a pattern like "\d+" written without the r? \d isn't one of Python's own escape sequences, so Python leaves it alone and the pattern happens to work. But you're relying on luck: recent versions of Python (3.12 and later) print SyntaxWarning: invalid escape sequence '\d' for it, and the plan is to turn that into an error in a future version.

The rule: write r"..." for every pattern, even ones with no backslash yet. You'll add one sooner or later, and the r will already be there.

Part 3: The Main Functions

The re module has a lot in it, but these five functions do almost all of the work:

3.1 re.search: The First Match, or Nothing

re.search scans the text from left to right and stops at the first place the pattern matches. It gives back a match object: a small object that holds the matched text and where it was found.

match = re.search(r"@\w+", caption)

if match:
    print(match)
    print("matched text:", match.group())
    print("position    :", match.start(), "to", match.end())
    print("span        :", match.span())

<re.Match object; span=(37, 43), match='@guido'>
matched text: @guido
position    : 37 to 43
span        : (37, 43)

What each line shows:

• print(match) shows the whole match object: what matched ('@guido') and its span.
• match.group() gives you just the matched text.
• match.start() and match.end() give the positions: the match begins at character 37 and ends just before character 43 (the end position is "one past the last character", the same way slicing works, so caption[37:43] is '@guido').
• match.span() gives both positions together as a tuple.

Always Check for None

That if match: in the code isn't just for show. When nothing matches, re.search returns None instead of a match object.

A match object counts as True in an if, and None counts as False (you saw this "truthy" and "falsy" idea in chapter 14), so a simple if is the normal way to check.

If you call .group() without checking, you get an error as soon as a search finds nothing. In this example, [0-9] means "any digit" (Part 4 covers this kind of pattern), so @[0-9]+ looks for a mention made only of digits, and there isn't one:

try:
    print(re.search(r"@[0-9]+", caption).group())    # no mention in the caption is all digits
except AttributeError as e:
    print("AttributeError:", e)

AttributeError: 'NoneType' object has no attribute 'group'

The error message is worth learning to recognise: 'NoneType' object has no attribute 'group' almost always means "your search found nothing, and you didn't check".

Analogy: asking re.search is like sending someone to fetch a parcel. If they come back empty-handed, you can't open the parcel. You have to check whether they're holding anything before you start unwrapping.

With the if in place, a missing match is just another case to handle:

for pattern in [r"@\w+", r"@[0-9]+"]:
    found = re.search(pattern, caption)
    if found:
        print(f"{pattern:<10} matched {found.group()!r}")
    else:
        print(f"{pattern:<10} no match")

@\w+       matched '@guido'
@[0-9]+    no match

(!r inside the f-string prints the value with its quotes, which makes it clear exactly what text matched.)

re.match and re.fullmatch

re.match is a different function, and its name is easy to misunderstand. It doesn't search the whole text. It only checks at the very start of the string.

re.fullmatch goes further: the pattern has to match the whole string, from the first character to the last. That's what you want when you're checking that a value has the right format. In the example below, \d{4} means "exactly four digits":

print("search    'Python'  :", re.search(r"Python", caption))
print("match     'Python'  :", re.match(r"Python", caption))
print("match     'Learning':", re.match(r"Learning", caption))
print()
print("fullmatch '2026'    :", re.fullmatch(r"\d{4}", "2026"))
print("fullmatch '2026a'   :", re.fullmatch(r"\d{4}", "2026a"))

search    'Python'  : <re.Match object; span=(10, 16), match='Python'>
match     'Python'  : None
match     'Learning': <re.Match object; span=(0, 8), match='Learning'>

fullmatch '2026'    : <re.Match object; span=(0, 4), match='2026'>
fullmatch '2026a'   : None

Look at each result:

• re.search(r"Python", caption) found Python at position 10, in the middle of the caption.
• re.match(r"Python", caption) returned None, because the caption starts with Learning, not Python.
• re.match(r"Learning", caption) found it, because Learning is right at the start.
• re.fullmatch(r"\d{4}", "2026") matched, because the whole string is four digits.
• re.fullmatch(r"\d{4}", "2026a") returned None, because of the extra a at the end.

Analogy: think of three ways a security guard could check a visitor list.

• search: "Is this name anywhere on the list?"
• match: "Is this name the first entry on the list?"
• fullmatch: "Is this name the only entry, with nothing else written on the page?"

   text: "Learning #Python and #pandas! ..."

   re.search("Python")     scans everywhere        ──► found at 10 ✓
   re.match("Python")      checks the start only   ──► "Learning..." ✗ None
   re.fullmatch(r"\d{4}")  must cover everything   ──► "2026" ✓   "2026a" ✗

3.2 re.findall and re.finditer

findall gives you a list of the matching text and nothing else. finditer gives you match objects one at a time, so you also get the positions.

finditer is also lazy, in the sense you met in chapter 12 on iterators: it only finds each match when you ask for the next one, instead of finding them all up front.

for found in re.finditer(r"#\w+", caption):
    print(f"  {found.group():<16} at characters {found.start()}-{found.end()}")

  #Python          at characters 9-16
  #pandas          at characters 21-28
  #100DaysOfCode   at characters 72-86

Use findall when you only want the text, and finditer when you also need to know where each match is (for example, to highlight it, or to show the surrounding context).

3.3 re.sub: Find and Replace

re.sub is like str.replace(), but it replaces anything that fits a pattern, not just one fixed piece of text. A common use is hiding personal data (like card or phone numbers) before it goes into a log file or a shared notebook:

record = "Card 4111 1111 1111 1234, phone 98765 43210"

print(re.sub(r"\d", "X", record))
print(re.sub(r"\d{4,}", "[redacted]", record))

Card XXXX XXXX XXXX XXXX, phone XXXXX XXXXX
Card [redacted] [redacted] [redacted] [redacted], phone [redacted] [redacted]

• \d matches one digit, so every single digit became an X.
• \d{4,} means "four or more digits in a row", so each block of 4 or 5 digits was replaced as one unit.

Replacing with a Function

The replacement doesn't have to be fixed text. It can also be a function. re.sub calls the function once for each match, passes the match object in, and puts whatever the function returns in its place. So the replacement can depend on what was matched.

Two new pieces in this pattern:

• \$ means "a real dollar sign". A bare $ has a special meaning in regex ("end of the string", covered in Part 4), so the backslash tells regex to treat it as an ordinary character.
• The round brackets around \d+\.\d\d capture the number, and found.group(1) reads it back. Part 5 covers groups properly.

prices = "Books $59.98, Electronics $45.00, Appliances $89.99"

def add_tax(found):
    amount = float(found.group(1))
    return f"${amount * 1.18:.2f}"

print(re.sub(r"\$(\d+\.\d\d)", add_tax, prices))

Books $70.78, Electronics $53.10, Appliances $106.19

Each price was found, passed to add_tax, multiplied by 1.18 (adding 18% tax), and the new value was written back in its place.

Analogy: a normal find-and-replace is like a stamp: every match gets exactly the same new text. A replacement function is like a person reading each match and writing a suitable replacement for that particular one.

3.4 re.split: Splitting on a Pattern

str.split() takes one fixed separator. re.split takes a pattern, which is what you need when the separator changes from place to place:

messy = "Books;Electronics, Appliances|Toys , Garden"

print("str.split:", messy.split(","))
print("re.split :", re.split(r"\s*[;,|]\s*", messy))

str.split: ['Books;Electronics', ' Appliances|Toys ', ' Garden']
re.split : ['Books', 'Electronics', 'Appliances', 'Toys', 'Garden']

The pattern \s*[;,|]\s* reads as:

• \s*: zero or more whitespace characters (spaces, tabs),
• [;,|]: then one of a semicolon, a comma or a pipe,
• \s*: then zero or more whitespace characters again.

So it treats ";", ", ", "|" and " , " as separators, and swallows the spaces around them too. That's why the results come out clean, with no extra spaces.

Part 4: The Building Blocks

Every pattern is built from three kinds of piece:

1. What to match (a digit? a letter? one of a few characters?)
2. How many of it (one? several? optional?)
3. Where it should be (at the start? at a word edge?)

Analogy: describing a pattern is like describing a car number plate to a friend: "two letters, then two digits, then four more digits, and it's on the back of the car". The "letters" and "digits" are the what, the "two" and "four" are the how many, and "on the back" is the where.

4.1 What to Match

Two patterns make these easy to remember:

• The capital letter always means the opposite: \d is a digit, \D is anything that isn't a digit.
• A ^ at the start of square brackets means "not": [abc] is one of these, [^abc] is anything except these.

Square brackets are called a character class. However many characters you put inside, a character class always matches exactly one character from the text.

One small extra detail: in Python, \d and \w understand more than English. \d also matches digits from other writing systems (such as Devanagari २०२६), and \w matches accented letters like the é in café. When you only want the plain digits 0 to 9, write [0-9].

sample = "Order A001 shipped on 2024-03-01 for $59.98"

print("digits      :", re.findall(r"\d", sample)[:6], "...")
print("numbers     :", re.findall(r"\d+", sample))
print("words       :", re.findall(r"\w+", sample)[:4], "...")
print("not digits  :", re.findall(r"[^\d\s]+", sample))
print("vowels      :", re.findall(r"[aeiou]", sample))

digits      : ['0', '0', '1', '2', '0', '2'] ...
numbers     : ['001', '2024', '03', '01', '59', '98']
words       : ['Order', 'A001', 'shipped', 'on'] ...
not digits  : ['Order', 'A', 'shipped', 'on', '-', '-', 'for', '$', '.']
vowels      : ['e', 'i', 'e', 'o', 'o']

Look at what each pattern picked out:

• \d found single digits, one at a time (we print only the first six).
• \d+ found whole numbers, because + keeps going while there are more digits.
• \w+ found whole words. A001 counts as one word, because digits are word characters.
• [^\d\s]+ found runs of characters that are neither digits nor spaces: words, dashes, the $ and the dot.
• [aeiou] found each lowercase vowel. The capital O in Order isn't there, because regex is case-sensitive unless you tell it not to be (Part 7 shows how).

4.2 How Many

These are called quantifiers. A quantifier applies only to whatever comes right before it. In A\d+, the + applies to \d, not to the A, so it means "an A, then one or more digits".

In the code below, str(...) turns True/False into text so that :<6 can line up the columns:

codes = ["A1", "A001", "A", "AB12345", "2024-03-01"]

for code in codes:
    is_order_id = bool(re.fullmatch(r"A\d+", code))
    is_date = bool(re.fullmatch(r"\d{4}-\d{2}-\d{2}", code))
    print(f"  {code:<12} A + digits: {str(is_order_id):<6} date shape: {is_date}")

  A1           A + digits: True   date shape: False
  A001         A + digits: True   date shape: False
  A            A + digits: False  date shape: False
  AB12345      A + digits: False  date shape: False
  2024-03-01   A + digits: False  date shape: True

• A\d+ accepts A1 and A001 (an A and at least one digit), but not A alone (no digits) or AB12345 (a B where a digit should be).
• \d{4}-\d{2}-\d{2} describes the shape of a date: four digits, dash, two digits, dash, two digits. Only 2024-03-01 has that shape.

Remember that fullmatch needs the whole string to fit, which is why AB12345 fails instead of matching some part of it.

? Makes Something Optional

? means "zero or one", which is a neat way to accept two spellings with one pattern:

for value in ["color", "colour", "colouur"]:
    print(f"  {value:<10} {bool(re.fullmatch(r'colou?r', value))}")

  color      True
  colour     True
  colouur    False

In colou?r, the ? applies only to the u. So the u can appear once or not at all: color and colour both match. colouur has two, so it doesn't.

4.3 Where: Anchors

These are called anchors. They match a position in the text, not a character, so they don't use up any of the text.

Notice that ^ has two different jobs depending on where it is: at the start of a pattern it means "start of the string", and just inside square brackets ([^...]) it means "not". The position tells you which one it is.

One small detail about $: it also matches just before a newline at the very end of the string. So re.search(r"^\d{4}$", "2026\n") finds a match, even though the string has a newline after the digits. re.fullmatch is stricter and says no, which is another reason to prefer it when checking a value.

\b is the anchor you'll use most. It's what stops you matching inside a longer word:

text = "cat catalogue concatenate the cat sat"

print("without \\b:", re.findall(r"cat", text))
print("with    \\b:", re.findall(r"\bcat\b", text))

without \b: ['cat', 'cat', 'cat', 'cat']
with    \b: ['cat', 'cat']

Without \b, cat was found four times: twice as the real word, once inside catalogue and once inside concatenate. With \bcat\b, there must be a word edge on both sides, so only the two real cats match.

   "cat catalogue concatenate the cat sat"
    ───                           ───      ← \bcat\b : whole words only
        ───          ───                   ← plain "cat" also hits these
                   (inside catalogue, inside concatenate)

Analogy: \b is like the spaces between words in a crossword clue. It doesn't match a letter itself; it marks where one word ends and the next begins.

4.4 Alternation: "This or That"

The | character means "or". There's a catch: it splits the whole pattern in two, unless you put round brackets around the part that should have the choice. Compare these two:

log = "level=ERROR level=INFO level=WARNING note=WARNING"

print("no brackets  :", re.findall(r"level=ERROR|WARNING", log))
print("with brackets:", re.findall(r"level=(ERROR|WARNING)", log))

no brackets  : ['level=ERROR', 'WARNING', 'WARNING']
with brackets: ['ERROR', 'WARNING']

   r"level=ERROR|WARNING"        means   (level=ERROR)   OR   (WARNING anywhere)
   r"level=(ERROR|WARNING)"      means   level=  followed by  (ERROR or WARNING)

• Without brackets, the pattern means "level=ERROR, or WARNING anywhere at all". So it also matched the WARNING after note=, which we didn't want.
• With brackets, the choice is only between ERROR and WARNING, and level= is required either way.

Analogy: it's like the difference between ordering "pasta with tomato or cheese sauce" and "pasta with tomato, or cheese". Without the brackets, the waiter might bring you a plate of plain cheese.

You might also notice that the second version returned only ERROR and WARNING, not level=ERROR. That's because round brackets have a second job, capturing, which is what the next part is about.

Part 5: Groups: Pulling Out the Pieces

Round brackets do two jobs:

1. They keep part of a pattern together (like the (ERROR|WARNING) above).
2. They capture whatever matched inside them, so you can read it back afterwards.

A date is a good first example, because often you only want part of it, like the year:

found = re.search(r"(\d{4})-(\d{2})-(\d{2})", sample)

if found:
    print("whole match:", found.group())      # group 0
    print("year       :", found.group(1))
    print("month      :", found.group(2))
    print("all groups :", found.groups())

whole match: 2024-03-01
year       : 2024
month      : 03
all groups : ('2024', '03', '01')

• found.group() (also written found.group(0)) is always the whole match.
• found.group(1) is what the first pair of brackets captured, found.group(2) the second, and so on.
• found.groups() gives you all the captured pieces together as a tuple.

Groups are numbered from left to right, in the order their opening brackets appear:

   (\d{4})-(\d{2})-(\d{2})
   └──1──┘ └──2──┘ └──3──┘

   "2024-03-01"  ──►  group(1) = "2024"   group(2) = "03"   group(3) = "01"

Analogy: capturing groups are like the labelled boxes on a paper form. The whole form is group 0, and each box (year, month, day) holds one piece you can read separately.

5.1 Named Groups

Counting brackets gets confusing quickly. Was the log level group 3 or group 4? (?P<name>...) gives a group a name instead, and .groupdict() gives you all of them back as a dictionary:

line = "2024-03-01 09:14:22 WARNING Payment retry 1 for order A002"

pattern = r"(?P<date>\S+) (?P<time>\S+) (?P<level>\w+) (?P<message>.*)"
found = re.search(pattern, line)

if found:
    print(found.group("level"))
    for field, value in found.groupdict().items():
        print(f"  {field:<8} {value}")

WARNING
  date     2024-03-01
  time     09:14:22
  level    WARNING
  message  Payment retry 1 for order A002

Here's what the pattern says, piece by piece:

• (?P<date>\S+): one or more non-space characters, saved as date,
• a space,
• (?P<time>\S+): the next run of non-space characters, saved as time,
• a space,
• (?P<level>\w+): a word, saved as level,
• a space,
• (?P<message>.*): everything else on the line, saved as message.

The pattern is longer than one with numbered groups, but anyone reading it can now see what each field is. And found.group("level") is much clearer than found.group(3). Use named groups for any pattern you'll come back to.

5.2 Non-Capturing Groups

In Part 4.4, the brackets around ERROR|WARNING were only there to contain the |. They captured as a side effect, which is why findall returned only the level.

(?:...) makes a group that groups without capturing. Then findall gives you the whole match again:

print("capturing    :", re.findall(r"level=(ERROR|WARNING)", log))
print("non-capturing:", re.findall(r"level=(?:ERROR|WARNING)", log))

capturing    : ['ERROR', 'WARNING']
non-capturing: ['level=ERROR', 'level=WARNING']

Analogy: normal brackets are like a highlighter: they group the words together and mark them to be copied out. (?:...) is like a pencil circle: it groups the words together, but marks nothing to be copied.

5.3 The findall Trap

findall changes what it returns depending on how many capturing groups the pattern has:

• no groups: a list of whole matches,
• one group: a list of just that group,
• two or more groups: a list of tuples, one value per group.

text = "orders: A001, A002, B003"

print("no groups :", re.findall(r"[AB]\d+", text))
print("one group :", re.findall(r"[AB](\d+)", text))
print("two groups:", re.findall(r"([AB])(\d+)", text))

no groups : ['A001', 'A002', 'B003']
one group : ['001', '002', '003']
two groups: [('A', '001'), ('A', '002'), ('B', '003')]

Nothing warns you about this. A pattern that returned strings yesterday can return tuples today, just because someone added a pair of brackets to it. If you want the whole match but still need brackets for grouping, either make them non-capturing with (?:...), or use finditer and read .group() yourself.

Part 6: Greedy and Lazy

Quantifiers like + and * are greedy: they grab as much text as they possibly can, and only give some back if they have to. This surprises a lot of people the first time they try to match an HTML tag:

html = '<p class="lead">Regex</p> and <b>bold</b>'

print("greedy:", re.findall(r"<.+>", html))

greedy: ['<p class="lead">Regex</p> and <b>bold</b>']

One match, and it's the whole string! Here's what happened, step by step:

1. < matched the first < in the text.
2. .+ (one or more of any character) ran all the way to the end of the string, grabbing everything.
3. Then the pattern still needed a >, so the engine stepped back one character at a time until it found one. The first > it found going backwards was the last > in the string.

Every > in between was simply skipped over.

Analogy: greedy matching is like a hungry person at a buffet who piles everything onto their plate first, and then puts things back one at a time only if they've taken too much. They'll always end up with the biggest plate that still fits the rules.

Lazy Matching with ?

Putting ? after a quantifier makes it lazy: it takes as little as possible, and only grabs more if it has to.

print("lazy  :", re.findall(r"<.+?>", html))

lazy  : ['<p class="lead">', '</p>', '<b>', '</b>']

Four matches, which is probably what you expected the first time.

   text:   <p class="lead">Regex</p> and <b>bold</b>

   greedy  ├───────────────────────────────────────┤   <.+>  : one huge match
   lazy    ├──────────────┤     ├──┤     ├─┤    ├──┤   <.+?> : four small matches

Every quantifier has a lazy version: *?, +?, ??, {2,4}?. When a pattern matches far more than you meant, greediness is usually the reason.

Analogy: a lazy quantifier is the careful person at the buffet who takes one spoonful at a time and stops as soon as the plate is acceptable.

Often Better: Say What You Don't Want

There's a third option that's often better than either: describe what the match can't contain. [^>]+ means "one or more characters that aren't >", so it can never run past the end of a tag:

print("negated:", re.findall(r"<[^>]+>", html))

negated: ['<p class="lead">', '</p>', '<b>', '</b>']

Same result as the lazy version, but the pattern itself says exactly where a tag stops. There's no "grab and step back" at all, which makes it both clearer and usually faster.

Part 7: Compiling, and Flags

7.1 re.compile

re.compile turns a pattern into a pattern object that you can keep and reuse. The object has the same matching methods as the module: .search(), .findall(), .sub() and so on, without the pattern argument.

The re module already remembers recently used patterns behind the scenes, so compiling doesn't make much difference to speed. The real reason to do it is readability: the pattern gets a name, and it's written once instead of being retyped in every place it's used.

HASHTAG = re.compile(r"#\w+")
ORDER_ID = re.compile(r"\b[AB]\d{3}\b")

print(HASHTAG.findall(caption))
print(ORDER_ID.findall("orders A001, B003 and the number 1234"))

['#Python', '#pandas', '#100DaysOfCode']
['A001', 'B003']

Notice ORDER_ID uses \b on both sides. That's why 1234 wasn't matched: it doesn't start with A or B, and the \bs stop the pattern from matching part of a longer word.

Writing the names in CAPITALS is a common Python habit for values that are set once and never changed, like these patterns.

Analogy: compiling a pattern is like saving a phone number as a contact. You could type the full number every time, but giving it a name makes it quicker to use and much easier to read later.

7.2 Flags

Flags change how a pattern is read. Three are worth knowing.

re.IGNORECASE

It does exactly what it says: uppercase and lowercase letters count as the same.

print(re.findall(r"python", caption))
print(re.findall(r"python", caption, re.IGNORECASE))

[]
['Python']

The caption has Python with a capital P, so plain python found nothing. With re.IGNORECASE, it found Python.

re.MULTILINE

Normally ^ means "the start of the whole string" and $ means "the end of the whole string". re.MULTILINE makes them match at the start and end of each line instead.

A triple-quoted string ("""...""") can span several lines, which is handy for test data like this sample log:

log_text = """2024-03-01 09:14:22 INFO Order A001 placed
2024-03-01 09:15:03 WARNING Payment retry 1 for order A002
2024-03-01 09:15:40 ERROR Payment failed for order A002
2024-03-01 09:16:00 INFO Order A003 placed"""

print("without MULTILINE:", re.findall(r"^\S+", log_text))
print("with MULTILINE   :", re.findall(r"^\S+", log_text, re.MULTILINE))

without MULTILINE: ['2024-03-01']
with MULTILINE   : ['2024-03-01', '2024-03-01', '2024-03-01', '2024-03-01']

^\S+ means "non-space characters at the start". Without the flag, there's only one "start" (the start of the whole text), so only the first date was found. With re.MULTILINE, every line has its own start, so we got four.

   without MULTILINE                with MULTILINE
   ^2024-03-01 09:14:22 INFO ...    ^2024-03-01 09:14:22 INFO ...
    2024-03-01 09:15:03 WARN ...    ^2024-03-01 09:15:03 WARN ...
    2024-03-01 09:15:40 ERROR ...   ^2024-03-01 09:15:40 ERROR ...
    2024-03-01 09:16:00 INFO ...    ^2024-03-01 09:16:00 INFO ...
   (one start)                      (a start on every line)

re.VERBOSE

re.VERBOSE lets you spread a pattern over several lines, with comments explaining each part. Whitespace inside the pattern is ignored (except inside square brackets), and # starts a comment. Any pattern that someone else will have to read and maintain should be written like this:

LOG_LINE = re.compile(r"""
    (?P<date>\d{4}-\d{2}-\d{2})     # 2024-03-01
    \s
    (?P<time>\d{2}:\d{2}:\d{2})     # 09:14:22
    \s
    (?P<level>[A-Z]+)               # INFO, WARNING, ERROR
    \s
    (?P<message>.+)                 # the rest of the line
""", re.VERBOSE)

found = LOG_LINE.search(log_text)
if found:
    print(found.groupdict())

{'date': '2024-03-01', 'time': '09:14:22', 'level': 'INFO', 'message': 'Order A001 placed'}

Compare this with the one-line version in Part 5.1. The two do the same job, but this one is far easier to understand at a glance, because every part has a comment showing an example of what it matches. It's also stricter: \d{4}-\d{2}-\d{2} only accepts something shaped like a date, where \S+ would accept any word.

(.search() stopped at the first matching line. Part 8 uses .finditer() to go through every line.)

Analogy: a one-line regex is like a recipe written as one long sentence. re.VERBOSE lets you write it as a proper recipe card, with one ingredient per line and a note beside each.

One Rule to Remember with re.VERBOSE

Because ordinary spaces inside the pattern are ignored, you have to write a real space differently:

• \s matches any whitespace (that's what the pattern above uses),
• [ ] or \ matches exactly one space.

The same goes for #: since it starts a comment in verbose mode, a pattern that needs a real # (like a hashtag) must write it as \# or [#].

7.3 re.escape: When the Pattern Comes from Data

Anything you put into a pattern is read as a pattern. That's fine when you type the pattern yourself. But a search term typed by a user, or read from a file, will sooner or later contain a character that means something special to regex:

term = "a.c"                            # a literal string someone typed
haystack = "abc and a.c"

print("used as a pattern:", re.findall(term, haystack))

used as a pattern: ['abc', 'a.c']

Someone searched for the exact text a.c, but the dot means "any character" in regex. So it matched abc too, even though abc doesn't contain the search term at all.

re.escape puts a backslash in front of every special character, so the text matches itself and nothing else:

print("escaped     :", re.escape(term))
print("used escaped:", re.findall(re.escape(term), haystack))

escaped     : a\.c
used escaped: ['a.c']

a\.c means "an a, a real dot, a c", so now only a.c matches.

Analogy: re.escape is like putting quotation marks around someone's words when you write them down. It makes clear that these are the exact words they said, not instructions to follow.

Use it whenever the pattern is built from a value instead of typed by you: search terms, filenames, product codes, or anything else that might contain a dot, a bracket or a plus sign.

Part 8: Real-World Examples

8.1 Reading a Log File

The LOG_LINE pattern from Part 7 plus chapter 15's Counter and defaultdict turns a whole log into a summary. A lot of real log-processing scripts follow this same shape:

from collections import Counter, defaultdict

levels = Counter()
by_level = defaultdict(list)

for entry in LOG_LINE.finditer(log_text):
    fields = entry.groupdict()
    levels[fields["level"]] += 1
    by_level[fields["level"]].append(fields["time"])

print("counts:", levels.most_common())
print()
for level, times in sorted(by_level.items()):
    print(f"  {level:<8} {times}")

counts: [('INFO', 2), ('WARNING', 1), ('ERROR', 1)]

  ERROR    ['09:15:40']
  INFO     ['09:14:22', '09:16:00']
  WARNING  ['09:15:03']

How it works:

1. LOG_LINE.finditer(log_text) goes through the log and gives back one match per line.
2. entry.groupdict() turns each match into a dictionary like {"date": ..., "time": ..., "level": ..., "message": ...}.
3. levels[fields["level"]] += 1 counts how many lines there are at each level.
4. by_level[fields["level"]].append(fields["time"]) collects the times for each level.

The regex does the hard part (splitting each line into named fields), and the collections tools do the counting and grouping.

8.2 Cleaning a Currency Column

Amounts typed into spreadsheets come in whatever format the person felt like using. One simple approach is to remove everything that isn't a digit or a dot, and then convert what's left to a number. Watch the last value closely:

raw_amounts = ["$59.98", "1,299.00 INR", "USD 45", "  89.99  ", "free", "Rs. 500"]

for raw in raw_amounts:
    digits = re.sub(r"[^\d.]", "", raw)
    print(f"  {raw!r:<16} -> {float(digits) if digits else None}")

  '$59.98'         -> 59.98
  '1,299.00 INR'   -> 1299.0
  'USD 45'         -> 45.0
  '  89.99  '      -> 89.99
  'free'           -> None
  'Rs. 500'        -> 0.5

[^\d.] means "any character that isn't a digit or a dot" (inside square brackets, a dot is just a real dot). Replacing those with "" deletes them.

The first five came out right. "free" has no digits at all, so digits is an empty string and we print None instead of crashing.

But Rs. 500 became 0.5. The dot after Rs is kept too, because it's a dot, which leaves the text .500, and float(".500") is 0.5. Nothing raised an error.

This is an important lesson: a regex does exactly what the pattern says, not what you meant. Always check the output on real data before you trust a cleaning rule.

Analogy: a cleaning pattern is like a sieve. It lets through exactly what the holes allow. If a bit of grit happens to be the same size as the grain you want, it goes straight through with it.

8.3 Checking Emails, and Knowing When to Stop

Regexes are good at rejecting obvious rubbish, and bad at deciding what's truly valid. Email addresses are the classic example. Here's the kind of pattern people usually write first:

SIMPLE_EMAIL = re.compile(r"[\w.]+@\w+\.\w+")

for address in ["shikhar@example.com", "first.last@example.co.uk",
                "a+tag@example.com", "not-an-email", "@example.com"]:
    print(f"  {address:<28} {bool(SIMPLE_EMAIL.fullmatch(address))}")

  shikhar@example.com          True
  first.last@example.co.uk     False
  a+tag@example.com            False
  not-an-email                 False
  @example.com                 False

The pattern [\w.]+@\w+\.\w+ reads as "letters, digits, underscores or dots; then @; then a word; then a real dot; then another word".

It correctly rejects the two obvious failures (not-an-email has no @, and @example.com has nothing before the @).

But it also rejects two perfectly normal addresses that real people use:

• first.last@example.co.uk, because the part after @ has two dots, and the pattern only allows one.
• a+tag@example.com, because + isn't allowed before the @ in this pattern.

A pattern that follows the full official rules for email addresses is thousands of characters long. And even that would only prove that the address is written correctly, not that anyone actually reads mail there.

In practice, the sensible approach is: check for an @ with something on each side, and confirm the rest by sending an email to it (that's why websites send you a "confirm your email" link).

8.4 Splitting Text into Sentences and Words

review = "The book arrived fast! Was it worth 59.98? Yes... mostly. I'd buy again."

sentences = re.split(r"(?<=[.!?])\s+", review)
for s in sentences:
    print(f"  {s}")

print("\nwords:", re.findall(r"\b[\w']+\b", review.lower())[:8], "...")

  The book arrived fast!
  Was it worth 59.98?
  Yes...
  mostly.
  I'd buy again.

words: ['the', 'book', 'arrived', 'fast', 'was', 'it', 'worth', '59'] ...

Two new ideas here.

The lookbehind. (?<=[.!?]) is called a lookbehind. It says "right before this point, there must be a ., ! or ?", but it doesn't use up that character. So the split happens on the spaces that come after sentence-ending punctuation, and the punctuation stays attached to the end of its sentence.

   "The book arrived fast! Was it worth 59.98? ..."
                         ↑                   ↑
           split here: a space with ! or ? right before it

   "59.98" is not split: the dot inside it isn't followed by a space

Analogy: a lookbehind is like checking over your shoulder before stepping forward. You look at what's behind you, but you don't pick it up.

The judgement calls. Look carefully at the results:

• Yes... was split off from mostly.. Is Yes... mostly. one sentence or two? That's a judgement call, and the pattern quietly made it for you.
• The word pattern \b[\w']+\b (word characters or apostrophes, so I'd stays whole) broke 59.98 into 59 and 98, because a dot isn't a word character. Whether that's right depends on what you're counting.

Patterns make these choices silently, so always look at the output instead of assuming it's right.

Part 9: When a Regex Is the Wrong Tool

A regex is great at matching text that follows a simple, regular shape. Plenty of everyday text doesn't follow one.

9.1 Don't Read HTML with a Regex

The patterns from Part 6 looked convincing on a tidy example. Give one some real-world HTML, and it falls apart:

page = '<a href="/p1">first</a> <!-- <a href="/hidden">skip</a> --> <a\n   href="/p2">second</a>'

print("regex   :", re.findall(r'<a href="([^"]+)">', page))

regex   : ['/p1', '/hidden']

The page has three <a> tags, the pattern found two matches, and it made two mistakes:

1. It returned /hidden, which is inside an HTML comment (<!-- ... -->). Comments aren't shown on the page, so that isn't a real link at all.
2. It missed /p2 completely, because that tag has a line break between <a and href (the \n in the string). That's perfectly valid HTML, but it doesn't fit the pattern.

Neither mistake would be noticed without checking. Fixing those two cases means adding rules for comments, attribute order, single versus double quotes, and extra whitespace, and every fix uncovers the next gap.

An HTML parser already knows all of those rules. Chapter 18 uses one called BeautifulSoup, which reads this page correctly without a single pattern.

Analogy: reading HTML with a regex is like trying to understand a sentence's grammar by looking only for certain letters. It works on simple sentences, then falls apart the moment someone uses a comma in a new way. A parser actually understands the grammar.

9.2 Don't Read CSV with a Regex

Cases like this are why chapter 10's csv module exists:

row = 'A001,"Sharma, Alice",Books,59.98'

print("str.split:", row.split(","))
print("re.split :", re.split(r",", row))

import csv
import io

print("csv      :", next(csv.reader(io.StringIO(row))))

str.split: ['A001', '"Sharma', ' Alice"', 'Books', '59.98']
re.split : ['A001', '"Sharma', ' Alice"', 'Books', '59.98']
csv      : ['A001', 'Sharma, Alice', 'Books', '59.98']

The comma inside the quoted name "Sharma, Alice" is part of the name, not a separator. Both str.split and re.split broke it in two. Only the csv module understands the quoting rules and keeps it whole.

(io.StringIO(row) makes the string behave like a file, because csv.reader expects to read from a file.)

9.3 Don't Use a Regex Where a String Method Will Do

re.search(r"^https://", url) is a slower, harder-to-read way of writing url.startswith("https://"). For fixed text, string methods are clearer:

One small difference in the second row: $ also matches just before a final newline (Part 4.3), so the regex accepts "xabc\n" where endswith doesn't. The string method is the more exact of the two.

The rule of thumb: use re when you're describing a shape ("some digits, then a dash"). Use string methods when you're looking for a fixed piece of text ("https://").

9.4 Patterns That Can Freeze Your Program

Some patterns slow down enormously on text that almost matches. (a+)+b run against a string made only of as is the standard example.

Here's why. The pattern needs a b at the end, and there isn't one, so it can never match. But before giving up, the engine tries every possible way of sharing the as between the inner + and the outer +. With 3 as there are only a few ways to share them, but the number of ways doubles with every extra character. This is called catastrophic backtracking, and it has taken real websites down.

Analogy: imagine trying to open a combination lock by trying every combination. A 3-digit lock is quick. Each extra digit multiplies the work, and soon you'd be there for days. The engine is stuck doing the same thing, trying every combination before it can say "no match".

We can watch it grow safely by timing short strings:

import time

def time_it(n):
    start = time.perf_counter()
    re.search(r"(a+)+b", "a" * n)
    return time.perf_counter() - start

short, longer = time_it(18), time_it(22)

print("4 more characters made it more than 5 times slower:", longer > short * 5)

4 more characters made it more than 5 times slower: True

The exact times depend on your computer, so the cell only prints whether the slowdown happened. On a typical machine, every 2 extra characters make it about 4 times slower, so a string of 40 as would take hours.

   characters:   18     20     22     24    ...    40
   time:          1x    ~4x   ~16x   ~64x   ...   hours

The fix: don't nest quantifiers. Never put a + or * around a group that already contains one. Here, a+b describes the same text and finishes instantly.

Part 10: Redoing Chapter 2, Properly

Here's everything this guide started with, written the way you'd actually ship it: compiled, named patterns inside small functions.

HASHTAG = re.compile(r"#\w+")
MENTION = re.compile(r"@\w+")

def parse_caption(text):
    return {"hashtags": HASHTAG.findall(text),
            "mentions": MENTION.findall(text)}


def slugify(text):
    return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")


print(parse_caption(caption))
print(parse_caption("No tags here, just words."))
print()

for q in ["How to learn Python in 2026?  (beginner guide)",
          "  Data Science & AI: where to start!  ",
          "already-a-slug"]:
    print(f"  {q!r:<48} -> {slugify(q)!r}")

{'hashtags': ['#Python', '#pandas', '#100DaysOfCode'], 'mentions': ['@guido', '@raymondh']}
{'hashtags': [], 'mentions': []}

  'How to learn Python in 2026?  (beginner guide)' -> 'how-to-learn-python-in-2026-beginner-guide'
  '  Data Science & AI: where to start!  '         -> 'data-science-ai-where-to-start'
  'already-a-slug'                                 -> 'already-a-slug'

• parse_caption returns empty lists instead of failing when there's nothing to find.
• slugify handles punctuation, repeated spaces, spaces at the start and end, an & sign, and text that's already a slug.

Neither function needed a single special case for any of those. That's the real strength of a regex: you describe the rule, and the rule covers cases you never listed.

Part 11: Common Mistakes

Notice how many of these fail quietly: an empty list, a wrong number, a missing match, with no error message. With regex, the habit that protects you most is simple: print the result and look at it before you trust it.

Quick Reference Summary

The functions

import re

re.search(pattern, text)      # first match object, or None
re.match(pattern, text)       # same, but anchored at the start
re.fullmatch(pattern, text)   # the whole string must match
re.findall(pattern, text)     # list of strings
re.finditer(pattern, text)    # iterator of match objects
re.sub(pattern, repl, text)   # replace; repl may be a function
re.split(pattern, text)       # split on a pattern
re.compile(pattern, flags)    # a reusable pattern object
re.escape(text)               # treat text as literal, not as a pattern

The match object

found.group()        # the whole match
found.group(1)       # the first capture group
found.group("name")  # a named group
found.groups()       # every group as a tuple
found.groupdict()    # named groups as a dict
found.start(), found.end(), found.span()

What to match

A pipe between two options means "or", as in (?:ERROR|WARNING), and (?:...) groups without capturing.

How many

Where

Groups

r"(\d{4})-(\d{2})"                    # numbered: .group(1), .group(2)
r"(?P<year>\d{4})-(?P<month>\d{2})"   # named: .group("year"), .groupdict()
r"(?:ERROR|WARNING)"                  # grouped, not captured

Flags

Patterns you'll write again

r"#\w+"                             # hashtag
r"@\w+"                             # mention
r"\d{4}-\d{2}-\d{2}"                # ISO date
r"[^a-z0-9]+"                       # runs of punctuation, for slugs
r"[^\d.]"                           # everything that is not part of a number
r"<[^>]+>"                          # a tag, without overrunning