Modules and Virtual Environments in Python: A Complete Beginner's Guide

By the end, you will be able to split your own code into several files and reuse it, install code that other people wrote, and set up a project so that it still runs on someone else's computer. These are the skills that turn a single practice script into a real project.

Here is what we will cover:

1. What a module is, the different ways to import one, and what import does behind the scenes
2. Writing your own module and importing it
3. if __name__ == "__main__", and why so many Python files end with it
4. Packages, which are folders of modules
5. The standard library, the modules you get for free
6. Third-party packages, the terminal, virtual environments and pip
7. How to read the error when an import fails
8. How a real Python project is laid out
9. A real-world example: turning one messy script into a tidy project

Before you start: in this guide we write files and then import them, and Python remembers everything it has already imported. So if you run notebook cells out of order, you might see something different from the outputs shown here. If that happens, restart the kernel and run all the cells from the top (in JupyterLab: Kernel → Restart Kernel and Run All Cells). Part 1 explains why this happens.

Part 1: What Is a Module?

A module is simply a file of Python code that other code can use. Any file that ends in .py is a module. The import statement is how one file uses the code inside another file.

Analogy: think of a module as a recipe book on a shelf. You don't rewrite the recipe for bread every time you want bread. You take the book off the shelf, open it, and use the recipe that is already written. import is the act of taking the book off the shelf.

Python comes with hundreds of these books already on the shelf. That's why you could write import csv without ever creating a file called csv.py yourself.

Here's one in action:

import math

math.sqrt(144)

12.0

One line, and you get a square-root function that you didn't have to write. Someone else wrote it, tested it, and shipped it with Python, and you get to use it for free.

A Module Is Just an Object

There's no magic here. Once you import a module, it becomes an object, just like a list or a string is an object. The functions and values inside the module are its attributes, which is why you reach them with a dot (math.sqrt).

You can check this yourself. It also explains why the dir() and help() tools work on modules in the same way they work on everything else:

print(type(math))          # a module really is just an object
print(math.__name__)       # every module knows its own name
print("sqrt" in dir(math))  # dir() lists the names a module contains

<class 'module'>
math
True

Let's read each line of that output:

• type(math) says <class 'module'>. So a module is a real Python object with its own type.
• math.__name__ says math. Every module knows its own name.
• "sqrt" in dir(math) says True. dir() gives a list of all the names inside the module, and sqrt is one of them.

A Module Is Also a Real File

A module is not only an object in memory. It is also a real file sitting on your computer. Every module that came from a .py file remembers where that file is, in an attribute called __file__:

import statistics
from pathlib import Path

print(Path(statistics.__file__).name)   # the full path differs per computer, so print just the file name

statistics.py

So the statistics module is a file called statistics.py, sitting somewhere inside your Python installation. If you print statistics.__file__ by itself, you'll see the full path. You can even open that file in a text editor if you're curious. Inside, you'll find normal Python code: functions, comments and all, written the same way you write yours.

A few modules that need to be really fast, like math, are written in another language called C and then compiled. So there's no math.py file to open. You still import them and use them in exactly the same way, so for now you can ignore that difference.

Looking Around Inside a Module

Two built-in tools let you explore a module you don't know yet, without leaving your editor:

• dir(module) lists every name inside it.
• help(module.thing) prints the documentation for one of those names.

Try help(math.floor) in a cell to see what it looks like. When you're wondering "what does this function do?", it's often quicker than searching online.

Modules You've Already Used

Here are the main modules used so far in this course, and what each one was used for:

None of these are built into the Python language itself. They all live in modules that come with Python, and import is how you get to them.

Three Ways to Import

You'll see three import styles all the time. The only real difference between them is what name you end up with in your file.

Analogy: imagine borrowing from a friend's toolbox. You can borrow the whole toolbox and say "the hammer from the toolbox" each time (style 1). You can borrow just the hammer and call it "hammer" (style 2). Or you can borrow the whole toolbox but give it a short nickname (style 3).

   import math                 from math import floor         import math as m
   ┌──────────────┐            ┌──────────────┐               ┌──────────────┐
   │ math         │            │              │               │ m            │
   │  ├─ pi       │            │   floor      │               │  ├─ pi       │
   │  ├─ floor    │            │              │               │  ├─ floor    │
   │  └─ sqrt ... │            │              │               │  └─ sqrt ... │
   └──────────────┘            └──────────────┘               └──────────────┘
   use: math.floor(9.7)        use: floor(9.7)                use: m.floor(9.7)

Style 1: import module. You get the whole module, and you use a dot to reach inside it.

import math

print(math.pi)
print(math.floor(9.7))

3.141592653589793
9

Style 2: from module import name. You get one specific thing (or a few things), and you don't need the dot anymore.

from math import pi, floor

print(pi)
print(floor(9.7))

3.141592653589793
9

Style 3: import module as alias. This is the same as style 1, but you give the module a shorter name.

People don't do this just to save typing. Some library names are long, and in data science everyone uses the same short names, so you'll see import pandas as pd and import numpy as np in pretty much every tutorial and every job. Using the same nicknames as everyone else makes your code easier for others to read.

import statistics as stats

stats.mean([10, 20, 30, 40])

25

The Fourth Style You Should Avoid: from module import *

This pulls everything from the module straight into your file. It looks handy, but it causes real bugs:

from math import *      # don't do this

The problem is that you can no longer tell where any name came from. If two modules both have a function called sqrt, the one you imported last quietly wins, and Python doesn't warn you. Even worse, if you wrote your own function called floor, from math import * will replace it without telling you.

Analogy: it's like emptying a whole box of someone else's tools onto your workbench, on top of your own tools. Some of their tools have the same names as yours. Now, when you reach for "the hammer", you can't be sure whose hammer you're holding.

Here's that bug in just a few lines. We write our own floor() that chops the decimals off a price, and then an innocent-looking import * replaces it:

def floor(price):
    return int(price)           # our version: just chop the decimals off

print("our floor:", floor(-9.99))

from math import *              # looks harmless -- but math has a floor() too

print("after import *:", floor(-9.99))

our floor: -9
after import *: -10

We called the same function name twice and got two different answers, with no warning at all:

• Our floor() uses int(), which chops the decimal part off and moves toward zero, so -9.99 becomes -9.
• math.floor() always rounds down (toward negative infinity), so -9.99 becomes -10.

In a short cell you'd spot this right away. In a 300-line file, with the import * at the top and the call at the bottom, it's really hard to find. So keep things clear. When you come back to your code in a few months, you'll want to see where every name came from.

So Which Style Should You Use?

If you're not sure, just use import module. Then you can always tell where a name came from.

What import Does Behind the Scenes

When you write import statistics, Python does three things, in this order:

1. It finds the module. It looks through a list of folders, one by one, for a file called statistics.py (or a compiled module with that name). A few modules, like sys, are built right into Python itself, so there's nothing to look for.
2. It runs the file once, from top to bottom. Every line runs. Functions get defined, and any code sitting at the top level of the file (not inside a function) runs too.
3. It caches the module, which means it saves it for later. The finished module goes into a dictionary called sys.modules. If you import it again, Python hands you the saved copy. That's instant, and the file does not run a second time.

In fact, Python checks that cache before step 1. If the module is already in sys.modules, it skips finding and running altogether, as the diagram below shows.

   import statistics
         │
         ▼
   ┌─────────────────────────────┐
   │ Already in sys.modules?     │── YES ──► hand back the saved copy (file is NOT run again)
   └─────────────────────────────┘
         │ NO
         ▼
   1. FIND    search the folders in sys.path for statistics.py
         │
         ▼
   2. RUN     execute the file once, top to bottom
         │
         ▼
   3. CACHE   save the finished module in sys.modules
         │
         ▼
   the name "statistics" now points to the module

Analogy: think of a library. The first time you ask for a book, the librarian searches the shelves, finds it, and photocopies it for you. She also keeps a copy on her desk. The next time anyone asks for that same book, she just hands over the copy from her desk. She doesn't search the shelves again.

That third point (the caching) surprises a lot of people. We'll prove it in Part 3 with a module of our own, where you can watch it happen. For now, just know that the cache is a real dictionary, and you can look inside it:

import sys

print("statistics" in sys.modules)               # it is in the cache...
print(sys.modules["statistics"] is statistics)   # ...and the cached object is the very module we are using

True
True

The first True says statistics is in the cache. The second True says the object in the cache is (the very same object as) the statistics module we've been using.

Why Caching Matters

Caching does more than make imports fast. Within one running program, a module only runs once, no matter how many files import it. So any setup it does at the top level (like opening a connection, loading a settings file or building a lookup table) happens one time, and everything that imports the module shares it.

It also has a side effect you need to know about. If you change a module while Python is still running, importing it again won't pick up your changes. This is the most common mix-up in notebooks: you fix a bug in your .py file, run the import again, and nothing changes. To fix it, restart the kernel (or use importlib.reload). It's also why this guide works best when you run everything from the top after a restart, like the note at the beginning says.

Where Python Looks: sys.path

The list of folders Python searches is called sys.path. Python goes through it in order and stops at the first match. (Built-in modules like sys are found before this search even starts, because they live inside Python itself.)

   sys.path  (searched top to bottom, first match wins)

   1. the folder your script is in      ◄── checked FIRST
   2. folders from the PYTHONPATH setting (if you set any)
   3. the standard library folders
   4. site-packages (where pip installs third-party packages)

The first folder on the list is the one your script is in (for a notebook, the folder you opened it from). That's why a module sitting right next to your file just works, with no setup. The same rule also causes one of the most common beginner bugs, and we'll trigger it on purpose in Part 7.

We'll need sys.path in a minute, because we're about to put a module inside a subfolder.

Part 2: Writing Your Own Module

Once a project grows past a couple of hundred lines, keeping everything in one file gets uncomfortable. You scroll up and down all day looking for things. The fix is to move related functions into their own file and import them.

Here's a good rule for deciding how to split things: group code by what it's about, not by what kind of thing it is. A file called text_utils.py with all your text functions is easy to find your way around. A file called functions.py with everything in it is the same mess you started with, just in a different file.

Analogy: in a kitchen, you keep spices in one drawer and cutlery in another. You don't make one giant drawer labelled "small things". Each drawer has one clear purpose, so you know where to look.

To keep things tidy, everything we make in this guide goes into a folder called module_demo/. First we delete that folder if it's already there, so every run starts with an empty folder:

import shutil
from pathlib import Path

demo = Path("module_demo")
if demo.exists():
    shutil.rmtree(demo)          # start clean, so re-running gives the same output
demo.mkdir()

print(f"{demo}/ is ready")

module_demo/ is ready

Now we'll write a real module. The %%writefile line at the top of the next cell is a Jupyter shortcut. Instead of running the cell, it saves the rest of the cell into a file. It's an easy way to make a .py file without leaving the notebook.

Everything under that first line is normal Python, exactly what you'd type into text_utils.py in a regular editor:

%%writefile module_demo/text_utils.py
# Small text helpers, the kind every project accumulates.

def clean_title(title):
    # "  the   MATRIX  " -> "The Matrix"
    return " ".join(title.split()).title()


def slugify(title):
    # "The Matrix Reloaded" -> "the-matrix-reloaded"
    return "-".join(clean_title(title).lower().split())


def shorten(text, limit=30):
    # Cut long text down for a preview, without cutting mid-word.
    if len(text) <= limit:
        return text
    return text[:limit].rsplit(" ", 1)[0] + "..."

Writing module_demo/text_utils.py

Let's understand the three functions before we use them:

• clean_title(title) uses title.split() to break the text into words (this also throws away all the extra spaces), then " ".join(...) glues the words back together with single spaces, and .title() capitalises each word. So " the MATRIX " becomes "The Matrix".
• slugify(title) first cleans the title, then lowercases it and joins the words with hyphens. A "slug" is the web-friendly version of a title that you see in URLs.
• shorten(text, limit=30) returns the text as it is if it's short enough. Otherwise it cuts the text at the limit, then uses rsplit(" ", 1)[0] to keep only the part before the last space. That throws away the word that got cut in half, and then "..." is added at the end. So "...processing genuinely pleasant" cut at 30 characters ends in "processing g", the stray g is dropped, and you get "Python makes text processing...".

The file is on disk now. But to import it, Python has to be able to find it, and module_demo/ is a subfolder, not the folder we're working in. So we add that folder to sys.path, the list of places Python searches:

import sys

sys.path.insert(0, "module_demo")

import text_utils

print(text_utils.clean_title("  the   MATRIX  "))
print(text_utils.slugify("The Matrix Reloaded"))
print(text_utils.shorten("Python makes text processing genuinely pleasant", 30))

The Matrix
the-matrix-reloaded
Python makes text processing...

All three functions work, exactly as if they had come with Python.

About that sys.path.insert line: you wouldn't need it in a normal project. If text_utils.py were in the same folder as the file importing it, the import would just work, because Python always searches the script's own folder first. We only need it here because we put the module in a subfolder on purpose, to keep things organised. insert(0, ...) puts our folder at the front of the list, so it's searched first.

The Mysterious __pycache__ Folder

Now look inside module_demo/ again. There's something in there that we didn't write:

import os

sorted(os.listdir("module_demo"))

['__pycache__', 'text_utils.py']

That __pycache__/ folder holds bytecode, saved to disk. Bytecode is the simpler, lower-level form of your code that Python actually runs. To import text_utils.py, Python had to translate it into bytecode first, and it kept that translation so it can skip the step next time.

You'll see one of these folders next to any module you import. You can safely delete it (Python just makes it again), and you never need to share it with anyone.

If you use VS Code, it might underline import text_utils and say "Import could not be resolved". VS Code checks your code by reading it, not by running it, so it never sees the sys.path.insert line that tells Python where to look. Your code is fine. In a normal project, where the module sits next to the file that imports it, you won't get this warning.

Your Modules Follow the Same Rules

Your own modules follow exactly the same import rules as math, so the from ... import ... style works here too:

from text_utils import slugify, shorten

print(slugify("Breaking Bad Season Five"))
print(shorten("A very long description that needs trimming for the card", 25))

breaking-bad-season-five
A very long description...

So the standard library isn't special. statistics.py is a file someone wrote and shipped with Python. text_utils.py is a file you wrote and put in a folder. Python imports both of them the same way.

Every module also knows its own name. Python stores it automatically in a variable called __name__:

text_utils.__name__

'text_utils'

Keep this __name__ variable in mind. It's the key to the next part.

Part 3: if __name__ == "__main__"

This is one of the most common patterns in Python. You'll find it at the bottom of thousands of files. It exists because of something we saw earlier: importing a module runs every line in it.

For function definitions, that's fine. Defining a function doesn't do anything visible; it just creates the function. But for anything else, it's a problem. Watch what happens when a module has a print sitting at the top level:

%%writefile module_demo/greet.py
def welcome(name):
    return f"Welcome back, {name}!"

# This line is NOT inside a function, so it runs the moment the file is imported.
print("*** greet.py is running ***")

Writing module_demo/greet.py

import greet

print(greet.welcome("Shikhar"))

*** greet.py is running ***
Welcome back, Shikhar!

See the *** greet.py is running *** line? We only wanted the welcome function, but importing the module also ran its print. Imagine if that line was something slow, like downloading a file or training a model. Every program that imported this module, even one that only wanted a tiny helper function, would have to sit through that work.

Proof That Modules Are Cached

Now for the proof promised in Part 1. Import the same module a second time:

# Already in sys.modules, so the file does not run a second time
import greet

print("...and this time the module did not run.")

...and this time the module did not run.

No *** greet.py is running *** this time. Python found greet already in sys.modules and gave us the saved copy without opening the file again. (If you run the cell above again, the same thing happens. You'll only see that line again after you restart the kernel.)

One File, Two Jobs

So how do you write a file that works both as a module you can import and as a script you can run by itself?

That's what __name__ is for. Python sets this variable automatically in every file, and that includes your notebook. Run this to see what your own code is called:

print(__name__)

__main__

__main__ means "this is the program that's running right now", and a notebook is exactly that.

Python sets __name__ to:

• "__main__" when you run the file directly, and
• the module's own name when the file is imported by another file.

   python3 report.py                      import report   (from another file)
          │                                       │
          ▼                                       ▼
   __name__ == "__main__"                  __name__ == "report"
          │                                       │
          ▼                                       ▼
   the if-block RUNS                       the if-block is SKIPPED
   (demo code executes)                    (only the functions are loaded)

Analogy: think of a chef who also writes cookbooks. When the chef is in their own restaurant (the file is run directly), they cook full meals for guests. When someone just borrows their cookbook (the file is imported), nobody wants the chef to start cooking in their kitchen. They only want the recipes.

So by checking __name__, you can mark a block of code that only runs when the file is run as a program:

%%writefile module_demo/report.py
def summarise(titles):
    return f"{len(titles)} titles, longest is '{max(titles, key=len)}'"


if __name__ == "__main__":
    # Only runs when you execute:  python3 report.py
    # Skipped entirely when another file does:  import report
    demo_titles = ["Stranger Things", "Wednesday", "The Crown"]
    print(summarise(demo_titles))

Writing module_demo/report.py

Import it, and the code inside the if block doesn't run. We only get what we asked for:

import report

print(report.summarise(["Dune", "Arrival"]))

2 titles, longest is 'Arrival'

Now let's run the same file as a script.

The ! at the start of the next cell tells Jupyter to send the rest of the line to your computer's terminal instead of to Python. (The terminal is the window where you type commands; Part 6 explains it properly.) So !python3 module_demo/report.py does exactly what typing python3 module_demo/report.py into a terminal does: it starts a brand-new Python and runs that file as the main program.

On Windows, the command is usually python (or py) instead of python3. If the cell below says python3 wasn't found, change it to !python module_demo/report.py.

!python3 module_demo/report.py

3 titles, longest is 'Stranger Things'

It's the same file, but it behaves differently depending on how you use it. When imported, it quietly offers its summarise function. When run directly, it also runs its little demo. That's why you'll see if __name__ == "__main__": at the bottom of so many Python files.

A simple rule to follow: put your functions and classes at the top level, and put anything that actually does something inside the if block.

The main() Function Pattern

When files get bigger, people often move the code from inside the if block into a function called main(). Then the bottom of the file is just two lines:

def main():
    ...                      # everything the script does when it is run

if __name__ == "__main__":
    main()

It works exactly the same way. It just keeps the bottom of the file tidy, and it gives the "run me" code a clear name. The project at the end of this guide uses this pattern.

Part 4: Packages, Folders of Modules

One file is a module. A folder of modules is a package.

Say your project has fifteen helper functions. Splitting them into text_utils.py, date_utils.py and money_utils.py helps, but now your project folder is getting crowded. A package puts related modules together under one name.

Analogy: if a module is a single recipe book, a package is a labelled shelf holding several related books. Instead of having all your books scattered on the floor, you put the "Italian cooking" books together on one shelf with a label.

A folder becomes a package when it has a file called __init__.py inside it. The file can be completely empty. Just having it there marks the folder as a package.

To be exact, Python 3.3 and newer will import a folder even if it has no __init__.py. It treats it as a namespace package, which is a feature meant for very large projects spread over several folders. So if you forget the file, you usually won't get an error. Add it anyway. It's what makes a folder a normal package, and it's where the package's own setup code goes. We'll use that in a minute.

Here's the package we're about to build:

   module_demo/
   └── netflix/              ← the package (a folder)
       ├── __init__.py       ← marks the folder as a package
       ├── catalog.py        ← a module: formats runtimes
       └── ratings.py        ← a module: averages and labels scores

First, the folder:

Path("module_demo/netflix").mkdir(exist_ok=True)   # exist_ok: no error if the folder is already there

print("created module_demo/netflix/")

created module_demo/netflix/

Then the __init__.py file that marks it as a package. For now it only contains comments:

%%writefile module_demo/netflix/__init__.py
# This file marks the folder as a package.
# It can be empty. Code placed here runs when the package is imported.

Writing module_demo/netflix/__init__.py

Now the first module inside the package. divmod(minutes, 60) divides by 60 and gives back two numbers at once: the whole hours, and the minutes left over. So divmod(154, 60) gives (2, 34):

%%writefile module_demo/netflix/catalog.py
def format_runtime(minutes):
    hours, mins = divmod(minutes, 60)
    if hours == 0:
        return f"{mins}m"
    return f"{hours}h {mins}m"

Writing module_demo/netflix/catalog.py

And the second module. average() returns 0.0 for an empty list (so we never divide by zero), and label() turns a number into a friendly description:

%%writefile module_demo/netflix/ratings.py
def average(scores):
    if not scores:
        return 0.0
    return round(sum(scores) / len(scores), 2)


def label(score):
    if score >= 4.5:
        return "Critically acclaimed"
    elif score >= 3.5:
        return "Well received"
    return "Mixed reviews"

Writing module_demo/netflix/ratings.py

Now you can import from the whole folder as one unit. Write the package name, then the modules you want from it:

from netflix import catalog, ratings

print(catalog.format_runtime(154))
print(catalog.format_runtime(47))

scores = [4.8, 4.6, 4.9, 4.5]
print(ratings.average(scores), "->", ratings.label(ratings.average(scores)))

2h 34m
47m
4.7 -> Critically acclaimed

You can also go straight to one function. This is the style you'll see most often. The dot in netflix.ratings separates the package from the module inside it, a bit like / separates a folder from a file in a path:

from netflix.ratings import label

label(3.9)

'Well received'

What Goes in __init__.py?

Ours is (nearly) empty, and that's completely normal. But this file runs when the package is imported, so you can use it as a front door for the package.

Instead of making people remember which module each function lives in, you import the useful functions into __init__.py, and then people can get them straight from the package.

Analogy: think of a big shop with many departments. You could tell customers "go to aisle 7, second shelf" for every product. Or you could put the most popular items right at the front entrance. __init__.py is that front entrance.

That's exactly why import pandas as pd gives you pd.DataFrame directly, even though DataFrame is actually defined several files deep inside the pandas package.

%%writefile module_demo/netflix/__init__.py
# Pull the useful names up to the package level, so users can write
#     from netflix import average
# instead of having to know that average() lives in netflix/ratings.py
from netflix.ratings import average, label
from netflix.catalog import format_runtime

Overwriting module_demo/netflix/__init__.py

To see the change, we have to run this in a fresh Python process. This notebook already imported netflix, and as we saw earlier, Python saves modules and won't run the file again.

This new script doesn't need a sys.path line, because it lives inside module_demo/, right next to the package it imports:

%%writefile module_demo/front_door.py
import netflix          # not netflix.ratings -- the package hands us the names directly

print(netflix.format_runtime(96))
print(netflix.label(4.7))

Writing module_demo/front_door.py

!python3 module_demo/front_door.py

1h 36m
Critically acclaimed

Same functions, shorter import. Now the package has one simple front door, and you can move the files behind it around without breaking anyone's code. That's the main reason big libraries do this.

The New Words, in One Place

Part 5: The Standard Library, Batteries Included

For years, Python's slogan was "batteries included". Like a new toy that comes with batteries in the box, Python comes with a big collection of useful modules. So even a fresh install can read CSV files, work with dates, do maths, and manage files and folders, without installing anything else.

You've already used more than a dozen of these modules. Here's a new one, random:

import random

random.seed(42)                     # fixes the "randomness" so results repeat

print(random.randint(1, 100))
print(random.choice(["Netflix", "Spotify", "Amazon"]))

82
Netflix

Why random.seed() Matters

Random numbers on a computer aren't truly random. They come from a formula that starts from some value, called the seed. Setting the seed fixes that starting value, so you get the same sequence of "random" numbers every time you run the code.

Analogy: it's like shuffling a deck of cards in exactly the same way every time. The order looks random, but if you start from the same position and shuffle the same way, you always get the same order.

This is more useful than it sounds. It's how machine learning experiments are made repeatable, so another person can run your code and get exactly your results. It's also why you'll see random_state=42 all over scikit-learn.

Doing Maths with Dates

Next, datetime. You may have only used it to stamp the current time, but it can also do maths with dates. Subtracting one date from another gives you the gap between them:

import datetime

release = datetime.date(2024, 3, 15)
report_date = datetime.date(2024, 6, 1)   # a fixed date, so this cell prints the same thing every time

print("Released:", release.strftime("%d %B %Y"))   # %d = day, %B = month name, %Y = year
print("Days since release:", (report_date - release).days)

Released: 15 March 2024
Days since release: 78

strftime() (short for "string format time") turns a date into text in whatever format you like. %d is the day, %B is the full month name, and %Y is the four-digit year. And (report_date - release).days gives the number of days between the two dates.

A Rough Map of the Standard Library

Some of these get their own proper coverage later in the course: re in chapter 16, and collections, itertools, functools and datetime in chapter 15.

Part 6: Third-Party Packages and Virtual Environments

The standard library is big, but it doesn't have everything. There's no pandas, no requests and no scikit-learn in it. Those are third-party packages: code written by other people and shared on the Python Package Index, or PyPI (you say it "pie-pea-eye"), at pypi.org. The tool that downloads and installs them is called pip, and it comes with Python.

Analogy: if the standard library is the set of tools that came with your house, PyPI is a huge free hardware store with hundreds of thousands of tools made by people all over the world. pip is the delivery van that brings the tools you order to your house.

Once a third-party package is installed, you import it like anything else:

import requests                   # no different from "import csv"

Python doesn't care who wrote a module. The standard library, your own file, something you installed: it's the same import, with the same rules.

What causes trouble is where an installed package ends up. That's what the rest of this part is about.

Why Every Project Gets Its Own Environment

Say you build a project in March using version 2.28 of requests. In June you start a new project that needs version 2.31. If both projects use the same Python installation, then upgrading for the June project means the March project is now running on a version you never tested it with. Now imagine that with twenty packages and several projects. Things start breaking each other.

A virtual environment is a private set of packages for one project, kept apart from everything else. Each project gets its own folder of packages, so nothing leaks from one project into another.

Analogy: imagine several chefs sharing one kitchen and one spice rack. One chef swaps the salt for a new brand, and suddenly another chef's famous dish tastes wrong. Now give every chef their own small kitchen with their own spice rack. Each chef can change anything they like, and nobody else is affected. Each small kitchen is a virtual environment.

   WITHOUT virtual environments            WITH virtual environments

   ┌───────────────────────────┐           project_march/        project_june/
   │   one shared Python       │           ┌──────────────┐      ┌──────────────┐
   │   requests 2.31  ◄── June │           │ .venv/       │      │ .venv/       │
   │                   upgrade │           │ requests 2.28│      │ requests 2.31│
   │   (March project now      │           └──────────────┘      └──────────────┘
   │    runs on an untested    │             each project has its own packages,
   │    version!)              │             so neither can break the other
   └───────────────────────────┘

On a lot of systems, this isn't even optional anymore. If you installed Python with Homebrew on macOS, or you're on a recent version of Ubuntu or Debian, a plain pip install outside a virtual environment gets refused with an externally-managed-environment error. That's there to stop projects from overwriting each other, and to protect the operating system's own tools, some of which are written in Python.

So always do it in this order: first create the environment, then install packages into it.

A Quick Introduction to the Terminal

You create environments in the terminal. It's a window where you type commands for your computer instead of clicking on things. All the commands in the rest of this part go there, not in a notebook cell or a .py file.

How to open one:

A terminal is always "in" one folder at a time. That folder is called the working directory, and your commands act on it. Before doing anything else, use cd (short for change directory) to go into your project folder:

cd path/to/Python-Programming-Code      # go to the project folder
pwd                                     # print the folder you are in now

The text before your cursor, something like ~/Python-Programming-Code $, is called the prompt. It's just the terminal telling you it's ready for a command. You don't type it yourself.

python3 or python? On macOS and Linux, the command is python3. On Windows it's usually python or py. Wherever this guide says python3, use the one that works on your computer.

Creating and Using a Virtual Environment

Here is the whole life cycle of an environment, in four steps:

# 1. Create it, inside your project folder (this makes a folder called .venv)
python3 -m venv .venv

# 2. Activate it
source .venv/bin/activate          # macOS / Linux
.venv\Scripts\activate             # Windows

# 3. Your prompt now starts with (.venv): installs go into .venv/, and only there
python3 -m pip install requests

# 4. When you are done
deactivate

Let's go through it slowly:

1. Create. python3 -m venv .venv makes a new folder called .venv inside your project. It contains its own python3 command (on macOS and Linux this is a shortcut, or link, to your main Python program; on Windows it's a small copy) and a private folder for packages, which already has pip installed in it. The dot at the start of .venv is just a convention that hides the folder in file browsers on macOS and Linux.
2. Activate. Activating tells this terminal window to use the environment's Python and pip. You'll know it worked because your prompt changes to start with (.venv).
3. Install. Anything you install now goes into .venv/, and only there.
4. Deactivate. deactivate switches the terminal back to normal. Nothing is deleted; you can activate it again later.

What does the -m mean? The -m in python3 -m venv and python3 -m pip means "run the module called...". It tells this exact Python to run its own venv or pip module.

This matters because a plain pip command might belong to a different Python than the one you meant (and on some Macs, plain pip doesn't exist at all). That's the classic way people end up installing a package somewhere their code can't see it. python3 -m pip always installs for the Python you typed. Once an environment is active, plain pip install ... also points to the right place, and most tutorials use it.

While the environment is active, python3 means the environment's own Python. It can see the standard library and whatever you installed into .venv/, and nothing installed anywhere else. So nothing you do here affects your other projects.

On Windows PowerShell, activating might fail with an error about execution policies. Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once, type Y when it asks, and then activate again.

The pip Commands You'll Use Most

All of these are run with the environment active:

python3 -m pip install requests              # install the latest version
python3 -m pip install requests==2.31.0      # install one exact version
python3 -m pip install --upgrade requests    # upgrade to the newest
python3 -m pip uninstall requests            # remove it
python3 -m pip list                          # everything installed in this environment
python3 -m pip show requests                 # details about one package

Your Notebook and Your Environment

This one trips up almost everyone. When you activate .venv in a terminal, it changes what that terminal runs. It does not change the Python behind a notebook that's already open. So you can install a package in the terminal, import it in the notebook, and get ModuleNotFoundError for a package you installed a minute ago.

Analogy: it's like stocking a new spice in the kitchen next door, then being surprised you can't find it in your own kitchen.

There are two ways to keep them in sync:

• Start Jupyter from inside the environment. With .venv active, run python3 -m pip install jupyterlab, then jupyter lab. Any notebook you open this way runs on the environment's Python.
• In VS Code, click the kernel name in the top-right corner of the notebook and pick the interpreter inside .venv.

If you want to install something from inside a notebook, run %pip install requests in a cell. Unlike !pip, %pip always installs into the Python that the notebook is running on.

If you're ever unsure which Python you're using, ask Python itself:

import sys

print(sys.executable)                                            # the Python program running this notebook
print("Inside a virtual environment:", sys.prefix != sys.base_prefix)

/opt/anaconda3/bin/python
Inside a virtual environment: False

Your output will be different from the one shown here, because this one came from the author's computer.

• The first line is the full path to the Python program running your code. If it points inside your project's .venv folder, your notebook and your environment match.
• The second line compares where this Python keeps its packages (sys.prefix) with where the original Python installation keeps them (sys.base_prefix). They're only different when you're inside a virtual environment.

(Conda environments, which come up below, are kept apart too, but they still print False here, because conda sets up its environments in a different way.)

Sharing an Environment: requirements.txt

An environment isn't much use if nobody else can rebuild it. You can't just email someone your .venv folder (more on why in a moment). Instead, the usual fix is a plain text file called requirements.txt that lists what the project needs.

Analogy: instead of mailing someone your whole kitchen, you send them a shopping list. With the list, they can stock their own kitchen exactly like yours.

You can make one in two ways.

1. Write it by hand, listing only the packages your code imports. That's what the course repository does. Here are the packages that chapters 17 to 19 need:

from pathlib import Path

for line in Path("requirements.txt").read_text().splitlines():
    if line and not line.startswith("#"):      # skip blank lines and comments
        print(line)

requests>=2.31
beautifulsoup4>=4.12
pytest>=8.0
mypy>=1.8

The loop reads the file line by line and skips blank lines and comments (lines starting with #). In these lines, >= means "this version or newer". Using == instead would lock the package to one exact version.

2. Save a list of everything that's installed:

python3 -m pip freeze > requirements.txt

pip freeze prints every package in the environment with its exact version. The > is the terminal's way of saying "put the output in this file instead of showing it on screen".

You'll get more lines than you expect, because packages bring their own dependencies (other packages they need) along with them. If you install just requests and beautifulsoup4, you get something like:

beautifulsoup4==4.12.3
certifi==2024.2.2
charset-normalizer==3.3.2
idna==3.7
requests==2.31.0
soupsieve==2.5
urllib3==2.2.1

A hand-written file is easier to read, and a frozen file rebuilds the environment exactly. You'll see both in real projects.

For someone else (or you, later) to rebuild the environment:

python3 -m venv .venv
source .venv/bin/activate                  # Windows: .venv\Scripts\activate
python3 -m pip install -r requirements.txt

Those three commands give them the same packages the project was built with. The -r flag tells pip to read the package list from a file.

For this course: chapters 12 to 16 don't need anything installed. Chapter 17 is the first one that needs one of the packages above, and it walks you through this setup again when you get there. If you'd like to get it done now, open a terminal in the folder that has the course notebooks, run those three commands, then run python3 -m pip install jupyterlab, and open the course again with jupyter lab.

Two Rules to Follow from Day One

1. Share requirements.txt, but never .venv/. The file is just a few lines. The folder has thousands of files, is tied to your operating system, and you can rebuild it from the file any time. Most programmers keep their projects in git, a tool that keeps a history of a folder and can upload it to sites like GitHub. Saving a version in git is called a commit. A file called .gitignore lists what git should skip, and .venv/ and __pycache__/ should always be in it.
2. Use one environment per project. They're cheap to make, so delete them and rebuild them whenever you like. That's the whole point.

You might also come across conda. It does the same job and is popular in data science, because it can also install tools that aren't written in Python. The idea is the same: separate environments, one per project.

Part 7: When an Import Fails

Most import problems come down to one of three errors, and Python tells you which one it is, as long as you read the message carefully.

   Your import failed. Which error did you get?
   │
   ├── ModuleNotFoundError: No module named 'x'
   │       └── Python can't find the module at all
   │           → typo? not installed? installed into a different Python?
   │
   ├── ImportError: cannot import name 'y' from 'x'
   │       └── the module exists, but 'y' isn't inside it
   │           → check the real name with dir(x) or help(x)
   │
   └── AttributeError: module 'x' has no attribute 'z'
           └── (when you're SURE x has z) one of YOUR files is named x.py
               → rename your file

Error 1: ModuleNotFoundError

This means Python can't find the module at all.

try:
    import panads
except ModuleNotFoundError as e:
    print(type(e).__name__, "->", e)

ModuleNotFoundError -> No module named 'panads'

This usually means one of three things:

1. There's a typo in the name, like panads instead of pandas above.
2. It isn't installed, so you need python3 -m pip install <name>, or %pip install <name> in a notebook.
3. It's installed somewhere else. Most often it went into a different Python or environment from the one running your code. sys.executable, from Part 6, tells you which Python that is.

Error 2: ImportError

This means Python found the module, but the name you asked for isn't in it.

try:
    from math import square_root      # math has sqrt, not square_root
except ImportError as e:
    print(e)

cannot import name 'square_root' from 'math' (/opt/anaconda3/lib/python3.12/lib-dynload/math.cpython-312-darwin.so)

Python found math, and it even tells you which file it loaded (the part in brackets will look different on your computer). But there's nothing in it called square_root. If you check with dir(math) or help(math), you'll find the real name is sqrt.

These Two Errors Are Related

ModuleNotFoundError is a more specific kind of ImportError, in the same way that ZeroDivisionError is a kind of ArithmeticError. In other words, ModuleNotFoundError is a child class of ImportError. So except ImportError catches a missing module too:

try:
    import panads
except ImportError as e:                                  # the parent class...
    print("Caught as ImportError:", type(e).__name__)    # ...catches the child

Caught as ImportError: ModuleNotFoundError

   ImportError                  ← "something went wrong while importing"
   └── ModuleNotFoundError      ← "...specifically, the module wasn't found"

Making an Extra Package Optional

A failed import is just an exception, so you can catch it like any other exception. Libraries use this trick to offer extra features that only switch on if you have another package installed:

try:
    import fancy_plotting_library   # a made-up name: no package by that name exists
    HAS_PLOTTING = True
except ImportError:
    HAS_PLOTTING = False

if HAS_PLOTTING:
    print("Charts enabled.")
else:
    print("Charts disabled. Install with: python3 -m pip install fancy_plotting_library")

Charts disabled. Install with: python3 -m pip install fancy_plotting_library

The program doesn't crash. It just turns off the charts feature and tells the user how to turn it on. Many real libraries work this way: they run fine with the basics, and unlock extras when an optional package is present.

Error 3: One of Your Own Files Is Hiding a Real Module

Remember, Python looks in your script's own folder first. So if you save a file called random.py, every import random in that folder finds your file instead of the real one from the standard library. Beginners run into this all the time, and the error you get never mentions your file by default.

Analogy: imagine you put a homemade jar labelled "Salt" on the front of your shelf, but it actually contains sugar. Every recipe that reaches for "Salt" grabs your jar first, because it's the closest one, and the dish comes out wrong. The recipe didn't fail because salt doesn't exist; it failed because the wrong jar had the right label.

Let's cause this bug on purpose, in its own folder:

Path("module_demo/shadow").mkdir(exist_ok=True)

print("created module_demo/shadow/")

created module_demo/shadow/

%%writefile module_demo/shadow/random.py
# A practice file, innocently saved as random.py
print("Running module_demo/shadow/random.py -- not Python's random module!")

Writing module_demo/shadow/random.py

%%writefile module_demo/shadow/dice.py
import random    # meant: Python's random module. Gets: random.py in this folder.

try:
    print(random.randint(1, 6))
except AttributeError as e:      # caught only so the message prints cleanly
    print("AttributeError:", e)

Writing module_demo/shadow/dice.py

!python3 module_demo/shadow/dice.py

Running module_demo/shadow/random.py -- not Python's random module!
AttributeError: module 'random' has no attribute 'randint'

The first line of the output shows that Python ran our random.py, and of course our file has no randint. The error itself doesn't mention the file name at all, and that's why this bug is so confusing the first time you see it.

Python 3.13 and newer are kinder here. They add a hint to the end of the same error message, telling you exactly which file to rename. On those versions, the second line looks like this:

AttributeError: module 'random' has no attribute 'randint' (consider renaming '.../module_demo/shadow/random.py' since it has the same name as the standard library module named 'random' and prevents importing that standard library module)

The fix is easy: don't name your own files after a module you use, like random.py, csv.py, math.py, requests.py or pandas.py. Rename the file (for example, to random_practice.py), and the real module works again.

Part 8: How a Real Project Is Laid Out

Put everything from this guide together, and a small, well-organised Python project looks roughly like this:

netflix_analysis/
├── .venv/                  <- the virtual environment (never committed)
├── .gitignore              <- lists .venv/, __pycache__/, data files
├── requirements.txt        <- the packages this project needs
├── README.md               <- what this project is and how to run it
├── main.py                 <- the entry point you actually run
├── data/
│   ├── raw/                <- data as downloaded, never edited
│   └── processed/          <- cleaned data your code produced
└── netflix/                <- your own package
    ├── __init__.py
    ├── catalog.py
    └── ratings.py

Python doesn't force any of this on you. These are habits that most people follow, and each one has a good reason:

• main.py is the entry point. It's the one file you actually run. It imports from netflix/ and does the actual work, inside if __name__ == "__main__":.
• Nobody edits the files in raw/. If your cleaning code has a bug, you want to be able to start over from the original download. Treat raw data like the original copy of an important document: you make copies, you never write on the original.
• Your own package sits next to main.py, so importing it doesn't need any sys.path tricks. Python searches the script's own folder first, remember?
• requirements.txt and README.md let anyone (including future you) set up and understand the project quickly.

You don't need any of this for a ten-line script. But once a project lasts longer than the afternoon you wrote it in, you'll want it.

Real-World Example: Splitting a Script into a Project

This is the change this whole guide has been building up to.

Here's the kind of script everyone writes first. It's one file that holds the data, cleans up the titles, does the maths, formats the runtimes and prints a report, all mixed together in one loop. It works, but you can't reuse any of it anywhere else, and you can't check any piece of it on its own:

%%writefile module_demo/messy_report.py
# Everything in one file: data, cleaning, maths, formatting and printing.
SHOWS = [
    {"title": "  stranger   THINGS ", "runtime": 51, "scores": [4.8, 4.9, 4.7]},
    {"title": "the   crown", "runtime": 58, "scores": [4.2, 4.0, 4.3]},
    {"title": "  WEDNESDAY", "runtime": 47, "scores": [3.4, 3.6, 3.2]},
]

print("NETFLIX CATALOGUE REPORT")
print("-" * 58)
for show in SHOWS:
    title = " ".join(show["title"].split()).title()
    score = round(sum(show["scores"]) / len(show["scores"]), 2)

    hours, mins = divmod(show["runtime"], 60)
    if hours == 0:
        runtime = f"{mins}m"
    else:
        runtime = f"{hours}h {mins}m"

    if score >= 4.5:
        label = "Critically acclaimed"
    elif score >= 3.5:
        label = "Well received"
    else:
        label = "Mixed reviews"

    print(f"{title:<20} {runtime:>6}  {score:.2f}  {label}")

Writing module_demo/messy_report.py

!python3 module_demo/messy_report.py

NETFLIX CATALOGUE REPORT
----------------------------------------------------------
Stranger Things         51m  4.80  Critically acclaimed
The Crown               58m  4.17  Well received
Wednesday               47m  3.40  Mixed reviews

It works fine. But notice how the cleaning, the maths, the formatting and the labelling are all tangled together inside one loop. If another project needed just the runtime formatting, you'd have to copy and paste it out of here.

Now here's the same report, built from the module and the package we wrote in this guide. The main file now just connects the pieces:

%%writefile module_demo/main.py
# The entry point. Notice how little it does itself: it imports the real work.
# No sys.path line needed here: main.py sits in the same folder as the modules
# it imports, and a script's own folder is always searched first.
import text_utils
from netflix import catalog, ratings

SHOWS = [
    {"title": "  stranger   THINGS ", "runtime": 51, "scores": [4.8, 4.9, 4.7]},
    {"title": "the   crown", "runtime": 58, "scores": [4.2, 4.0, 4.3]},
    {"title": "  WEDNESDAY", "runtime": 47, "scores": [3.4, 3.6, 3.2]},
]


def build_report(shows):
    lines = []
    for show in shows:
        title = text_utils.clean_title(show["title"])
        score = ratings.average(show["scores"])
        lines.append(
            f"{title:<20} {catalog.format_runtime(show['runtime']):>6}  "
            f"{score:.2f}  {ratings.label(score)}"
        )
    return lines


def main():
    print("NETFLIX CATALOGUE REPORT")
    print("-" * 58)
    for line in build_report(SHOWS):
        print(line)


if __name__ == "__main__":
    main()

Writing module_demo/main.py

!python3 module_demo/main.py

NETFLIX CATALOGUE REPORT
----------------------------------------------------------
Stranger Things         51m  4.80  Critically acclaimed
The Crown               58m  4.17  Well received
Wednesday               47m  3.40  Mixed reviews

The output is exactly the same, down to the last character. What changed is how the code is organised. There are now four files, and each one has one clear job:

   main.py  (connects the pieces and prints the report)
     │
     ├── imports ──► text_utils.py          knows about text, nothing about Netflix
     │                 └── clean_title()
     │
     └── imports ──► netflix/  (package)
                       ├── catalog.py       formats runtimes
                       │     └── format_runtime()
                       └── ratings.py       scores things
                             ├── average()
                             └── label()

• text_utils.py knows about text, and nothing about Netflix.
• netflix/catalog.py formats runtimes.
• netflix/ratings.py scores things.
• main.py connects them and prints the report.

What you get out of this is code you can reuse and test. You can use text_utils.clean_title in any project you ever write, and you can check each piece on its own. That's exactly what chapter 19 does with real tests.

Analogy: the messy script is like a meal where everything was cooked in one pot. The organised project is like a kitchen with separate stations: one person chops, one person grills, one person plates. Each station can be improved, checked or reused on its own, and the final plate looks exactly the same.

Quick Reference Summary

Importing

The new words

Virtual environments (in a terminal, inside the project folder)

python3 -m venv .venv                         # create
source .venv/bin/activate                     # activate (macOS/Linux)
.venv\Scripts\activate                        # activate (Windows)
python3 -m pip install -r requirements.txt    # restore from file
deactivate                                    # leave

pip (with the environment active)

python3 -m pip install <package>
python3 -m pip install <package>==<version>
python3 -m pip uninstall <package>
python3 -m pip list
python3 -m pip freeze > requirements.txt

In a notebook, use %pip install <package>. To see which Python is running your code: import sys; print(sys.executable).

When an import fails

The pattern to remember

def main():
    ...

if __name__ == "__main__":
    main()

The code in main() runs when you run the file directly, and stays quiet when the file is imported.