Web Scraping — BeautifulSoup & Playwright
Scrape static HTML with BeautifulSoup, automate dynamic JavaScript pages with Playwright, handle pagination, and respect robots.txt.
Part 1: What You Will Learn
- Parse static HTML with BeautifulSoup.
- Select elements with CSS selectors.
- Automate JavaScript-rendered pages with Playwright.
- Handle pagination responsibly and respect site rules, rate limits, and robots.txt.
Part 2: Key Concepts
Static scraping downloads HTML that already contains the desired data. Dynamic pages may create content with JavaScript after loading; a browser automation tool such as Playwright can render those pages. Always check a site’s terms, robots directives, and rate limits before scraping.
Part 3: Topic-Specific Code Examples
# Install once: pip install beautifulsoup4
from bs4 import BeautifulSoup
html = """
<div class="book" data-id="101">
<h2>Python Made Easy</h2><span class="price">RM49.90</span>
</div>
<div class="book" data-id="102">
<h2>AI Coding Basics</h2><span class="price">RM59.90</span>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
for card in soup.select(".book"):
book_id = card.get("data-id")
title = card.select_one("h2").get_text(strip=True)
price = card.select_one(".price").get_text(strip=True)
print(book_id, title, price)# Install once:
# pip install playwright
# playwright install chromium
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://example.com", wait_until="domcontentloaded")
print("Page title:", page.title())
print("Heading:", page.locator("h1").inner_text())
browser.close()import time
for page_number in range(1, 6):
url = f"https://example.com/catalog?page={page_number}"
print("Would request:", url)
# fetch_and_parse(url)
time.sleep(1) # Be polite; follow the site's actual rules and limits.Part 4: How the Example Works
BeautifulSoup works directly with HTML and CSS selectors. Playwright starts a real browser engine, which is heavier but can access content that appears only after JavaScript executes. For real sites, avoid aggressive loops: request only what you need, pause between pages when appropriate, and obey the site’s published access rules.
Part 5: Hands-On Practice
Mini project — Local Course Catalog Scraper. Create a local HTML file containing six course cards. Parse course name, duration, and fee with BeautifulSoup, convert the results into a list of dictionaries, and save them as CSV using the standard csv module.
Part 6: Next Steps
Run and modify the examples in Visual Studio 2026, then continue to Lesson 29. Return to Python Tutorial Home to review the complete curriculum.