Scrapy

🔢	Topic
1	What Scrapy is good at
2	Debian + dedicated virtualenv setup
3	Project layout & core commands
4	Best practices (performance, ethics, structure)
5	Markdownification pipeline (BeautifulSoup/markdownify idea)
6	/home/dv/scrapez reference toolkit

1️⃣ What Scrapy is good at

• High‑throughput crawling of many pages/sites
• Structured extraction (items, pipelines, middlewares)
• Resilience (retries, backoff, error handling)
• Extensibility (plugins for JS rendering, proxies, storage)

Use Scrapy when you need:

• Repeated crawls (daily/weekly jobs)
• Strong separation of concerns (spider vs pipelines vs storage)
• Politeness controls and observability

2️⃣ Debian + dedicated virtualenv setup

🧱 Recommended pattern (toolbox per directory):

• One project/toolkit directory (e.g. ~/scrapez)
• One virtualenv dedicated to that toolkit (e.g. ~/scrapez/.venv)
• Keep all spiders/pipelines/config inside that directory

Minimal Scrapy toolkit setup (Debian):

mkdir -p ~/scrapez
cd ~/scrapez
 
python3 -m venv .venv
source .venv/bin/activate
 
python -m pip install --upgrade pip
python -m pip install scrapy beautifulsoup4 markdownify
 
# optional: freeze for reproducibility
python -m pip freeze > requirements.txt

Why dedicated virtualenv?

• Isolates Scrapy + deps from system Python
• Lets you pin versions per toolkit
• Easy to nuke/recreate without touching other projects

To activate later:

cd ~/scrapez
source .venv/bin/activate

3️⃣ Project layout & core commands

Canonical Scrapy layout (conceptual):

• scrapy.cfg – project entrypoint (settings mapping)
• myproj/ – Python package
  • __init__.py
  • items.py – item models
  • pipelines.py – processing/storage
  • settings.py – config
  • spiders/ – spiders live here

CLI workflow (inside activated venv):

# create new project in current dir
scrapy startproject myproj .
 
# generate spider
scrapy genspider quotes quotes.toscrape.com
 
# run spider
scrapy crawl quotes
 
# inspect HTML interactively
scrapy shell "https://quotes.toscrape.com/page/1/"

For the /home/dv/scrapez toolkit we directly provide a ready-made Scrapy project instead of running startproject manually.

4️⃣ Scrapy best practices (Debian, FOSS)

• Dedicated venv per toolkit (e.g. ~/scrapez/.venv)
• Run Scrapy only inside activated venv so imports + scripts resolve correctly
• Keep spiders small & focused (one concern/target per spider)
• Push data shaping to pipelines, not spiders (spiders = fetch + minimal extraction)
• Use items to define schema (explicit fields)
• Respect robots.txt and ToS (ROBOTSTXT_OBEY = True)
• Throttle politely:
  • CONCURRENT_REQUESTS lowish (e.g. 8)
  • DOWNLOAD_DELAY > 0
  • AUTOTHROTTLE_ENABLED = True
• Set clear User-Agent with contact email
• Centralize settings per project (not in spider) for easy tuning
• Use Scrapy shell to design selectors before coding
• Log to file for long-running jobs (via -s LOG_FILE=... or settings)

5️⃣ Markdownification (HTML → Markdown) concept

Goal: crawl pages with Scrapy, then convert HTML to Markdown for storage / diffing / git.

Pattern:

• Spider yields item containing:
  • url
  • title
  • raw html (e.g. response.text)
• Pipeline converts to Markdown via library, saves .md file + returns enriched item

Typical tools:

• beautifulsoup4 for HTML cleanup/targeted extraction
• markdownify (Python lib) or html2text for HTML → Markdown

Pipeline idea (conceptual):

• MarkdownPipeline:
  • Build clean HTML snippet (optionally using BeautifulSoup)
  • Convert with markdownify
  • Generate slug from title
  • Write output/<slug>.md
  • Attach markdown field to item

6️⃣ /home/dv/scrapez reference toolkit

This repo is a ready-to-go Scrapy + Markdown toolkit with:

• Dedicated venv location: ~/scrapez/.venv
• Scrapy project: scrapez_project
• Example spider: quotes_markdown scraping quotes.toscrape.com
• Pipeline: converts full HTML pages → Markdown files in output/
• Runner script: run_quotes_markdown.py to launch spider via CrawlerProcess

Layout (once files are created):

• /home/dv/scrapez/
  • requirements.txt (Scrapy, BS4, markdownify)
  • scrapy.cfg
  • run_quotes_markdown.py
  • scrapez_project/
    • __init__.py
    • items.py (PageItem with url/title/html/markdown/timestamp)
    • pipelines.py (MarkdownPipeline → output/*.md)
    • settings.py (Debian-friendly defaults, polite crawling)
    • spiders/
      • __init__.py
      • quotes_markdown.py

Basic usage:

cd ~/scrapez
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
 
# option 1: via Scrapy CLI
scrapy crawl quotes_markdown
 
# option 2: via runner script
python run_quotes_markdown.py
 
# Markdown files will appear under: output/

This cheat sheet + /home/dv/scrapez form a minimal yet powerful Debian-native FOSS toolkit for Scrapy-based HTML → Markdown scraping.