Graph View

Debian Scraping Guide 🕸️

🔢Topic
1Scope, ethics, stack overview
2Debian base setup
3Python stack (Scrapy, Playwright, BS4)
4Node/TS stack (Crawlee, Playwright, Puppeteer)
5CLI/Unix scraping toolbox
6Headless browsers & rendering
7Architecture, robustness, scheduling
8Tool selection matrix & resources

1️⃣ Scope & mental model

🧠 Goal: FOSS-only toolkit for robust, maintainable web scraping on Debian.

Think in layers:

  • Transport: HTTP client or full browser
  • Extraction: HTML parser, selectors (CSS/XPath), data modeling
  • Orchestration: crawling, queues, backoff, retries, persistence
  • Ops: scheduling, monitoring, logging, packaging, containers

⚠️ Web scraping is legally and ethically sensitive. Always:

  • Respect robots.txt and site ToS where applicable
  • Avoid collecting PII or sensitive data without clear legal basis
  • Throttle requests (rate limits, random jitter) to avoid DoS
  • Prefer official APIs when available
  • Log what you access and why (audit trail)

On Debian, add basic protections:

  • Central config for concurrency/ratelimits per target
  • Global User-Agent with contact email
  • Per-target denylist/allowlist configuration

3️⃣ Base Debian setup

📦 System packages (typical starting point):

sudo apt update
sudo apt install -y \
  python3 python3-venv python3-pip python3-dev \
  build-essential libffi-dev libssl-dev \
  libxml2-dev libxslt1-dev zlib1g-dev \
  curl wget git

Optional extras (CLI scraping helpers):

sudo apt install -y jq pup w3m html-xml-utils xmlstarlet parallel

Python venv pattern:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Node.js (for Crawlee / Playwright / Puppeteer) – recommended via nvm:

# install nvm (if not present); then
nvm install --lts
nvm use --lts

4️⃣ Python stack (FOSS)

🕷️ Scrapy – heavy-duty crawler framework

Best for:

  • Large crawls, multi-domain projects, structured pipelines
  • High throughput, robust retrying, extension ecosystem

Install in venv:

python -m pip install scrapy

Minimal project:

scrapy startproject demo_spider
cd demo_spider
scrapy genspider quotes quotes.toscrape.com
scrapy crawl quotes -o quotes.json

Key Scrapy concepts:

  • Spiders: crawl logic (start URLs, parsing)
  • Items: structured data containers
  • Pipelines: post-processing (cleaning, DB/CSV export)
  • Middlewares: cross-cutting concerns (proxies, headers)
  • Settings: concurrency, delays, caching, autothrottle

Scrapy best-practice highlights:

  • Use pipelines for cleaning and persistence; keep spiders focused on extraction
  • Centralize settings per project and per-domain
  • Use DOWNLOAD_DELAY, AUTOTHROTTLE_ENABLED, small CONCURRENT_REQUESTS_PER_DOMAIN
  • For JS-heavy sites, combine with Splash, Playwright, or Selenium-based solutions only where needed

Example spider (single file, quick use):

import scrapy
 
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
 
    def parse(self, response):
        for q in response.css("div.quote"):
            yield {
                "text": q.css("span.text::text").get(),
                "author": q.css("small.author::text").get(),
                "tags": q.css("a.tag::text").getall(),
            }
 
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

📜 Requests + BeautifulSoup / lxml – quick scripts

Best for:

  • Simple pages, ad‑hoc scripts, one-off exports

Install:

python -m pip install requests beautifulsoup4 lxml

Example script:

import time
import random
 
import requests
from bs4 import BeautifulSoup
 
HEADERS = {"User-Agent": "DebianScraper/1.0 (+contact@example.com)"}
 
resp = requests.get("https://example.com", headers=HEADERS, timeout=15)
resp.raise_for_status()
 
soup = BeautifulSoup(resp.text, "lxml")
for link in soup.select("a[href]"):
    print(link["href"].strip())
 
# polite crawling
time.sleep(1 + random.random())

Tips:

  • Use session = requests.Session() to reuse connections
  • Centralize headers, timeouts, error handling

🎭 Playwright for Python – JS-heavy sites

Best for:

  • Modern SPAs, complex JS rendering, infinite scroll

Install:

python -m pip install playwright
playwright install chromium

Minimal example:

from playwright.sync_api import sync_playwright
 
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="networkidle")
    html = page.content()
    browser.close()
 
    # parse html with BeautifulSoup or lxml

Tips:

  • Use headless mode on servers; keep visible for debugging
  • Block analytics/ads via route intercept to reduce noise and load

🧪 Selenium – only if you must

Use when:

  • You need real browser automation semantics (e.g., reusing Selenium infra)

Install:

python -m pip install selenium
sudo apt install -y chromium-driver  # or geckodriver for Firefox

Selenium is heavier and slower than Playwright; prefer Playwright for new projects.

5️⃣ Node.js / TypeScript stack

🕸️ Crawlee – high-level JS/TS scraping framework

Best for:

  • Node/TS shops
  • Headless browser or HTTP-based crawlers with robust anti‑bot handling

Install:

npm init -y
npm install crawlee

HTTP-based example:

// src/main.ts
import { CheerioCrawler } from "crawlee"
 
const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 50,
  requestHandler: async ({ request, $, enqueueLinks, log }) => {
    log.info(`URL: ${request.url}`)
    $("a").each((_, el) => {
      const href = $(el).attr("href")
      if (href) console.log(href)
    })
    await enqueueLinks()
  },
})
 
await crawler.run(["https://example.com"])

Headless/browser-based example (PlaywrightCrawler):

import { PlaywrightCrawler } from "crawlee"
 
const crawler = new PlaywrightCrawler({
  launchContext: { headless: true },
  requestHandler: async ({ page, request, enqueueLinks, log }) => {
    log.info(`URL: ${request.url}`)
    const title = await page.title()
    console.log(title)
    await enqueueLinks()
  },
})
 
await crawler.run(["https://example.com"])

Crawlee strengths:

  • Pluggable storages (local FS, key-value stores, datasets)
  • Built-in proxy rotation hooks and session management
  • Rich queueing and backpressure support

🎭 Playwright / 🤖 Puppeteer (Node-level)

Playwright (Node):

npm install playwright
npx playwright install chromium

Puppeteer:

npm install puppeteer

Use raw Playwright/Puppeteer when:

  • You need low-level control over browser automation
  • Or you are building custom tools that may embed a crawler framework later

6️⃣ CLI / Unix web-scraping toolbox

For quick jobs, piping, prototyping.

🌐 curl / wget

Simple download:

curl -A "DebianScraper/1.0" -L "https://example.com" -o page.html
wget --user-agent="DebianScraper/1.0" -O page.html "https://example.com"

🧩 HTML filtering (pup, htmlq, xidel)

Using pup (CSS selectors):

curl -s https://example.com | pup 'a attr{href}'

Using htmlq (if installed):

curl -s https://example.com | htmlq 'a' --attribute href

Using xidel (XPath/CSS, often via apt or manual install):

xidel https://example.com -e "//a/@href"

🔗 JSON processing with jq

curl -s https://api.example.com/data | jq '.items[] | {id, name}'

⚙️ Parallelization (GNU parallel)

echo "https://example.com/1" > urls.txt
echo "https://example.com/2" >> urls.txt
 
cat urls.txt | parallel -j4 'curl -s {} | pup "title text{}"'

7️⃣ Headless browsers & rendering layer

On Debian:

  • Chromium, Firefox ESR available via apt
  • Headless mode supported by both

Examples:

  • Playwright/Node: Chromium, Firefox, WebKit managed via npx playwright install
  • Playwright/Python: managed via playwright install

Recommendations:

  • Prefer Playwright for predictable, cross-browser automation
  • Restrict full-browser scraping to targets where static HTML is insufficient
  • Harden browser: disable images/fonts/analytics when not needed to reduce bandwidth

8️⃣ Architecture & robustness

🧱 Typical scraping architecture

  • Input: URL seeds, sitemap, search results
  • Crawler: Scrapy or Crawlee (or custom asyncio/Playwright)
  • Extraction: CSS/XPath selectors; strict data models
  • Storage: PostgreSQL, SQLite, Parquet/CSV, or object storage
  • Orchestration: cron/systemd timers, Docker, possibly message queues

🛡️ Hardening against failures

  • Retries with backoff (Scrapy/Crawlee have built-ins)
  • Global timeouts for all network + browser operations
  • Circuit-breaker behaviour per target domain
  • Catch and log parse failures with sample HTML for post-mortem
  • Snapshot changes in target HTML (diffing DOM) when selectors break

🕰️ Scheduling on Debian

cron example:

crontab -e
# run every night at 02:15
15 2 * * * /usr/bin/flock -n /tmp/my_scraper.lock \
  /usr/bin/bash -lc 'cd /opt/my_scraper && source .venv/bin/activate && scrapy crawl main >> logs/run.log 2>&1'

systemd timer (more robust than cron):

  • service unit runs your script/spider
  • timer unit defines cadence

🧱 Containers

  • Use Docker/Podman for isolated, reproducible scraping environments
  • Bake all Python/Node deps + system libs into the image
  • Mount volume for logs and result datasets

9️⃣ Tool selection matrix (Debian, FOSS)

Quick decision helper:

SituationRecommended stack
Large crawl, many domains, high structureScrapy (Python) or Crawlee (Node)
Small/simple HTML, one-off exportrequests + BS4 (Python) or curl+pup+jq
Heavy JS SPAPlaywright (Python or Node), optionally via Crawlee PlaywrightCrawler
Low-level browser automationPlaywright / Puppeteer (Node) or Playwright (Python)
Shell-only environmentcurl/wget + pup/htmlq/xidel + jq + parallel
Needs strong async + pipelines in PythonScrapy, possibly extended with Playwright/Splash/Selenium

🔟 Security, privacy, and ops tips

  • Centralize all credentials (DBs, proxies) via env vars or secret managers
  • Rate limit per target; add global safety limits for outbound connections
  • Monitor disk usage and log rotation for long-running crawlers
  • Obfuscation is not a security model; operate assuming your scraper can be observed

1️⃣1️⃣ Key FOSS projects & docs

Python:

Node/TS:

General scraping knowledge:

With these pieces you can build a fully FOSS, Debian-native scraping stack from tiny one-off scripts to large distributed crawlers.

Properties3
descriptionA comprehensive guide to setting up and running a robust, FOSS-based web scraping stack on Debian.
tagsdebian, web-scraping, foss, python
aliasesdebian-scraping, foss-scraping

Backlinks