🎵 Volkslieder Scraper

German folk songs archive → clean markdown

📍 Location

/home/dv/scrapez/volkslieder/

📊 Quick Stats

Files	HTML	Markdown	Combined
11,643	2.9 GB	58 MB	31 MB

🚀 Commands

# Full pipeline
cd /home/dv/scrapez/volkslieder && ./run_scrape.sh
 
# Individual steps
source /home/dv/scrapez/.venv/bin/activate
cd /home/dv/scrapez/volkslieder/scripts
python collect_urls.py    # 📋 Get URLs from sitemaps
python scrape_pages.py    # 🕷️ Async scrape (~1h)
python convert_to_md.py   # 📄 HTML → Markdown
python fix_links.py       # 🔗 Fix internal links
python consolidate.py     # 📚 Single file + TOC

📁 Output Files

File	Description
output/	1000+ folders by song/category
volkslieder-complete.md	Single 31MB consolidated
urls.json	All 11,646 URLs
raw_html/	Original HTML backup

🔧 Key Scripts

Script	Purpose
config.py	Base URL, selectors, rate limits
collect_urls.py	Parse 59 sitemaps
scrape_pages.py	Async fetch with retries
convert_to_md.py	BeautifulSoup + markdownify
fix_links.py	URL→local path mapping
consolidate.py	Build TOC + merge

💡 Content Types

• 🎶 Lieder - 10,800 folk songs
• 📖 Encyclopedia - Composers, poets
• 👶 Kinderreime - Children’s rhymes
• 📜 Liedergeschichten - Song histories