Volkslieder Archive Reference

German folk songs archive from volksliederarchiv.de

📊 Stats

Metric	Value
🎶 Songs	11,594
👤 With author	9,131 (79%)
📅 With year	10,880 (94%)
📁 Categories	835
💾 Clean output	55 MB
🗄️ Raw HTML	2.9 GB

📁 Directory Structure

/home/dv/scrapez/volkslieder/
├── clean/              # ✨ Cleaned markdown (V2)
│   ├── kinderlieder/   # 609 songs
│   ├── soldatenlieder/ # 1,140 songs
│   ├── liebeslieder/   # 859 songs
│   ├── geistliche-lieder/ # 639 songs
│   └── ... (835 categories)
├── output/             # 📄 Original conversion
├── raw_html/           # 🗄️ Original HTML (2.9 GB)
├── scripts/            # 🔧 Pipeline scripts
├── index.json          # 🔍 Search index
├── authors.json        # 👤 Top 100 authors
├── categories.json     # 📁 Category counts
├── years.json          # 📅 Decade distribution
└── volkslieder-complete.md  # 📚 Consolidated

🔧 Scripts

Script	Purpose
collect_urls.py	Parse sitemaps → urls.json
scrape_pages.py	Async HTTP fetch with retries
convert_to_md.py	HTML → Markdown (parallel)
cleanup_v2.py	🆕 Extract metadata, clean links
generate_index.py	🆕 Build JSON search index
fix_links.py	URL → local path
consolidate.py	Single file + TOC

🚀 Quick Commands

# Activate environment
source /home/dv/scrapez/.venv/bin/activate
cd /home/dv/scrapez/volkslieder/scripts
 
# Full pipeline
./run_scrape.sh
 
# Individual steps
python collect_urls.py     # 📋 ~12K URLs from sitemaps
python scrape_pages.py     # 🕷️ Async download (~1h)
python convert_to_md.py    # 📄 HTML → MD (parallel)
python cleanup_v2.py       # ✨ Enhanced cleanup
python generate_index.py   # 🔍 Build search index

📝 Markdown Format (V2)

---
title: "Song Title"
url: https://www.volksliederarchiv.de/slug/
themes: ["Kinderlieder", "Weihnachtslieder"]
periods: ["19. Jahrhundert"]
text_author: "Hoffmann von Fallersleben"
musik_author: "Volksweise"
year: 1840
---
Song lyrics here...
(lexikon links converted to plain text)

🔍 Index Files

index.json - Song Search

{
  "title": "Der Mond ist aufgegangen",
  "path": "abendlieder/der-mond-ist-aufgegangen.md",
  "url": "https://...",
  "incipit": "Der Mond ist aufgegangen...",
  "themes": ["Abendlieder", "Geistliche Lieder"],
  "text_author": "Matthias Claudius",
  "musik_author": "Johann Abraham Peter Schulz",
  "year": 1779
}

authors.json - Top 100 Authors

{
  "Verfasser unbekannt": 3877,
  "Hoffmann von Fallersleben": 156,
  "Hermann Löns": 76,
  ...
}

categories.json - All Categories

{
  "Soldatenlieder": 1140,
  "Liebeslieder": 859,
  "Geistliche Lieder": 639,
  "Kinderlieder": 609,
  ...
}

📊 Top Categories

Category	Count
🪖 Soldatenlieder	1,140
💕 Liebeslieder	859
⛪ Geistliche Lieder	639
👶 Kinderlieder	609
🍺 Trinklieder	430
🚶 Wanderlieder	287
🏠 Heimatlieder	281
💃 Tanzlieder	280
⚒️ Arbeiterlieder	271
📖 Balladen	253

👤 Top Authors

Author	Songs
Verfasser unbekannt	3,877
Hoffmann von Fallersleben	156
Hermann Löns	76
Ernst Moritz Arndt	37
Ludwig Uhland	30
Georg Herwegh	25
Erich Mühsam	23
Wilhelm Müller	23
Theodor Körner	23

⚙️ Config

Edit /home/dv/scrapez/volkslieder/scripts/config.py:

CONFIG = {
    "base_url": "https://www.volksliederarchiv.de",
    "download_delay": 0.3,      # Rate limiting
    "concurrent_requests": 4,   # Parallel fetches
    "content_selector": "article",
    "remove_selectors": [       # Elements to strip
        "nav", "header", "footer", "aside",
        ".ad-inserter", ".yarpp", ...
    ],
}

🔗 Useful Queries

# Find songs by author
grep -l "Hoffmann von Fallersleben" clean/**/*.md
 
# Count songs per category
jq 'to_entries | sort_by(-.value) | .[:20]' categories.json
 
# Search by title
jq '.[] | select(.title | test("Mond"; "i"))' index.json
 
# Songs from specific decade
jq '.[] | select(.year >= 1840 and .year < 1850)' index.json