Langfuse Clickhouse Migration

🚀 🚀 Quick Start Langfuse v3 replaces the Postgres-only core with a ClickHouse-centric data stack. Self-hosted deployments must provision ClickHouse, Redis/Valkey, an S3-compatible Blob Store, and an asynchronous Worker Container.

🔧 🔧 Configuration Required environment variables for linking the new stack:

CLICKHOUSE_URL="http://clickhouse:8123" # Must be version 24.3+
CLICKHOUSE_MIGRATION_URL="http://clickhouse:8123" # Dedicated migration endpoint
CLICKHOUSE_USER="default"
CLICKHOUSE_PASSWORD="securepassword"
CLICKHOUSE_CLUSTER_NAME="my_cluster" # Explicitly set if using external cluster
NODE_OPTIONS="--max-old-space-size=8192" # Required to prevent OOM during migration
LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE="2024-11-01" # Skips unnecessary reads

📦 📦 Dependencies

• ClickHouse: Version >= 24.3 (Avoid specific 25.5.2+ releases due to memory issues).
• Redis / Valkey: Queuing and caching tier.
• S3 / Blob Store: Offloads large objects to cold storage.
• Worker Container: Processes asynchronous background events.

💡 💡 Best Practices

• Ingestion Optimization: Set LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE prior to your project creation date. This dramatically reduces read loads during high-throughput ingestions.
• Read-Side Deduplication: ReplacingMergeTree deduplicates asynchronously. Design queries to expect fluctuating counts and implement read-side deduplication.
• Separation of Concerns: Split the langfuse-web deployment into separate ingestion and UI/API services under heavy load.
• Data Tiering: Use ClickHouse native TTL features to move old data to S3. This avoids scaling expensive block storage.

🚨 🚨 Gotchas / Warnings (Anti-Patterns)

• “Dirty Database” State: Failed migrations lock the schema_migrations table. You must manually clean this table in ClickHouse to retry.
• Batch Size Overflows: Default Postgres background_migrations batch sizes often cause HTTP 413 Payload Too Large errors. Edit the args column to reduce batchSize to 100 or 250.
• Memory Exhaustion: Migrations require at least 8 GiB memory per replica. Missing NODE_OPTIONS memory flags guarantees OOM crashes.
• Single-Shard Limitation: Multi-shard ClickHouse configurations are unsupported. Forcing multi-shard setups corrupts data consistency and breaks migrations.
• ENABLE_ANALYZER Errors: Upgrading ClickHouse can trigger CAST AS Map query errors. These stem from the new ClickHouse analyzer; verify GitHub issues for version-specific flags.

🔍 🔍 Research / References

• Official Self-Hosting Docs
• Langfuse GitHub Repository