🔍 Pi Langfuse Tracing

Target: npm:@hdkiller/pi-langfuse Core Concept: Production-grade Langfuse tracing integration for Pi Coding Agent mapping conversational AI lifecycles (Prompt -> Turn -> LLM/Tools) into hierarchical, observable Langfuse structures.

🧬 Architectural Patterns

• Hierarchical Agent Observation Tree:
  • Trace (pi-agent): Root object encapsulating the entire conversational loop, storing global config (cwd, model, provider, release, environment).
  • Span (agent.prompt): Groups multi-turn execution flows mapped to a single user instruction.
  • Span (agent.turn): Tracks reasoning loops consisting of tool calls and LLM evaluation steps.
  • Generation (llm-response): Captures complete prompt arrays, streaming outputs, latency, token usage, and cost matrices.
  • Span (tool:<name>): Captures detailed execution of discrete tools (e.g., bash, read, write), nesting dynamically underneath turns.
• Stateful Event Reconciliation: Uses decoupled Map structures (activeTurns, activeTools) to correctly correlate asynchronous parallel tool executions and non-linear Pi hooks back to their parent traces.
• Deep-Copy Context Retention: Hooks into Pi’s context event to clone raw message arrays before the generation phase. This ensures Langfuse receives the exact structured payload (system prompts + history) natively.
• Dynamic Tag Injection: Automatically injects structural metadata tags: project:<cwd_basename>, provider:<name>, model:<name>, and session:<reason>, mapping unstructured workspaces into filterable dashboards.

⚠️ Anti-Patterns (What to Avoid)

• Raw Output Dumping (The Payload Crash): Piping raw read_file, web_search, or massive bash execution results directly into traces.
  • Correction: Implement localized truncation mechanisms (trace-input-max-chars, tool-output-max-chars defaults: ~1200) before emitting trace updates.
• Dangling Spans on Abort: Failing to close active generations and tool spans when the core loop crashes, truncates, or triggers early.
  • Correction: Enforce a finalizePrompt() sweep algorithm that iterates through unresolved Maps, closing abandoned spans/generations with an abandoned: true metadata flag and marking them errored.
• Flattening Chat Arrays: Passing the prompt history as a single concatenated string into the input of a Generation.
  • Correction: Pass the raw message array so the Langfuse UI can natively render chat roles.
• Ignoring Cache Metrics: Discarding cacheRead and cacheWrite stats provided by the LLM provider, leading to inaccurate cost analysis.
  • Correction: Explicitly integrate caching tokens into total/input sums and push discrete usageDetails.

💡 Best Practices

• Isolate “Thinking” from Output: Explicitly intercept thinking_delta events natively, segregating them from text_delta. Store thinking logic in metadata.thinking while logging finalized text as primary output.
• Real-time Streaming Overlays: If tracking real-time streams (captureMessageUpdates), use Langfuse’s .update() with { partial: true } to observe thinking blocks dynamically before the turn resolves.
• Session Lifecycle Alignment: Derive Langfuse sessionId dynamically from the underlying Pi session file name (.jsonl), creating a unified 1:1 mapping between local agent sessions and remote traces.
• Scored Token Tracking: Utilize Langfuse .score() events for critical LLM telemetry (input_tokens, output_tokens, total_cost) mapping directly back to the active Generation ID, enabling immediate dashboard aggregation.
• Aggressive Config Reloading: Tie telemetry configuration directly to the agent’s live settings view. Use event listeners (pi-extension-settings:changed) to hot-reload Langfuse clients seamlessly without restarting the core agent.