Job Details

Job Description

Pay Range: $70hr - $75hr

Responsibilities

• Set up and deploy Langfuse v3 in a production environment.
• Architect and implement the upgrade from Langfuse v2 to v3 within the LamBots framework, ensuring backward compatibility and performance optimization.
• Design modular components for prompt management, tracing, metrics, evaluation, and playground features using Langfuse v3.
• Leverage Langfuses full feature set, including:
• • Prompt Management: versioning, templating, and optimization.
  • Tracing: end-to-end visibility into GenAI workflows.
  • Metrics: performance, latency, and usage analytics.
  • Evaluation: automated and manual scoring of model outputs.
  • Playground: interactive testing and debugging of prompts.
• Integrate Azure AI Evaluation SDK into LamBots to enable scalable enterprise-grade evaluation pipelines/workflows.
• Build reusable components and templates for evaluation across diverse GenAI use cases.
• Collaborate with cross-functional teams to integrate evaluation capabilities into production pipelines and systems.
• Ensure scalability and reliability of evaluation tools in both offline and online environments.
• Define and enforce evaluation standards and best practices for GenAI agents, Retrieval-Augmented Generation (RAG) pipelines, and multi-agent orchestration.
• Collaborate with product, engineering, and data science teams to align evaluation metrics with business KPIs.
• Drive observability, debugging, and traceability features for GenAI workflows.
• Stay current with emerging GenAI evaluation tools, frameworks, and methodologies.

Required Skills/Experience

• Hands-on experience with Langfuse (including v3 features) and integrations.
• Experience with other GenAI observability tools (e.g., TruLens, W&B, Helicone).
• Knowledge of Retrieval-Augmented Generation (RAG), fine-tuning, and multi-agent orchestration.
• Strong understanding of Azure AI services, especially the Evaluation SDK.
• Deep expertise in LLMOps, prompt engineering, and GenAI lifecycle management.
• Proficiency in Python, TypeScript, or similar languages used in GenAI frameworks.
• Experience with cloud-native architectures (Azure preferred).
• Familiarity with tracing tools, observability platforms, and evaluation metrics.
• Excellent communication and documentation skills.