Building Reliable AI Agents in Production

Building an AI agent is easier than building one people trust in production. In this talk, I'll share lessons from building and operating production AI copilots and autopilot systems for high-volume financial operations using specialized AI agents, deterministic decision logic, and continuous feedback loops. I'll explain how we earned trust through a copilot-to-autopilot rollout strategy, subject matter expert validation, analyst override feedback, and continuous LLM evaluations (LLM-as-Judge) that monitor quality and detect regressions in production. You'll leave with practical design patterns for building reliable AI agents, establishing robust evaluation pipelines, and safely deploying AI systems in regulated, high-stakes environments.