From Vibes to Metrics: Live Prompt Optimization with DSPy

Live demo transforms a fragile LLM pipeline into an optimized DSPy workflow, comparing baseline and optimized accuracy, latency, and cost with reproducible code.

Overview

Prompting by gut feel doesn’t scale. I’ll start with a brittle LLM pipeline and a small eval set, show baseline accuracy, latency, and cost, then “compile” it with DSPy. We’ll define Signatures and Modules, run an optimizer to generate candidates and tune prompts, switch metrics, re-optimize, and compare before and after on held-out cases. I’ll show the code and dataset wiring so people can reuse the pattern in their own agents.

Links

https://github.com/Mahir-Isikli/ai-tinkerers-dspy-talk
DSPy optimizes GPT-4.1-mini prompts automatically using MIPROv2 for performance.
https://dspy.ai/
DSPy is a declarative framework for building and optimizing modular LLM programs.

Tech stack