Service
AI Product Leadership
Turn an AI experiment into a workflow people can use, with clear quality criteria and a way to learn from real use.
What you get
- A workflow assessment: where AI could help, what should stay with people, and how to judge the value.
- Evaluations (evals): reference examples, quality criteria, and tests that reveal where outputs fall short.
- Context design: the business knowledge, rules, and information the system needs, with appropriate access.
- Workflow orchestration: how tools, systems, and human review fit together, including exceptions and approvals.
- An adoption and feedback plan that turns observed failures into better tests, context, and workflows.
Why this matters
- The Audubon work started by examining disagreement in human labels before judging AI output.
- In insurance, putting case evaluation into a usable Slack workflow reduced production turnaround from two days to under ten minutes.
- In campaign optimization, preserving campaign managers’ control helped increase adoption. Model capability was only part of the product decision.
In practice · Pixability · CTV classification
An evaluated approach. A deadline the team could meet.
The team had 30 days to classify four million additional CTV videos and no available CTV models.
I worked with data science to evaluate GPT-3.5, secured higher API quota, and gave the partner visibility into daily delivery progress.
Four million additional videos classified in 30 days, meeting the partner’s deadline.
Read the full case studyEstablish the approach
Classification evaluation
- Hand-scored reference set
- Separate scores and spot-checks
Deliver at scale
Production pipeline
- GPT-3.5 and increased API quota
- Shared delivery dashboard
Evaluation informed the delivery decision.
How I work with the delivery team
I lead the product work we agree on: understanding the workflow, setting priorities and quality criteria, building prototypes, and reviewing what happens in use. With your product and engineering leads, we define who owns implementation, integration, release, and ongoing support.
If the work needs additional development capacity, we can discuss Stravika, the two-person product and engineering studio I run with Valentina Kroshilina. We agree on that scope and ownership before starting the build.
Meet the Stravika studio ↗More examples from the work
AI adoption & change management
Increasing adoption by preserving campaign control
+25 points
Adoption increase after the first release
AI operations & workflow design
From two days to under ten minutes for case evaluations
2 days → <10 min
Production case-evaluation turnaround
AI evaluation & discovery
Before you can trust the AI, find out if your humans agree
63%
Agreement between two human-labeled datasets
Where is AI getting stuck in the work?
Bring an experiment, a quality concern, or a workflow the team wants to improve to a Clarity Call.
Book a Clarity Call