Classifying four million CTV videos in 30 days

Selected work from my time as VP of Product at Pixability (2020–2024).

The problem

Our partner needed a classified connected-TV content library that customers could buy to advertise against. We had 30 days to deliver.

Pixability had ended its relationship with a classification vendor after that vendor was acquired. When the opportunity arrived, we had no CTV models available. The data science team tried adapting our existing models, but accuracy on CTV was in the 70s. Retraining improved the results only marginally, and the team expected further work to take time we did not have.

Establish the approach

Classification evaluation

• Hand-scored reference set
• Separate scores and spot-checks

→

Deliver at scale

Production pipeline

• GPT-3.5 and increased API quota
• Shared delivery dashboard

Evaluation informed the delivery decision.

What I did

Give data science room to own a new approach

I had independently tested GPT-3.5 and seen promising results, but the approach was not initially popular. I introduced the idea and gave data science room to develop and propose the approach themselves.

The head of data science led the prompt-engineering work. I worked with product, data science, and engineering to turn the approach into a production pipeline. We classified each video across IAB Tier 1 and Tier 2 content categories, made-for-kids, brand safety, and brand suitability.

Secure API capacity before it blocked delivery

I initially assumed we could buy as much API capacity as we needed. GPT-3.5 quota was scarce. I requested increases repeatedly before we hit the limit and found a contact who helped us secure an increase beyond the available tiers. Capacity remained a constraint.

Evaluate classifications

We evaluated classifications against a hand-scored reference set, with separate scoring for each prediction type and manual spot-checks. Internal testing showed approximately 85% overall accuracy. Given the deadline and the alternatives available, we proceeded with that approach.

Track delivery with the partner

The shared Looker dashboard I created showed daily prediction volumes and progress toward content-library and channel goals. The partner could follow delivery throughout the assignment.

The outcome

We completed classification of four million additional videos within 30 days, growing the classified library from roughly one million to five million assets.

Pixability was the only partner to meet the deadline. The others delivered late or never completed the assignment. The partner was very pleased with the delivery, which strengthened the relationship.