AI evaluation & discovery
Audubon Corkscrew Swamp Sanctuary
Before you can trust the AI, find out if your humans agree
Evaluating existing wildlife tools and correcting reference labels led to a platform the sanctuary adopted to clear its image backlog.
Service: AI Product Leadership
6
Platforms researched
63%
Agreement between two human-labeled datasets
90K+
Images the recommended platform went on to classify
The problem
Corkscrew Swamp Sanctuary's trap cameras generated images faster than the team could review them. Images were downloaded by hand daily and reviewed three times each: twice by volunteers and once by staff, with results tracked in a Google Sheet. The workflow depended on volunteer availability and could not keep pace with the backlog.
I initially expected to build something to help. Research shifted the work toward evaluating whether an existing platform could handle the sanctuary's images reliably.
Check the reference
Human labels first
- Compare volunteer datasets
- Build a staff-reviewed reference set
Evaluate the fit
Existing wildlife tools
- Detection and species identification
- Recommend Wildlife Insights
Staff corrections improved the reference labels.
What I did
Check the labels before judging the models
I researched six platforms, considering classification capabilities, supported media, storage, data-sharing requirements, and fit with the sanctuary's workflow.
Comparing two volunteer-labeled datasets across 2,026 images, I found that they agreed on 63% of labels. Frames labeled blank accounted for much of the disagreement. This was agreement between the original datasets, before I created the evaluation reference set.
I created a gold-standard reference set for the platform evaluations. Sanctuary staff reviewed it and corrected labels I had gotten wrong.
Compare what each platform could do
I evaluated selected tools, including Wildlife Insights, WildTrax, and MegaDetector, against the reference set. Detecting an animal in a frame and identifying its species were different tasks; ranking those capabilities with a single accuracy score would have been misleading.
I recommended that the team try Wildlife Insights, informed by the evaluation and staff review of the reference labels.
The outcome
The sanctuary adopted Wildlife Insights. The recommended approach went on to classify 90,000+ images, reducing manual review and getting the team, in Alex Meinders' words, “caught up for the first time ever.”
The team also reported that Corkscrew used the evaluation and approach to help other local nature preserves adopt the same method.
“Don's AI solution helped us tackle 90K+ images, using leading wildlife identification platforms to cut review time and get caught up for the first time ever — likely saving us weeks of work.”