01.1 / audio + vision
Whales and PhotoID
In 2013 I detected right-whale calls in hydrophone recordings by turning audio into spectrograms and using a convolutional network designed for images. The model did well in Kaggle's whale-detection challenge. Then researchers at Oregon State University supplied much messier continuous recordings.
My shiny model was lousy on them. I had some listening to do. The competition examples had been preselected by another detector, so many confusing sounds were missing from training. I started again with a few hundred examples. I listened to the worst mistakes, labelled them, and fed them back into training. Then I repeated the process. The whale post explains what happened.
PhotoID began because I'm bad at identifying birds and wanted to learn their names. I built the recognition backend and public Twitter bots for birds, butterflies, and wildflowers. Asked to identify a moorhen, it answered mallard. Its second guess was moorhen.