It's been a while since I started my little adventure of building an image recognition pipeline that would reliably identify animal and plant species from photos. I'm pretty happy with what the results so far, and these days I'm even successfully scratching my own itch with it. Which is how I came up with the idea in the first place: I suck at identifying birds, butterflies, and plants, but I want to learn about them. And knowing a species' name is really the gateway to learning more about it.
So here's a painless way of identifying these species. Say you want to identify one bird that you see. You take a photo of it and send it to @id_birds on Twitter using Twitter's own image upload. After a minute or two, you'll get the automatic reply, with hopefully the right species name. @id_birds can with a remarkably high accuracy identify yours between the 256 different birds that it knows.
There's two other bots that are also available to use for free: @id_butterflies knows around 50 butterfly species, and @id_wildflowers around 200 wildflowers. All three bots know species occurring natively in Europe only, at least for now.
"But does it actually work?" I hear you say. And the answer is: it works pretty well, especially if you're sending it good quality photos where the animal or plant is clearly visible, is not too blurry, has good white balance etc.
Seeing is believing though, so to prove that it works well, let's try and identify the ten birds that are in The Wildlife Trust's river bird spotter sheet.
Let's start with the first bird, a grey heron. To the left we have me addressing the Twitter bot and sending the image. To the right is @id_birds' reply. And it's correct! Ardea cinerea is indeed the scientific name for the grey heron:
Notice how @id_birds will always respond with the two most likely guesses. The second guess here was the demoiselle crane (Anthropoides virgo), which looks sort of similar.
The next bird on the river bird spotter sheet is the common moorhen (Gallinula chloropus). Let's try to ID it:
And already we have @id_birds' first mis-classification. It thinks that this is a mallard (Anas platyrhynchos). Only the second guess was correct. Thus so far we're counting one correct, one error.
The next bird we want to identify is incidentally a mallard:
And the response is correct. Notice how the second guess (Coracias garrulus) seems a little weird. I guess it's the similarities in color here. We don't count the second guesses here though, so it's two correct, one error.
What about the kingfisher? That's a bird with some pretty unique appearance among birds native to Europe. So it's probably one of the easier birds to identify:
And sure enough, @id_birds knows what it is. The latin name for the kingfisher is Alcedo atthis.
The fifth bird on the spotter sheet is the mute swan (Cygnus olor). Everyone probably knows this one, but does @id_birds?
Yes, the answer is correct. Note that the second guess is also a swan, namely the whooper swan, quite similar in appearance. A good job!
So out of the first five birds we got four correct, not bad. But let's see about the rest of the birds from the spotter sheet.
I'll post the responses to the remaining five birds without further comment. But do note that of the remaining ones, only the coot (Fulica atra) is mis-classified, that's the third one below:
So all in all we have two mis-classifications out of ten. That's 80% accuracy, which is pretty remarkable when you consider that differences between these birds can sometimes be minuscule (grey wagtail versus yellow wagtail, anyone?). As mentioned, @id_birds knows 256 bird species. Which means that if you made guesses at random, then you'd have 0.4% accuracy.
I'll write about how the computer vision algorithm works under the hood another time. I call the stack that I've developed and that's running behind @id_birds and the other bots PhotoID. So far PhotoID has proven to have very competitive performance and be remarkably flexible, that is, I can use it as a basis for different kinds of classification problems without too much adaptation work.
Let's conclude this post with two other IDs, one from @id_butterflies, the other from @id_wildflowers:
(Plug: Feel free to contact me through my company if you have a challenging image recognition problem that you want me to help solve.)