We’ve been drinking Nespresso Capsule coffees for a while and it’s an absolute pain choosing from all the variations available. Yes, the names are written on the capsules but not everyone knows what “fortissio lungo” tastes like.
Here’s how the finished demo looked like:
For this solution I used Tracking.js. Tracking.js can track images, camera streams and videos and can recognise colours and faces in them.
I started the project off by creating a capsule colour and tasting note definition object. This is what I use to lookup the flavours from.
When the application is opened the camera image of the Android phone or laptop appears in an HTML video element through the MediaStream API. Tracking.js then starts watching the image and looking for large areas appearing filled with colours that match our definition. The simplest form of this would be something along these lines:
Registering new colours into Tracking.js wasn’t straight forward. Here’s the piece of code from the documentation which unfortunately didn’t work with thresholds:
Here's what I had to do to extend it with threshold functionality:
After this Tracking.js fires an event and calls a function each time a colour from that camera image is matched with a colour from the capsule definition. When this happens a panel slides in with the image of the capsule and reads the tasting notes using the WebSpeech API.
I know the WebSpeech API might be a little bit too much but it ALWAYS gives that wow factor to my projects. People love talking apps and robots. Take Geroge, the talking plant for instance who was an instant hit in the office.
Here’s how the Nespresso Scanner speaks:
The main problem with the current implementation is that colours appearing in the camera stream are different when light conditions change. They might look brighter or darker but the biggest problem is white balance. You know when you take a photo and everything looks slightly blue or slightly red. This obviously all affect Trakcing.js.
The current implementation could be improved by adding a 'global white balance reference colour' that would be used to calibrate the app. In perfect light conditions the grey colour has equal amounts of red, green and blue in it, for example: 150, 150, 150. If the grey colour is 180, 130, 150 under another light condition then each colour has to be adjusted by the same amount: red + 30, green - 20, blue + 0. With this solution you would still need to re-calibrate the app each time but instead of re-calibrating all the colours you only do one.
At one point I might also build the Arduino powered scanner with my colour recognition module. The good thing about that are the 4 bright LEDs which cancels out the tinted environmental light so colours always look the same.