SnapDish
Photograph an ingredient, get recipes. A Core ML classifier I trained, running entirely on device.
Overview
A solo project with a paper attached. I trained an image classifier in Create ML on 706 photos of four ingredients, ran it through Vision on device, and used the predicted label as the search term against TheMealDB. The write-up is an ACL-style report covering the data, the training, the results and the limits.
Problem
Deciding what to cook from what is in front of you is a vision problem and a retrieval problem stacked on each other. Neither half is hard alone; the interesting part is that the output of the first is literally the input of the second.
Trade-off
Inference stays on device instead of calling a hosted vision API. That means no API key, no per-call bill, no photo of someones kitchen leaving their phone, and a classifier that works with the network off — the connection is only needed at the end, to fetch the recipe. Because the backbone ships with the operating system, the whole trained model is a 49 KB file.
What it does
Stack
What I took from it
The first model only knew banana and apple and scored a perfect 100% on both splits — which is mostly a statement about how narrow the problem was. Widening it to four ingredients dropped the numbers, and that drop is the more honest measurement of the two.
