♰ Universitat Politecnica de Catalunya ✦ Massachusetts Institute of Technology ✥ Qatar Computing Research Institute Abstract In this work we train a neural network to learn a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Moreover, we demonstrate that regularization via the addition of a high-level classification objective both improves retr