Effects of segmentation granularity on nutrient prediction from real-world food images

Abstract

Diet quality is central to preventing non-communicable diseases, yet traditional dietary assessment methods are burdensome and error-prone. Estimating nutrients directly from food images offers a scalable alternative, but most methods treat a meal as a single visual entity and estimate calories and mass more accurately than individual macronutrients. It has not yet been systematically evaluated whether decomposing a meal into its ingredients - predicting nutrients per ingredient and aggregating to the dish level - improves estimation. We compared three input representations for nutrient regression from overhead food images: direct regression from the original image, regression from binary food-region masks, and regression from fine-grained ingredient-level masks. To enable this, we annotated a 1,056-image subset of the Nutrition5K dataset with 2,922 ingredient-level segmentation masks across 40 categories, and used an identical regression model for all conditions. We also evaluated a realistic setting with automatically generated masks. Contrary to our hypothesis, finer segmentation did not improve overall performance: the unsegmented and binary food-region segmentation conditions performed comparably, while ingredient-level segmentation resulted in the highest error. Ingredient-level predictions preserved the relative balance of macronutrients well per ingredient, but errors in each ingredient's estimated nutrient quantities accumulated once aggregated to the dish level, leading to a general, though not strict, tendency toward worse performance as dish complexity increased. Ingredient-level regression stayed competitive for simpler dishes with visually separable ingredients, but struggled when ingredients were highly mixed. These findings indicate that the principal limitation of ingredient-level nutrient estimation is accurate quantity estimation rather than modeling of nutrient composition. Future work should prioritize explicit quantity cues, such as depth or volume, to complement ingredient-level decomposition. We release our annotations and code to support reproducible research.