Turning a Meal Photo into a Food Log

The client runs a guided health programme. Participants follow a plan with daily tasks, and one of them is logging every meal, so their nutritionist can review what they actually eat and adjust the advice.

Food is also the hardest thing to log. Water is one tap, and sleep can come from a wearable. A meal means remembering what was on the plate, guessing the portion, and looking up the calories. When logging felt like homework, participants skipped meals or stopped altogether, and nutritionists were left reviewing half-empty diaries.

The goal was simple to say and hard to do: point the camera at a plate and get a usable food log in seconds, detailed enough for a nutritionist to review. The AI should do the tedious part (naming the items, estimating weights, adding up calories), and the participant should stay in charge of the final answer.

The Problem

A photo has no sense of scale

A picture of rice looks the same at 100 grams and at 300 grams. Without a sense of size, any calorie number is a guess, and a confident wrong guess is worse than no number at all.

People want different depths on different days

Some participants only want to log the meal quickly and move on. Others manage diabetes, follow keto, or watch their sodium, and want the full breakdown themselves. One fixed screen would be too slow for the first group and too shallow for the second.

The log has a second reader

A meal log isn't only for the participant. A nutritionist reads it later, so a wrong portion or a missing item quietly turns into wrong advice. Speed couldn't come at the cost of an accurate record.

AI answers are not always tidy

The model is asked for structured data, but it sometimes returns something slightly broken: a missing bracket, a comment in the wrong place, a field in an old shape. If any of that reached the app, the screen would fail just as the person was waiting for their result.

Requirements we had to meet

• Recognise the food from one photo, taken live or picked from the gallery.
• Offer a quick mode and a detailed mode, and let people switch between them.
• Let people correct the AI and get recalculated numbers back.
• Never leave someone stuck: when recognition fails, offer another way to log the meal.
• Produce a record a nutritionist can review: named items, portions, and calories for every meal.

The Solution

How a scan works

1. Take or pick a photo. The camera screen remembers whether the person last used Quick or Detailed mode. Gallery photos keep the date they were taken, so a meal photographed yesterday is logged to yesterday.
2. Shrink it first. The server resizes the photo before the AI sees it. Recognising the food doesn't need full resolution, and a smaller image makes the round trip faster and cheaper.
3. Ask the AI to measure, not guess. The instructions ask the model to look for a reference object (a fork, the edge of the plate) to judge size, estimate each item's volume, and convert that to grams. Fruit and eggs are counted instead ("2 pcs", "4 slices"), because that's how people think about them.
4. Clean up the answer in code. The server checks and tidies the AI's reply before anything reaches the app (more on that below).
5. Review and save. The person sees the meal name, each ingredient with its portion, and the calories, then saves it to the day's food log.

Quick and Detailed modes

Quick returns ingredients, portions and calories, which is enough for the day's log and for the nutritionist to see what was eaten.

Detailed goes further for each item: sugar, carbohydrates, fibre, protein, fat, sodium, cholesterol, potassium and glycaemic index. It also flags allergens and health notes, and marks whether the meal suits diabetic, keto or heart-healthy diets and whether it is oil-, sugar- or grain-free. Where it helps, it suggests a healthier swap.

A quick result isn't a dead end. A Get Full Analysis button turns it into the detailed version, building on the ingredient list the person has already seen, so they don't need to take the photo again.

Keeping the AI's answer trustworthy

Three small decisions do a lot of the work:

• Short answers. The AI replies in a compact format with short field names. That keeps responses fast and leaves the model less room to wander. The server then expands the reply into the full, readable shape the app uses.
• Code does the maths. The AI estimates each item, and the server adds up the meal totals itself. Calories are rounded to the nearest 5, and anything under 5 counts as zero. Totals stay consistent, and nobody sees false precision like "347 kcal" for a plate of food.
• Repair before failing. If the reply is slightly broken, the server tries to repair it before giving up. If it still can't be read, the person gets an empty result rather than an error screen.

When the AI gets it wrong

The person always has the last word:

• Edit and recalculate. People can rename an item or change a portion ("that was half a bowl, not a full one"). The AI then recalculates the meal from their corrections. In Detailed mode it looks at the original photo again alongside the edits.
• Describe it instead. If nothing is recognised, the app says so plainly and offers a text box: "I ate a bowl of salad with lettuce, cherry tomatoes, shredded carrots and balsamic dressing." A tip underneath nudges people toward weights, cups, bowls or spoonfuls, which give the AI something concrete to work with.
• Offline still logs. Without a connection the AI is skipped, but the photo is kept and the meal is saved as a plain entry. It catches up once the phone is back online (see the offline-first case study).

Key Outcomes

• From photo to food log in one step. Most meals need no typing at all, so participants keep logging.
• Two depths, one flow. Quick for the daily habit, Detailed for people managing a condition, with a one-tap upgrade between them.
• Numbers people can correct. Edits are recalculated instead of overwritten by hand, so totals stay consistent.
• No dead ends. A failed scan leads to text description or manual entry, not an error screen.
• Better material for nutritionists. Meal diaries arrive with named items and portions instead of gaps and vague notes.

What Changed for Users

Before	With AI food recognition
Type every ingredient and look up calories	Take a photo and review the result
Guess portion sizes	Portions estimated from visual scale, adjustable by hand
Same depth for everyone	Quick for the daily log, Detailed for diet and health questions
"Food not found" ends the attempt	Describe the meal in words, or enter it manually
Nutritionist reviews a patchy diary	Nutritionist reviews itemised meals with portions

Trade-offs & Lessons

What worked: Splitting recognition into Quick and Detailed matched how people actually log food. Most days are quick days, and a quick log that actually happens beats a detailed one that doesn't. Asking the model for compact answers and doing the arithmetic in code made results more consistent than asking the AI for totals. And treating every AI reply as "probably right, check it anyway" prevented a whole class of broken screens.

What we tried and stepped back from: Giving the AI the person's meals from the past few days as context, so it could recognise their usual dishes. It added more confusion than accuracy at this stage, so it is switched off for now.

What we'd harden next:

• Tighter, consistent instructions for swap suggestions, which are the part of the answer that drifts most often.
• Learning from corrections: when people often change the same estimate, feed that back into the instructions.
• Showing confidence. Some meals (stews, mixed bowls) are much harder to estimate than others, and the result could say so.

---

Stack: React Native · Expo Camera · TanStack Query · NestJS · OpenAI vision model · Sharp