[six2seven]

This is exactly what I needed and was thinking about building a similar app. But to be really useful and automated, there's a bit long way I'm afraid. The biggest deal breaker is the time spend for the data entry and updates, leaving a pen-and-paper / notes solution still the most easy to use and just visiting local groceries and shops. I was collecting receipts and playing a bit with scanning of these, but the amount of noise in the data and inconsistencies between shops were a bit pushing away (unless you want to dedicate time cleaning the data and training own tailored information extraction models).

One of the options to solve the data entry could be: - good quality automatic scanning of receipts (not only individual barcodes) from shops using OCR possibly supported with image recognition for double-checking (can happen that products will be mis-labelled or without quantities, etc) - when ordering on-line, the receipt should be available, so should be also much easier Yet, not always one will have a meaningful receipt available...

Solved the data entry and being able to predict own's supply needs would be also great to have a up-to date management of the inventory. Here are even more challenges on the tracking of the available goods at home, where these are and how many items (and in what state, expiration date, etc) would require most probably implementing different solutions from IoT (connected cameras, sensors, etc.).

Then, having a connected home with own groceries supplies under control, one can then automate further the shopping process with feeding-back the information about own's demand to on-line groceries one is subscribed to. This can enable customer subscription plans, and for retailer keeping a possible continuous flow of goods. This could be really really useful especially for upcoming months, when it seems like we are expected to spend a bit more time at home rather than usual, hopefully not fighting in the local shops for the last rolls of the new white paper gold.

[chipperyman573]

If you shop at walmart, there is an option to add a receipt to your account by scanning the barcode in the app (or it shows up by itself when you use walmart pay). These can be retrieved online from anywhere (not just the app) and is how I do it, way easier than you think. It basically comes down to

$(".icon-button-children").each((index, item) => ($(item).click())) // expand all items

$(".LinesEllipsis ").each((index, item) => console.log(item.innerText)) // record the stuff

on https://www.walmart.com/account

[davchana]

If you pay from Walmart Pay app, with any card, receipts get added to your account automatically.

[pc86]

But then you have to shop at Walmart.

[NamTaf]

Absolutely agree with everything you've said. I had dreamed of a Libib[1] for my kitchen, but knew I would have to do all the painful data adding and it was too much of a bother.

The other big challenge I never resolved was how you'd account for e.g. using 1/4 cup of flour out of a bigger volume. Or taken to its extreme, cooking oil. How do you know how much your 'splash' is? You can't predict the remaining volume without a lot of fiddling to measure it and that defeats the purpose.

In the end, I opted for manual databases too, but they're pain to keep up-to-date. I still think there's a lot of value in a database for all-or-nothing style ingredients, but it was enough to deter me. I'm glad someone is less lazy.

[1]: https://www.libib.com/

[dylan604]

Isn't this why "Alexa, add flour and olive oil to my shopping list" supposed make it so useful? You know when you are using something that you are almost out, and easy to say out loud to add to a list. It doesn't need to be so granular so something gets a reorder when there is only 2oz of olive oil left.

[six2seven]

> The other big challenge I never resolved was how you'd account for e.g. using 1/4 cup of flour out of a bigger volume. Or taken to its extreme, cooking oil. How do you know how much your 'splash' is? You can't predict the remaining volume without a lot of fiddling to measure it and that defeats the purpose.

Definitely. Or, even, just taking some snacks to another room, and later placing it back to an 'intelligent' cupboard, let's say. In extreme cases, one may need be under a constant tracking at home.

But, simplifying, another option may be to use an improved voice assistant when cooking. One would need then to interact with the VA to note and confirm the stuff used. Improved VA, in essence, would try to 'understand' the activities and interact, instead of being a passive one (as most of currently are), that one needs to say commands. This would require adding it access to camera and maybe other sensors available at 'smart' home. Definitely a lot of interesting and challenging problems to tackle not forgetting too about the user privacy part.

[byproxy]

My naive solution, assuming a sufficiently "smart" home, would be to have each shelf be an accurate scale. If it's sensitive enough, it should be able to detect the weight of what was consumed.

[urzav]

A shelf for each item?

[urzav]

What you are talking about is called taking inventory. Even if automated tracking is being performed, i.e., by scanning the boxes or containers when they are moved out and updating the quantity in the database, a periodic inventory audit is performed in the warehouse to reconcile the actual number of the product on the shelf with the one derived from reflecting any additions or subtractions of the product.

In the household setting, the most realistic system are we r would be to keep track of the addition (purchase/replenishment) ołl the containers/packages of each product and subtraction when the entire container is used-up and discarded. Obviously, this system would require maintaining at least one spare container for each product and reorder replacement when the remaining inventory count is 1 or whatever number of stock of a specific product one would want to maintain. Partial expenditures (splashes or table spoons of cooking oil, pinch of silt, a number of eggs from the pack, cups of sugar or flour from the container, a serving of cereal from box etc.) of the product from the container wouldn’t be tracked. The inventory would only be updated when the container/package has been fished up or purchased and added to the home inventory. For those items which one wishes not stock more than one container or packaging, a solution would be to label such containers, so that you can re-order the product (via Alexa or Google Assistant or Siri) when the remaining amount of the product is below certain level, depending on the product. I.e., you wouldn’t stock more than one container of milk or eggs. But when you got below half full, you would know to reorder. If certain products are used exclusively by the kids, they should learn to talk to Alexa or let parents know that they would soon run out of the product (they would learn if they want to have the product available). If one would follow this system of continuous replenishment, it might be unnecessary to keep the track of the available stock/inventory. But updating of the inventory database could be done automatically, using designated commands to the voice assists which invoke IFTT engine action sequence in which the product is ordered or added to the weekly order or shopping list and updated in the inventory database as being ordered. Further, the delivery of the orders could be automatically detected by the IFTT email analysis extension, but for accessing the order in the notification and updating the inventory of items delivered in a given order, one would need to write a bit of the code, for each of the suppliers, i.e., Amazon, Jet, Target, Chewy etc. The high turnover items with no reserve (fresh groceries) do not justify keeping track of their inventory, just remember to reorder them or add to the shopping list and complete the recurring task of doing grocery shopping, of which the voice assistants and personal devices could be reminding you about.

[oxymoran]

For items like oil and flour or other staples, maybe the best option would be to track and learn how often you are buying those items and have it added to your list when the time is approaching.

I think the key is the meal planning. The recipes need to bulk manage the inventories and list as meals are prepared and upcoming. The recipes would need to be cleaned up and normalized.

[golem14]

Just plan for whole bottles / bags and have at least 2 in the pantry. Then record when you run out of a box and buy the replacement.

[ozzyman700]

all shelves with scales all over them for each thing you use unmeasured

[6510]

An idea I had was coasters for bars and restaurants. You could easily glue those to shelves.

[urzav]

And what’s the point of these coasters?

[nogabebop23]

I built chowcaster to (a) learn jquery mobile and (b) solve a simpler problem: keeping track of what was in the deep freeze in my dettached garage. We used a barcode scanner for data entry, but even this was too much hassle and was eventually abandoned.

[sedgjh23]

A barcode scanner with a scale solves this

[bauerd]

Re data entry: was thinking this through a lot recently. Basically what I'd like to build requires either (1) product recognition vision like Google Lens offers, or (2) a barcode scanner and an extensive barcode product database. Both can be combined of course.

My solution would be to read frames from a smartphone's camera until a barcode is detected. This can be achieved with eg Firebase ML, on-device. If a barcode database lookup gives a product, put it on the in/out list. If not, send the frame to a product recognition vision service. This could be Google Cloud Vision AI, but they don't give you access to their product set that backs Google Lens.

Finally, provide controls to adjust the number of items on the in/out list

I thought about OCRing supermarket receipts too but these differ so greatly in layout etc. per country I figured it's not the way to go

Edit: Problem you run into as not all of this can be done on-device is privacy concerns of coursem just my thoughts on what interface I'd like to have

[jka]

The 'Open Food Facts' and 'Open Product Data' projects could be of interest for barcodes:

https://world.openfoodfacts.org/

https://product-open-data.com/

I just tested Product Open Data using the barcode from a can of Coconut Milk from my cupboard, and it returned the correct product (including a photograph) and the name of the brand and their owner.

Edit: updated to add Open Food Facts, given that Open Product Data appears no-longer-maintained

[bauerd]

Yeah tried it with a variety of products. It doesn't know about enough products, makes entry unreliable. Think of utmost importance here is that the entry works very reliably, if I have to fallback to manual entry on every fifth product, that sucks

[belthesar]

If you contribute to Open Food Facts though, you're helping contribute to the database, making it better for everyone. An API to contribute directly from the app would make that super awesome.

[jka]

Hmm, yep. That's a problem, especially if there isn't a straightforward way to contribute missing products and corrections.

Do you know of any alternative data sources?

Edit: since writing these comments, I've learned that Product Open Data may be unmaintained[0].

[0] - https://discuss.okfn.org/t/can-we-resurrect-product-open-dat...

[bauerd]

Yep it's unmai rained and I couldn't find any good alternatives. Which is why I put the whole grocery tracking thing on hold. Barrier here is the lack of accessible data for me

[jka]

That's a shame. Let's hope better alternatives exist and/or develop :)

I did try Open Food Facts[0] which correctly identified the same product for me -- and allows anyone to enter missing products, a bit like Wikipedia.

User /u/homarp mentions[1] an app which can integrate Open Food Facts with Grocy.

[0] - https://world.openfoodfacts.org/

[1] - https://news.ycombinator.com/item?id=22648741

[jacobsenscott]

Unless you can build a system that can identify the product as fast as you can pull it out or put in in the fridge or pantry, without holding a phone in one hand, it will be unusable.

[bauerd]

Have you tried Google Lens? It offers an experience I'd tolerate, given that one finds a way to remove the need to tap the screen …

[dakial1]

"I thought about OCRing supermarket receipts too but these differ so greatly in layout etc. per country I figured it's not the way to go"

People usually shop at the same places and the same things, so they could identify the products in the receipts once and then only update the new ones. Also,You could crowdsource it.

[bauerd]

Yep but I was looking for a turnkey solution. What you describe are workarounds. I tried several grocery management apps, and a lot of these did just that: Prompt me for data when a barcode did not result in a matched product. This is not the experience I personally tolerate and am looking for. For an app to get integrated in my grocery shopping, product identification needs to work seamlessly and reliably

[toss1]

Just discovered a potential workaround with one vendor.

It turns out that with an Amazon Prime account and shopping at Whole Foods the entire history is available down the the SKU, quantity, and date. Discovered this when going to order a delivery for the first time in the COVID-19 outbreak - aside from the generic shopping selection, we can also pick from our own previous purchases.

I haven't checked if there are other methods to access the data (e.g., with the app, or some history list), but there's the potential for at least some screen-scraping, and maybe they'll make the history available in a downloadable file if we pester them? It'd certainly help both this app and making their stores a bit 'stickier'.

[starpilot]

With the image recognition, I think the dream might be: take a picture of your shopping cart or basket, recognize any product that is at no more than 50% occluded. Put a red square around any item not recognized, let the user pick it up and rotate it until it's recognized.

[jka]

If you were going to tackle the ingredient data-entry problem, what would your preferred system design be?

On the other end of the system, I'm hoping to implement a way to bulk load recipes into grocy, using the open source recipe-scrapers[0] library.

[0] - https://github.com/hhursev/recipe-scrapers

[aindilis]

There are also lots of MealMaster recipes available online at sites like this: http://thehoseys.org/buster/recipes.html

[zebnyc]

I would imagine that all this tracking & automation would hit have to address the concerns of security/privacy.

Also, costs would simply explode making the whole solution a non-starter for a lot of households.

[homarp]

there is https://github.com/Forceu/barcodebuddy that integrates with grocy: "If already in Grocys system, it will consume/add/open the product in Grocy. If an unknown barcode is passed, the product name will be looked up and a corresponding product can be chosen in the Web UI. Tags can be saved - if a new product contains the tag in the name, the product will be already preselected in the drop-down list."

[MidgetGourde]

Me too, was thinking about creating simple web app. for myself. Purely from the view of food wastage. I live alone and it's pretty difficult sometimes being able to plan meals when some food packs don't last. Freezable things are OK, but sometimes the salad goes in the bin. Tracking these things avoids purchasing any unnecessary items from the shop. Will check this out.

[cheez]

I would be OK if we can take a receipt, or an order email, dump it into the system and it takes best guesses. For automation, at least.

[huffmsa]

Sound like you stole my notebook on this subject.