Validate everything - why I no longer trust my supermarket

The above is from the receipt when visiting a supermarket earlier in the year.
Yes, it's an entry for 88.297 KiloGrams of loose fruit at £1.95 per KG, coming to a total of £172.18!

That's almost 200 pounds and the weight of a grown adult!
It's also (to the best of my memory) more than I've ever spent in a supermarket at one time.

tldr1: Don't blindly trust ML/AI algorithms without adding extra data validation.

tldr2: Double check your receipts!

So, how did this happen?

During the lockdown, I made my once-a-week trip to the supermarket for essentials.

I used the "Smart Shop" app on my phone to scan and pack items as I shopped.
There are several possible benefits to using the app, but, to me, the biggest was reduced time with a cashier when we should be doing "social distancing."

The following is an explanation of a specific issue I've observed with the app.
I'm realistic enough to know the app won't be perfect, especially with my high standards.

There are some other issues that I regularly encounter (most times, I use the app), but I don't think they are worth exploring here. (If you're interested, these include the "scan" button disappearing and not allowing me to scan other products, and the scan button being unresponsive, leading me to press it again, which leads to the scan mode briefly being displayed before automatically closing again.)

I was doing my weekly shop, navigating my way around the store, and got to the confectionary aisle. I was choosing some treats for my children and saw something I thought my wife would like. I picked it up, scanned it. I wasn't paying attention to the app but heard the positive "beep" of a successful scan and put it in the trolley without looking at the screen.
A few minutes later, I noticed the running balance was significantly larger than what was reasonable based on my trolley. I was only shopping once a week, so spending a bit more than I had in the past, but this was way beyond anything reasonable.

I was able to scroll back through the list of everything I'd purchased and found the rogue entry. Based on the items before and after it in the list, I worked out what item was missing based on what I'd put in my trolley and where the erroneous entry had come from.

Cautiously I tried scanning the item for my wife again, and this time it showed up correctly. 
I tried removing the incorrect entry, but the app wouldn't let me without scanning it again--which was obviously impossible.

When I had gathered all my shopping and was ready to pay, I found a staff member to explain what had happened. They tried to remove the entry but required managerial approval because of the high price of the item. (Aside: why is managerial approval needed to void an item that doesn't require managerial approval to purchase?)
The item was eventually removed, and I paid for my shopping. Nobody in the store was interested in finding out more about how or why this had happened, and so I went home. (I'll save my rant on front-line customer service workers who aren't interested in escalating technical issues for another time.)

This experience weighed on my mind and left me with two big questions.

1. How did this item end up on my list?
2. How is this item even possible?

In reverse order.

How is this item possible?

The entry is for a product the weight of a grown adult.

The scales in the store don't support that much.
It's not physically possible to balance that many items on the scales.

How is this deemed valid?

It's a tremendous amount, so it should have raised an exception.

Why is there no default flagging or limits on the weight and total price of an individual item?

There was a check when removing the item, so why not on adding it?

Perhaps there's an implied check based on what the scales can measure, so a check wasn't deemed necessary. If the scales/printer can't print a barcode label for an item that heavy or expensive, why is it necessary to validate the value?

As we shall see, the system is based on the fact that it relies on accurate scanning, and my experience is that what is in a barcode and what the software reads can be different.


The troublesome item has what I'll refer to as a 'self-weighed barcode.' (It's a few years since I worked in a supermarket, and I don't know the terminology they use.)

These self-weighed barcodes contain two parts: an identifier for the product and the price of this particular item. This is how the correct price is charged without creating an entirely unique barcode for every weighed item.

This system is at risk of abuse, but that abuse is presumably deemed acceptable. If you've ever heard of someone weighing and printing the label for a small handful of grapes and then sticking the label on a bag containing a much larger number, you'll understand the risk. But, most people are honest, and so the amount lost by the supermarkets in this way is less than they save by not having to have everything preweighed or paying for staff to weigh them.

So, hopefully, you can see what the system thought I'd scanned, but...


How did this item end up on my list?

Or, as a bigger, more general question, how did a different item from what I'd scanned show up on my list?


Firstly, let's consider if this matters.

If it's a different product.

The store's inventory management will be out. They may think they've sold more (or less) of products. They probably allow for small discrepancies, so this may not be a problem. Whether 200 pounds of fruit counts as a small discrepancy, I can't say.

For the individual making the purchase, there are two potential consequences. Firstly they will have no proof of purchase of the product they put in their trolley/bag and so won't be able to return it or get assistance/replacement if there's a problem. The second consequence is if they are asked to verify that their shopping contents match their receipt. This may be a security check or a random check applied to all users of self-scanning systems. At best, this might require the rescanning of all items or, at worst, may be seen as an attempt at shoplifting. Good luck proving the issue was with their software if you're accused by them of shoplifting.

What if this affects an age-restricted product?

Customers may have to get their purchases checked unnecessarily. But, potentially more concerning is people buying age-restricted products without appropriate verifications being made. If my discovery has found a flaw in the supermarket's software, could this be exploited to purchase age-restricted products by people below that age? What are the legal implications of this?

If the prices are different.

If the price charged is less than it should be:

• The store will lose money and may try and accuse you of deliberately underpaying. It might be hard for them to prove guilt, but it'll be harder for you to prove innocence. At any rate, it has the potential to be an unwelcome and stressful exchange. Again, a small discrepancy may not be worth the company's effort to change their systems to address.
• If you're undercharged, then you gain. Bonus.

But if the price is higher than it should be...

• The company won't mind. For them, the occasional over or undercharged item will probably cancel out.
• Legislators might be interested, but I don't know enough about this area to say for sure.
• For the individual, this could be a big deal. If, as in my case, it's a large difference, this could have a big impact on a person's finances. A large discrepancy is likely to be spotted, but multiple smaller differences might not be. What if you were regularly being overcharged?

How many times has this happened before, but the price difference wasn't large enough for me to notice? - It's a worrying thought.


So, what's happening when the wrong product/price show up?

Let's start by looking at the process as a whole. Here's how it's supposed to work.

1. I scan the product with the app on my phone.
2. The app decodes the image to identify a barcode.
3. The data (barcode value) is sent to the server.
4. The server responds with the product and price.
5. Repeat for each product.
6. At the end, the device/app sends all data to the till.
7. I pay at the till. - Done.

The possible causes of the issue are either in the app, on the server, or in communication between them.

• If it's a communication issue, it could be one of data corruption or a man-in-the-middle attack causing data corruption. I don't think either of these is the cause of the issue I encountered.
• It could be that the server sent the response to the wrong device. Perhaps somewhere else, someone did scan the product that showed up on my device, and the messages got mixed up. Perhaps someone else scanned all that fruit and was only charged for some chocolate. This again seems extremely unlikely.
• Perhaps the server was misconfigured, and it thought the barcode for the chocolate was the expensive fruit. That I scanned the chocolate again and it showed up correctly suggests this isn't the case. Of course, an invalid server configuration could have been corrected between the two times I scanned, but I think this is very unlikely.

I think the problem was with the software in the app that "reads" the barcode.

The app is running on my phone. My phone doesn't have a dedicated barcode scanner, but it does have a camera. The app uses the camera to take an image (well, it probably scans frames of video) and looks for a barcode image from which it can extract the value. (Years of building software that prints and scans barcodes have taught me a thing or two about how this works ;)

I think the "barcode scanner" software extracted the wrong value from the image/barcode, and it just happened to be something that the server considered valid.

I think this is what happened because when using the app, I regularly (it probably happens every other week) "scan" an item and receive an error message that the item is not recognized. I can "scan" the barcode again, and it goes through correctly. Is it that the barcodes are added to the system between the times I scan them? Or, is it more likely that the app sends invalid data because it "misread" the barcode?

Based on my knowledge of barcodes and image recognition, I think the app is "reading" the barcodes incorrectly. 

Don't get me wrong, I think the image recognition is impressively good. Sometimes it even reads barcodes that my knowledge of barcodes and laser scanners surprises me as I thought they'd be unreadable.

But, I know that the "barcode scanner software" isn't "reading" the barcode the same way that a laser scanner does. It uses AI (a machine-learning-based algorithm) to work out the best guess for an image's barcode value.

In this instance, I think I was "lucky" and encountered an incorrect barcode read that happened to be a value the server recognized but had a price difference enough for me to notice. This then combined with my knowledge of client-server software and barcode scanning to make an informed guess about what happened.


What could the supermarket have done to prevent this?

I have a few recommendations:

• Continue to invest in improved accuracy of barcode "scanning" based on phone cameras.
• Add validation for extremes of weight and price when reading values from 'self-weighed barcodes.'
• Ensure encrypted connections between the app and server to avoid accidental corruption, modification during transmission, or responding to a different client than the sender.
• Introduce a process for automatically logging and investigating exceptional values or errors.

What I'm doing because of this?

• I now double-check everything I scan.
• I verify the contents of my receipt matches what I put in my basket/trolley.

Not doing these could lead to me being overcharged and/or charged for products I didn't buy.


Bonus validation issue I found with their software for everyone who has read this far.

After several months of shopping with their phone app to scan items as I go round the store, I've discovered that there are a few aisles where the network signal is feeble. This leads to a potential issue with the app communicating with the server.

I regularly see the busy/progress spinner being displayed for longer than elsewhere in the store in these aisles. On one occasion, the delay was significantly longer than I'd encountered previously. It stopped me from scanning further items, and because I'm inclined to wonder when software misbehaves, I started counting seconds until it finished what it was doing. 

I counted for about a minute (I know I'm bad at estimating seconds, and it's not a skill I see value in spending time improving) but more concerning was that the list of products now displayed that I'd scanned two of the item I'd only scanned once. Was this another instance of the app scanning something other than I had? No, here's what I think happened.

I think the app sent the details to the server but didn't get a response. It then automatically retried (timeout and retry periods are typically either 60 or 100 seconds--both of which could have been met in this instance) and sent the request again. I think the server received both requests, but the response to the first request was not received. When the server sent the second response, it included the total number of that item that had been scanned, but this was actually the number of requests it had received, and due to the automatic retry, these weren't the same.

Having seen many, many codebases, I know it's common for developers to include automatic retry logic but not account for the server receiving a request but the response being lost.

I find it very disappointing when a large, multi-million-pound business has software that doesn't account for basic and expected connectivity scenarios.


If you're wondering which supermarket this was, why not leave a guess in the comments. ;)