I had to build a web scraper to buy groceries

Of all the grocery companies in my area, only Walmart has it right: Choose an available pick up time, <i>then</i> start online shopping.<p>All the others let you fill your cart, then at checkout tell you there are no slots available until the end of time. Waste of time, energy, and hope. F U, Target, Safeway, Albertsons, and Kroger.

Why is grocery delivery considered a ‘safer’ option than going to the store yourself. I actually prefer to do it myself. I’d rather be the one to lay hands on all my own products.<p>Why would you trust an unknown group of people to take all the proper precautions? Who picked the item by hand? Who is this bringing it to your door? Are you <i>sure</i> they washed hands? Did they accidentally sneeze on anything? Is the vehicle sanitary? Also, I would imagine people doing these jobs are probably at higher risk of being exposed, just by nature of traveling &amp; being in public all the time.<p>I would rather trust my own skills to be responsible and protect myself, rather than other unknown entities to touch all my stuff. All the supermarkets I’ve been to have someone at the door sanitizing carts as the hand them too you. I am hyper aware of avoiding touch anything unnecessary. I can keep as much distance from people as I want. I know my hands are clean when I start, and then avoid touching my face until I sanitize after.

Young healthy people shouldn’t be doing this, the slots should go to the ill and elderly, people who are in immediate danger from catching the illness.

It seems like a lot of demand issues highlighted by the pandemic could be solved by queues. Instead of saying &quot;there are no slots available&quot;, put the user in the queue. Process the queue in order. The improvement in wait time and demand accuracy would be a win for both consumers and the business.

I had to do the same thing to get an Amazon Fresh order. Well, not quite the same thing, I ended up writing an Selenium project that automated the Amazon Fresh checkout process till it got to a delivery window page that didn’t complain there were no slots available.<p>If I can’t get groceries, it’s not the end of the world for me, since I can call on family if necessary, but I’d rather not make their stressful life any worse right now, since I’m not the only one in my family’s circle that needs help.<p>There are no great answers, since I might have taken a slot from someone that can’t write a web tool to scrape Amazon, and doesn’t have a family to fall back on. It sucks on many levels.

Amazon Fresh (until a few days ago) was releasing slots 3 days out each night at midnight, so I&#x27;ve been saying up to place orders. Now it seems incremental throughout the day, but I&#x27;ve gotten lucky and nicked a spot. However, they don&#x27;t have everything I need even normally. Now, it&#x27;s even worse, so I have to order from multiple places. The local Mart&#x27;s that deliver are still opening their slots at midnight, though usually for 8 days out, and again items ordered often go out of stock before delivery. It feels like a sureal video game.

&gt;&gt; everybody stays at home and especially avoids crowded indoor places like supermarkets.<p>I know I shouldn&#x27;t do that, my doctor has advised me against it, but I have a headache today and this made it worse and I can&#x27;t help myself so: this is an oxymoron. If everybody is staying at home, then supermarkets cannot be crowded. If supermarkets are crowded, then everybody is not staying at home. Both things can&#x27;t be happening at once.

Shameless but very relevant plug: Monitoring a website for changes and sending data somewhere else is exactly what we do at <a href="https:&#x2F;&#x2F;monitoro.xyz" rel="nofollow">https:&#x2F;&#x2F;monitoro.xyz</a><p>It’s the difference between maintaining a bunch of scripts and making sure they don’t stop working abruptly, and having the data you wanted without headaches.<p>It also wouldn’t make for such an elaborate article, but that’s the cost of simplicity if you just want to get things done.

We recently ran into this problem at Cooklist in the US. We ended up setting up the ability for our users to place orders up to 3 days out.<p>Our system then books their pickup order at midnight when the new pickup slots are opened. We are seeing that most pickup slots are filled for the next day by 9am.<p>It is interesting that the stores don&#x27;t allow booking several days in advance anymore. It makes sense though because even with booking for the next day we are seeing that 25% to 50% of the order is substituted due to out of stocks.

I&#x27;ve done the same thing in The Netherlands, as I&#x27;m doing the shopping for my grandparents. I noticed that I don&#x27;t need it yet, but I did see that I might need it in the future since I need to stay awake until 23:59 at night, so I can snatch a slot at 00:00 at the most famous supermarket here.<p>My script isn&#x27;t fully ready, but it&#x27;s 75% complete and I&#x27;ll complete the final 25% when I really do need it.<p>Also, these things are quite easy to make, so I expect a lot of techies to eventually do this.

Few months ago someone had posted open source scraper that got 1k+ votes (or maybe not) - it was used by NYTimes for realtime alerts.<p>Someone remind me please.

Tips: the first session cookie is probably returned if you do a GET on the homepage of the website, so you can simplify your script by<p>Declaring the requests session, doing get on the homepage and then doing get on the endpoint<p>It will also prevent issue with session expiration

I’ve done a few of those scrapers and mail to notify projects in the past, but I’ve recently switched from using email to a telegram bot, it’s pretty great. Very easy to implement and doesn’t spam your inbox.

I must say, from the title I was expecting to be disappointed - but this is a very nice and clear example of a useful little hack combining python requests module and a little web page introspection.<p>I think maybe I&#x27;d have gone down to every 5 minutes, to be a little nicer to the end-point (especially when sharing the code) - otoh once a minute isn&#x27;t to much of a hammer, I guess.

I had a similar idea on Saturday and wrote a puppeteer based script which monitors Tesco.com and then pushes a screenshot to my phone if it finds a slot: <a href="https:&#x2F;&#x2F;github.com&#x2F;paulmaunders&#x2F;delivery-slot-bot" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;paulmaunders&#x2F;delivery-slot-bot</a>

The supermarket is a post WWII invention. Labour was expensive, so why not have the customer walk around and pick. Typically before WWII shop clerks handed items to waiting customers across the counter.<p>As any invention it can be reinvented.<p>Organic farmers started doing deliveries 20 years ago, to compete with the big warehouse boxes.

Would be simpler to just plan four days in advance. Shopping used to be done once a week, or even once a month for bulk and frozen goods. Now it seems to a be what people do instead of going for a walk in the countryside. Makes even less sense now that you don&#x27;t have to leave your bed to do it.

Greasemonkey seems a much better weapon.

The picture with his cookies contains the suspicious keys &quot;hphone&quot; and &quot;hmail&quot;, which likely show md5 sums of his phone number and mail respectively.<p>Especially the phone one can easily be brute-forced (small search space). I wonder why they hash this in the first place?<p>I hope they&#x27;re salted...

Everyone should be ordering online regardless of age or increased risk. Limiting contact with individuals is the only way to stop the spread of this disease. The answer is that the surge in demand for delivery should quickly be filled on the supply side by delivery companies. The increase in business should go to providing hazard pay, PPE, and benefits for the supplying workers. This limits the number of people in contact with others while supplying jobs for those who have been unemployed during this time while increasing pay and benefits for those workers due to their increased risk vs reward. This is how capitalism is supposed to work in theory at least, to quickly adjust to market needs.

I literally build the same thing here in the UK. I knew I couldn&#x27;t have been the only one lol.<p>Good job.

The best part of &quot;blood in the streets&quot; is old school hacker news posts like this. Kpi of creative progress to come!

next, booking your grocery pickup as a service &#x2F;s

I couldnt find an appointment slot with safeway this past week. Ended up writing a quick script on lunch that did the trick. Ran it as a schedule task to run every 5 mins. Pretty hacky and unpolished but worked, got a text message a day later and checked out. Not sure if anyone could use it but here it is. <a href="https:&#x2F;&#x2F;github.com&#x2F;christru&#x2F;safeway-delivery-check" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;christru&#x2F;safeway-delivery-check</a>

I’ve just been going to the grocery store as usual and it works great. You show up in the morning and there’s no line yet and shelfs are full. Later in the day they meter entry so you can keep distance inside.<p>Don’t be that ass who makes others do your shopping just coz you’re too afraid to.

Hehe, I had to do the same a week ago. The only difference is Slack’ing instead of mailing and also cart prepopulation because those delivery slots are taken out almost instantly in my city. This is in the EU btw, not specifying the country or chain to, well, to be able to survive longer.