TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

I had to build a web scraper to buy groceries

244 pointsby lord-bazookaabout 5 years ago

26 comments

reaperducerabout 5 years ago
Of all the grocery companies in my area, only Walmart has it right: Choose an available pick up time, <i>then</i> start online shopping.<p>All the others let you fill your cart, then at checkout tell you there are no slots available until the end of time. Waste of time, energy, and hope. F U, Target, Safeway, Albertsons, and Kroger.
评论 #22716622 未加载
评论 #22717527 未加载
评论 #22716810 未加载
评论 #22717260 未加载
评论 #22718075 未加载
评论 #22718701 未加载
评论 #22716992 未加载
评论 #22717416 未加载
评论 #22718241 未加载
评论 #22716773 未加载
40fourabout 5 years ago
Why is grocery delivery considered a ‘safer’ option than going to the store yourself. I actually prefer to do it myself. I’d rather be the one to lay hands on all my own products.<p>Why would you trust an unknown group of people to take all the proper precautions? Who picked the item by hand? Who is this bringing it to your door? Are you <i>sure</i> they washed hands? Did they accidentally sneeze on anything? Is the vehicle sanitary? Also, I would imagine people doing these jobs are probably at higher risk of being exposed, just by nature of traveling &amp; being in public all the time.<p>I would rather trust my own skills to be responsible and protect myself, rather than other unknown entities to touch all my stuff. All the supermarkets I’ve been to have someone at the door sanitizing carts as the hand them too you. I am hyper aware of avoiding touch anything unnecessary. I can keep as much distance from people as I want. I know my hands are clean when I start, and then avoid touching my face until I sanitize after.
评论 #22718947 未加载
评论 #22718784 未加载
评论 #22718725 未加载
评论 #22718900 未加载
评论 #22718847 未加载
评论 #22718985 未加载
评论 #22718767 未加载
评论 #22720648 未加载
Brakenshireabout 5 years ago
Young healthy people shouldn’t be doing this, the slots should go to the ill and elderly, people who are in immediate danger from catching the illness.
评论 #22715800 未加载
评论 #22715849 未加载
评论 #22715793 未加载
评论 #22716749 未加载
评论 #22717833 未加载
评论 #22717007 未加载
评论 #22717596 未加载
评论 #22715857 未加载
评论 #22717449 未加载
评论 #22715787 未加载
somebodynewabout 5 years ago
It seems like a lot of demand issues highlighted by the pandemic could be solved by queues. Instead of saying &quot;there are no slots available&quot;, put the user in the queue. Process the queue in order. The improvement in wait time and demand accuracy would be a win for both consumers and the business.
评论 #22715746 未加载
评论 #22716691 未加载
评论 #22718071 未加载
评论 #22717439 未加载
评论 #22715697 未加载
评论 #22720690 未加载
评论 #22715926 未加载
banana_giraffeabout 5 years ago
I had to do the same thing to get an Amazon Fresh order. Well, not quite the same thing, I ended up writing an Selenium project that automated the Amazon Fresh checkout process till it got to a delivery window page that didn’t complain there were no slots available.<p>If I can’t get groceries, it’s not the end of the world for me, since I can call on family if necessary, but I’d rather not make their stressful life any worse right now, since I’m not the only one in my family’s circle that needs help.<p>There are no great answers, since I might have taken a slot from someone that can’t write a web tool to scrape Amazon, and doesn’t have a family to fall back on. It sucks on many levels.
评论 #22717100 未加载
评论 #22717108 未加载
ineedasernameabout 5 years ago
Amazon Fresh (until a few days ago) was releasing slots 3 days out each night at midnight, so I&#x27;ve been saying up to place orders. Now it seems incremental throughout the day, but I&#x27;ve gotten lucky and nicked a spot. However, they don&#x27;t have everything I need even normally. Now, it&#x27;s even worse, so I have to order from multiple places. The local Mart&#x27;s that deliver are still opening their slots at midnight, though usually for 8 days out, and again items ordered often go out of stock before delivery. It feels like a sureal video game.
评论 #22715928 未加载
YeGoblynQueenneabout 5 years ago
&gt;&gt; everybody stays at home and especially avoids crowded indoor places like supermarkets.<p>I know I shouldn&#x27;t do that, my doctor has advised me against it, but I have a headache today and this made it worse and I can&#x27;t help myself so: this is an oxymoron. If everybody is staying at home, then supermarkets cannot be crowded. If supermarkets are crowded, then everybody is not staying at home. Both things can&#x27;t be happening at once.
评论 #22718813 未加载
评论 #22718926 未加载
评论 #22718381 未加载
omneityabout 5 years ago
Shameless but very relevant plug: Monitoring a website for changes and sending data somewhere else is exactly what we do at <a href="https:&#x2F;&#x2F;monitoro.xyz" rel="nofollow">https:&#x2F;&#x2F;monitoro.xyz</a><p>It’s the difference between maintaining a bunch of scripts and making sure they don’t stop working abruptly, and having the data you wanted without headaches.<p>It also wouldn’t make for such an elaborate article, but that’s the cost of simplicity if you just want to get things done.
vitiell0about 5 years ago
We recently ran into this problem at Cooklist in the US. We ended up setting up the ability for our users to place orders up to 3 days out.<p>Our system then books their pickup order at midnight when the new pickup slots are opened. We are seeing that most pickup slots are filled for the next day by 9am.<p>It is interesting that the stores don&#x27;t allow booking several days in advance anymore. It makes sense though because even with booking for the next day we are seeing that 25% to 50% of the order is substituted due to out of stocks.
mettamageabout 5 years ago
I&#x27;ve done the same thing in The Netherlands, as I&#x27;m doing the shopping for my grandparents. I noticed that I don&#x27;t need it yet, but I did see that I might need it in the future since I need to stay awake until 23:59 at night, so I can snatch a slot at 00:00 at the most famous supermarket here.<p>My script isn&#x27;t fully ready, but it&#x27;s 75% complete and I&#x27;ll complete the final 25% when I really do need it.<p>Also, these things are quite easy to make, so I expect a lot of techies to eventually do this.
jveabout 5 years ago
Few months ago someone had posted open source scraper that got 1k+ votes (or maybe not) - it was used by NYTimes for realtime alerts.<p>Someone remind me please.
评论 #22717150 未加载
poloteabout 5 years ago
Tips: the first session cookie is probably returned if you do a GET on the homepage of the website, so you can simplify your script by<p>Declaring the requests session, doing get on the homepage and then doing get on the endpoint<p>It will also prevent issue with session expiration
评论 #22717761 未加载
top_kekeroni_m8about 5 years ago
I’ve done a few of those scrapers and mail to notify projects in the past, but I’ve recently switched from using email to a telegram bot, it’s pretty great. Very easy to implement and doesn’t spam your inbox.
e12eabout 5 years ago
I must say, from the title I was expecting to be disappointed - but this is a very nice and clear example of a useful little hack combining python requests module and a little web page introspection.<p>I think maybe I&#x27;d have gone down to every 5 minutes, to be a little nicer to the end-point (especially when sharing the code) - otoh once a minute isn&#x27;t to much of a hammer, I guess.
pmukabout 5 years ago
I had a similar idea on Saturday and wrote a puppeteer based script which monitors Tesco.com and then pushes a screenshot to my phone if it finds a slot: <a href="https:&#x2F;&#x2F;github.com&#x2F;paulmaunders&#x2F;delivery-slot-bot" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;paulmaunders&#x2F;delivery-slot-bot</a>
评论 #22786540 未加载
mymythisisthisabout 5 years ago
The supermarket is a post WWII invention. Labour was expensive, so why not have the customer walk around and pick. Typically before WWII shop clerks handed items to waiting customers across the counter.<p>As any invention it can be reinvented.<p>Organic farmers started doing deliveries 20 years ago, to compete with the big warehouse boxes.
kwhitefootabout 5 years ago
Would be simpler to just plan four days in advance. Shopping used to be done once a week, or even once a month for bulk and frozen goods. Now it seems to a be what people do instead of going for a walk in the countryside. Makes even less sense now that you don&#x27;t have to leave your bed to do it.
评论 #22718585 未加载
评论 #22718393 未加载
6510about 5 years ago
Greasemonkey seems a much better weapon.
评论 #22793964 未加载
chmod775about 5 years ago
The picture with his cookies contains the suspicious keys &quot;hphone&quot; and &quot;hmail&quot;, which likely show md5 sums of his phone number and mail respectively.<p>Especially the phone one can easily be brute-forced (small search space). I wonder why they hash this in the first place?<p>I hope they&#x27;re salted...
xarthnaabout 5 years ago
Everyone should be ordering online regardless of age or increased risk. Limiting contact with individuals is the only way to stop the spread of this disease. The answer is that the surge in demand for delivery should quickly be filled on the supply side by delivery companies. The increase in business should go to providing hazard pay, PPE, and benefits for the supplying workers. This limits the number of people in contact with others while supplying jobs for those who have been unemployed during this time while increasing pay and benefits for those workers due to their increased risk vs reward. This is how capitalism is supposed to work in theory at least, to quickly adjust to market needs.
adamleithpabout 5 years ago
I literally build the same thing here in the UK. I knew I couldn&#x27;t have been the only one lol.<p>Good job.
qiguaiabout 5 years ago
The best part of &quot;blood in the streets&quot; is old school hacker news posts like this. Kpi of creative progress to come!
chris1993about 5 years ago
next, booking your grocery pickup as a service &#x2F;s
ct520about 5 years ago
I couldnt find an appointment slot with safeway this past week. Ended up writing a quick script on lunch that did the trick. Ran it as a schedule task to run every 5 mins. Pretty hacky and unpolished but worked, got a text message a day later and checked out. Not sure if anyone could use it but here it is. <a href="https:&#x2F;&#x2F;github.com&#x2F;christru&#x2F;safeway-delivery-check" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;christru&#x2F;safeway-delivery-check</a>
Swizecabout 5 years ago
I’ve just been going to the grocery store as usual and it works great. You show up in the morning and there’s no line yet and shelfs are full. Later in the day they meter entry so you can keep distance inside.<p>Don’t be that ass who makes others do your shopping just coz you’re too afraid to.
评论 #22715826 未加载
评论 #22716755 未加载
评论 #22718517 未加载
评论 #22715962 未加载
punarintaabout 5 years ago
Hehe, I had to do the same a week ago. The only difference is Slack’ing instead of mailing and also cart prepopulation because those delivery slots are taken out almost instantly in my city. This is in the EU btw, not specifying the country or chain to, well, to be able to survive longer.