Headless Chrome support in Cloud Functions and App Engine

333 点作者 idoco将近 7 年前

22 条评论

I wrote a collection of Dockerfiles for images running Python 2.7 or Python 3.6 + Selenium with either Chrome or Firefox and using Xvfb for the X display (necessary for running Selenium headlessly).<a href="https://github.com/seanpianka/docker-python-xvfb-selenium-chrome-firefox" rel="nofollow">https://github.com/seanpianka/docker-python-xvfb-selenium-ch...</a>Using this, in conjunction with AWS Step Functions, Lambda, and ECS, it became merely cents a month to run a headless scraper task in the cloud.

评论 #17796618 未加载

mostlystatic将近 7 年前

I tried to use Headless Chrome on Cloud Functions for a project I'm working on, but even on the fastest instances loading pages was sometimes really slow (pages timing out after waiting for 60s).It seems sometimes JS execution was taking a long time, so I guess that was preventing requests from being made. In a single CPU cloud function you have network requests, JavaScript execution, rendering, and the Node process controlling the browser all competing for resources.That being said, it was super simple to get started!

评论 #17797626 未加载

评论 #17796225 未加载

iamjustlooking将近 7 年前

Among other things with puppeteer we do screenshot generation using GKE on Google Cloud @ <a href="https://screenshots.cloud/" rel="nofollow">https://screenshots.cloud/</a> scaling up and down running instances depending on demand. We keep browser instances running constantly as the startup time is significant. I will be interested to see what the startup time is for puppeteer on this, will definitely be giving it a try.

评论 #17797056 未加载

bergie将近 7 年前

Nice to see this concept get into the big cloud platforms. We built something similar couple of years ago, primarily to get a sandbox for some compute jobs we were running on Heroku:<a href="https://github.com/flowhub/jsjob" rel="nofollow">https://github.com/flowhub/jsjob</a>

k__将近 7 年前

Does this mean that HC is preinstalled in the runtime?Becauee as far as I know you can already run HC with other FaaS solutions, but having this out of the box would be really nice.

评论 #17795974 未加载

mrskitch将近 7 年前

If Google cloud ain’t your jam then checkout browserless (<a href="https://browserless.io/" rel="nofollow">https://browserless.io/</a>). It can be considerably cheaper under certain situations, and we’ve been up and running for almost a year. Happy to answer questions if anyone has any.EDIT: We’ve got stuff on GH: <a href="https://github.com/joelgriffith/browserless" rel="nofollow">https://github.com/joelgriffith/browserless</a>, and startup is under 100ms most of the time. Fonts and other things “just work” as well, plus there’s a slew of REST APIs for common stuff as well. Selenium webdriver support landing soon!

评论 #17796333 未加载

评论 #17796335 未加载

schappim将近 7 年前

I wish Google would support Ruby!If you do too checkout the petition over at <a href="https://www.serverless-ruby.org" rel="nofollow">https://www.serverless-ruby.org</a>

评论 #17796396 未加载

评论 #17796595 未加载

评论 #17796398 未加载

antoncohen将近 7 年前

I think it would be pretty cool to use Cloud Functions as a Selenium Grid, sort of like Zalenium (<a href="https://github.com/zalando/zalenium" rel="nofollow">https://github.com/zalando/zalenium</a>) does with Kubernetes. If you could parallelize end-to-end tests enough, you could get massive burstable capacity to run parallel tests.

wslh将近 7 年前

Not perfect but HtmlUnit anyone? I used it for scraping in the past with mixed experiences.

评论 #17796646 未加载

tnolet将近 7 年前

I've been screwing around with running Headless Chrome & Puppeteer on Lambda/Serverless/FAAS solutions. It's all a bit of a mixed bag. You CAN run Headless Chrome on AWS Lambda, but the cost involved is pretty crazy as you need ~1500Mb in RAM to comfortably run any code with Chrome.Google Cloud of course has "inside knowledge" and I would love to switch to them for my SaaS <a href="https://checklyhq.com" rel="nofollow">https://checklyhq.com</a>, were it not that Google Cloud Functions is just offered in four (!) regions...

评论 #17796282 未加载

pwaai将近 7 年前

that's it...im moving to GCPsorry but Rekognition rekt it for any type of computer vision on AWS.Great infrastructure...after all I do have an AWS Solution Architect Associate certification....which means jack shitGreat move by GCP, I'm also very pleased with Firebase and it's integration with cloud functions....BUT my biggest reservation still in 2018 when it comes to serverless is the cold start up time...I built a token based API on AWS Lambda and registering, signing up took forever when the app was not at peak. that was 2014 tho.

评论 #17798086 未加载

评论 #17806909 未加载

guiomie将近 7 年前

Ok, I was just looking into this 1 week ago and was gonna spin up a VM to do headless. Now I get to keep my firebase project in cloud functions only. Much cleaner architecture.

defied将近 7 年前

My company provides a similar service, with both Chrome and FireFox headless support for Automated Testing/Screenshots: <a href="https://testingbot.com/support/getting-started/headless.html" rel="nofollow">https://testingbot.com/support/getting-started/headless.html</a>We run each test in a new VM, running on our own private cloud (dedicated servers).Note: we use the Selenium protocol for this, not yet Puppeteer.

patd将近 7 年前

I'm currently using a headless Chrome for my latest project www.blockedby.com (still in alpha stage, looking for feedback)I've been looking at a non-local solution. I'm using Python and this article hints that Puppeteer is not the only way to invoke this. But I don't see any documentation on the DevTools protocol.Anyone knows if it's supported ? Or any providers that do ?

评论 #17799204 未加载

评论 #17798909 未加载

kwerk将近 7 年前

I’ve tested HC on GCF and GAE standard with the IO launch. Sadly they’re 3-10x slower than App Engine Flex (same vm size on App Engine Flex vs Standard). Even a screenshot of google.com takes 6+ seconds on GCF / GAE Standard vs 2 seconds for Flex. I hope they fix this as spinning to zero is important for me but the latency is too high right now.

nojvek将近 7 年前

I don’t know what this means for browserless.io but I hope he still retains a strong niche and has a desirable product.

eknkc将近 7 年前

Tried HC just yesterday on cloud functions. For some reason, it runs extremely slow. Did some comparisons to AWS lambda with similar memory / cpu sizes and basic “load page and screenshot” jobs would take 2x - 3x more time on google cloud functions.I’ll dig deeper soon but this is a bad start.

评论 #17796235 未加载

dakom将近 7 年前

Can someone please explain what the advantage of running a snapshot service via GCF would be vs. AppEngine Standard (w/ node)?

评论 #17798284 未加载

isuckatcoding将近 7 年前

I wonder how feature/pricing compete with Browserless?

评论 #17796274 未加载

techsin101将近 7 年前

How Google is behaving recently... I can feel it becoming Oracle. I want to stay 10 miles far from it. Learned my lesson with Google maps.

benatkin将近 7 年前

I'm not buying it, Google Cloud just moved to node 8 earlier this year as the post says, but now it's node 10. It's just not good tech, it's unnecessary lock-in. Docker on anything is better, this is similar: <a href="https://zeit.co/blog/serverless-docker" rel="nofollow">https://zeit.co/blog/serverless-docker</a>

评论 #17797316 未加载

jancurn将近 7 年前

It's nice to see serverless platforms adding support for headless Chrome. But there's still one problem with AWS Lambda / Cloud Functions / Zeit Now - the run time is limited to a few minutes. If you want to run any longer job, e.g. a web crawler, you need to either spin up the instances yourself or use platform like Apify, which allows running arbitrary-long jobs, provides pre-built Docker images for headless Chrome or XVFB, and provides SDK to simplify state persistence, access to proxies etc.For example, a simple actor to convert HTML to PDF looks like this:<a href="https://www.apify.com/jancurn/url-to-pdf" rel="nofollow">https://www.apify.com/jancurn/url-to-pdf</a>More info:<a href="https://www.apify.com/docs/actor" rel="nofollow">https://www.apify.com/docs/actor</a><a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest" rel="nofollow">https://www.apify.com/docs/sdk/apify-runtime-js/latest</a><a href="https://www.apify.com/library?type=acts" rel="nofollow">https://www.apify.com/library?type=acts</a>Disclaimer: I'm a co-founder of Apify