TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Show HN: Pings 8000 servers in 11 seconds: Parallel HTTP/SSH/TCP/Ping library

22 pointsby jeffpeiytover 9 years ago

2 comments

jeffpeiytover 9 years ago
Hi Uberneo, yes, we exactly use it for HTTP, but also others. here is a 20 lines example to extract from a lot of HTTPs to elastic search. <a href="https:&#x2F;&#x2F;github.com&#x2F;eBay&#x2F;parallec-samples&#x2F;blob&#x2F;master&#x2F;sample-apps&#x2F;src&#x2F;main&#x2F;java&#x2F;io&#x2F;parallec&#x2F;sample&#x2F;app&#x2F;http&#x2F;Http3WebAgrregateToElasticSearchMinApp.java" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;eBay&#x2F;parallec-samples&#x2F;blob&#x2F;master&#x2F;sample-...</a><p>Parallec has a special super convenient response context let you pass in&#x2F;out any object when handling the response. Now you can conduct scalable API calls, then pass aggregated data anywhere to elastic search, kafka, MongoDB, graphite, memcached, etc.<p>Python has global interpreter lock so if it is computational expensive, you have to use multi-process to use more than 1 core. Parallec can let the handler to run your onComplete() function either in worker before aggregation (parallel) or in manager after aggregation
评论 #10509534 未加载
uberneoover 9 years ago
Can this be used to scrape data from website parallely like scrapy?