TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

A Case Study in LASP and Distribution at Scale [video]

117 点作者 cmeiklejohn超过 7 年前

7 条评论

elcritch超过 7 年前
This is great! I&#x27;m continuously nonplussed by deployment systems which require bootstrapping a master to bootstrap the system. Docker Swarm, Kubernetes, Rancher, Project FIFO, etc all require setting up hosts machines, designating some coordinator node(s), and then turning on the system.<p>Working a lot in IoT fields, or even with onsite deployments of services, this pattern becomes a blocker. It&#x27;s more difficult than traditional &quot;setup a server, turn it on, install software&quot; approach, especially for small deployments. There&#x27;s a lot of need (IMHO) for various orchestration systems which are self-boostrapping and fully P2P for smaller teams deploying on clouds, IoT devices in a given locale, research labs etc.<p>I&#x27;ve been watching your work for a while, and it&#x27;s great you&#x27;re moving toward usable systems! Is there a good place to watch your work and&#x2F;or engage with the community? I&#x27;m starting work on some new IoT devices, and while this won&#x27;t be directly applicable I&#x27;d like to play around with some of the core ideas.
评论 #15760887 未加载
评论 #15761207 未加载
cmeiklejohn超过 7 年前
Related work:<p>PPDP &#x27;17 paper on scaling: <a href="http:&#x2F;&#x2F;christophermeiklejohn.com&#x2F;publications&#x2F;ppdp-2017-preprint.pdf" rel="nofollow">http:&#x2F;&#x2F;christophermeiklejohn.com&#x2F;publications&#x2F;ppdp-2017-prep...</a><p>Ensuring monotonicity via types: <a href="http:&#x2F;&#x2F;prl.ccs.neu.edu&#x2F;blog&#x2F;2017&#x2F;10&#x2F;22&#x2F;monotonicity-types-towards-a-type-system-for-eventual-consistency&#x2F;" rel="nofollow">http:&#x2F;&#x2F;prl.ccs.neu.edu&#x2F;blog&#x2F;2017&#x2F;10&#x2F;22&#x2F;monotonicity-types-to...</a>
评论 #15760609 未加载
shalabhc超过 7 年前
Very interesting - LASP looks like it is based on CRDTs, which I understand work by attaching location and time metadata to the data operations (add&#x2F;remove) and then being able to merge history consistently.<p>Is there more work that generalizes this idea? E.g. attach metadata to more generic &#x27;messages&#x27; that operate on &#x27;processes&#x27; and then have the processes define how (or if) the merge occurs, given the history of all messages at different cloned replicas? Can LASP support something like this?
评论 #15760919 未加载
fenollp超过 7 年前
&gt; 10k EUR in 9 monts<p>Did you take a look at Scaleway&#x27;s &quot;Dedicated ARM cores&quot;? They start at ~3EUR for 4c&#x2F;2GB. <a href="https:&#x2F;&#x2F;www.scaleway.com&#x2F;pricing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.scaleway.com&#x2F;pricing&#x2F;</a><p>Their instances come up in under a minute. They have a nice client <a href="https:&#x2F;&#x2F;github.com&#x2F;scaleway&#x2F;scaleway-cli" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;scaleway&#x2F;scaleway-cli</a> and they are from the EU which is a great place to spend EU grants ;)<p>Not affiliated. Just a happy customer.
binarytemple超过 7 年前
Nice work. Distributed computation is the future. Also glad to see someone scaling Erlang node count.
评论 #15760748 未加载
rad_gruchalski超过 7 年前
I&#x27;ve been looking forward to watching the video and learn a little bit more about LASP itself. Been following your work closely on Twitter a while ago. Unfortunately, got a little bit lost about half way through. I am not exactly sure what was the point you are trying to put across and I still don&#x27;t really know what LASP is. It seems that it&#x27;s somehow about disseminating the state in distributed systems? However, the digs at Mesos, distributed Erlang, the use case of launching a cluster within a short period of time suggest that disseminating state isn&#x27;t the core of the talk.<p>I&#x27;ve done some work on gossip systems in the past, <a href="http:&#x2F;&#x2F;gossiperl.com" rel="nofollow">http:&#x2F;&#x2F;gossiperl.com</a> is the result of my research. Gossiperl was based on work I&#x27;ve done at Technicolor Virdata (shut down nearly couple of years ago). We&#x27;ve built a distributed device management &#x2F; IoT &#x2F; data ingestion platform consisting of over 70 VMs (EC2, OpenStack, SoftLayer). That was before Docker became popular, virtually everyone was thinking in terms of instances back then. These machines would hold different components of the platform: ZooKeeper, Kafka, Cassandra, some web servers, some hadoop with Samza jobs, load balancers, Spark and such. Our problem was the following: each of these components have certain dependencies. For example, to launch Samza jobs, one needs Yarn (the Hadoop one) and Kafka, to have Kafka, one needs ZooKeeper. If we were to launch these sequentially, that would take significant amount of time considering that each node would&#x27;ve get bootstrapped every single time from zero (base image with some common packages installed) using Chef and installing deb &#x2F; rpm packages from the repos. What we put in production was a gossip layer written in ruby, 300 lines or so. Each node would announce just a minimum set of information: what role it belongs to, what id within the role it has, the address. Each component would know the count of the dependency it requires within the overlay to bootstrap itself. For example, in EC2, we would request all these different machines at once. Kafka would be bootstrapping at the same time as ZooKeeper, Hadoop would be bootstrapping alongside. Each machine, when bootstrapped, would advertise itself in the overlay and the overlay would trigger a Chef run with a hand crafted run list for the specific role it belonged to. So each node would effectively receive a notification about every new member and decide to take an action, or not. Once 5 ZKs are up, Kafka nodes would configure themselves for ZooKeeper and launch. Eventually Kafka cluster was up. Similar process would&#x27;ve happen on all other systems, eventually leading to a complete cluster of over 70 VMs running (from memory) about 30 different systems being completely operational. Databases, dashboards, MQTT brokers, TLS, whatnot. We used to launch this thing at least once a day. The system would usually become operational within under half an hour, unless EC2 was slacking off. Our gossip layer was trivial. In this sort of platform there are always certain nodes that should reachable from outside: web server, load balancer, mqtt broker. Each of those would become a seed, any other node would contact one of those public nodes and start participating.<p>From the capabilities perspective, the closest thing resembling that kind infrastructure today, is HashiCorp Consul. Our gossip from Virdata is essentially what the service catalog in Consul is, our Chef triggers is what watches in Consul are. With these two things, anybody can put up a distributed platform like what you are describing in your talk and what we&#x27;ve built at Virdata. There are obviously dirty details like, one needs to have a clear separation of installation, configuration and run of the systems within the deployment. The packages can be installed concurrently on different machines, application of the configuration triggers the start (or restart), system becomes operational.<p>Or do I completely miss the point of the talk. I&#x27;d like to hear more about your experiences with Mesos. You&#x27;re not the first person claiming that it doesn&#x27;t really scale as far as the maintainers suggest.<p>By the way, HyParView, good to know, I&#x27;ve missed this in my own research. Maybe it&#x27;s time to dust off gossiperl.<p>* edit: wording
评论 #15763399 未加载
polskibus超过 7 年前
How does lasp stack compare to Akka&#x27;s?