TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

How to do distributed locking (2016)

244 点作者 yusufaytas7 个月前

10 条评论

jojolatulipe7 个月前
At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.
评论 #41902284 未加载
评论 #42010620 未加载
评论 #41898031 未加载
eknkc7 个月前
I tend to use postgresql for distributed locking. As in, even if the job is not db related, I start a transaction and obtain an advisory lock which stays locked until the transaction is released. Either by the app itself or due to a crash or something.<p>Felt pretty safe about it so far but I just realised I never check if the db connection is still ok. If this is a db related job and I need to touch the db, fine. Some query will fail on the connection and my job will fail anyway. Otherwise I might have already lost the lock and not aware of it.<p>Without fencing tokens, atomic ops and such, I guess one needs a two stage commit on everything for absolute correctness?
评论 #41899374 未加载
评论 #41895763 未加载
antirez7 个月前
I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog.<p>Btw, things to note in random order:<p>1. Check my comment under this blog post. The author had missed a <i>fundamental</i> point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points.<p>2. It is not true that you can&#x27;t wait an approximately correct amount of time, with modern computers an APIs. GC pauses are bound and monotonic clocks work. These are acceptable assumptions.<p>3. To critique the auto release mechanism in-se, because you don&#x27;t want to expose yourself to the fact that there is a potential race, is one thing. To critique the algorithm in front of its goals and its system model is another thing.<p>4. Over the years Redlock was used in a huge amount of use cases with success, because if you pick a timeout which is much larger than: A) the time to complete the task. B) the random pauses you can have in normal operating systems. Race conditions are very hard to trigger, and the other failures in the article were, AFAIK, never been observed. Of course if you have a super small timeout to auto release the lock, and the task may easily take this amount of time, you just committed a deisgn error, but that&#x27;s not about Redlock.
评论 #41895393 未加载
评论 #41895296 未加载
anonzzzies7 个月前
I am updating my low level and algo knowledge; what are good books about this (I have the one written by the author). I am looking to build something for fun, but everything is either a toy or very complicated.
评论 #41896260 未加载
egcodes7 个月前
Once I wrote a dist. lock blog using this resource. Here it is: <a href="https:&#x2F;&#x2F;medium.com&#x2F;sahibinden-technology&#x2F;an-easy-integration-of-distributed-lock-4b19a704ce49" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;sahibinden-technology&#x2F;an-easy-integration...</a>
dataflow7 个月前
&gt; The lock has a timeout (i.e. it is a lease), which is always a good idea (otherwise a crashed client could end up holding a lock forever and never releasing it). However, if the GC pause lasts longer than the lease expiry period, and the client doesn’t realise that it has expired, it may go ahead and make some unsafe change.<p>Hold on, this sounds absurd to me:<p>First, if your client <i>crashes</i>, then you don&#x27;t need a timed lease on the lock to detect this in the first place. The lock would get released by the OS or supervisor, whether there are any timeouts or not. If both of <i>those</i> crash too, then the connection would eventually break, and the network system should then detect that (via network resets or timeouts, lack of heartbeats, etc.) and then invalidate all your connections before releasing any locks.<p>Second, if the problem becomes that your client is <i>buggy</i> and thus holds the lock <i>too long</i> without crashing, then shouldn&#x27;t some kind of supervisor detect that and then kill the client (e.g., by the OS terminating the process) before releasing the lock for everybody else?<p>Third, if you <i>are</i> going to have locks with timeouts to deal with corner cases you can&#x27;t handle like the above, shouldn&#x27;t they notify the actual program somehow (e.g., by throwing an exception, raising a signal, terminating it, etc.) instead of letting it happily continue execution? And shouldn&#x27;t those cases wait for some kind of verification that the program was notified before releasing the lock?<p>The whole notion that timeouts should somehow permit the program execution to continue ordinary control flow sounds like the root cause of the problem, and nobody is even batting an eye at it? Is there an obvious reason why this makes sense? I feel I must be missing something here... what am I missing?
评论 #41897034 未加载
评论 #41897032 未加载
hoppp7 个月前
I did distributed locking with Deno, and Deno KV hosted by Deno Deploy.<p>Its using foundationdb, a distributed db. The deno instances running on local devices all connect to the same Deno KV to acquire the lock.<p>But using postgres, a select for update also works, the database is not distributed tho.
jroseattle7 个月前
We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke.<p>Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It&#x27;s something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a request if one is available, assign some metadata from the request to the allocated ticket, and remove it from consideration for future client requests.<p>We had failed campaigns in the past (over-allocation, under-allocation, duplicate allocation, etc.) so our concern was accuracy. Clients would connect and request a ticket; we wanted to exclusively distribute only the set of tickets available from the pool. If the number of client requests exceeded the number of tickets, the system should protect for that.<p>We tried Redis, including the naive implementation of getting the lock, checking the lock, doing our thing, releasing the lock. It was ok, but administrative overhead was a lot for us at the time. I&#x27;m glad we didn&#x27;t go that route, though.<p>We ultimately settled on...Postgres. Our &quot;distributed lock&quot; was just a composite UPDATE statement using some Postgres-specific features. We effectively turned requests into a SET operation, where the database would return either a record that indicated the request was successful, or something that indicated it failed. ACID transactions for the win!<p>With accuracy solved, we next looked at scale&#x2F;performance. We didn&#x27;t need to support millions of requests&#x2F;sec, but we did have some spikiness thresholds. We were able to optimize read&#x2F;write db instances within our cluster, and strategically load larger&#x2F;higher-demand campaigns to allocated systems. We continued to improve on optimization over two years, but not once did we ever have a campaign with ticket distribution failures.<p>Note: I am not an expert of any kind in distributed-lock technology. I&#x27;m just someone who did their homework, focused on the problem to be solved, and found a solution after trying a few things.
评论 #41895829 未加载
评论 #41895977 未加载
评论 #41896180 未加载
评论 #41896833 未加载
评论 #41895681 未加载
评论 #41897029 未加载
评论 #41896281 未加载
评论 #41897993 未加载
galeaspablo7 个月前
Many engineers don’t truly care about the correctness issue, until it’s too late. Similar to security.<p>Or they care but don’t bother checking whether what they’re doing is correct.<p>For example, in my field, where microservices&#x2F;actors&#x2F;processes pass messages between each other over a network, I dare say &gt;95% of implementations I see have edge cases where messages might be lost or processed out of order.<p>But there isn’t an alignment of incentives that fixes this problem. Ie the payment structures for executives and engineers aren’t aligned with the best outcome for customers and shareholders.
评论 #41895586 未加载
评论 #41895051 未加载
评论 #41895407 未加载
评论 #41894888 未加载
评论 #41897997 未加载
jmull7 个月前
This overcomplicates things...<p>* If you have something like what the article calls a fencing token, you don&#x27;t need any locks.<p>* The token doesn&#x27;t need to be monotonically increasing, just a passive unique value that both the client and storage have.<p>Let&#x27;s call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a hash of all the data in the store, though that&#x27;s probably not practical.) The logic becomes:<p>(1) client retrieves the current version token from storage, along with any data it may want to modify. There&#x27;s no external lock, though the storage needs to retrieve the data and version token atomically, ensuring the token is specifically for the version of the data retrieved.<p>(2) client sends the version token back along with any changes.<p>(3) Storage accepts the changes if the current token matches the one passed with the changes and creates a new version token (atomically, but still no external locks).<p>Now, you can introduce locks for other reasons (hopefully goods ones... they seem to be misused a lot). Just pointing out they are&#x2F;should be independent of storage integrity in a distributed system.<p>(I don&#x27;t even like the term lock, because they are temporary&#x2F;unguaranteed. Lease or reservation might be a term that better conveys the meaning.)
评论 #41895448 未加载
评论 #41895513 未加载
评论 #41895264 未加载
评论 #41895192 未加载
评论 #41895382 未加载
评论 #41895475 未加载