TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: WAL Implementation in Golang

59 点作者 stym0611 个月前
I wrote this simple WAL library in Golang that I use to write data that my kafka producer fails due to errors like Broker going down or some other issue. Took inspiration from etcd/wal

11 条评论

while1malloc011 个月前
Cool library. Two small generic Go library issues:<p>1. The rebuf.Init function panics. I almost never want a library to call panic, and when it does, I want the library function to denote that. The convention I’ve seen most often is to start the function name with Must, so MustInit instead of Init. In this case though, I think it’d be safe to be a little more lenient in what you accept as input and trim the trailing slash.<p>2. I never (not almost, actually never) want library code to call any of the fmt.Print functions unless the library is explicitly for writing output, or that behavior is strictly opt in. If the library really must print things, it should take a user supplied os.Writer and write to that. Let the user control what gets printed or not.
评论 #40912589 未加载
vlowther11 个月前
Having written one of these, a few optimizations will go a long way:<p>1. syscall.Iovec allows you to build up multiple batches semi independently and then write them all in a single syscall and sync the file with the next one. It is a good basis for allowing multiple pending writes to proceed in independent go routines and have another one have all the responsibility for flushing data.<p>2. It is better to use larger preallocated files than a bunch of smaller ones, along with batching, fixed size headers and padding write blocks to a known size. 16 megabytes per wal and a 128 byte padding worked well for me.<p>3. Batching writes until they reach a max buffer size and&#x2F;or a max buffer age can also massively increase throughput. 1 megabyte max pending write or 50 ms time passed worked pretty well for me for batching and throughput to start with, then dynamically tuning the last bound to the rolling average of the time the last 16 write+sync operations (and a hard upper bound to deal with 99th percentile latency badness) worked better. Bounded channels and a little clever math makes parallelizing all of this pretty seamless.<p>4. Mmap&#x27;ing the wals makes consistency checking and byte level fiddling much easier on replay. No need to seek or use a buffered reader, just use slice math and copy() or append() to pull out what you need.
评论 #40912598 未加载
tjungblut11 个月前
Besides what Phil mentioned below, I can&#x27;t write more than one record to the WAL. You&#x27;re closing the file after every write, the second time you write the error `seek data&#x2F;rebuf.tmp: file already closed` is returned.<p>I also think your rotation will delete the wrong segment when you have more than ten segments - imagine you&#x27;re writing rebuf-1 to rebuf-10 - what&#x27;s the &quot;oldest file&quot; to delete now? Besides, should you really delete those files?
评论 #40909059 未加载
Smaug12311 个月前
This is one of the absolutely classic cases where I&#x27;d expect a very small amount of property-based testing to flush out a very large number of bugs, by the way.
评论 #40912599 未加载
stym0611 个月前
OP here! Pls feel free to raise any bugs you encounter! I&#x27;ll be doing the following immmediate fixes:<p>1. Use fsync for durable writes in case of system crashes<p>2. Fix log-rotation-purging logic<p>3. Fix `file already closed` bug on consecutive writes<p>4. Add CRC checksum
0xjnml11 个月前
Another simple WAL: <a href="https:&#x2F;&#x2F;pkg.go.dev&#x2F;modernc.org&#x2F;file#WAL" rel="nofollow">https:&#x2F;&#x2F;pkg.go.dev&#x2F;modernc.org&#x2F;file#WAL</a>
DLion11 个月前
Having some tests is necessary to avoid mostly of the bugs that other comments are pointing out.<p>Perhaps it&#x27;s just me, but I don&#x27;t trust code that hasn&#x27;t been tested.
评论 #40912604 未加载
eatonphil11 个月前
Did I miss it or is there no call to os.File.Sync(), i.e. fsync, anywhere?<p>Since you mention etcd&#x2F;wal:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;etcd-io&#x2F;etcd&#x2F;blob&#x2F;v3.3.27&#x2F;wal&#x2F;wal.go#L671">https:&#x2F;&#x2F;github.com&#x2F;etcd-io&#x2F;etcd&#x2F;blob&#x2F;v3.3.27&#x2F;wal&#x2F;wal.go#L671</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;etcd-io&#x2F;etcd&#x2F;blob&#x2F;v3.3.27&#x2F;pkg&#x2F;fileutil&#x2F;sync.go">https:&#x2F;&#x2F;github.com&#x2F;etcd-io&#x2F;etcd&#x2F;blob&#x2F;v3.3.27&#x2F;pkg&#x2F;fileutil&#x2F;sy...</a>
评论 #40908591 未加载
neonsunset11 个月前
This reminds me of ZoneTree which is persistent LSM tree project based on top of WAL, written in C#: <a href="https:&#x2F;&#x2F;github.com&#x2F;koculu&#x2F;ZoneTree">https:&#x2F;&#x2F;github.com&#x2F;koculu&#x2F;ZoneTree</a><p>Similar to RocksDB.
评论 #40912612 未加载
Smaug12311 个月前
golangci-lint finds three errors in rebuf.go at commit 615209d. It&#x27;s never safe to write golang without the linters!
drgo11 个月前
in line 200 of rebuf.go, did you mean to return err (instead of returning nil even when an error occurs)?
评论 #40912609 未加载