TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Pipe Viewer – A Unix Utility You Should Know About

236 点作者 lnyan6 个月前

19 条评论

mort966 个月前
Pipe viewer is excellent. I use it all the time.<p>As of version 1.8.10[1], which includes my merge request[2] to add an &#x27;--output&#x27; option, it has even completely replaced my use of &#x27;dd&#x27; for writing disk images: &#x27;sudo pv -Yo &#x2F;dev&#x2F;mmcblk0 whatever.img&#x27; is nicer, has much better progress indication, automatically selects a more sensible buffer size, <i>and</i> begets fewer groans from UNIX neckbeards, than the old &#x27;sudo dd of=&#x2F;dev&#x2F;mmcblk0 if=whatever.img&#x27;. (The &#x27;-Y&#x27; causes pv to sync after each write, which greatly improves progress indication in Linux.)<p>Though it&#x27;s useful for much more of course. I use it for progress when compressing files (&#x27;pv blah | gzip ...&#x27;), when uploading files to the web (&#x27;pv blah | curl --upload-file - ...&#x27; — curl doesn&#x27;t show progress when uploading for whatever reason), or just when I wanna see that <i>something</i> is happening with an operation which would otherwise take a while (even things like a slow &#x27;du -h &#x2F;some&#x2F;path | sort -h&#x27; benefits from a &#x27;pv&#x27; squeezed in the middle just to indicate that something is happening).<p>[1] <a href="https:&#x2F;&#x2F;codeberg.org&#x2F;a-j-wood&#x2F;pv&#x2F;releases&#x2F;tag&#x2F;v1.8.10" rel="nofollow">https:&#x2F;&#x2F;codeberg.org&#x2F;a-j-wood&#x2F;pv&#x2F;releases&#x2F;tag&#x2F;v1.8.10</a><p>[2] <a href="https:&#x2F;&#x2F;codeberg.org&#x2F;a-j-wood&#x2F;pv&#x2F;pulls&#x2F;90" rel="nofollow">https:&#x2F;&#x2F;codeberg.org&#x2F;a-j-wood&#x2F;pv&#x2F;pulls&#x2F;90</a>
heinrich59916 个月前
There&#x27;s also `progress` which works for tools mainly operating on a single file, but unlike `pv`, you don&#x27;t have to start the tool differently. It&#x27;d e.g. work nicely for the `gzip` example. Just call `progress` on a different terminal while `gzip` is running.
评论 #42199327 未加载
评论 #42200650 未加载
arghwhat6 个月前
&gt; This example shows that the access.log file is being read at the speed of 37.4MB&#x2F;s but gzip is writing data at only 1.74MB&#x2F;s. We can immediately calculate the compression rate. It&#x27;s 37.4&#x2F;1.74 = 21x!<p>No you cannot:<p>1. Compression algorithms buffer a lot and will have have tendencies to perform large burst writes and in particular large writes on final flush. Instantaneous measurements will therefore not be particularly useful.<p>2. Compression ratio refers to the average across an entire file, as entropy is not uniform across the input unless you are compressing just noise or just zeros. Some sections might compress extremely well while others do not and end up consuming more space than the input due to overhead.
评论 #42203733 未加载
jcul6 个月前
pv is great.<p>It has a limit parameter so you can limit the speed. Great if you don&#x27;t want to saturate some link or have additional costs for uploading above a certain rate per hour&#x2F;day.<p>Also useful for testing behaviour on slow filesystem &#x2F; connections.<p>It can take a pid argument too, -d IIRC, which will get it to display progress info for all the open file descriptors of a running process.<p>Really useful as a quick way to check what a IO process is doing if appears to be stuck.
darkwater6 个月前
Pipe viewer? What&#x27;s that? Let me check the post...oh, it&#x27;s good old pv! Never noticed it had a full name, damn Unix utilities with their short names!
评论 #42198740 未加载
NelsonMinar6 个月前
I love pv but how much does adding the pipe affect overhead? I feel like most of my big jobs I want to measure are on things where you want the program to have direct access to the underlying file or storage. `pv somefile | dd` is going to be slower than `dd somefile`. At least I think so? I have no idea what modern Linux I&#x2F;O can optimize.<p>Also does pv necessitate doing single threaded I&#x2F;O?
codetrotter6 个月前
I used to love pv, but I had ZFS send and recv hang many times when I was using pv in the pipeline and I was never sure why but after I stopped using pv in the pipeline and started using the verbose flag of the ZFS command on the receive side instead which provides enough output for me to see that it’s progressing and haven’t had those kinds of problems since.<p>Searching now it seems indeed that up until recently this was a problem other people have been having too. For example the recent forum thread <a href="https:&#x2F;&#x2F;forums.freebsd.org&#x2F;threads&#x2F;zfs-send-hanging.94994&#x2F;" rel="nofollow">https:&#x2F;&#x2F;forums.freebsd.org&#x2F;threads&#x2F;zfs-send-hanging.94994&#x2F;</a> where they were discussing which version this was fixed in and someone saying that the latest available version to them from the packages was still a bit too old.
throwaway1274826 个月前
I like to use pv as a quick and dirty operations per second counter. Sometimes I will write a program or script that does a bunch of things in parallel (e.g. RPCs to a service I&#x27;m working on), and prints one line of output for every operation completed. Then I pipe that output to pv using the --lines option to count only lines. It shows how many lines are being printed per second, which roughly counts operations per second. (IIRC, also need to pipe to &#x2F;dev&#x2F;null to prevent pv&#x27;s fancy output from clobbering the tool&#x27;s output).<p>Fun stuff! Especially when combined with GNU parallel, in cases where the thing I&#x27;m measuring isn&#x27;t already parallelized, and I want to be lazy.
6c696e75786 个月前
A little more typing, but I find dd present on most systems already, so I tend to do this:<p><pre><code> tar ... | dd status=progress | ...</code></pre>
评论 #42198684 未加载
评论 #42200618 未加载
评论 #42201324 未加载
评论 #42198757 未加载
emptiestplace6 个月前
&gt; The obvious way to do it is:<p>&gt; $ gzip -c access.log &gt; access.log.gz<p>Is it?
评论 #42199308 未加载
ElevenLathe6 个月前
See also vipe(1) from the wonderful moreutils: <a href="https:&#x2F;&#x2F;manpages.debian.org&#x2F;stretch&#x2F;moreutils&#x2F;vipe.1.en.html" rel="nofollow">https:&#x2F;&#x2F;manpages.debian.org&#x2F;stretch&#x2F;moreutils&#x2F;vipe.1.en.html</a>
评论 #42202348 未加载
sirjaz6 个月前
We have that in powershell also show-progress
评论 #42198895 未加载
thanatos5196 个月前
Yes! My `,pv` is approximately: (probably a better way to make the k, but I stop once something is adequate; maybe I just need to make a `,,kof`)<p><pre><code> tar cpS &quot;$@&quot; --sort=name | pv -bratpes $(du -cks &quot;$@&quot;|sed -n &#x27;&#x2F;.total$&#x2F; s&#x2F;.total$&#x2F;&#x2F;p&#x27;)k </code></pre> Which gives me progress bars for big copies like:<p><pre><code> ,pv files&#x2F; | tar xp -C &#x2F;destination ,pv files&#x2F; | ssh overthere tar xp -C &#x2F;destination</code></pre>
kazinator6 个月前
If you want the drill into text flowing through a pipeline, I made a different tool called Pipe Watch (pw).<p><a href="https:&#x2F;&#x2F;www.kylheku.com&#x2F;cgit&#x2F;pw&#x2F;about" rel="nofollow">https:&#x2F;&#x2F;www.kylheku.com&#x2F;cgit&#x2F;pw&#x2F;about</a>
dustfinger6 个月前
In addition to pv, I also recommend learning about watch [1], which allows you to watch things change over time.<p>[1]: <a href="https:&#x2F;&#x2F;linux.die.net&#x2F;man&#x2F;1&#x2F;watch" rel="nofollow">https:&#x2F;&#x2F;linux.die.net&#x2F;man&#x2F;1&#x2F;watch</a>
dazzawazza6 个月前
Also available on FreeBSD <a href="https:&#x2F;&#x2F;www.freshports.org&#x2F;sysutils&#x2F;pv" rel="nofollow">https:&#x2F;&#x2F;www.freshports.org&#x2F;sysutils&#x2F;pv</a>
0823498723498726 个月前
tangentially, <i>tee</i>(1) can also be useful for deciding which update in the middle of a long pipeline broke things
c0deR3D6 个月前
Got me wondering, how does it works?
mac3n6 个月前
see also<p><a href="https:&#x2F;&#x2F;gitlab.com&#x2F;mac3n&#x2F;pipeleak" rel="nofollow">https:&#x2F;&#x2F;gitlab.com&#x2F;mac3n&#x2F;pipeleak</a>