TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

All my servers have an 8 GB empty file on disk

1345 点作者 sonicrocketman大约 4 年前

146 条评论

rgj大约 4 年前
In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason.<p>One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few percent of disk space.<p>Turned out that we had a number of people who, as soon as the disk had some free space, created large files in order to reserve that free space for themself. About half the capacity of the fileserver was taken up by balloon.txt files.
评论 #26585300 未加载
评论 #26585874 未加载
评论 #26585073 未加载
评论 #26585690 未加载
评论 #26586598 未加载
评论 #26585489 未加载
评论 #26585270 未加载
评论 #26590036 未加载
评论 #26588554 未加载
评论 #26584918 未加载
评论 #26588104 未加载
评论 #26587201 未加载
评论 #26592617 未加载
geocrasher大约 4 年前
For everyone saying &quot;This isn&#x27;t a real solution!&quot; I&#x27;d like to explain why I think you&#x27;re wrong.<p>1) It&#x27;s not intended to be a Real Solution(tm). It&#x27;s intended to buy the admin some time to solve the Real Issue.<p>2) Having a failsafe on standby such as this will save an admin&#x27;s butt when it&#x27;s 2am and PagerDuty won&#x27;t shut up, and you&#x27;re just awake enough to apply a temp fix and work on it in the morning.<p>3) Because &quot;FIX IT NOW OR ELSE&quot; is a thing. Okay, sure. Null the file and then fill it with 7GB. Problem solved, for now. Everybody is happy and now I can work on the Real Problem: Bob won&#x27;t stop hoarding spam.<p>That is all.
评论 #26585247 未加载
评论 #26585215 未加载
评论 #26586311 未加载
评论 #26589019 未加载
评论 #26592079 未加载
评论 #26585678 未加载
评论 #26585688 未加载
评论 #26585274 未加载
SethTro大约 4 年前
Same idea as this game development legend<p><a href="https:&#x2F;&#x2F;www.dodgycoder.net&#x2F;2012&#x2F;02&#x2F;coding-tricks-of-game-developers.html" rel="nofollow">https:&#x2F;&#x2F;www.dodgycoder.net&#x2F;2012&#x2F;02&#x2F;coding-tricks-of-game-dev...</a><p>&gt; he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it&#x27;s really needed.
评论 #26584694 未加载
评论 #26585276 未加载
评论 #26586563 未加载
评论 #26601871 未加载
评论 #26584444 未加载
评论 #26584452 未加载
ryandrake大约 4 年前
A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I&#x27;d like to highlight the statement: &quot;The disk filled up, and that&#x27;s one thing you don&#x27;t want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.&quot;<p>As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pegged CPU. We typically see the bug in our triage queue and think in our minds &quot;Oh! out of disk space: Edge case. P3. Punt it to the backlog forever.&quot; This is how we get in this place where every tool in the toolbox simply stops working when there&#x27;s zero disk space.<p>Especially on today&#x27;s mobile devices, running out of disk space is common. I know people who install apps, use them, then uninstall them when they&#x27;re done, in order to save space, because their filesystem is choked with thousands of pictures and videos. It&#x27;s not an edge case anymore, and should not be treated as such.
评论 #26586764 未加载
评论 #26589819 未加载
评论 #26585696 未加载
评论 #26589658 未加载
评论 #26585572 未加载
评论 #26590807 未加载
herpderperator大约 4 年前
The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as &quot;reserved blocks&quot;.[0] That means if a non-root user uses all of the available space, it wasn&#x27;t really all of the space -- root still has access to 5% free space within that partition. That&#x27;s exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of disk space, you actually have 5% left for root. So log in as root and fix the issue. Simple.<p>Also:<p>&gt; On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there&#x27;s very little you can do to free up large chunks of space.<p>Why would memory swap to disk when the disk is full? I feel like the author is conflating potential memory pressure issues with disk issues.<p>How many serious production-grade servers even use swap, which usually just causes everything to grind to a halt if memory becomes full?<p>[0] <a href="https:&#x2F;&#x2F;ma.ttias.be&#x2F;change-reserved-blocks-ext3-ext4-filesystem-linux&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ma.ttias.be&#x2F;change-reserved-blocks-ext3-ext4-filesys...</a>
评论 #26591087 未加载
评论 #26589098 未加载
评论 #26589560 未加载
评论 #26589103 未加载
评论 #26589031 未加载
评论 #26588443 未加载
AlisdairO大约 4 年前
One other option is increasing the reserved block count ( <a href="https:&#x2F;&#x2F;ma.ttias.be&#x2F;change-reserved-blocks-ext3-ext4-filesystem-linux&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ma.ttias.be&#x2F;change-reserved-blocks-ext3-ext4-filesys...</a> ). This has the nice side effect of increasing the space available for critical daemons.<p>If you haven&#x27;t customised this, in a pinch you can still lower it down a bit to buy some time.
评论 #26584385 未加载
评论 #26584226 未加载
评论 #26584174 未加载
评论 #26584185 未加载
scottlamb大约 4 年前
&gt; On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there&#x27;s very little you can do to free up large chunks of space.<p>This reasoning doesn&#x27;t make sense. On Linux, swap is preallocated. This is true regardless of whether you&#x27;re using a swap partition or a swap file. See man swapon(8):<p>&gt; The swap file implementation in the kernel expects to be able to write to the file directly, without the assistance of the filesystem. This is a problem on files with holes or on copy-on-write files on filesystems like Btrfs.<p>&gt; Commands like cp(1) or truncate(1) create files with holes. These files will be rejected by swapon.<p>I just verified on Linux 5.8.0-48-generic (Ubuntu 20.10) &#x2F; ext4 that trying to swapon a sparse file fails with &quot;skipping - it appears to have holes&quot;.<p>Now, swap is horribly slow, particularly on spinning rust rather than SSD. I run my systems without any swap for that reason. But swapping shouldn&#x27;t fail on a full filesystem, unless you&#x27;re trying to create &amp; swapon a new swapfile after the filesystem is filled.
评论 #26584720 未加载
评论 #26585748 未加载
评论 #26585662 未加载
terramex大约 4 年前
Ah, the classic &#x27;speed-up loop&#x27; approach: <a href="https:&#x2F;&#x2F;thedailywtf.com&#x2F;articles&#x2F;The-Speedup-Loop" rel="nofollow">https:&#x2F;&#x2F;thedailywtf.com&#x2F;articles&#x2F;The-Speedup-Loop</a><p>About the blogpost itself:<p><i>The disk filled up, and that&#x27;s one thing you don&#x27;t want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.</i><p>I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full system reboot. I was also unable to save any open file, even to external disk and suffered minor data loss every time due to that.<p>What is the proper way of dealing with such issue on macOS? (or other systems, if they behave the same way)
评论 #26584952 未加载
评论 #26584262 未加载
评论 #26584893 未加载
评论 #26597388 未加载
anonymousisme大约 4 年前
One thing that many Linux&#x2F;Unix users do not know is that all commonly used filesystems have a &quot;reserved&quot; amount of space to which only &quot;root&quot; can write. The typical format (mkfs) default is to leave 5% of the disk reserved. The reserved space can be modified (by root) any time, and it can be specified as a block count or a percentage.<p>As long as your application does not have root privileges, it will hit the wall when the free+reserved space runs out. Instead of the clumsy &quot;spacer.img&quot; solution, one could simply (temporarily) reduce the reserved space to quickly recover from a disk full condition.
评论 #26586647 未加载
raldi大约 4 年前
This reminds me of Perl’s esoteric $^M variable. You assign it some giant string, and in an out-of-memory condition, the value is cleared to free up some emergency space for graceful shutdown.<p>“To discourage casual use of this advanced feature, there is no English long name for this variable.”<p>But the language-build flag to enable it has a great name: -DPERL_EMERGENCY_SBRK, obviously inspired by emergency brake.
评论 #26585462 未加载
评论 #26585219 未加载
评论 #26585338 未加载
john_moscow大约 4 年前
I happen to run a couple of small servers myself and here&#x27;s a better version of this approach. Create a cron job that will run a simple self-testing script once every few hours. My self-test does this:<p>1. Checks that all domains can be accessed via HTTP and HTTPS. If not, DNS might have died.<p>2. Checks that a few known CMS-generated pages contain some phrases they should contain. If not, SQL might have died.<p>3. Checks that the HTTPS certificate has enough runway left. If not, certbot might have died.<p>4. Sends a basic email message from my domain to a gmail account. Receives it via IMAP and sends a reply. Then, verifies the reply. This catches a whole bunch of mail-related issues.<p>5. Checks the free RAM and disk space. Updates an internal &quot;dashboard&quot; page and sends me an email if they are off.<p>It only took a couple of hours to hack this together and I must say, I get a much better night time sleep ever since.
评论 #26587561 未加载
kristjansson大约 4 年前
Lots of comments assailing this approach as a poor replacement for monitoring miss the point. Of course monitoring and proactive repair are preferable - but those are systems that can also fail!<p>This is a low cost way to make failure of your first line of defense less painful to recover, and seems like a Good Idea for those managing bare-metal non-cattle systems.
prewett大约 4 年前
This reminds me of an old gamedev story that I have no idea how to find. The project was getting near to shipping, they had cut all the space they could cut, but they still needed another megabyte of space. After a week of this, the senior dev told the narrator to meet him in his office, and he closed the door. He opened one of the project files and deleted a 1 MB static array. &quot;At the beginning of development I always reserve space for just this occasion,&quot; he said. Shortly afterwards he emerged from his office, announced that he had been able to find some extra space, and was lauded as a hero.
评论 #26588692 未加载
moistbar大约 4 年前
When I worked at SevOne, we had 10x500 MB files on each disk that were called ballast files. They served the same purpose, but there were a couple nice tools built in to make sure they got repopulated when disk space was under control, plus alerting you whenever one got &quot;blown.&quot; IIRC it could also blow ballast automatically in later versions, but I don&#x27;t remember it being turned on by default.
badcc大约 4 年前
This trick has certainly saved me more times than I am willing to admit! I usually roll with: `fallocate -l 8G DELETE_IF_OUT_OF_SPACE.img`
评论 #26586325 未加载
CrLf大约 4 年前
This is why the invention of LVM was such a good idea even for simpler systems (where some people claimed it was useless overhead). In my old sysadmin days I <i>never</i> allocated a full disk. The &quot;menace&quot; of an almost full filesystem was usually enough to incentivize cleanups but, when necessity came, the volume could be easily expanded.<p>I guess a big file is not a bad idea either.
tyingq大约 4 年前
Careful how you create it. Several ways to create large files can make a sparse file, which I don&#x27;t think removing will actually help later.
评论 #26585832 未加载
jedberg大约 4 年前
Since the late 90s, this was always my solution:<p><pre><code> tune2fs -m 2 &#x2F;dev&#x2F;hda1 </code></pre> That sets the root reserve on a disk. It&#x27;s space that only root can use, but also you can change it on the fly. So if you run out of userland space you can make it smaller, and if a process running as root fill your disk, well, you probably did something real bad anyway. :)<p>But yeah, this is a pretty good hack.
tshaddox大约 4 年前
How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!
评论 #26585072 未加载
评论 #26584974 未加载
评论 #26587853 未加载
评论 #26585290 未加载
评论 #26585041 未加载
评论 #26585821 未加载
评论 #26585210 未加载
gherkinnn大约 4 年前
An architect once told me that he always plans for a solid gold block hidden away in the cellar.<p>Once the project invariably goes over budget, he drops the plans for the gold and frees up extra funds.<p>Edit: I think it was a large marble slab. Same thing.
评论 #26584703 未加载
danielrhodes大约 4 年前
My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.
评论 #26585064 未加载
评论 #26587896 未加载
zeta0134大约 4 年前
If you happen to use ext as your default filesystem, check the output of tune2fs; it&#x27;s possible your distro has conveniently defaulted some 2-5% of disk space as &quot;reserved&quot; for just such an occasion. As the root user, in a pinch, you can set that to 0% and immediately relieve filesystem pressure, buying you a little bit more time to troubleshoot whatever the real problem is that filled the disk in the first place.
diego大约 4 年前
This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you&#x27;re low on fuel, but your server doesn&#x27;t if you&#x27;re low on critical resources.
评论 #26585130 未加载
评论 #26585733 未加载
评论 #26585322 未加载
评论 #26584682 未加载
评论 #26585303 未加载
评论 #26585351 未加载
johbjo大约 4 年前
No. Careful partitioning is the solution to this problem. Monitor the growth of your partitions and make sure nothing on rootfs or other sensitive partitions grow significantly.
评论 #26584915 未加载
评论 #26587387 未加载
SpaceInvader大约 4 年前
To extend space in any filesystem in the root volume group on AIX you need space in &#x2F;tmp. Years ago while working for some major bank I proposed to create such dummy file in &#x2F;tmp exactly for the reason of extending filesystem. It saved us several times :)
rags2riches大约 4 年前
Back in my early university days the disks always seemed to be full at inconvenient times on the shared Unix systems we used. Some students resorted to &quot;reserving&quot; disk space when available. Which of course made the overall situation even worse.
pritambarhate大约 4 年前
All my servers have an alarm when disk space goes above 70%. It sends an email every hour once the disk usage goes above 70%. Never had a server go down because of disk space issue after adopting this practise.<p>Also one of the main reasons server disks go full is generally log files. Always remember to &quot;logrotate&quot; your log files and you will not have this issue that much.<p>Yes one more thing, for all user uploaded files use external storage like NFS or S3.
评论 #26584950 未加载
评论 #26585408 未加载
评论 #26585545 未加载
评论 #26584880 未加载
cmckn大约 4 年前
This really goes to show, there is more than one way to skin a cat. Yeah the guy could probably overhaul his entire approach to system administration, but also...this works. Well-placed hacks are maybe my favorite thing.
davidelettieri大约 4 年前
I always thought that database files should be on a different drive from the os. If the db fills up the HD, the os is still running smoothly.
评论 #26586406 未加载
aidenn0大约 4 年前
This won&#x27;t work with ZFS, as it may be impossible to delete a file on ZFS when disk is full. The equivalent in ZFS is to create an empty dataset with reserved space.
评论 #26585708 未加载
评论 #26586030 未加载
评论 #26585825 未加载
Saris大约 4 年前
It&#x27;s interesting to me that linux doesn&#x27;t natively reserve a little space to allow basic commands like directory listing and file deletion to function even with a full disk.<p>Because really the biggest problem when I&#x27;ve had a partition get full, is I sometimes can&#x27;t even delete the offending log file.
评论 #26585637 未加载
londons_explore大约 4 年前
Linux has this built in...<p>By default, only root can use the last 5% of disk space.<p>That means you can fire up a root shell and know you have a buffer of free space to resolve the issue.
评论 #26585237 未加载
评论 #26585288 未加载
nycdotnet大约 4 年前
This is an old trick for when you need to deploy to media with a fixed size - floppy&#x2F;CD-ROM&#x2F;etc. Make a file that is 5-10% the size of your media and don&#x27;t remove unless you&#x27;re running out of space in crunch time.
clipradiowallet大约 4 年前
An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv&#x27;s, add them to your vg&#x27;s, and finally expand your lv&#x27;s.<p>some reading if LVM is new and you want to know more: <a href="https:&#x2F;&#x2F;opensource.com&#x2F;business&#x2F;16&#x2F;9&#x2F;linux-users-guide-lvm" rel="nofollow">https:&#x2F;&#x2F;opensource.com&#x2F;business&#x2F;16&#x2F;9&#x2F;linux-users-guide-lvm</a><p>edit to add: pv=physical volume, vg=volume group, lv=logical volume
评论 #26584399 未加载
评论 #26584405 未加载
评论 #26585086 未加载
评论 #26585082 未加载
deeblering4大约 4 年前
if you are on an ext filesystem, reducing the reserved percentage on the full filesystem can save the day. its more or less this same trick built in to the filesystem<p>IIRC 5% is reserved when the filesystem os created, and if it gets full you can run:<p>tune2fs -m 4 &#x2F;dev&#x2F;whatever<p>which will instantly make 1% of the disk available.<p>of course should be used sparingly and restored when finished
fggg444大约 4 年前
it feels like setting your watch 5 minutes fast, it&#x27;s not a real solution
评论 #26585389 未加载
ineedasername大约 4 年前
A great idea, but it still leaves the possibility for performance issues prior to an admin&#x27;s ability to address is. Some like two 4gb blocks might work better: if you get within, say, 200mb of storage limits you remove the first one and trigger an email&#x2F;text&#x2F;whatever to the admin, that way they can address it before it goes further. It&#x27;s an early warning and automated solution. Then, if the situation continues, the second 4gb block is also automatically removed with another message send to the admin. Nothing fails silently.
kchoudhu大约 4 年前
This is why I insist on data and root partitions on all the machines I administer. Go ahead and kill the data partition, at least the root partition will keep the system up and running.
erlkonig大约 4 年前
For ext* filesystems, you can use tune2fs to change the reserved block percentage to accomplish this in what might - depending on your preferences - be a more graceful way.<p>Basically it lets you knock 8 GB or more (although it&#x27;s a percentage instead, 5% by default) off of the disk space available to non-root users.<p>When it hits 100% and things start breaking, that reserve can be used by root to do compression safely, move things around, and so on. Alternatively the reserve percentage can be changed with a single command (by root), to allow non-root processes more space while the admin contemplates what do do next.<p>One nice aspect of using the reserve instead of a file is that it prevents runs of &quot;du&quot; for including the file in their results. Another is that it&#x27;s pretty much impossible to accidentally remove the reserve (or for some other admin to find it and decide it&#x27;s superfluous).<p>This is less effective at sites that have a lot of services running as root, in which case only your approach is fully effective. I want to say &quot;But who <i>does</i> that nowadays...&quot;, but it happens.<p>tune2fs apparently also supports allowing members of a certain unix group or user to have access instead of solely root.<p>The core command for all this is:<p><pre><code> tune2fs -m &lt;reserved-percent&gt; &lt;device&gt; </code></pre> One other thing you might want to worry about: inode exhaustion. tune2fs has an inode reserve % as well - and trying to emulate this by creating a few hundred thousand files instead would be... inelegant.
nijave大约 4 年前
The real question... Why does Linux or at least the common filesystems get stuck so easily running out of disk space? Surely normal commands like `rm` should still function.
评论 #26585506 未加载
评论 #26585363 未加载
AYBABTME大约 4 年前
Once upon a time, I wanted to cache large and expensive to pull files on many thousands of servers. Problem is the disk space on these servers was at premium and meant to be sold for customer use. The servers did have scratch space on small disks, but that was used by the OS.<p>So I wrote an on-disk cache system that would monitor disk usage, and start to evict and shrink its disk space usage. It would take up to N gigabytes of disk (configurable) for the purpose of caching, and maintain an M gigabytes free-disk-space buffer.<p>Say you had a 100 GiB total space on a partition, with 8 GiB used for cache with a 2 GiB headroom. As legitimate&#x2F;regular (customer) space usage increased and reached 91 GiB, the cache would see 9 GiB available, and removing the 2 GiB buffer, would start to evict items to resize to 7 GiB, and so on until it had evicted everything.<p>When this system deployed, it started to trigger low-disk-space alerts earlier than before. At first that seemed like a problem, but the outcome is that we were now getting low-disk-space alerts with more advance warning, and the cache bought some time as it kept resizing down to free up space. It kind of, in a way, served the same purpose as described in this blog post.<p>Overall this cache was pretty neat and still is, I bet. There&#x27;s probably ways to do similar things with fancy filesystems (or obscure features) but this was a quick thing to deploy across all servers without having to change any system setting or change the filesystem.<p>I sometime wish I had done this in open-source, because it would be convenient to use locally on my laptop, or on many servers.
评论 #26593808 未加载
rektide大约 4 年前
hope you&#x27;re not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don&#x27;t actually get much space back!<p>i do wonder how many FS would actually allocate the 8GB if you, for example, opened a file, seeked to 8GB mark, and wrote a character. many file systems support &quot;sparse files&quot;[1]. for example on btrfs, i can run &#x27;dd if=&#x2F;dev&#x2F;zero of=example.sparse count=1 seek=2000000&#x27; to make a &quot;1GB&quot; file that has just one byte in it. btrfs will only allocate a very small amount in this case, some meta-data to record an &quot;extent&quot;, and a page of data.<p>i was expecting this article to be about a rude-and-crude overprovisioning method[2], but couldn&#x27;t guess how it was going to work. SSDs notably perform much much better when they have some empty space to make shuffling data around easier. leaving a couple GB for the drive to do whatever can be a colossal performance improvement, versus a full drive, where every operation has to scrounge around to find some free space. i wasn&#x27;t sure how the author was going to make an empty file that could have this effect. but that&#x27;s not what was going on here.<p>[1] <a href="https:&#x2F;&#x2F;wiki.archlinux.org&#x2F;index.php&#x2F;sparse_file" rel="nofollow">https:&#x2F;&#x2F;wiki.archlinux.org&#x2F;index.php&#x2F;sparse_file</a><p>[2] <a href="https:&#x2F;&#x2F;superuser.com&#x2F;questions&#x2F;944913&#x2F;over-provisioning-an-ssd-does-it-still-hold&#x2F;944915" rel="nofollow">https:&#x2F;&#x2F;superuser.com&#x2F;questions&#x2F;944913&#x2F;over-provisioning-an-...</a>
评论 #26584028 未加载
评论 #26584760 未加载
评论 #26584015 未加载
skytreader大约 4 年前
Oh man, reminds me of a Game Dev war story I read years ago. This purportedly happened in those console days with very limited memory capabilities.<p>In some game studio, as a project neared its release, the team was still struggling with memory issues. No matter what they did, they had a surplus of just about 2MB. The artists have reduced their polygon counts drastically, the programmers have checked every possible leak, have optimized algorithms and buffers the best they could but the 2MB surplus just kept haunting them.<p>That&#x27;s when the VP of Engineering stepped-in. Calling the TL of the project into a closed-doors optimization code-review, they had the source code on a large screen and the TL talked the VP through everything the team has done so far to stay within the memory budget.<p>As the TL finished the walkthrough, the VP opens some mother-of-all files and deletes a cryptic variable declaration to the effect of:<p><pre><code> int toLiveBuffer[2000000]; </code></pre> The VP then explains that he hid this declaration in their codebase after a project that had to optimize drastically late into the development cycle. But first he wanted to make sure that the team did their homework.<p>And poof. They emerge from the closed-doors meeting jubilant and victorious. The game is ready for prime time!
评论 #26590396 未加载
ultrafez大约 4 年前
A fun problem on a Mac is that if you&#x27;re using APFS for your filesystem, if it fills up, you can&#x27;t delete any files. It&#x27;s caught me out a handful of times, and each time, the only way to recover is to reboot, and thankfully I&#x27;ve had more free disk space after a reboot.<p>I&#x27;m not going to try to understand the logic as to why APFS requires free space in order to delete files (via any method, including dd)
评论 #26594539 未加载
cwt137大约 4 年前
In theory, this is a good idea, but doesn&#x27;t protect you in all cases. I have had instances on a few of my application servers where an event happened that dumped GB&#x27;s worth of log data to the log files in a matter of a couple of minutes and filled up the drive (Thanks fast SSDs!). If I employed the strategy in the article, it would have only bought me a couple of more minutes worth of time, if that!
1123581321大约 4 年前
This is clever. Our shaky version of this, historically, has been to run ncdu and hastily delete the first large log file we see. It’s not ideal.
评论 #26584084 未加载
评论 #26584575 未加载
评论 #26584326 未加载
beervirus大约 4 年前
&gt; even deletions can fail or take forever<p>&gt; in a moment of full-disk crisis I can simply delete it and buy myself some critical time to debug and fix the problem<p>Uhh...
monksy大约 4 年前
To prevent the root fs block from filling up. That&#x27;s why I always partition my home+var+opt partition away from the root partition.
Blikkentrekker大约 4 年前
&gt; <i>On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk</i><p>I don&#x27;t understand this. Swap is either a swap partition, or a specific swap file, all of which allocated in advance, so the fullness of the storage should have no bearing.
KernelMC大约 4 年前
I keep my databases on a separate filesystem from root, var, or anything system critical for this reason. Even with the 8GB space waster, if you aren&#x27;t on top of your disk usage you&#x27;d have down time when you fill up the filesystem containing the DB. I might be missing something here, but this does not seem like a good solution to this problem.
joana035大约 4 年前
Mind you can also use tune2fs. It has an option &quot;-m&quot; that one can tune how much reserved space is dedicated to root user.
Something1234大约 4 年前
I have an empty leader on my hard drive so that I can recover if I accidentally nuke the front of it with dd while making a live usb. So it&#x27;s not a bad idea, and it&#x27;s super effective so far it hasn&#x27;t been tested, and hopefully I never will need to.
评论 #26584510 未加载
Diesel555大约 4 年前
&gt;The disk filled up, and that&#x27;s one thing you don&#x27;t want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.<p>I found a bug with time machine where it wouldn&#x27;t delete local copies properly and filled my hard drive until I couldn&#x27;t do anything. The OS slowly stopped working. At first I couldn&#x27;t copy or save anything, then deleting files made more files. It was so bad that the `rm` command eventually wouldn&#x27;t work from recovery or the local OS. I could do nothing. I had to format.<p>It happened again and I learned to manually delete the time machine local snapshots, but it was crazy how hard it was to recover once it took all my storage. That bug is fixed now.
arendtio大约 4 年前
With virtual servers this should not be necessary, as it is easy enough to add some disk space. After all, this should not be a common issue in production environments, but more like a once in a decade problem.<p>With physical servers it might be a different story and might be a good idea. I tend to size filesystems to the requirements I have and enlarge them when required (it gives you a periodic reminder to think about what waste you have accumulated). That way, I can still add space even if the filesystem has been filled up. However, if you do it just to have some space when you need it, it probably is overkill and to have an empty buffer file is a lot easier to handle.
gkarthik92大约 4 年前
So far, my first stop to temporarily get more disk space was to reduce the size of the swapfile which on a lot of servers seems to be allotted &gt;1x the requirement.<p>Will be switching to this hack! Perfect illustration of the KISS principle (Keep it simple, stupid).
评论 #26584253 未加载
crabmusket大约 4 年前
This reminds me of a similar story in a classic Gamasutra article[1] (the section is &quot;The Programming Antihero&quot;, and I&#x27;d recommend the other pages of the article for a few good chuckles). Apocryphal or not, it makes for a good story.<p>&gt; I can see how sometimes, when you&#x27;re up against the wall, having a bit of memory tucked away for a rainy day can really make a difference. Funny how time and experience changes everything.<p>[1] <a href="https:&#x2F;&#x2F;www.gamasutra.com&#x2F;view&#x2F;feature&#x2F;132500&#x2F;dirty_coding_tricks.php?page=4" rel="nofollow">https:&#x2F;&#x2F;www.gamasutra.com&#x2F;view&#x2F;feature&#x2F;132500&#x2F;dirty_coding_t...</a>
njacobs5074大约 4 年前
As hacks go, it&#x27;s a good one. I also like it because you don&#x27;t have to be root to implement it and you don&#x27;t have to reconfigure your file system params in ways that might or might not be great for other reasons.
_0w8t大约 4 年前
This reminded me of embedded Java project that I worked 20 years ago. The VM had only 10MB of RAM and properly dealing with out-of-memory exceptions was a must. The most effective strategy was to preallocate like 200K array. Then on any memory exception the code released that array and set a global flag. The flag was queried through out the code to aggressively minimize memory usage until it drops to tolerable limit.<p>The preallocated buffer was essential. Without it typical result was recursive out-of-memory that eventually deadlocked&#x2F;crashed the VM with no recovery.
benibela大约 4 年前
I have a dual-boot laptop with windows and linux, and use the ntfs partition to share data between them<p>Recently, I extracted a large archive with Linux on the ntfs, and the partition was full<p>Then Windows did not start anymore<p>Linux would only mount the partition as read-only, because it was marked dirty after the failed start. Finally I found a tool to reset the mark, and delete the files.<p>Now Windows starts again, but my user account is broken. It always says &quot;Your Start menu isn&#x27;t working. We&#x27;ll try to fix it the next time you sign in.&quot;, then I sign out, and it is still broken<p>I had to make a new user account
评论 #26585790 未加载
freeone3000大约 4 年前
This sounds like you should, instead, use the &quot;Filespace Reserved for Root&quot; functionality of your filesystem, which exists specifically for this contingency. The default for ext3 is 5%.
noxer大约 4 年前
Just because HN likes to bashes Windows. I tell you that Windows runs pretty much normal if the disk is full. Had that happen many times and intentionally did this for tests as well.<p>Even disconnecting the disk technically doesn&#x27;t break the OS. Because of the &quot;Windows To Go&quot; feature, the OS can detects this and pauses.<p>(Note: Windows To Go is officially removed from current versions but the code that freezes is still there. However, whether that works with your hardware is basically a gambling... so yeah dont try at home&#x2F;work.)
davidmoffatt大约 4 年前
Dumb idea. Read the man page for tunefs. The file system has some thing called min free which does the same thing. However this does not interfer with wear leveling. Dummy data does.
评论 #26586049 未加载
xen2xen1大约 4 年前
I will be doing this. Marvelous idea.
ohazi大约 4 年前
Ah yes... good ole&#x27;<p><pre><code> in_case_of_fire_break_glass.bin</code></pre>
00deadbeef大约 4 年前
Isn’t using LVM and holding some space back a better solution for this?<p>Also I keep databases on their own partition so that nothing else can accidentally fill up the space and lead to data loss.
评论 #26584162 未加载
评论 #26584108 未加载
评论 #26584115 未加载
layla5alive大约 4 年前
Would be better to leave 8GB unpartitioned and then expand the partition. An 8GB file on an SSD is removing 8GB worth of blocks from being able to participate in wear leveling.
评论 #26589366 未加载
otikik大约 4 年前
Reminds me of the cron task I set up once, long time ago, on a bare metal server. It would kill and relaunch a web service every 4 hours.<p>The service in question didn&#x27;t require high availability (it was a mailing list processing&#x2F;interface thing, if I remember correctly) but it had some memory leak which would eventually devour all the memory in the server, in about 2 days.<p>This hack served its purpose well, until the service was eventually replaced by something else.
t0mmel大约 4 年前
What I don’t understand about this approach is why you think it actually does anything for you ? What you do instead of this is to setup an alert to monitor disk space at the right threshold for you, and then have a contingency plan for how to add more space to your environment.<p>It seems like you have sort of done that, but in this case you are actually allowing your system to get into a bad state before you react.<p>Perhaps it’s better to be proactive instead of reactive.
评论 #26589571 未加载
评论 #26590294 未加载
1970-01-01大约 4 年前
Tell all your SecDevOps friends how this file can also pull double duty as a ransomware canary.<p><a href="https:&#x2F;&#x2F;blog.urbackup.org&#x2F;371&#x2F;ransomware-canary" rel="nofollow">https:&#x2F;&#x2F;blog.urbackup.org&#x2F;371&#x2F;ransomware-canary</a><p><a href="https:&#x2F;&#x2F;support.huntress.io&#x2F;article&#x2F;136-ransomware-canaries-technical-details" rel="nofollow">https:&#x2F;&#x2F;support.huntress.io&#x2F;article&#x2F;136-ransomware-canaries-...</a>
评论 #26638198 未加载
jordinebot大约 4 年前
This reminded me of that joke about two guys who meet in the middle of the Savanna. One is carrying a phone booth and the other one an anvil. So, the one with the anvil asks:<p>- Why do you carry a phone booth around? - Oh, you see, it&#x27;s for the lions. If I see a lion, I drop the booth, step inside and I&#x27;m safe. What&#x27;s with the anvil? - It&#x27;s for the lions too. If I see a lion, I drop the anvil and I can run way faster!<p>Good trick anyway!
EricE大约 4 年前
I&#x27;ve been doing this for years too. I also learned instead of one big file, halving several files is also useful so you can release space in &quot;chunks&quot;. If you need everything at once you can wildcard the delete.<p>Such flexibility has been invaluable over the years. Thankfully with block storage and modern operating systems&#x2F;file systems growing volumes can be significantly easier for most servers.
larusso大约 4 年前
I maintain a small fleet of CI machines (mostly macs) and run into this issue as well from time to time. The free space idea is nice but I ram into the problem that under very critical disk space I can’t even shh or when delete a file because there is simply not enough space to execute the simple command. A reboot to get rid of some temp files helps me in these situations to get some control back.
davesque大约 4 年前
On the subject of &quot;inverted&quot; thinking like this, I recently added a test to a test suite that is intended to fail some day. The test will eventually fail when a bug (for which we developed a workaround and added the aforementioned test to confirm the fix) is fixed in one of our open source dependencies. When the test fails, we&#x27;ll know to remove the workaround (and the test)!
louwrentius大约 4 年前
Please use LVM (Logical Volume Manager) if you really are afraid of filling up disks.<p>If the disk would ever fill up:<p><pre><code> 1. Buy an additional virtual disk 2. Add the disk to the LVM volume group 3. Expand the Logical volume </code></pre> A really good primer on LVM:<p><a href="https:&#x2F;&#x2F;wiki.archlinux.org&#x2F;index.php&#x2F;LVM" rel="nofollow">https:&#x2F;&#x2F;wiki.archlinux.org&#x2F;index.php&#x2F;LVM</a>
评论 #26586070 未加载
cafard大约 4 年前
In the days of minicomputers, Data General&#x27;s first 16-bit operating system, RDOS, required that the main file be &quot;contiguous&quot;. Not only that, there was some model of disk they sold where the OS file had to be close to the edge for speed in loading. Prudent sysadms would create empty contiguous files in the favored space against the next upgrade.
rafaelvasco大约 4 年前
This is an old technique. For example, some game developers back in the early days used to put dummy files in the game data space, and code the entire game with less space so that if later more space was needed, it was just a matter of deleting the dummy files. In that context, it kinda forces you to be smarter about your game assets and code.
tomaszs大约 4 年前
Interesting. I would try to notify myself when space is getting low. But I like the solution because of it&#x27;s simplicity
hardwaresofton大约 4 年前
Proof that the future is here, but just unevenly distributed -- we have technology for dynamic disk expansion, but implementation &amp; integration just isn&#x27;t present&#x2F;slick enough to make it available to even tech-inclined hosting consumers just yet.<p>Guess this is another place of differentiation that some of these platforms could offer.
capableweb大约 4 年前
I&#x27;ve had my disks so full that a `rm` command doesn&#x27;t even work, would this workaround work in those cases too?
评论 #26584956 未加载
da_big_ghey大约 4 年前
Full disc problem in linux macine has been a problem in partialy solved in past many decades. We have had seperated partition &#x2F;home, &#x2F;tmp, &#x2F;var, &#x2F;usr in each its own partition. This is reduce problem if not completly removing. This is small desadvantage: there is reducion in fungability for a disc space.
ck2大约 4 年前
Another trick you can use is to adjust the size of the FS tables, NFS4 can do this very quickly and free up space.<p>However on a sketchy drive this is obviously not a wise move.<p>Actually wait, NFS2&#x2F;3&#x2F;4 has reserved block counts you can free.<p><pre><code> # tune2fs -m 3 &#x2F;dev&#x2F;md2 # Setting reserved blocks percentage to 2%</code></pre>
KernelMC大约 4 年前
I keep my databases on a separate file system from root, var, or anything system critical for this reason. Wouldn&#x27;t you still have down time when you fill up the filesystem with the 8GB space waster in place? I might be missing something here, but this doesn&#x27;t seem like a good solution.
truetraveller大约 4 年前
This is not the right solution. It&#x27;s like setting your clock 5 minutes ahead, to trick yourself into thinking it&#x27;s 9:00 am, when it&#x27;s really 8:55 am. It doesn&#x27;t work.<p>The better solution is simple monitoring. Alert when limit is passed. Increase limit to 16gb disk space remaining if paranoid.
superjan大约 4 年前
We write a data intensive desktop app, and when you are close to disk full, we reduce functionality so you can’t make the problem worse, or lose work because of the disk full situation. The thing is that we know that more than half of that user’s data is ours, so our data is often the cause.
tammerk大约 4 年前
I&#x27;ve seen in Cockroach db documents last month : <a href="https:&#x2F;&#x2F;www.cockroachlabs.com&#x2F;docs&#x2F;v20.2&#x2F;cockroach-debug-ballast" rel="nofollow">https:&#x2F;&#x2F;www.cockroachlabs.com&#x2F;docs&#x2F;v20.2&#x2F;cockroach-debug-bal...</a><p>This technique even has a name : ballast file
michelb大约 4 年前
Showing off that i’m not a sysadmin, but wouldn’t a monitoring daemon work? Once disk usage grows past a certain uncomfortable threshold you get an email&#x2F;notification to see what’s up. I mean you obviously are monitoring other server vitals anyway right?
评论 #26585711 未加载
评论 #26584758 未加载
Denvercoder9大约 4 年前
&gt; Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk<p>That&#x27;s only a problem if your memory is full as well, and even then, I&#x27;ve never encountered a server that uses a swapfile instead of a swap partition.
评论 #26584145 未加载
评论 #26584491 未加载
评论 #26584125 未加载
TazeTSchnitzel大约 4 年前
I am reminded of a tweet that suggested adding a sleep() call to your application that makes some part of it needlessly slow, so that you can give users a reason to upgrade when there&#x27;s a security fix (it&#x27;s 1 second faster now)!
评论 #26586124 未加载
pantalaimon大约 4 年前
Doesn&#x27;t ext[2,3,4] reserve 5% of the space on the disc for this very reason?<p>This can be adjusted with tune2fs -m &lt;percentage&gt; &#x2F;dev&#x2F;sda1<p>You can check the reserved blocks with sudo tune2fs -l &#x2F;dev&#x2F;sda1 | grep &#x27;Reserved block count&#x27;
ussrlongbow大约 4 年前
This approach is not unique to IT. Some companies are overstaffed for similar reasons
k__大约 4 年前
I remember a discussion here about a dude who did this with memory in game development. People didn&#x27;t like the idea very much.<p>To me it has a taste of domain squatting or GPU scalping, but you don&#x27;t do it with strangers, but your team.
AcerbicZero大约 4 年前
In most VMware clusters that use resource pools extensively I&#x27;ve always maintained a small emergency CPU reservation on a pool that would never use it, just in case I had to free up some compute without warning.
znpy大约 4 年前
That&#x27;s a dumb idea?<p>Iirc some filesystems allow you to reserve a percentage of blocks for this particular use case (recovery by root).<p>Ext2&#x2F;3 for sure, ext4 probably too.<p>Not sure you can do that on linode on the rootfs, since the filesystem is mounted, tho.
评论 #26586139 未加载
grconner大约 4 年前
Really good idea. After looking at the linked article about dd, I guess this wouldn&#x27;t work as well if one was using a file system with compression. In that case maybe &#x2F;dev&#x2F;urandom would be better?
eecc大约 4 年前
That’s what tune2fs is for <a href="https:&#x2F;&#x2F;www.unixtutorial.org&#x2F;commands&#x2F;tune2fs" rel="nofollow">https:&#x2F;&#x2F;www.unixtutorial.org&#x2F;commands&#x2F;tune2fs</a>
liaukovv大约 4 年前
I&#x27;m sorry for meta comment, but this font is barely readable.
评论 #26584291 未加载
评论 #26584252 未加载
bobbydreamer大约 4 年前
I had used this technique in Dev and ist servers precisely 11 years back. Get storage would be a days task which would stall current activity. This helped. 1.5gb of 5 files.
rrauenza大约 4 年前
I thought this was gonna be about the obscenely large sparse file &#x2F;var&#x2F;log&#x2F;last.<p>I really wish they would move it from a sparse memmap() file to a btree or something.
rswail大约 4 年前
mkfs has an option to reserve a %age or # of blocks&#x2F;inodes for root of a file system. It&#x27;s the file system equivalent of empty files.<p>Usually when free space is exhausted, it&#x27;s for non-root users. You get that same &quot;time to fix stuff by deleting the file&quot; by using tunefs to change that root reserved space to zero.<p>Plus have &#x2F;var&#x2F;log on a separate file system and make sure that your log rotations are based on size as well as time.
tanseydavid大约 4 年前
I am surprised WIN did not make the short list for OS-lost when disk space is almost gone.<p>I have been there a couple of times and it is a land of crazy unpredictable behavior.
sriku大约 4 年前
On a Mac, I&#x27;m often puzzled why the OS says both &quot;you&#x27;re low on disk space&quot; *and* &quot;you have 23GB available&quot; on thr same disk.
arthurmorgan大约 4 年前
It&#x27;s a really bad problem on iOS where a full disk won&#x27;t allow you to delete anything and a reboot puts your phone in a boot loop.
trissylegs大约 4 年前
I suppose if you have deduplication (eg ZFS) you&#x27;d have filter out that file or it&#x27;ll probably get converted to a sparse file.
评论 #26589295 未加载
sloucher大约 4 年前
The great thing about Windows is that there&#x27;s so much bloatware you&#x27;ll always find something to delete when you need space.
reph2097大约 4 年前
This is stupid. Just don&#x27;t make your servers use up all space. That&#x27;s why ext can reserve space for root, 5% by default.
867-5309大约 4 年前
8GB seems like an arbitrary number, especially with no explanation or percentage of total. would be no good on e.g. a 10GB slice
bluedino大约 4 年前
It’s nice today to be able just grow a disk using whatever virtualization platform you’re on. Another 50GB? Click. Click. Click.
mulle_nat大约 4 年前
This may not work on a filesystem with compression. That empty file might only take up a block, even though its size is 8 GB.
robin21大约 4 年前
This is a great idea. Hit this so many times.
dmingod666大约 4 年前
It sounds &#x27;cool&#x27; and all for 1995.. but, what about one script that&#x27;ll email you when the disk is at 80%?
评论 #26585356 未加载
评论 #26585360 未加载
igammarays大约 4 年前
Does anyone know if disk resizing works from the Digital Ocean console when the disk is full? Kindof worried now.
midasuni大约 4 年前
I do similar, I keep multiple files though - 4GB, 2GB, 1GB and 100M, which I also use for testing speed
tilolebo大约 4 年前
Or you could use &quot;tune2fs -m&quot; to temporarily reduce the percentage of reserved blocks
godber大约 4 年前
All my disks have 5% space reserved for root by default. This reserve can be changed at will.
indigodaddy大约 4 年前
This isn’t the worst idea in the world. It’s just not a long term or future looking plan..
kaydub大约 4 年前
Stories like this and my own past memories make me so happy to work somewhere big.
pabs3大约 4 年前
Hmm, I thought empty&#x2F;sparse files didn&#x27;t take up any space on disk.
asdofindia大约 4 年前
I just delete my docker logs when the disk is full and get a 5GB space :)
h4waii大约 4 年前
This is like carrying around a pound of beef because you refuse to look up the address of a McDonald&#x27;s 7 minutes away.<p>Setup quotas or implement some damn monitoring -- if you&#x27;re not monitoring something as simple and critical as disk usage, what else are you not monitoring?
评论 #26585571 未加载
评论 #26584779 未加载
hitul-mistry大约 4 年前
Why to have empty files?<p>Can&#x27;t you put disk size monitoring?<p>Regards, Hitul Mistry digiqt.com
DeepYogurt大约 4 年前
This is why reserved blocks in fileystems exist no?
johnchristopher大约 4 年前
It&#x27;s the google photos thumbnails db. &#x2F;s
gautamsomani大约 4 年前
fallocate is a better command then dd. Check it out. It doesn&#x27;t even cause any disk IO. I use it all the time for such cases.
andimm大约 4 年前
ot: the first two links are &quot;swapped&quot;
评论 #26585644 未加载
glacials大约 4 年前
On the chat team at Twitch in the early days after the Twitch Plays Pokemon crisis [1], we started artificially doubling all chat traffic through our systems, then dropping the doubles just before they would be sent to users. [2]<p>Not only did it give us a &quot;big red button&quot; to press during emergencies like OP, but it revealed important logical scaling issues before they became real problems.<p>[1]: tldr; 1 million people playing a single game of Pokemon Red by using chat to send button presses<p>[2]: <a href="https:&#x2F;&#x2F;www.twitch.tv&#x2F;videos&#x2F;93572955" rel="nofollow">https:&#x2F;&#x2F;www.twitch.tv&#x2F;videos&#x2F;93572955</a>
pikseladam大约 4 年前
Simple and beautiful. Thanks for sharing.
CodeBeater大约 4 年前
The gastric balloon of linux servers
notorandit大约 4 年前
I would keep 4x2gb. Just in case...
GameOfKnowing大约 4 年前
You should probably delete it.
linuxhansl大约 4 年前
ext4 has a default &quot;Reserved block count&quot; of 5%, only usable by root.
harperlee大约 4 年前
At work, OneDrive does not sync by policy if there is less than 30Gb free space. Apparently for ensuring space for updates when they come...
aye01大约 4 年前
as a sysadmin, i hate this... but I&#x27;m also thinking of using it.....
o-__-o大约 4 年前
Tunefs -m<p>Been doing this since the UFS days
rhdxmr大约 4 年前
Simple but clever idea
nix23大约 4 年前
Hmmm, Quotas anyone?
alvarlagerlof大约 4 年前
Sorry, but I cannot read this at all. Please increase the font thickness.
dominotw大约 4 年前
sysadmin version of setting the clock 5 mins ahead?
mito88大约 4 年前
also disable file compression on the 8gb file.
pgray大约 4 年前
tune2fs -m not good enough?
lameiam大约 4 年前
summary: people suck
dalu大约 4 年前
Just don&#x27;t use cloud, get a cheap dedicated server and never run out of disk space
gfody大约 4 年前
having an 8gb file you know you can delete isn&#x27;t really all that helpful if everything has already gone disk-full-fracked. you should really have an alarm on free space, especially if you&#x27;re an indie.
评论 #26584438 未加载
mgarfias大约 4 年前
Why not keep an eye on the disk and expand the fs before it goes south?
评论 #26584588 未加载
loloquwowndueo大约 4 年前
If you don’t have monitoring to tell you when the disk is more than X% full, then you’re at risk for more failure scenarios than just a full disk (usually trivial to buy time by deleting old logs).
评论 #26584207 未加载
评论 #26584173 未加载
IncRnd大约 4 年前
This solution is what creates the problem. If you want warnings when the free disk space is low, set-up warnings for when the free disk space is low.
fractal618大约 4 年前
I think i know where this is going without even reading. Any attempt from the outside to pull this 8b Gb file would be a very noticeable red flag.
评论 #26584369 未加载