Yes "D" is not for disk/drive/device/...<p>It comes from the DD (data definition) statement of OS/360 JCL, and hence why dd has the unusual option syntax compared to other unix utils<p>BTW if you are using dd to write usb drives etc. it's useful to bypass the Linux VM as much as possible to avoid systems stalls, especially with slow devices.
You can do that with O_DIRECT. Also dd recently got a progress option, so...<p><pre><code> dd bs=2M if=disk.img of=/dev/sda... status=progress iflag=direct oflag=direct
</code></pre>
Note dd is a lower level tool, which is why there are some gotchas when using for higher level operations. I've noted a few at:<p><a href="http://www.pixelbeat.org/docs/coreutils-gotchas.html#dd" rel="nofollow">http://www.pixelbeat.org/docs/coreutils-gotchas.html#dd</a>
Most of the time it's much better (as in, faster) to just use cat (or pv, to get a nice progress bar) for writing a file to a block device, because it streams, and lets underlying heuristics worry about block sizes and whatnot.<p>So:<p><pre><code> cat foobar.img > /dev/sdi
</code></pre>
will stream the file rather than what dd does, i.e. read block, write block, read block, write block and so on.<p>Usually I also lower the vm.dirty_bytes and vm.dirty_background_bytes to 16 resp. 48 MB (in bytes) beforehand, which limits the buffer sizes to those amounts. Else it will seem that the progress bar indicates 300MB/s is written, and when it completes you still have to wait a really long time for things to have been written out.
Afterwards I restore back vm.dirty_ratio and vm.dirty_background_ratio to respectively 10 and 5 - the defaults on my system.<p>I wish that all of those projects, tutorials etc. that explain how to write their image to a block device, like an sdcard, would start advise using cat, because there is no reason to use dd, it's just something that people stick with because others do it too.<p>I only use dd for specific blocks, like writing back a backup of the mbr, or as a rudimentary hex editor.
"Legend has it that the intended name, “cc”, was taken by the C compiler, so the letters were shifted by one to give “dd”."<p>Legend is wrong, this clearly derived from the mainframe JCL DD command. This is also why the syntax is so non unix-like.<p><a href="https://en.wikipedia.org/wiki/Job_Control_Language#In-stream_input" rel="nofollow">https://en.wikipedia.org/wiki/Job_Control_Language#In-stream...</a>
I've seen people try to use dd to clone a bad disk before tossing it or attempting recovery - while dd is not the right tool for the job, ddrescue [0] is.<p>ddrescue gives you options for error handling and will skip past bad blocks, it handles read errors much more gracefully.<p>[0]: <a href="https://www.gnu.org/software/ddrescue/" rel="nofollow">https://www.gnu.org/software/ddrescue/</a>
There are likely a number of reasons for using dd in tutorials designed for beginners:<p>- it is available by default on all Unix systems<p>- it distinguishes between input and output (i.e. if= and of=)<p>- it reports results<p>- it avoids using a common command for a dangerous operation that the user may not understand<p>dd also has another benefit: the ability to select a range of blocks to copy from and to. This isn't the most common scenario, but it certainly pops up on some devices.
> This belief can make simple tasks complicated. How do you combine dd with gzip? How do you use pv if the source is raw device? How do you dd over ssh?<p>These operations are not that complicated. Behold the magic of UNIX pipes:<p>dd if=/dev/sdb | pv | gzip -c | ssh name@host "gzip -dc | dd of=/dev/sdc"
+ "On UNIX, the adage goes, everything is a file."<p>- not all the things on unix are abstracted as files (or 'byte streams' to be more accurate). however, i/o resources and some ipc facilities are defined so. an operating system provides many other abstractions in addition to these such as processes, threads, non-stream devices, concurrency and synchronization primitives, etc.; thus it's absolutely wrong to say that everything is a file on unix.
The unique advantage of dd over cat, cp and friends: you can specify the block size.<p>Just try (on OS X) dd if=disk.img of=/dev/disk1... first speedup is gained by using rdisk1, but the real improvement comes with the bs=1m. 2 vs 16 vs 300 MB/s on my machine, when cloning via USB-SATA adapter.
More importantly: dd is not a benchmarking tool. I can't count how many times people have complained about dd being slow on a distributed filesystem. Well, yeah. When you're writing only one block at a time with no concurrency/parallelism at all, over a network interface that has much higher latency than your disk interface, of course it's slow. When you're using tiny block sizes, as about 80% of these people do, the effect is only magnified. Unless your use case <i>for a distributed system</i> is a single user who doesn't even multitask, use iozone or fio instead.
Hello! I'm the author of this article.<p>Sorry for the confusion: dd is still a very useful tool for copying disks.<p>The point is that you should not feel like you have to shoehorn dd into any command dealing with disks, because only dd is somehow "raw" or "low level" enough to access them.<p>For example, if you have a command like this:<p><pre><code> pv file1 | gzip | ssh host "gzip -d > file2"
</code></pre>
and you want to make it work with disks, just replace file1 and file2 with /dev/yourdisks and it's fine.
Indeed<p><pre><code> $ man dd
NAME
dd - convert and copy a file
SYNOPSIS
dd [OPERAND]...
dd OPTION
DESCRIPTION
Copy a file, converting and formatting according to the operands.
...
</code></pre>
More at <a href="http://man7.org/linux/man-pages/man1/dd.1.html" rel="nofollow">http://man7.org/linux/man-pages/man1/dd.1.html</a><p>See the part after "Each CONV symbol may be:"
I came from Solaris land originally and it always surprises me how seldom people use filesystem level utilities to copy such stuff, i.e. ufsdump | ufsrestore or dump | restore<p>Works a treat and using the fs level tool you know everything will be properly copied, much safer.
Can we have a single-purpose tool for getting the text between two delimiter strings?<p>I know it's possible with regex, but given how frequently that parsing logic is needed, and the difficulty of getting sed right, I think a "tb" tool would be very helpful.
Yes, but you can't set block size to n bytes in cat.<p>Also, it's useful to have in the back of your mind that dd can very easily mean Disk Destroyer, specially because of it's sui generis syntax
It also doesn't stand for 'destroy-disk' as was thought by a junior admin I employed once, and eventually had to fire because the level of incompetence was getting to the point of almost being destructive.<p>Nope, that's not hyperbole. I had to stop the kid from almost installing software that would have connected to a known botnet to help a user connect a personal computer to the VPN. He passed enough checks during the interview we figured "Okay, we can train him in the rest of the things"<p>Lesson learned.
If you're building disk images with blank space in them (say, for an 8GB EMMC, but your root is only 2GB) you may want to use bmap-tools [1] [2]<p>This way only actual data is written to the device, blocks of zeros can be skipped.<p>1: <a href="https://lwn.net/Articles/563355/" rel="nofollow">https://lwn.net/Articles/563355/</a>
2: <a href="https://github.com/01org/bmap-tools" rel="nofollow">https://github.com/01org/bmap-tools</a>
My favorite thing to do with 'dd' is to break up multi-gigabyte log files into, say, 500MB chunks, so I can easily view and search them in XEmacs (this is 'csh' syntax as I use 'tcsh'):<p><pre><code> foreach i (0 1 2 3 [...])
dd <big.log >big.log.$i bs=500m count=1 skip=$i
end
</code></pre>
(XEmacs is very fast at reading large files but has a 1GB limit.)
> It’s unique in its ability to issue seeks and reads of specific lengths, which enables a whole world of shell scripts that have no business being shell scripts. Want to simulate a lseek+execve? Use dd!<p>How would one simulate a call to execve with dd? Seems like a totally different problem domain.
dd is for converting EBCDIC to ASCII and vice versa :)<p><pre><code> $ echo how now brown cow > text.ascii
$ dd conv=ebcdic < text.ascii > text.ebcdic
0+1 records in
0+1 records out
18 bytes copied, 0.000261094 s, 68.9 kB/s
$ od -xc text.ebcdic
0000000 9688 40a6 9695 40a6 9982 a696 4095 9683
210 226 246 @ 225 226 246 @ 202 231 226 246 225 @ 203 226
0000020 25a6
246 %
0000022
$ dd conv=ascii < text.ebcdic
how now brown cow
0+1 records in
0+1 records out
18 bytes copied, 0.000140529 s, 128 kB/s</code></pre>
What? Yes dd absolutely is a disk writing tool, although that is not the only or even the tool's intended primary purpose.<p>It is useful for generating serial IO for a variety of purposes. For example, writing data with specific target block size; allocating contiguous blocks for use by an application (be it zeroing out a thin LUN before partitioning, or a file system); or simply dumping the content of one device to another (or to a file).<p>Good luck stretching out a thin LUN or creating an empty file that allocates contiguous space with cat.
> How do you combine dd with gzip?<p># dd < /dev/ada0 bs=8m | gzip -c -9 > /mnt/file.raw.gz<p>> How do you use pv if the source is raw device?<p># dd < /dev/ada0 bs=8m | pv | dd > /dev/ada1 bs=8m<p>> How do you dd over ssh?<p># dd < /dev/ada0 bs=8m | gzip -9 | pv | ssh user@host 'dd > /dev/da1 bs=8m'<p>> This belief can make simple tasks complicated.<p>As master Dennis Ritchie once said - "UNIX is very simple, it just needs a genius to understand its simplicity."
The best thing about dd is that you can use it with conv=noerror, which will let you recover as much data as possible from an otherwise damaged device.
So what is the best way to clone a disk(or in my case a raspberry pi sd cars)?<p>I tried to backup one of my cards last week using dd > .iso file and then tried to put it on a new card. I tried with /dev/Rdisk (faster) but none of the new cards was bootable.<p>So this is saying just use copy.<p>(I ended up just creating a second boot disk, and ftping the files over which seems less than ideal...)
In my experience dd is great for binary data. Yes, pretty much everything on Unix operates on files, as does dd. But so many utilities are either line based or don't handle null bytes, and it's a pain to have determine how a given program handles binary data when I know dd will at least not mess with it.
"Not a disk-writing tool?!?"<p>Who knew?<p>Now we have pretty good confirmation that this little utility is performing way more effectively than designed.<p>Software itself could probably benefit from some of the same approaches that allowed this little computer program to outperform its original design goals, in ways that might not have been anticipated.
One of those Unix tools that would deserve to be better known is dcfldd (<a href="http://dcfldd.sourceforge.net/" rel="nofollow">http://dcfldd.sourceforge.net/</a>). It is basically dd with extra powers including on-the-fly hashing, progression, multiple outputs...
>> The reason why people started using it in the first place is that it does exactly what it’s told, no more and no less.<p>Oh yes indeed. And for this exact reason, "dd" is commonly backronymed to "Data Death" (or, indeed, "<i>Disk</i> Death").
The statement<p><pre><code> cp myfile.iso /dev/sdb
</code></pre>
is simply wrong. That will overwrite the device node for your flash drive with the ISO, instead of writing the data to the drive.