TL/DR: xsv is probably what you want, or maybe zsv and/or awk<p>awk can do this super easily. Here's an example snippet that not only shards, but compresses your shards.
```
(NR - 1) % shard_size == 0 { # ready to start a new shard
current_n = current_n + 1
output_file = sprintf("%s%04d.%s.bz2", target_prefix, current_n, file_type)
print "writing to " output_file > "/dev/stderr"<p><pre><code> # close any prior-opened output_command (else will err on too many open files)
if(output_command != "")
close(output_command)
output_command = "bzip2 > " output_file
# print header
print headrow | output_command</code></pre>
}<p>NR != 1 {
print $0 | output_command
}<p>```<p>This of course assumes that each line is a single record, so you'll need some preprocessing if your CSV might contain embedded line-ends. For the preprocessing, you can use something like the `2tsv` or `select -e ...` command of <a href="https://github.com/liquidaty/zsv" rel="nofollow">https://github.com/liquidaty/zsv</a> (disclaimer: I'm its author) to ensure each record is a single line.<p>You can also use something like `xsv split` (see <a href="https://lib.rs/crates/xsv" rel="nofollow">https://lib.rs/crates/xsv</a>) which frankly is probably your best option as of today (though zsv will be getting its own shard command soon)