How To Open and Manipulate Large CSV Files On A Mac

The counter-intuitive part is that a 100MB file is considered large on a machine with 8-16GB RAM.<p>It&#x27;s definitely not the problem of the resources, but only of the architecture of applications.<p>(Indeed, the right application is a DBMS, not a spreadsheet.)

You could get interested in:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;xsv" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;xsv</a>

When I saw &quot;large&quot;, I expected &gt;10GB. And &quot;big data&quot; if it needs to be batch processed or distributed processed.

No mention of vi or Sqlite? While I&#x27;m no vi expert it&#x27;s a great tool for working with big files when you want to browse around without grep. And Sqlite is similarly ubiquitous and capable of crunching large files.

Two of my favorite tools for this kind of thing:<p><a href="http:&#x2F;&#x2F;visidata.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;visidata.org&#x2F;</a><p>And<p><a href="http:&#x2F;&#x2F;recs.pl&#x2F;" rel="nofollow">http:&#x2F;&#x2F;recs.pl&#x2F;</a>

&gt; Excel for Mac performed well but it is a paid solution so I did not consider it viable for many developers<p>Because developers handling gigabyte size data, and wanting to reliably manipulate it in a GUI, cannot possibly be expected to pay&#x2F;afford the $7&#x2F;month to Microsoft.<p>That said, the recommended solution is probably the best option for developers, not bedside because it&#x27;s free, but for the ability to run complex SQL statements, and visualize the results.<p>If I were to edit this article, that&#x27;d be my takeaway: use tool X for snappy vitalization of SQL queries, even on multi gigabyte sized CSVs

I wonder how well would Table Tool [1] perform with your large dataset? This is an open source CSV editor for Mac from the developer of Postico, my favorite PostgreSQL client for Mac [2]<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;jakob&#x2F;TableTool" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jakob&#x2F;TableTool</a><p>[2] <a href="https:&#x2F;&#x2F;eggerapps.at&#x2F;postico&#x2F;" rel="nofollow">https:&#x2F;&#x2F;eggerapps.at&#x2F;postico&#x2F;</a>

Python + Pandas

I use XSV: A fast CSV command line toolkit written in Rust.<p><a href="https:&#x2F;&#x2F;formulae.brew.sh&#x2F;formula&#x2F;xsv" rel="nofollow">https:&#x2F;&#x2F;formulae.brew.sh&#x2F;formula&#x2F;xsv</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;xsv" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;BurntSushi&#x2F;xsv</a>

I am not clear what &quot;manipulate&quot; means here -- what is the author trying to do with the comma separated values? FWIW, I can accomplish csv manipulation using a handful of Unix utilities: sed, awk, cut, and call it a day.

<a href="https:&#x2F;&#x2F;kothar.net&#x2F;csview" rel="nofollow">https:&#x2F;&#x2F;kothar.net&#x2F;csview</a><p>A fast viewer a friend of mine created to view large CSVs in a GUI - might be useful to someone.

Sublime Text and VSCode work fine with 1GB files in my experience.

There seem to be literally dozens of solutions to do read-only operations but very few to enable comfortable editing of the files in-place in a Unix &#x2F; command line environment.<p>Seems like a real gap in the software ecosystem atm:<p><pre><code> - fast - no limit on file size - spreadsheet style layout - command line - easily edit and update individual cell =&gt; save </code></pre> I&#x27;ve tried VIM&#x27;s CSV plugins many times and have never been satisfied.

@alecdibble Could you try LibreOffice&#x27;s Calc [1]? It&#x27;s my daily driver in Linux, it mostly works well as an excel replacement but I&#x27;m interested to see how well it does on a Mac.<p>[1] <a href="https:&#x2F;&#x2F;www.libreoffice.org&#x2F;download&#x2F;download&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.libreoffice.org&#x2F;download&#x2F;download&#x2F;</a>

Was SequelPro trying impute any field types? That could have been causing the crash when faced with a large file.

tad is absolutely great for this, i tested it for the exact use case mentioned. I had to file a ticket and wait for him to add an export to csv function. tad is built over sqlite and can filter&#x2F;sort&#x2F;pivot&#x2F;aggregate and export the result, which is all my business team partners ever need. can handle GB files, i didnt test with TB size.<p><a href="https:&#x2F;&#x2F;www.tadviewer.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.tadviewer.com&#x2F;</a>, its a desktop app<p>edit: one note after seeing other comments based on cli (xsv, sed, awk, etc) — the OP’s use case is something that marketing&#x2F;pm&#x2F;business stakeholders can use. my favorite tools are cli-based, however this does not fly with business teams so forget that option.

As a huge fan of OpenRefine, I want to say thanks for introducing me to Sequel Pro.

Used VIM in the past to open several GB of SQL files to edit them without problems.

Memory mapped file.. Intermediate binary representation..

Why do you want to do that?<p>Use cat, pipe, grep, awk. Problem solved.

I&#x27;m always astounded that there doesn&#x27;t seem to be a decent general purpose CSV editor&#x2F;viewer application. Excel is atrocious - it&#x27;s always dog slow, and it mangles any CSV I&#x27;ve ever opened by trying to interpret the data to format it &quot;smartly&quot;.<p>Having to build a table in a database and import the CSV into that feels a bit like hitting a house fly with a sledgehammer, but it&#x27;s the most effective way I&#x27;ve seen.