I use R a lot, and nothing in this post rings even the slightest bit true to me. This guy's use case for R seems to be very unlike almost anyone I've ever heard of. Towards the end of the article, when he's talking about how all he really needs from R is the summary() and boxplot() functions, it really becomes clear that he's never done anything more than dip the very tip of his pinky toe into R.<p>There are lots of valid criticisms of R, but this article doesn't touch on them. It's so off-base that it's the proverbial "not even wrong".
I use R on a daily level. A significant portion of our climate change adaptation research code and decision planning systems (which are used in various utilities around the world to support decision makers) are built using R.<p>I understand what the author is stating, but I just feel like this is from inexperience with R and ignoring the vast amount of packages available within it that are specifically targeted at data science. There are some valid issues and criticisms of R, but I think this article only focuses on the application of R in a single context. A significant portion of data science is about cleaning the data and an entire suite of packages known as tidyverse solves these problems (for me anyways) while also being very simple and easy to understand. I mean tidyverse supports piping, which is exactly what this article is saying to use.<p>Obviously your mileage may vary, but this post irks me the wrong way.
For historical context, R was not initially intended to be a standalone scripting language runnable via POSIX conventions and didn’t gain these features until circa 2010. R is designed to be used interactively, like the S language it was based on, a style later made familiar to students via MATLAB and TI graphing calculators. Before Rscript it was common to hack together shell scripts that cope with the expectation of human TTY input to run R in production, which functioned well as a “you must be this tall to ride” sign for people who might not appreciate how unreliable R code can be compared to a traditional language.
The precursor to R, S, was designed as a Unix tool, and pretty much implicitly relied on awk as its data-cleaning preprocessor.<p>For those who come from the world of large enterprise statistical and data reporting tools such as SAS, awk shares an exceedingly strong resemblance to the SAS DATA step, whilst R effectively provides a host of analysis and graphics tools the correspond to numerous other SAS procedure and products.<p>The hacks for pipelining R are cool and useful. Thanks.
> In my experience, just about anything beats R when it comes to cleaning dirty data.<p>> I used to resent R, it was shoved upon me as a strange tool that promised to replace Python, but failed miserably.<p>well I'm glad the author found a way to make R work for their purposes, but this just reeks of inexperience with R..
"R is only great if your data is already great."<p>Disagree. R has an incredible ecosystem for parsing, cleaning, and manipulating data. Even ignoring the tidyverse, base R provides more than enough functionality to clean and analyze data--you just need to spend some time learning it. If you use the tidyverse, it's even easier. The only other ecosystem that comes close to R is Julia, which was designed taking many of the best parts of R into consideration.
R has lots of packages available for data cleaning. They just aren't necessarily included in base R. Most of the ones I'm aware of are in the Tidyverse. Even if you're dealing with a hand-written Excel file with multiple tables in the same sheet and various information encoded in text/background colors and such, there's they tidyxl package to help you do that.
I haven't worked with logs, but I do find R a joy to work with in general, especially the tidy verse is a joy to use, but slow and memory hungry on very large datasets.<p>I haven't found a good way around very inconsistently formatted csv files in any language (a row only represents column 3,4 and 6 if it starts with a comma, all other rows have all columns, but are space separated and values may contain comma, etc, etc)
R isn't a joy, but it is powerful. This is the best blog post I've read on the matter: <a href="https://www.talyarkoni.org/blog/2012/06/08/r-the-master-troll-of-statistical-languages/comment-page-1/" rel="nofollow">https://www.talyarkoni.org/blog/2012/06/08/r-the-master-trol...</a>
If you want to treat R like awk, you should really check out the littler package, a super useful R package which provides an alternative to both Rscript and R CMD BATCH designed for writing one-liners <a href="https://github.com/eddelbuettel/littler" rel="nofollow">https://github.com/eddelbuettel/littler</a>
To be honest I don't find Python to be very useful for data cleansing. Full disclosure I use R quite a bit and use Python sparingly.<p>If you want to use R for data analysis I suggest the R package Tidyverse or use a product like OpenRefine.<p>If you just need to run summary statistics with an application then using Python is just fine.
I get what the author is saying -- I use R in shell scripts too. It's really useful and composes well with other shell tools.<p>I also get what the commenters are saying, because R is a useful interactive language too, and it's also pretty good for data cleaning. Although it's significantly slower than Python, which is why I do all cleaning that cuts down the data before loading it into R.<p>As an example, I generate some benchmarks with every release of Oil:<p><a href="https://www.oilshell.org/release/0.7.pre5/benchmarks.wwz/osh-parser/" rel="nofollow">https://www.oilshell.org/release/0.7.pre5/benchmarks.wwz/osh...</a><p>and the tables are manipulated with R, but running the benchmarks is done with shell, and creating the HTML is done with Python:<p><a href="https://github.com/oilshell/oil/blob/master/benchmarks/report.R" rel="nofollow">https://github.com/oilshell/oil/blob/master/benchmarks/repor...</a><p>(and yes this page is meant to be motivation to speed up my principled but slow shell parser)<p>R and tidyverse are the best tools for manipulating tables by far. I wrote an intro here:<p><i>What Is a Data Frame? (In Python, R, and SQL)</i> <a href="http://www.oilshell.org/blog/2018/11/30.html" rel="nofollow">http://www.oilshell.org/blog/2018/11/30.html</a>
I used to use awk all the time whenever I wanted to evaluate specific portions of a columnar file (With a short one liner). Python comes close but, awk is much faster. In conjunction with SED, I think that is a little much and imo exceeds what I was doing. If I need to do any replacing or 'cleaning', I would then use something like Python or SQL.
So, this is not about the article. I've recently started using rpy2 as a python wrapper for R functions and libraries and I'm finding that it is not that bad.<p>There is some performance issues, but I am OK to trade that off for the convenience of using "try:" rather than R's "tryCatch". Having tryCatch as a function rather than built into the syntax is unacceptable to me. But there are some libraries in R that don't have elegant alternatives in Python or which you have multiple options or perhaps not the time to unlearn.
Here is how I like to use R, which can use the authors more introductory methods or full fledged data crunch:<p>Like I do all my other data sources, as code blocks inside an emacs-org notebook. If you are doing data science, you quickly find that it's management and combination of the various particular projects that becomes the most daunting (imho), and your data science notebook becomes the most important part of that organization. In that arena for me it's pretty much either jupyter or emacs org-mode.
The regex stuff in stringr, and read_csv from readr are adequate for almost anything if you're talking about ordinary delimited file import.<p>What does awk give you that these don't?
R is really overkill here. Simpler and nicer tools exist if you want to munge data on the command line and/or plot it. For instance:<p><a href="https://github.com/dkogan/feedgnuplot/" rel="nofollow">https://github.com/dkogan/feedgnuplot/</a>
<a href="https://github.com/dkogan/vnlog/" rel="nofollow">https://github.com/dkogan/vnlog/</a>
I’ve recently discovered the pipe command allowing R to consume the result of a terminal command’s output (link: <a href="https://youtu.be/RYhwZW6ofbI" rel="nofollow">https://youtu.be/RYhwZW6ofbI</a>). Quite useful for reading in compressed files and it’s made me learn a bit of awk to fix text files on the fly<p>Oh and there’s also Rio if you want to explore injecting R into your command line workflow.
If you're interested in learning how to do data cleaning and restructuring in R, I highly recommend the "Wrangle" chapters of Hadley Wickham's R for Data Science book, which you can read online here: <a href="https://r4ds.had.co.nz/wrangle-intro.html" rel="nofollow">https://r4ds.had.co.nz/wrangle-intro.html</a>
dwodri: Not sure if you're a MacOS user, but iTerm2 distributes a script to display images inline in the terminal. I've forgotten how to get R to output png data directly to stdout, but it could be nice to do that and display the images inline in the terminal<p><a href="https://www.iterm2.com/documentation-images.html" rel="nofollow">https://www.iterm2.com/documentation-images.html</a>
OP Here: I’m aware of some CSS issues on mobile and will work to fix them shortly.<p>UPDATE: fixed color palette issue, although it still looks quite bad.