Why there isn’t an Apache Arrow article in Wikipedia

111 pointsby riboflavinover 5 years ago

29 comments

Having worked on commercially-resold Apache projects, can't say I argue with Wikipedia a whole lot on this. It seems to me it should be let in, but it's a bit silly to go and call them out like this on a corporate blog, IMO.Dremio does benefit from Apache Arrow publicity and notoriety, even if they don't profit directly. Having a de-facto standard data format and open-source engines is a selling point for some. That's why Dremio explicitly calls it out on their own website. It also never hurts in the recruiting department. (edit: there's a reason the article was submitted by someone working in marketing & strategy)>> I’m wondering if Wikipedia can continue to be considered a reliable source of information for technical folks who want to learn more about the vast system of Apache open source software projects.Sign up for the Olympics, because that's a hell of a leap. You didn't get your page in, it's really not much of a reflection on the rest of Wikipedia. It's an open-source project. It should have it's own freely available documentation that fills much the same purpose anyway. If I want to learn about Apache X, I go straight to x.apache.org. They concede that it's not an end-user product anyway, so I'd think their key audience knows how to find an open-source project website. Lower the bar too far the other way, and there are plenty of semi-open-source project's marketing departments would be all over using Wikipedia to their own ends - I've seen my own former employer do this for their Apache projects.

评论 #21830072 未加载

评论 #21829813 未加载

评论 #21831387 未加载

评论 #21830504 未加载

评论 #21830641 未加载

mxfhover 5 years ago

It was a pain to get gitlab in 5 years ago after a "controversial" deletion, so it wasn't available for simple undeletion. Domain specific knowledge has it notouriously hard with wikilawyers who, at large, seemingly stopped adding new things to their world view 15 years ago.Then it becomes a game of jumping through hoops and hoping you end up with a kind wiki-landlord or knowing a friendly wikipedia admin.Doing the latter by anouncing your concern on social media and hoping a sympathetic admin picks it up, might be the easiest on human time and resources, just let them copy your reasonably well sourced article draft from your personal space and see what happens.

评论 #21828950 未加载

评论 #21829768 未加载

tptacekover 5 years ago

Open source projects are particularly tricky for Wikipedia. There are tens of thousands of them. Their owners are often passionate. They compete with each other, so there's incentive to write hard-to-adjudicate competing claims. Many have commercial backing, which further warps incentives. The projects themselves are highly technical; many, like Arrow, are software development tools and components. There are few authoritative sources that reliably track open source projects. Keeping up involves directly following bug trackers and message boards and then synthesizing a narrative, which is the definition of "original research", forbidden in the encyclopedia.It's likely that Arrow does deserve a WP article. But Arrow's sponsors misunderstand more about Wikipedia than Wikipedia does about Arrow. Writing a defensible article about their project will require work; in particular, they're going to need to spend the time tracking down authoritative sources for why Arrow is notable, and those claims will probably need to be something more persuasive than "hundreds of companies use it"; hundreds of companies use all sorts of things that don't, and shouldn't, be featured in their own encyclopedia articles.I understand the impulse behind "this project is important; it should have a Wikipedia article". But when you take a step back and accept what Wikipedia actually is, rather than what you think it should be, you're left with the question: do we really need to feature this particular piece of software in its own encyclopedia article? 20 years from now, will people still be getting value from it? Whatever value that might be, will it outweigh the 20 years of other people's volunteer efforts to maintain the article, keeping it free of vandalism and ensuring that it doesn't surreptitiously turn into a promotion piece for some company or another?The answers might be "yes". But I don't see much evidence in this piece considered the questions.Lots of things that don't seem deserving have in-depth Wikipedia coverage. Many of those things probably really don't belong in an encyclopedia! But there are two sides to this problem: the merit of the topic, and the cost, in volunteer time, of including them. A marginal topic can be defensible if it's easy to reliably cover it. A seemingly important technical topic might not be if the only way to say anything interesting about it is to write original research directly into its article.Late editA useful tip for getting your open source project covered in its own Wikipedia article: don't have the Chief Marketing Officer of the company that owns the project write the article.

评论 #21829223 未加载

评论 #21829263 未加载

评论 #21829071 未加载

评论 #21830031 未加载

评论 #21837014 未加载

jccalhounover 5 years ago

>Arrow is designed to serve as a shared foundation for SQL execution engines, data analysis systems, storage systems, and more – think Pandas, Spark, Parquet, etc. Engineers across the community are working together to establish Arrow as a standard for columnar in-memory processing.I like to think I'm fairly techy for a non-programmer but I have no idea what that means. That might be part of their problem if that is the description in their wikipedia entry.

评论 #21831484 未加载

tetromino_over 5 years ago

See <a href="https://en.wikipedia.org/wiki/Wikipedia:Notability" rel="nofollow">https://en.wikipedia.org/wiki/Wikipedia:Notability</a> - all you need to show is that Apache Arrow has received significant coverage in reliable sources that are independent of the subject.So: find conference papers/talks by people not affiliated with Apache or the Apache Arrow project and that discuss Apache Arrow. Figure out how to incorporate the tidbits about Arrow from those papers into the article text. Add sources in footnotes. Done.

评论 #21829557 未加载

评论 #21828165 未加载

xibalbaover 5 years ago

As a strategy for getting Dremio on the front page of HN and thus on the radar of a large group of tech people (i.e. Dremio's prospects), this is article is very clever.As a critique of Wikipedia, not so much.

评论 #21829635 未加载

Ninjaneeredover 5 years ago

Here's the link to the draft:<a href="https://en.wikipedia.org/wiki/Draft:Apache_Arrow" rel="nofollow">https://en.wikipedia.org/wiki/Draft:Apache_Arrow</a>And some possible additional sources:* <a href="https://www.forbes.com/sites/forbestechcouncil/2019/09/24/dremio-helps-reduce-proximity-to-your-data/amp/" rel="nofollow">https://www.forbes.com/sites/forbestechcouncil/2019/09/24/dr...</a>* <a href="https://www.businesswire.com/news/home/20180906005114/en" rel="nofollow">https://www.businesswire.com/news/home/20180906005114/en</a>* <a href="https://thesiliconreview.com/2016/02/apache-arrow-is-the-new-open-source-project-for-big-data" rel="nofollow">https://thesiliconreview.com/2016/02/apache-arrow-is-the-new...</a>

评论 #21829468 未加载

评论 #21830413 未加载

qwerty456127over 5 years ago

Once I witnessed awesome articles [others added and I used with delight] on open source frameworks as well as some minor facts [I added] on other subjects deleted for being "insignificant" I decided I'm not donating to Wikipedia until this bullshit ends.Wiki articles are not videos, they take humble disk space to host so I can't recognize any reason in dismissing "insignificant" information other than a stupid rule.IMHO whatever can be considered a piece of knowledge should be there.BTW nearly the same applies to StackOverflow - thanks to high reputation points I earnt during the early days I can see deleted questions and answers and I often see really interesting (having three-figure upvvote scores and dozens of stars) questions and very informative (also heavily upvoted) answers deleted.

oefrhaover 5 years ago

<a href="https://en.wikipedia.org/wiki/Draft:Apache_Arrow" rel="nofollow">https://en.wikipedia.org/wiki/Draft:Apache_Arrow</a>> REVIEWERS: Please note that the submitting editor is the chief marketing officer and vice president of strategy at this company.Yeah, sorry, big no no there.Disclosure: consider myself a Wikipedian to some extent, got a couple hundred edits on Wikipedia.

评论 #21828977 未加载

评论 #21828718 未加载

mistrial9over 5 years ago

sadly, I can jump in on the "Wikipedia fails" train here, also. In about five attempts to really change an article (different ones) in about five years, every single change was rejected, as far as I know. The changes were different, one was writing style and order of facts on a public historical event in this century; one was adding a lot of detail to the description of a popular fantasy fiction series; one was removing a controversial and provocative one-liner at the top of a page about people at the edge of (western) society; and another .. hmm I forget now, because I just gave up !My aging colleague tells me, just keep doing the changes, they cant stop everything. However, my direct (and limited) experience is.. they do stop everything (that I try). I was logged in twice and used anonymous three times, and added citation a bit, too.To the point of the article, FOSS projects in wikipedia ? hmm maybe there could be a clear category for that ? software projects are proliferating rapidly.. dunno

评论 #21828380 未加载

评论 #21828323 未加载

pradnover 5 years ago

It looks like there aren't enough independent, non-commercial articles to use as references. This is somewhat common for many newish technical projects. Add some academic papers, some usage numbers, some summary blog posts that aren't related to the project. Wiki editors are very suspicious of people from companies editing articles related to their work.

评论 #21829174 未加载

miklover 5 years ago

Why do you care about having a Wikipedia page for Arrow? Why is it important enough to whinge about on HN?Wikipedia is much like Stack Overflow these days, the community has become hostile to newcomers who fail to meet their somewhat arbitrary but very exacting standards for what is allowed on their site.Fortunately, you can just publish your own web site. No need to be bothered about not being on WP.

dredmorbiusover 5 years ago

For those who think that edit wars, content disagreements, and innacuracies are any special realm of Wikipedia, they're not.One of the best examples I've encountered demonstrating this is a 19th century edit revision war between the British and American publishers of Chamber's Encyclopaedia, on the topics of Free Trade, Protection Duties, Slavery, and certain salacious particulars concerning His Royal Highness, the Prince of Wales.<a href="https://old.reddit.com/r/dredmorbius/comments/4xe2k1/chambers_encyclopaedia_editorial_statement/" rel="nofollow">https://old.reddit.com/r/dredmorbius/comments/4xe2k1/chamber...</a>What's novel concerning Wikipedia is that these disputes (as with those of free software vs. proprietary software) tend to occur, or at least leave significant evidence, in the open public record.

thrower123over 5 years ago

The hard-line Wikipedia deletionists should be deleted themselves. The argument is always brought up, like StackOverflow, that they have to be ruthless or it turns into an Eternal September dumping ground of garbage, but the quality is already very uneven and gatekeeping like Cerberus doesn't help further that goal. There's already a toxic Dead Sea effect where the pedantry and politicking has chased out a lot of people that would contribute; who the hell wants to bother putting in some hours writing something up if it is just going to be summarily deleted?Bandwidth and hard drives are cheap.Just spitballing, but it'd be nice if Wikipedia worked a little more like Linux distro repositories. Keep the tightly curated articles in a "core", but leave room for "community" or "nonfree" collections if you want to turn them on.

评论 #21830285 未加载

评论 #21830941 未加载

julianlamover 5 years ago

The whole concept of "notability" in Wikipedia-land is subjective as hell. Whether your article makes it in is simply a matter of rolling the dice the first time you submit the article.I created an article for NodeBB, a piece of forum software used worldwide by companies small and large (including several triple A gaming companies). We got AfD'd, and now every time someone creates an article for NodeBB, the AfD is brought up and the entire discussion ends as soon as it has begun.We even created an article the _suggested_ way, by submitting a draft for review. It got reviewed alright... instant rejection because they felt it looked like an ad. We made changes, but nobody ever took a second look at the article.Of course, a number of defunct open-source (and some proprietary) forum softwares with zero sources are still allowed on Wikipedia, simply due to the fact that they made it through when nobody was looking :)One could argue that we shouldn't be writing our own articles (and they'd be right), so we just quietly accepted our judgement and market NodeBB based on the merits of the software, instead of whether it appears in some arbitrary ranking of forum software.That said, it'd still be nice if we were listed in the Wikipedia list of forum softwares.... _sigh_, a guy can dream.

评论 #21840915 未加载

jabvigWeover 5 years ago

Add it to the Free Software Directory!<a href="https://directory.fsf.org/wiki/Main_Page" rel="nofollow">https://directory.fsf.org/wiki/Main_Page</a>

aaron695over 5 years ago

View the original declined drafts here -<a href="https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&action=history" rel="nofollow">https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arro...</a>Geez if you want to use Wikipedia as an ad, put a bit of effort in, when did marketing become so lazy and blame the platform.Although this meta ad is possibly a far better payoff.

michelppover 5 years ago

We're having the same issue getting the GraphBLAS API article to be accepted: <a href="https://en.wikipedia.org/wiki/Draft:GraphBLAS" rel="nofollow">https://en.wikipedia.org/wiki/Draft:GraphBLAS</a>. At first it was summarily deleted overnight, now we're stuck in Draft for who know how long.

nanoscopicover 5 years ago

This reminds me of my "war" to get an article for my parser XML::Bare.There was a time when there was a comparison page for XML parsers, and many parsers had articles.Still existing parsers on Wikipedia that should be removed; if they are to stay true to their war on having useful software info in Wikipedia:<a href="https://en.wikipedia.org/wiki/Category:XML_parsers" rel="nofollow">https://en.wikipedia.org/wiki/Category:XML_parsers</a>The original argument was that if you can find a citation in print you can have whatever it is on Wikipedia, but that ceased to be true years ago and it has become a popularity contest and power struggle with obnoxious Wikipedia editors.

scarejunbaover 5 years ago

<a href="https://en.wikipedia.org/wiki/Draft:Apache_Arrow" rel="nofollow">https://en.wikipedia.org/wiki/Draft:Apache_Arrow</a>This reads like it was written by the guy who wrote it. It can do this. It efficiently does that. It’s all promotional content. Not useful.

ggggtezover 5 years ago

I've never heard of it. Add my vote to removing the article.Cry more, company I never heard of either.

lmeyerovover 5 years ago

For context, some other companies contributing to it are in the GPU space, so orthogonal to CPU-centric Dremio: Nvidia, Blazing SQL, and Graphistry (us). Likewise, the pydata big guns intersect a bit here: conda, pandas, ... . This effort got a BOSSIE award for GPU dataframes this year and is taking off now that it is becoming usable for more than just framework devs. The reason we all really on it is because a standardized columnar IO streaming format is an awesome idea for compositional HPC.It does sounds like maybe Dremio's CMO wrote the original articles and it came off centered on them? (Did not have a chance to read.)

评论 #21829700 未加载

est31over 5 years ago

Hmmm this reminds me of the battle to get a Wikipedia page approved for Minetest, the biggest FLOSS voxel engine out there:<a href="https://en.wikipedia.org/wiki/Draft:Minetest" rel="nofollow">https://en.wikipedia.org/wiki/Draft:Minetest</a><a href="https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Minetest" rel="nofollow">https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletio...</a>

ForHackernewsover 5 years ago

I've literally never heard of this piece of software, and it's fair to say I'm much more interested in FLOSS than the average person on the internet. Why should this thing have its own article and not just appear in a list of Apache foundation projects?

评论 #21829204 未加载

aabbcc1241over 5 years ago

You're free to post to anywhere; And each site admin/helper/whatever-title are free to do their own censorship or moderation.It's the nature of the web.

ksecover 5 years ago

It was the same with 802.11ax aka WiFi 6.Someone decided all the technical information on the subject are irrelevant and deleted all Data Rate and Technical Improvement section. Another reason was because those details were not finalised.While it was a little frustrating that those useful information were gone as one could always found those in other source and media, but they also deleted the whole section on DensiFi [1], where all the major companies ( Apple, Broadcom, Cisco, Intel, Qualcomm, Huawei, Samsung and others ) behind the 802.11ax decided to do the work behind close door. TL;DR They were trying to push 802.11ax to the market earlier despite of all the un-resolved issues.So I decided to add only the DensiFi section, and it was constantly being deleted within 24 hours. After a few weeks of fun the page simply got back to the original, where Data Rate and Improvement are back but DensiFi section is totally gone. So it turns out it wasn't the technical section they were trying to get rid of.P.S We should be glad someone in the working group discovered this and called out on the action. The current WiFi 6 / 802.11ax situation and UX is much better than what we had when 802.11ac were shipped. Although this is at the expense of somewhat 2 years delay of the standard.[1] <a href="https://mlexmarketinsight.com/insights-center/editors-picks/antitrust/north-america/doj-probes-role-of-special-interest-group-in-new-wifi-standard" rel="nofollow">https://mlexmarketinsight.com/insights-center/editors-picks/...</a>

footaover 5 years ago

Can HN create a draft that would be accepted? :)

评论 #21829119 未加载

cdeilover 5 years ago

I think the reason this is discussed now is because yesterday I tried to re-submit the Apache Arrow article. Here's what I wrote: <a href="https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&oldid=931353298" rel="nofollow">https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arro...</a> It was rejected / reverted 10 minutes later by a Wikipedia editor. The blog post from Justin was in July 2019 (<a href="https://www.dremio.com/why-apache-arrow-wikipedia/" rel="nofollow">https://www.dremio.com/why-apache-arrow-wikipedia/</a>)There's many interesting and good points in the discussion here, thank you!To add my 2 cents:- Apache Arrow is notable, deserves a Wikipedia page. It might not have been when someone first tried to create a Wikipedia page for it in 2017 (see <a href="https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&action=history" rel="nofollow">https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arro...</a>), but in the three years since it has become a major project, see e.g. <a href="https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces46" rel="nofollow">https://blogs.apache.org/foundation/entry/the-apache-softwar...</a> Notability is clearly subjective, depends on what the author and reviewer find interesting. In the variant I submitted yesterday I tried to make it clear why it's notable - Apache arrow is a standard format that connects different languages, runtimes, data systems, communities, e.g. the Python and Java data communities. See e.g. <a href="https://wesmckinney.com/blog/apache-arrow-pandas-internals/" rel="nofollow">https://wesmckinney.com/blog/apache-arrow-pandas-internals/</a> - Apache Arrow is to my knowledge partly the brainchild of Wes McKinney, creator of pandas, it's his attempt (looking strongly like success) to resolve a major issue in data science. - I think it's a good point Justin made at <a href="https://www.dremio.com/why-apache-arrow-wikipedia/" rel="nofollow">https://www.dremio.com/why-apache-arrow-wikipedia/</a> that it's bad that Wikipedia editors reject articles on stuff they know nothing about - if you look at their profiles, they don't seem to have any knowledge or interest about technology or software. That's not a good system. - I haven't contributed to Wikipedia really before, and I don't understand the rules, I admit that. Probably what I did yesterday was just not following their process, and that's the reason my edit was reverted. I guess it's also true that Justin at first didn't do a great job at submitting an impartial, non-PR article. However, my understanding from looking at some drafts and the talk page is that he then took the editor comments into account, and the last variant of the page he tried to submit in July 2019 was OK. - So overall I think the answer to the question "Why isn't there a Wikipedia page on Apache arrow?" is that it's an unfortunate case of authors and editors not doing a great job. At least I'm pretty sure I didn't do a good job yesterday, I wanted to help, but only had an hour, not a day to learn how Wikipedia ticks and to do more research to find better references. I hope someone with more experience in Wikipedia and Arrow will try to re-write and re-submit the Wikipedia article in the future. - The rule to discourage (or forbid?) people involved with Apache Arrow from contributing to its Wikipedia page is unfortunate. I recently started to use it and learn about it, but I don't know much about it at this point. E.g. Wes McKinney has written at this point 8 high-quality blog posts about it (<a href="https://wesmckinney.com/archives.html" rel="nofollow">https://wesmckinney.com/archives.html</a>) - those don't count as references? Even if he or the Apache Arrow team wrote a paper about it, it wouldn't count because it's a primary source, and Wikipedia only wants secondary sources to establish notability? There are ~ 100 videos on YouTube, and many blog posts and a few podcasts (e.g. <a href="https://softwareengineeringdaily.com/2016/07/17/apache-arrow-with-uwe-korn/" rel="nofollow">https://softwareengineeringdaily.com/2016/07/17/apache-arrow...</a>) that mention Apache Arrow. Naturally almost all of them are from Apache Arrow contributors, or from companies using Apache Arrow. - Apache Arrow has an interesting story, and it has evolved over the past years and will keep evolving, so I think exactly for that reason a Wikipedia page would be good to have, since the current project page and old blog posts don't capture that well.

zevebover 5 years ago

One could perhaps be forgiven for wishing that the deletionists would … delete themselves.Seriously, though, bytes are cheap, and an article sitting somewhere in Wikipedia doing nothing and bothering no-one is pretty damned cheap too.

评论 #21829320 未加载