Algorithmic tagging of Hacker News or any other site

Looking at the hn demo, I&#x27;m impressed. There are definitely relevant tags being generated. Unfortunately there also some noisy tags which clutter the results. Taking one example, the post &quot;DevOps? Join us in the fight against the Big Telcos&quot; given the tags &quot;phone tools sendhub we&#x27;re news experience customers comfortable&quot;, I would say that &quot;we&#x27;re&quot; is unarguably noise. Another example, &quot;Questions for Donald Knuth&quot; with tags &quot;computer programming don i&#x27;ve knuth taocp algorithms i&#x27;m&quot; I would call out &quot;i&#x27;ve&quot; and &quot;i&#x27;m&quot;.<p>There are other words in both examples that I personally would not use as tags, but I can&#x27;t really say they would be universally not-useful. I think a vast improvement could be made just by having a dictionary blacklist filled with things like these - from this tiny sampling contractions seem to be a big loser.

Very interesting.<p>I have been doing some research towards automatic tagging lately, and I found several Python project coming close to this goal : <a href="https://pypi.python.org/pypi/topia.termextract/" rel="nofollow">https:&#x2F;&#x2F;pypi.python.org&#x2F;pypi&#x2F;topia.termextract&#x2F;</a> , <a href="https://github.com/aneesha/RAKE" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;aneesha&#x2F;RAKE</a> , <a href="https://github.com/ednapiranha/auto-tagify" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ednapiranha&#x2F;auto-tagify</a><p>but none of them is satisfying, whereas Algorithmic Tagging of HN looks pretty good.<p>I have been trying to implement a similar feature for <a href="http://reSRC.io" rel="nofollow">http:&#x2F;&#x2F;reSRC.io</a>, to automagically tag articles for easy retrieval through the tag search engine.

Well, it&#x27;s not that easy. The algorithms are very primitive and too full of noise to be useful.<p>For example, try this on restaurant reviews like <a href="http://www.yelp.com/biz/el-gaucho-seattle" rel="nofollow">http:&#x2F;&#x2F;www.yelp.com&#x2F;biz&#x2F;el-gaucho-seattle</a>. I get these tags:<p>steak reviews seattle food service gaucho restaurant review<p>Not useful, right?<p>The current state of the art would use much more sophisticated NLP for generating POS tags and use sentiment analysis. For example, check out MSR Splat at <a href="http://research.microsoft.com/en-us/projects/msrsplat/default.aspx" rel="nofollow">http:&#x2F;&#x2F;research.microsoft.com&#x2F;en-us&#x2F;projects&#x2F;msrsplat&#x2F;defaul...</a>.

This does well on the &#x27;T Shirt test&#x27; on some sites, e.g. <a href="http://www.riverisland.com/men/t-shirts--vests" rel="nofollow">http:&#x2F;&#x2F;www.riverisland.com&#x2F;men&#x2F;t-shirts--vests</a><p>This could be really useful in ecommerce for creating search keywords for category pages. The noise in the results matters not, so long as it gets &#x27;T-Shirt&#x27; and someone searches for &#x27;T-shirt&#x27; then all is well and good.<p>Are you looking to plug what you have into something such as the Magento e-commerce platform? The right clients could pay proper money for this functionality. It is something I would quite like to speak to you about.

LDA is very impressive. But it might be better to have an iterative algorithm that forms a linear-algebraic basis from several tags (and let people add more tags as vectors into the mix) and then every time people upvote something, you update their interests (points in the linear algebraic space) and then every time an article gets upvoted you update ITS tags ...<p>after a while the system converges to a very useful structure and new members can see correctly tagged articles and the system learns their interests by itself<p>do you know anything like this already existing?

This has real poetic potential:<p><pre><code> &quot;Erlang and code style&quot; process erlang undefined file write data true code</code></pre>

After watching &quot;Enough Machine Learning to Make Hacker News Readable Again&quot;[1] I thought of recommendation engine&#x2F;machine learning based linkshare&#x2F;discussion system (eg HN&#x2F;reddit style). Your frontpage would be continuously formed by your up&#x2F;down-votes. I&#x27;m not sure if the same could be applied to comment threads too, essentially creating automatic moderation. Algorithmic tagging would certainly be useful for that kind of site.<p>[1] <a href="https://news.ycombinator.com/item?id=7712297" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=7712297</a>

Not too impressed to be honest; singular&#x2F;plural forms are not treated equal; not familiar with LDA, but I&#x27;ve written and LSA implementation in the past, and it did a lot better than what is shown here.

Lol, this seriously took me by suprise. I&#x27;m currently developing a HackerNews with tags (you can self host it). I quickly generated this Google Form, if you are interested for being a beta user in the nearby future<p><a href="https://docs.google.com/forms/d/1UeSD11hrjwhsVbbPiv63VZBrEczzG5Tr4lwkuKAzY8A/viewform?usp=send_form" rel="nofollow">https:&#x2F;&#x2F;docs.google.com&#x2F;forms&#x2F;d&#x2F;1UeSD11hrjwhsVbbPiv63VZBrEcz...</a><p>PS. Screenshot included + it&#x27;s already in alpha in a company with 100 users.

I did that in 2012 for a pet project with a friend <a href="https://github.com/SnippyHolloW/HN_stats" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;SnippyHolloW&#x2F;HN_stats</a><p>Here is the trained topic model (Nov. 30, 2012) with only 40 topics (for file-size mainly) <a href="https://dl.dropboxusercontent.com/u/14035465/hn40_lemmatized.ldamodel" rel="nofollow">https:&#x2F;&#x2F;dl.dropboxusercontent.com&#x2F;u&#x2F;14035465&#x2F;hn40_lemmatized...</a><p>You can load it with Python:<p><pre><code> from gensim.models import ldamodel lda = ldamodel.LdaModel.load(&quot;hn40_lemmatized.ldamodel&quot;) lda.alpha = [lda.alpha for _ in range(40)] # because there was a change since 2012 lda.show_topics() </code></pre> Now if you can figure out what is this file: <a href="https://dl.dropboxusercontent.com/u/14035465/pg40.params" rel="nofollow">https:&#x2F;&#x2F;dl.dropboxusercontent.com&#x2F;u&#x2F;14035465&#x2F;pg40.params</a> I&#x27;ll pay you a beer next time you&#x27;re in Paris or I&#x27;m in the Valley. ;-)

After receiving 42 comments, I&#x27;ve ran their tagging algorithm on this page and got:<p>tags tagging hours link doppenhe reply ago lda<p>looks pretty promising!

LDA&#x2F;Topic Modeling is interesting stuff. I always feel like the way this data gets surfaced as &quot;tags&quot; is very ineffective. Any non-tech person would look at this and generally be confused. So this item is triggering my rants against tagging: - Tagging is like trying to predict the future. What word will help some future person to get to this content? - Tagging often tries to fill the hole left by bad search - There is no evaluation method to measure how good a set of tags are - Tags make very bad UI clutter.<p>Some of these points are related to encouraging users to tag content, but auto-tagging also seems problematic.<p>To me something more along the lines of entity extraction is more useful because it is a well defined problem, and can be used to improve a lot of other applications.

I like this project (i am creating something like this, so i&#x27;m pretty serious).<p>But doesn&#x27;t the auto-tagging feature make to much noise for a business use-case? For example, it tags a article of Amazon and includes Google in the tags. White-listing words wouldn&#x27;t fix this (Google is a whitelisted word if Amazon is).<p>I don&#x27;t know about LDA though. Perhaps a proper tag administration would fix this, but then you&#x27;d have to remove tags on the go.

Direct link to HN with tags: <a href="http://algorithmia.com/demo/hn" rel="nofollow">http:&#x2F;&#x2F;algorithmia.com&#x2F;demo&#x2F;hn</a>

Has anyone seen Open Calais [1]? It does tagging and categorization. It&#x27;s been around for years and seems pretty powerful. It&#x27;s a bit lower-level than Algorithmia (not href aware), but it seems more powerful, and a system like Algorithmia could be built on it.<p>[1] <a href="http://www.opencalais.com/about" rel="nofollow">http:&#x2F;&#x2F;www.opencalais.com&#x2F;about</a>

With german sites it does not work so well. There is no blacklist for to generic terms for other languages than english?

Doesn&#x27;t seem to handle pdfs properly. For the mtgox link it comes up with<p>&gt; stream rotate type&#x2F;page font structparents endobj obj endstream

I signed up. Not sure if i would use it, but the Algorithmia concept is pretty interesting.

@doppenhe - any hunch as to how well it would work on transcripts?

error: failed to find worker for algorithm

Maybe also take a look at AlchemyAPI

looks cool.