I'm really impressed that NYT took the time to document this. It's always interesting to see the different recommendation models evaluated and applied to real-world situations.<p>I've been pursuing a collaborative filtering approach to product recommendation lately ('people who bought this also bought that'), but perhaps LDA would let me model our products based on their metadata ('people who bought products broadly like this also bought products broadly like that').
We make a contextual recommendation engine as a service for online publishers at our startup ParallelDots. We discovered the problem of tags not really working well for recommendations on our clients websites too. We ended up using unsupervised word embeddings and auto encoders on top of them to solve the problem. We dont still use it for personalization though, just contextually similar articles. Great seeing some of similar problems being solved at New York Times too. :)
The way I see it, the primary thing to solve for any recommendation engine is to optimize for serendipity. I.e. allowing you to get information you didn't know you wanted.<p>This means basically also finding ex. articles that are not written by NYT.<p>Newspapers problem is that their primarily omnibus approach to whats relevant isn't really doing the waste amount of insightful information available that exist out there.<p>So the whole issue IMO with all newspapers/media these days. They are building silos where none should really exist and this is one of the primary the reason why people don't consider it valuable anymore.
Fun read. Topic modeling can be fascinating to work with.<p>Curious how they measured performance of their model, and whether they found a "best" number of topics for LDA where their model stopped getting much benefit by having more topics.<p>I'd imagine increased number of topics would have some interesting side effects where it would create too narrow of recommendations.
We have built this and anybody can use it <a href="https://Algorithmia.com/recommends" rel="nofollow">https://Algorithmia.com/recommends</a>. 2 lines of js to implement.Currently serving the geekwire.com recs. You can also modify it further (see blog.Algorithmia.com).<p>The article is awesome though good on NYT.
I think it'd be great if you'd have this kind of information in your help section later on, for anxious people like me who are very wary of even having a recommendation engine at a news paper. I was actually on my way to sign up for a subscription after reading "A Renegade Trawler, Hunted for 10,000 Miles by Vigilantes" by Ian Urbina - but held back for the moment to give it more thought.<p>That said, I guess I could see a point in it maybe retaining users / subscribers if it's good enough. (I'd still appreciated it a lot more if this functionality could be turned off for users who request it though).