I was inspired by https://news.ycombinator.com/item?id=20440079 thread regarding building a new search index. Technically it is feasible with CommonCrawl + cloud map reduce jobs. Clearly, you would be killed if you build google search, but I started thinking what kind of alternative data that you could build which could be valuable.<p>Here's some of my ideas:<p>- Ecommerce products ranking based on backlinks.<p>- Competitor analysis - most linked pages, total backlinks<p>- Most common scripts installed - you can see trending in integrations such as react, fb, intercom, etc. ( already been done, i know )<p>Comments? Any more ideas?
In 2015 'it will be killed off by google' sounded feasible in my mind. Today I use Duckduckgo for 90% of time and Google only for some very specific (mostly local) searches, Google seems to only hold 'early starter and low moral privacy policy' advantage. They also seem to target typical 'new Internet user' now more than ever, I don't remember last time I found something useful or 'smart' with Google.<p>That being said, let Google have general traffic search because it fails on everything else. I agree with your approach to find a generic niche, first question is - who's end user? Will it be 'I want to buy this item' (e-commerce)? Will it be 'I want to find out industry specific information? (competitor). Will it be 'I want to know how to use X' (your last one).<p>Each has its own problems. And, most likely, each already has a solution - it's just probably a) too commercialized (e-commerce - comparison sites) b) biased (competitors - analytics engines used by whole industry) c) too broad<p>There is abundance of data and it interconnects heavily, there is more and more alternative data (social apps, images, maps etc) too. I think it would be nice to think outside of algorithmic approach (how do I find X in most optimal way) and start thinking about contextual approach (how do I find if X is right).<p>Uff, to the point - I think starting small, with something annoyingly hard or time consuming to find, would be best. For example, I always spend DAYS reading about basic house appliances before buying it and most of info I find is heavily commercialized therefore useless (affiliate programs). I don't say 'Do that!', but there is definitely a need for niche-specific-search-algo/aggregator.