Just curious if anyone knows the answer to this question..I'm pretty sure when it first was started they were basically scraping these financial institutions, but I don't know for sure. If they were initially screen scraping, do you think by now they have their own special access to the financial institutions that they deal with?<p>The reason why I'm interested is that I'm currently working on a site that's similar in nature but for a totally different industry and I'm worried that the websites I'm scraping from will be updated and will no longer work...I'm not sure how often the websites in question are updated but my guess would be that as the industry evolves over time the websites will be getting updated more frequently and it's currently only a 1-man engineering team (me)
No, they used Yodlee initially, and now use the Intuit aggregator. However, Yodlee does some scraping, but rumor was (this was 2008) scraping was less than 10% of financial sites and they had more direct access to other 90%.
<i>I'm worried that the websites I'm scraping from will be updated and will no longer work</i><p>When I worked for Internet Brands, they bought a site called Tjoos (a coupon/deal site) that amassed what was at the time the largest number of merchant sites using a proprietary screen scraping technology. It is a very fragile method, and a good portion of the codebase was dedicated to config classes that needed to be updated with each change to a given site. The founders had to keep hiring developers to keep up with the demands of constantly updating scraping definitions, so the site had a lot of revenue, but little profit when IB bought it. Surely by now Internet Brands has updated the site to use the same engine that powers UlitmateCoupons, BensBargains, DealLocker, etc.
There was a previous discussion of this last year: <a href="http://news.ycombinator.net/item?id=1537825" rel="nofollow">http://news.ycombinator.net/item?id=1537825</a><p>I also recall (but can't find) another discussion of screen scraping where the consensus seemed to be 'be very careful of the legalities'. If you are small you might fly under the radar, but if you are successful you might just be opening yourself up to a whole can of worms.<p>Regardless of the outcome of any battle over this, the battle itself would be a severe distraction.