you should send a user agent that makes it look like a real browser. for eg. Microsoft.com responses are different based on user agent, which is why they do the redirect. Here is Microsoft.com from Chrome (it is diff again for IE 6/7/8 etc.), you get a lot more information:<p><pre><code> Cache-Control:public
Content-Encoding:gzip
Content-Length:23262
Content-Type:text/html; charset=utf-8
Date:Fri, 23 Jul 2010 07:27:32 GMT
ETag:634154400660000000
Expires:Fri, 23 Jul 2010 07:37:32 GMT
Last-Modified:Fri, 23 Jul 2010 07:01:06 GMT
P3P:CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Server:Microsoft-IIS/7.5
VTag:438638311400000000
Vary:Accept-Encoding
X-AspNet-Version:4.0.30319
X-Powered-By:ASP.NET</code></pre>