I get the point about getting to work with the best people in the world, I personally think that for the beginning of a company is better to be all in the same room. Remote working is fine for bigger companies imho.
After the release of the book "Remote", by 37signal, I decided to share my thoughts about Remote Working. I've been working remotely from a lot of time and this is my experience.