As the office manager of a relatively early stage startup, I think that company culture absolutely does matter. If people aren't having fun, or happy to be there, work suffers. If people are engaged with each other, sharing experiences and inside jokes, and pleased with the overall atmosphere of the office, it goes better. Why work at an early stage company at all if you don't want to shape the future of it?