As the office manager of a relatively early stage startup, I think that company culture absolutely does matter. If people aren't having fun, or happy to be there, work suffers. If people are engaged with each other, sharing experiences and inside jokes, and pleased with the overall atmosphere of the office, it goes better. Why work at an early stage company at all if you don't want to shape the future of it?
Definitely! If you're going to be a part of something in its early stages (or really, in any stage?), why not try to make it the best possible version of itself?