That depends on what you want to do with your career.
Generally, internships at startups are more flexible and open-ended. Internships at BigCos can be more rigid and less interesting.
If you believe you're going to spend your career in startups, you're obviously better off interning among startup people.
But, be careful: a serious internship at a large, technology- driven company (GS qualifies) is better than a poorly- managed internship at an inept startup.
Also, be aware that an internship at GS carries weight among SFBA startups, but an internship at a random SFBA startup probably carries no weight on Wall Street.
I've got a friend going to NYC for either Goldman or some other big firm; I can't recall which. He's super excited and for good reason! It's an awesome opportunity.
When it comes to connections/networking, what more could you want? Plenty of people with money, people in charge of money, people that will be hiring programmers for years; the lot!
As for knowledge or programming experience, I don't exactly know if it'll be more "rewarding" than a startup. It really depends on what you consider rewarding. You'll learn plenty about surviving a workforce and meeting deadlines in a work environment, that's for sure. I don't think you'll learn much, if anything at all, about web-development unless you're specifically working on some proprietary web-app that handles client interactions.
Can you give us a little more info on what division of the business you'd be working in?
Depends on the startup. There are top-tier startups, promising startups, and then there are random startups. Some "startups" like Pinterest, Airbnb, Dropbox, Quora, etc. have extremely strong engineering cultures and would probably be a ton better than GS.
Generally, internships at startups are more flexible and open-ended. Internships at BigCos can be more rigid and less interesting.
If you believe you're going to spend your career in startups, you're obviously better off interning among startup people.
But, be careful: a serious internship at a large, technology- driven company (GS qualifies) is better than a poorly- managed internship at an inept startup.
Also, be aware that an internship at GS carries weight among SFBA startups, but an internship at a random SFBA startup probably carries no weight on Wall Street.