It's the UK term for Wall Street. It doesn't have as much of the geographic connotation these days - working in Canary Wharf or a Mayfair hedge fund can all come under the phrase of 'working the in the city'
In the City, you get hired based on a "total comp" package that includes salary and a practically guaranteed bonus, that can be as much as your base salary or even more.
http://en.wikipedia.org/wiki/City_of_London