We're working very hard on making GitLab.com much faster. Right now, it's measurably (and more importantly: measured [0]) faster than it was a month ago.
If you would like to follow progress you can dive into it all here [1]. One of the spearpoints is Gitaly, which should significantly improve git access.