We had it running fine under FastCGI, as well as mod_python for a few quite large projects and about 30 active developers (and a few dozen moderately active users). Speed wasn't a problem in either of those modes, but I recall that when it was straight CGI or using it's own server it was close to unusable.
I have a trac 0.94 install running on a pentium 800 through Apache and its definitely fast enough. I am not using Fast CGI though, so it could potentially be faster.
Has anyone experienced a speedup by not using the built in server?