How did you determine that speed was what gave the site its first toehold? Did you collect some kind of data that supported your internal push to make that a priority?
It depends on the use case but loading time is very important. People will engage more with a site where they can move freely and focus on the content rather than a loading page. The longer the loading time the more likely users will click away.