5/4 isn't that challenging of a time signature. I would argue it's just uncommon and so most people associate it with Take Five. 3/4 was extremely popular in Europe for centuries and is not perceived as challenging.
5/4 is actually ambiguous because the "5" can be any of 3+2, 2+3, 2+1+2 etc. Similar issues for 7/4, etc. 4/4, 3/4 and 6/8 may have rhythmic variations too (e.g. the 3+3+2 tresillo rhythm common in rock-and-roll music) but these don't impact the metrical understanding of the music to anything near that extent.
I'm not saying it's THAT challenging. It's not 13/2, but it is 5/2. Most 5/2 songs will sound unnatural to most people . It's ironic to make a comment like that in the context of an article like this one.
The article is making the case for challenging music. You pointed out that Brubeck made challenging music (i.e. 5/4) accessible to most people. My argument is that this is a poor example as 5/4 is not that of a challenging time signature, and most non-musicians who heard Take Five probably didn't even perceive the uncommon 5/4, and just enjoyed the tune. 5/4 is popular in progressive rock, post rock, and electronic music. In my experience most people do not pay attention to that. Similar to 3/4 - it's an uncommon but easy time signature that people in the Western world have no problem with. Basically, there is no challenge.