Here is a problem: symbol macros. Symbol macro foo could expand to a form like (bar xyzzy) let cannot know whether foo is intended as a form (via expansion of to (bar xyzzy)) or whether it's intended as a new lexical variable shadowing the symbol macro foo. It doesn't matter whether let is a special form or macro: something at some point has to walk the code of let and make this decision, whether that is the expander for a let macro, or the code walker that traverses special forms in search of macros.
If you want foo to be interpreted as a form just wrap it in a progn.
(Personally, I think it's very bad practice to use a symbol macro that expands to anything other than a place, and it's certainly bad practice to define a symbol macro that expands to a form with side-effects. So this would never happen in my code. YMMV of course.)