I don't think it's being paranoid. It's a remotely controlled parser. Fuzzing has turned up some of bugs in irssi and weechat over the years. Things like malformed color codes, DCC filenames, or even basic protocol messages led to crashes.
Well, the difference is that someone could PoTenTiAlLY spawn a shell if they get their way. So between server access as a user and container access (if it has a shell), it does make a difference.
A good book on this was "Hacking: The Art of Exploitation".
My argument though is that irssi is that old, I think automatic file receiving (DCC) is off by default and it has sensible defaults and a long history of being reliable(?)
I personally use weechat inside nsjail on a raspberry pi (isolated rpi is enough here, but just for fun): https://github.com/google/nsjail/tree/master/configs