Would you mind expounding a bit on what open source options you've tried so far? I've got an itch for a side project and I'd be extremely interested in exploring this space.
This is the main one http://www.wowza.com/ -- I'm not sure if it's free anymore judging by the site but suffice to say it's much cheaper than Flash server. There is no CloudFormation stack for it AFAIK but there are existing AMIs.