In your opinion, other than for funsies, what would be a good situation for an open source project to pursue a custom IC vs say an FPGA or making due with a commercially available IC?
I think the only reason to do a custom IC, for now, is if you can't achieve your performance, power, area or price targets with FPGAs, microcontrollers, or other discrete components. It's still impractically expensive unless your volumes are enormous or your margins are very high. Even with a reasonable complex feature set, I think you can do better with individual components. You can get relatively cheap ARM cores, FPGAs, DSPs, microcontrollers, etc. I don't often see companies doing custom ICs for internal use.
That being said, I think the economies are changing. If you don't need a cutting edge process, the tapeout costs continue to come down. The problem that remains, though, is licensing cost of EDA tooling. There is a little bit of competition in that space, but not much. It is growing though, so hopefully that brings prices down.
You mentioned the cost of tools a couple of times. What's wrong with using MOSIS, which is free? Just asking, since I don't know anything about this area except that MOSIS was mentioned multiple times when I was in grad school back in the 1980s.
MOSIS is a fabrication service. The tools are totally separate, and you'd still have to pay to use them. Universities generally get free licenses though.
The thing is, with every chip generation the non recurring engineering costs are heavily swinging towards FPGAs staying competitive at ever higher volumes. If you're happy with a CPU+FPGA combo and want to take advantage of 12 nm but your design fits on an FPGA, then it almost never makes sense to design an ASIC.
Your design would in principle have to be an analog design or something that simply cannot take advantage of smaller nodes i.e. MEMS.
That being said, I think the economies are changing. If you don't need a cutting edge process, the tapeout costs continue to come down. The problem that remains, though, is licensing cost of EDA tooling. There is a little bit of competition in that space, but not much. It is growing though, so hopefully that brings prices down.