The heat itself is entirely absorbed by the walls. As well as the previously mentioned neutron flux.
Also, heres one of many sources- this was found as part of looking up HFIR's name.
https://arpa-e.energy.gov/sites/default/files/B03-Wirth-Fusi...