ARM SoCs aren't made in low numbers for just a bunch of dev boards, or when they are made that way it's for the first initial testing runs and not publicly available at actual prices.
All these ARM single-board computers use chips that are produced for other purposes in way larger numbers (some of the Raspberry Pi models might be an exception and actually have a slightly customized design just for them, but I'm not sure about that).
The rule of thumb is to not do an ASIC unless you're selling at least 100,000 units if you want it cheap per unit. The ARM SoC's that get used in a lot of products probably are selling in really high volume. To illustrate the effect, a product that cost $100,000 to design would break-even at $1 a unit in that volume but $1,000 a unit if only 100 were sold. That's idealized: you generally have to pay for the masks, a wafer per set of chips that fit on one (number varies), absorb/charge the cost of broken chips, IC packaging for the chips, shipping, storage, etc. That's not counting boards, assembly or whatever.
So, selling something in high volume like big-name ARM's gets the prices way down per unit. Getting to high volume is a marketing and product development problem more than a technical one. Good luck on the startup. :) The ecosystem benefits will give ARM-based solutions an advantage there for a while into the future. The RISC-V chips will cost more due to lower volume unless all the heavy costs are absorbed at a loss by whoever builds them. I've recommended Universities or foundations attempt that to get good, FOSS chips started on good nodes with them sold at material, assembly, and distribution cost from there.
OpenPITON is another one with potential since they have prototypes for 32-core, OpenSPARC CPU's at 32nm. Leon3 GPL was an older one in SPARC. J2 (SuperH-style) is a compact one that's about 3 cents a core in 180nm with who knows what performance, energy usage and pricing could be achieved at 28nm-32nm. CHERI, a capability-secure extension of MIPS, could be ported to one to give it a security advantage to secure sales from defense sector. Draper is aiming for that with SAFE architecture (crash-safe.org) added to RISC-V. If wanting max reliability (eg safety-critical), VAMP was a 32-bit DLX CPU formally-verified for correctness that could be done on older node with mods for lock-step, triplicated redundancy, or board-level fault-tolerance.
So, there's possibilities to justify higher prices long enough to recover upfront costs or get them down on simpler designs. The HiFive is a serious core on a relatively-recent node, though. It's not going to be cheap unless selling boatloads. They probably didn't expect this product to sell boatloads, either.
don't forget the cost of testing - that adds up, often people do simple testing on the wafer to avoid the cost of packaging bad die, then full testing once packaged - remember a bad chip for a semi manufacturer is a whole bad board for their customers
Most SBCs are based on SoCs that are pretty much end of life. They have already amortised their costs years ago by selling them in smartphones, tv boxes, tablets.
All these ARM single-board computers use chips that are produced for other purposes in way larger numbers (some of the Raspberry Pi models might be an exception and actually have a slightly customized design just for them, but I'm not sure about that).