A 3 karat diamond is still a crystal (approximately) the size of your pinkie nail that takes a fairly large energy input to create... and only the ones that've grown clear and without flaws are viable for jewelry purposes. The bigger it gets, the lower the yields are at the appropriate quality level, and so the more effort / waste is involved in getting one. (As opposed to industrial diamond uses, which mostly don't have those requirements and so have their price even more heavily impacted by synthetics.)
Diamond is not the energetically favorable form of carbon at room temperature and pressure, so to manufacturer there will always need to be expensive tricks. Not as expensive as digging out tons of kimberlite, but still not cheap.