Really. Vacuum casing is not even close to sufficient to set heat absorption to zero because of thermal radiation.
And you can't just make the walls reflective once the cold object gets smaller than the wavelength of the radiation. The colder the object, the longer that wavelength.
The way it works is that the entire assembly is in a vacuum. It kinda has to be as any gas which touches it will instantly condense to it or freeze to it. You then have a dual cryostat of liquid helium and liquid nitrogen cooling down the assembly (within the vacuum). The helium and nitrogen cryostat also have a vacuum shield. The nitrogen (liquid at 77K) is a sacraficial coolant which is far cheaper than liquid helium (liquid at 4K) that you need to get to these temperatures. Your're right that thermal radiation is an issue so you have to be careful with the placement of any windows or mirrors around the device.
Souce. I have a PhD in physics where I used equipment cooled to 4K.
Great, then we both have physics PhDs, and you'll know that none of that equipment has, or easily could be, sufficiently miniaturized, which is the topic of discussion ("extremely small cryocooler"). You can't put nested closed dewers of liquid nitrogen and helium on a O(1 mm^2) microchip, and the reason is exactly what I said: it will warm up too fast.
The topic is cooling small objects so that personal electronics (e.g., your phone) can compete with datacenters. Cold at scale (i.e., in datacenters) is comparatively easy.
And you can't just make the walls reflective once the cold object gets smaller than the wavelength of the radiation. The colder the object, the longer that wavelength.