JPEGs are intrinsically YCbCr not RGB, so I don't know how you would avoid some precision loss when you combine that colorspace conversion with things like the quality level and (potentially) subsampling.
JPEG can be YCbCr, RGB or even CMYK (usually from Photoshop or similar software). There can also be unusual subsample ratios for YCbCr, such as 4:2:1. I created some WASM Go bindings for jpegli, and I recall that such images cannot be represented in Go; therefore, for these images, I force RGBA output.
Also, XYB is an option; I use the adaptive quantisation option from jpegli (with progressive level 2) to get smaller files. I never bothered with XYB, as it looked complicated.