Then you'd have to use Svelte which not many organizations will, as React is more mature and Svelte is newer. Perhaps in a few years that will change but IMO moving to SvelteKit is orthogonal, most who are currently using React will want to continue using React based frameworks.
I played around with Next.js and SvelteKit. I don't recall the specific "that's a down/upgrade" but I switched to SvelteKit because "it just works". HMR, more or less plain HTML, (Tailwind) CSS and JS. Developing with Svelte(Kit) feels, and probably is faster than NextJS/React.
The only downside are missing UI libraries like RadixUI. Tried out everything and just "wasted" 2 weeks to build a component library on top of TailwindCSS inspired by Material UI.
SvelteKit comes navigation, sane state management, optional Server Side Rendering, Hot Module Reloading and super-fast pipeline with Vite. And some extra goodies like built-in animation utilities, debouncing, auto-prefetch links on hover, etc...