Try to determine which one you're using by running "M-x linum-mode" and then "M-x display-line-numbers-mode". Note that you can use both at the same time (which would make zero sense: it's only to troubleshoot which one is problematic for you).
Do not use linum-mode anymore.
I can display buffers with hundreds of thousands of lines while showing the line numbers (Emacs 29 / native compilation / display-line-numbers-mode / CPU is an AMD 3700X with 32 GB of RAM) and Emacs is ultra responsive.
This is the kind of thing which infuriates me the most about emacs. There are thousands of similar but not equivalent ways of doing the same thing, both StackOverflow and the emacs wiki contains some (but not all, and not the same, and usually not the right, and usually contradictory) ways of doing things. And it's impossible to figure out which way is the right one without asking around in IRC channels.
Why oh why couldn't they improve linenum-mode instead of introducing a new mode.
Do not use linum-mode anymore.
I can display buffers with hundreds of thousands of lines while showing the line numbers (Emacs 29 / native compilation / display-line-numbers-mode / CPU is an AMD 3700X with 32 GB of RAM) and Emacs is ultra responsive.