[jmfldn]

This is exactly right. You can't become a music buff by following a predefined map, or at least that wouldn't be a very satisfying journey.

Speaking as a music fan, former producer and DJ I say that the secret is to follow your instincts. Find some things you do like, can be anything even if you currently only like a few things. Explore from here. Explore the artist's catalogue, find similar artists, look at related genres or the genre's roots / DNA. Spotify and YouTube algorithms are great at finding related content so just fall down a recommendation hole and see here it takes you. You can quickly make some interesting transitions into other genres even like this. Don't overthink it but see which things make you hungry to explore further.

Don't seek to be a buff. Seek to explore your taste, follow your gut, follow your feelings and see where it takes you. You'll naturally end up knowing a lot but it's not about knowing, it's about the joy and fulfilment of music and nothing more.