It's a classic and highly regarded physics course although these days there are many options to choose from. It's not for median laypeople, though - the original audience was CalTech undergrads so there's a fair bit of mathematical (and, realistically, physics) pre-req.
I read somewhere (probably another HN comment) that most of the undergrads dropped the course, but Feynman didn't know because they were replaced by grad students.
This can't be right. You couldn't drop the course. Two years of physics in the freshman and sophomore years, taught from the Feynman Lectures, was a requirement for all Caltech undergrads, including those in humanities majors like economics, history and even English -- there were a few.
The whole two year physics sequence - the lectures where your entire class attended together, always in the same room, three times a week for two years, the homework sets, the sections, the weekly take-home quizzes - were a shared rite of passage, a central part of the Caltech experience for every student. It was difficult for some of us, but we had to persevere - it wasn't optional. A classmate called it 'intellectual boot camp'.