[olegp]

Very few pages have well formed mark-up. The few large scraping projects I've seen have started out with a mark-up based approach and then switched to regular expressions.

What experiences has everyone else had?

[Harkins]

I've had a lot of success with BeautifulSoup. It turns terrible HTML into a usable DOM tree.

[joseakle]

i am using beatiful soup, it works with malformed markup

[DenisM]

Uhm. I though any valid HTML is also valid SGML? Are you sure you're not confusing it with XML markup?

[ryanwaggoner]

I think he's saying that the HTML-parsing approach only works when the HTML is well-formed and for most sites, it isn't.