[scriptsmith]

In a similar vein, I have found success using request interception [1] for some websites where the HTML and API authentication scheme is unstable, but the API responses themselves are stable.

If you can drive the browser using simple operations like keyboard commands, you can get the underlying data reliably by listening for matching 'response' events and handling the data as it comes in.

[1] https://github.com/puppeteer/puppeteer/blob/main/docs/api.md...

[RockRobotRock]

For this use-case, selenium-wire for Python could be really useful.

[tylergetsay]

You can also inspect the application storage, monitor for cookie changes, etc using the dev tools protocol