Hey, Joel here, mostly responsible for this tool. Happy to answer questions — one thing not well covered is getting puppeteer to run in the browser, especially a webworker. Can talk more about it if there’s interest!
As a web dev generalist, I can usually understand how most things work under the hood.
But playing with chrome.browserless.io breaks that. You're streaming the web page in a <canvas> element, but how can I highlight text? When I load a youtube video page are you literally proxying the video through your infra, through <canvas> pixels to my browser?
Who dictates what IP the headless chrome is assigned to? Do you have a lot of IPs? I noticed on some pages I'd get the CloudFlare captcha which makes sense if browserless has to cycle through a limited set of IPs where other people have been using it to scrape another cloudflare page.
Yup, There’s a lot going on here. Currently the tool uses a fixed IP for the running browser. That’s why you’re seeing that Cloudflare issue.
As far as the hovering goes, the canvas element is “mirroring” interactions back through to the underlying page. When Devtools are active, this triggers chromium to render hover effects in its GUI. This then gets sent back to the canvas element in the debugging page.
It’s a lot of network traffic and Synchronization... but once everything is setup it works fairly seamlessly
can you explain how this is architected? you are running a browser on a server and its streaming the video to the canvas? where does the cloudflare bit come in?
You’ve got it: the core service listens for inbound Upgrade http requests, starts a browser, then funnels the connection into the browser.
This debugger simply sits on top of all that, and puts the code/execution context in the browser versus the server. Cloud flare is simply detecting that our server IP is a known headless chrome instance, and is serving their bot detection check
As a web dev generalist, I can usually understand how most things work under the hood.
But playing with chrome.browserless.io breaks that. You're streaming the web page in a <canvas> element, but how can I highlight text? When I load a youtube video page are you literally proxying the video through your infra, through <canvas> pixels to my browser?
Who dictates what IP the headless chrome is assigned to? Do you have a lot of IPs? I noticed on some pages I'd get the CloudFlare captcha which makes sense if browserless has to cycle through a limited set of IPs where other people have been using it to scrape another cloudflare page.