[wanderingmind]

Almost a year back, someone proposed about TaxyAI[1], a chrome extension for browser automation. TaxyAI looks more matured compared to this. Are there any other similar tools that exist for browser automation using large language models

[1] https://news.ycombinator.com/item?id=35344354

[suchintan]

We're also working in the space and just open sourced Skyvern

https://github.com/Skyvern-AI/Skyvern

[hamoodhabibi]

ah this is actually quite valuable because it utilizes CV

I'm kinda surprised why you chose to open source this instead of slapping AGPLv3 like all the YC funded github projects are doing

[suchintan]

Haha we are not unique there. We chose AGPL-3 as well -- become some would argue it's like an open source virus -- everything it touches must become open source! How exciting.

[hamoodhabibi]

suchintan are you on X by any chance

how can i contact you

[suchintan]

You can message me on our discord or email me [e-mail]

[torginus]

I did desktop UI testing a couple years ago on Windows apps, and the standards solution there is to use UI Automation, which itself works by sending messages to each app that makes them run internal queries to find elements.

It seems like quite the intuitive approach, but we quickly discovered that due to differing implementations, and the reliance on the apps actually cooperating with you, it's actually so much more reliable and much faster to use OpenCV to physically detect UI elements by appearance.

[nextaccountic]

Regarding "instead of": AGPL is open source too

[anonzzzies]

Might as well not be for many companies; I know many who are not allowed to even glance at AGPL code for fear of getting infected (and sued).

[littlestymaar]

There's an easy way to avoid being sued though: comply with AGPL and make your own work open-source as well.

The “problem” with AGPL is companies who want to use open source software to build proprietary stuff on top without contributing anything back. AGPL is purposely designed to avoid this kind of parasitic behavior, but that doesn't make it “not open source” quite the opposite: it's “forced open-source”.

It is indeed restricting companies' freedom though: their freedom to restrict their user's freedom.

[anonzzzies]

Sure and I am all for it; I am just saying what my clients say to us. So for them (and it’s most of them, even if they never have any intention of changing the source code, ever), if it’s this license, they won’t touch it. That’s coming from their lawyers, no matter what we/I say.

[orra]

That's a them problem. AGPL is clearly open source.

[anonzzzies]

Sure, but that doesn’t change the reality. I would say ‘their loss’, but I think it’s more nuanced than that.

[999900000999]

Anyway to get this to run inside of a lambda or in another server less framework?

[suchintan]

I'll create an issue to create a Docker file for Skyvern. Would make that much easier

[999900000999]

Thanks!

HN is awesome!

[suchintan]

Yep! It's just a standard python + postgres combo, so if you create a docker file for it it should run inside a lambda!

[hamoodhabibi]

It's always the same story with web scraping product building: On the surface it's very interesting work. There is joy in seeing the fruits of your work automating human hours. There is also pain in seeing race to the bottom in that its very tough to get a recurring client who is always looking to reduce the cost.

[LZ_Khan]

care to elaborate a bit? im thinking about getting into the space

[wanderingmind]

More similar tools from quick browsing

[1] AI Employe: https://github.com/vignshwarar/AI-Employe

[2] BrowserPilot: https://github.com/handrew/browserpilot

[abrichr]

https://github.com/OpenAdaptAI/OpenAdapt is designed for all desktop applications, including browsers. We're also working on a chrome extension to support reading/writing directly to DOM: https://github.com/OpenAdaptAI/OpenAdapt/pull/364

[jimmySixDOF]

I think openinterpreter [1] were one of the first teams in this space along with shroominic code interpreter api and afaik they started with selenium but have expanded to do a lot more os level work but wonder if having a more narrow specialization could help these newer projects be better at the one thing they are focused on.

[1] https://openinterpreter.com/

[fulafel]

The voice-ai-device startup Rabbit seems to have a lot of browser automation stuff in their research side, they're calling their stuff a Large Action Model: https://www.rabbit.tech/research

[anonzzzies]

But you cannot try/download etc it right? We need open source stufff for things that control computers via a layer of vague human language. In my opinion of course.