[sirmarksalot]

As with any of these LLM workflow automation tools, it raises a few questions about each potential use case, and the likely long-term outcomes.

1. Is this working around friction due to a lack of interoperability between tools? For example, is this something that would be more efficient if the owner of the website exposed a REST service? Will the existence of this tool disincentivize companies from exposing services when it makes sense?

2. If there is a good reason for the lack of a service endpoint, perhaps for security reasons, will your automation workflow be used to bypass those security measures? Could your tool be used by malicious actors to disable major services? Are you that malicious actor yourself? Will your tool be used by scalpers to prevent consumers from buying high-demand products?

3. If this is being used to work around deferred maintenance with internal tools and processes, will the existence of these kind of tools be used by management to justify further deferral of that maintenance? Will your tool become a critical piece of the support staff's workflow?

4. If your tool is being used in good faith to work around anti-patterns in website design, will the owner of the website be incentivized to break your workflow? Is your use case just a step in an arms race?

These are the thoughts that go through my head whenever I hear about software being laid on top of complicated processes, where instead of simplifying the underlying processes, we add another layer of complexity to sweep it under the rug. I'm sure that people will find your project useful, but I wonder what the longer-term effects will be.

[suchintan]

1. Yes absolutely. But the issue is a little bit more nuanced than that. Websites without APIs don't have them for one of two reasons: (1) They want to protect their data (LinkedIn) or (2) can't be bothered to make an API (boutique websites, government portals). This solves that problem, but also makes it so these websites never have to build an API (after LLM costs go down).

2. We don't want Skyvern to be used on websites that prohibit this kind of behaviour (LinkedIn is the obvious example). Specifically, we didn't open source any of our anti-bot or captcha related code because we get requests to make "Reddit upvote rings" and such. We don't want to support bad actors like that

(3) I think this is a net net good thing. AI browser automations= less need for APIs = no need to maintain both an API and UI = streamlined experience + less code = simpler systems

(4) I'm not 100% sure about this one. We usually just assume companies don't build APIs because they don't have budget for it. Ie for non malicious reasons. Companies like LinkedIn will likely thwart any attempts at automation, but we're not interested in participating in this cat mouse game

[rmbyrro]

> after LLM costs go down

I think 100 Gb of GPU memory will always cost multiples of CPU + regular memory.

Using LLMs and computer vision for these kinds of tasks only make sense in small scales. If the task is extensive and repeated frequently, you're better off using an LLM to generate a script using Selenium or whatever, then running that script almost for free (compared to LLM). O1 is very good at it, by the way. For the $0.10 of 1 page interaction charged by Skyvern, I can create several scripts using O1.