[Ldorigo]

I'm always stumped by comments like this. I'm at a point where ~70% of my code is AI-written, and the majority of the remaining is mostly because it would take too much time to provide enough context to the tool/LLM of choice for it to be able to produce the code I need.

Given the right context and the right choice of model/tools, I think ~90-95% of the code I write could be generated. And this is not for doing trivial CRUD; I work on a production app with 8 other people.

I'm really curious if you could give examples of problems that you tried and failed to use these tools for?

[joblemblem]

Two recent examples.

Please link to your history when you get one of these things to build my example so I can see how you managed to do it.

First, a friend without technical knowledge wanted to query an API using SQL. (At a previous firm he learned how to write “SELECT * FROM … WHERE …” statements.)

He asked one of these llms to do this, that he paid a premium for, and it suggested installing VSCode and writing an extension to ingest an API and to query it with python.

I am unfamiliar with VSCode so I’m unsure if this is even feasible, but after 3 days of trying to get it to work, he asked me, and I set up a python environment for him, and wrote a 5 line script to allow him to query the data ingested from the API into SQLite.

For me, the last time I tried, I asked one to write me a container solution using Docker that allowed for live coding of a Clojure project. I wanted it to give me solutions for each: deps.edn and lein.

I wasted hours, because it always felt “just around the corner”, trying to get it to output anything of use for either paradigm then when I abandoned the llm I quickly found, via a web search, a blog post someone wrote that did exactly what I asked, for a lein project of their own, and I just changed it to work for my project, and then again for the deps.edn version on my own.

[oxidant]

This isn't a knock on you. The prompt (including additional context like specific examples/docs) has an incredible influence on the output.

Do you know what your friend asked? "Query an API with SQL" sounds like you're sending SQL with POST api[.]com/query. What you built is more like

"Make a request to this endpoint, using these parameters, and store it in sqlite. This is the shape of the data coming back and this is what I want the table(s) to look like."

Gpt4o or Claude could easily write that five line script, if given the right information.

I find writing prompts and working with LLMs to be an entirely new way of thinking. My struggle has been to learn how to write out my tacit knowledge and be comfortable iterating.

Do you still have your conversation where you tried to build the Docker project?

[jcinau]

Share my experience. I tried 3-4 different LLMs, and one of them is outstanding.

For code samples, popular programming languages are much better than languages like Clojure.

Two examples: About a week ago, I had found Myers' string diff algorithm and asked to write some code and initially it spat out Python code. I asked it write a Common Lisp code, and it generated about 90% complete code. I rewrote it again and the whole thing took less than a day. It was my first time seeing 'quality' from machine generated code.

I experimented further. I found Automerge-Java and want to write a Clojure wrapper. So asked it how to parse Java source files and it showed a Python code. I ran it and gave some feedback than I could get almost perfect output, which is easy to process from Clojure side. After three days, I could write interface generator. From my experience, this type of work is time consuming process and three days is pretty good I think. I fed it concrete patterns and pointed mistakes less than ten times.

Overall, it still lacks of 'creativity' but for concrete examples with 'right' patterns, saves a huge amount of time.

[manishsharan]

Hold on..

In my experience I found that ChatGPT writes awesome Clojure code. So much so that most of my clojure code in the last few months were written by clojure. sure it gets some stuff wrong but overall it knows more clojure functions than I do.

My prompts start by asking it questions about appropriate functions to use and then write the code itself. The prompts have to be a bit verbose and give it instructions to evaluate and think before generating output.

[Ldorigo]

As the other commenter said, prompting is everything, and most LLMs are sycophants and will try to do anything you tell them without pausing to tell you "why the hell are you trying to query an API with SQL? That's not what SQL is for". While it's possible to build stuff with llms with little to no technical knowledge, it's still very hit and miss.

With that said, the space is moving incredibly fast and the latest Claude/GPT-o1 are far ahead of anything that was available 3-6 months ago. Unfortunately Claude doesn't allow sharing publicly like ChatGPT, but here is a gist of Claude's answer for +- the same question your friend asked:

https://gist.github.com/ldorigo/1a243218e00d75dd2baaf0634640...

I'm on mobile so it wasn't handy to quickly paste an example API request/documentation for the LLM to follow; so there's a chance it might have hallucinated some if the API parameters - but if I included that; in my experience the code would work on first shot 90% of the time.

Regarding your second query, I'm too unfamiliar with clojure and the two solutions you mentioned to really understand what you were trying to achieve, but if you explain just a little bit more, I'm happy to record a screencast of me figuring it out with llms/genai tools from the ground up. What do you mean with "a container solution that allows for live coding"?

[oxidant]

Not OP, but I believe Clojure has a REPL that lets you run and edit code, persisting the changes from the REPL.

Off the top of my head you would want a Dockerfile with the version of Clojure you're working in, a mounted volume for sharing between host/container data. My guess is the two different things they mentioned are dependency sources.

LLMs require a -fu, similar to the Googlefu of old, to get what you want out of them.

[acuozzo]

Not OP, but I've been trying to use Copilot to help me determine how to force Android's ConnectivityService to select the network I've created as its default.

The network shows up as the default network in netd. It shows up as a network when I dumpsys connectivity, but I cannot get it to be what ConnectivityService considers default.

I'm open to changing the code within AOSP as this is a research project, but even then Copilot just has no idea what to do and it keeps recommending the same non-working "solutions" over and over again.

FWIW, I'm using Copilot Pro.

[mehh]

It’s a niche thing you’re trying to do, and likely not seen code that does that, thus it can’t help … it can’t actually think its way around it.

[acuozzo]

Has it not ingested the entire AOSP codebase? I was under the impression that OpenAI had trained GPT-* on just about everything available to the public.

It's not necessarily niche either. The codebase already does this all the time. I just can't figure out why it won't do it for me.

FWIW, at least 80% of my time in writing software for an R&D laboratory is devoted to solving problems like this.