[kingrolo]

It feels to me as though LLMs should (eventually?) really shine at these kinds of tasks where the intent is already defined in code of some sort and the challenge of the task is lots of detailed legwork that humans find hard, more because it's time consuming and not interesting so hard to focus on, rather than because it's technically challenging.

So swapping languages, yeah maybe, but I expect of more practical use would be the situation where you inherit a legacy codebase in an ancient version of a language or framework that hasn't been loved in a long time. I saw this so many times when doing dev team for hire work.

Obviously you'd want to do boat loads of testing and there may well be manual work left to do afterwards, but I think it would be the kind of manual work that felt like you were polishing something new and clean and beautiful rather than trying to apply bits of sticky tape to something unmaintainable.

I also wonder about eventually being able to say to an LLM "take this codebase and make it look like my code", or maybe one of your favourite open source developer's code. Maybe everyone could end up with their own code style vector attached to their github profile describing their style. You could find devs with styles close to yours to work on your team, or maybe find devs with styles different to yours so you could go and argue about tabs vs spaces or something.

[tinco]

That's it precisely. I'm working on the exact same thing GPT-migrate is doing, but I'm approaching it from the other direction first. My project is trying to generate a test suite that aims to cover the full original functionality (bug for bug as they say). That way a tool like GPT-migrate has a much better chance of generating the translation without errors and whoever uses it can have more confidence in that the output will be correct.

I'm a bit intimidated that Josh came so close in just a week of work but it's also inspiring confidence that this is the right track and it's actually going to work when all the puzzle pieces fall into place.

edit: damn, this project actually creates rudimentary tests as well. It's such a lean approach, makes me feel like I'm still coding in 2022 when Josh is firmly in 2023.

[arach]

I agree! This is a pretty elegant approach and while like you I haven't fully internalized the 2023 way of building AI native product, I'm inspired and increasingly confident in how much can be accomplished in a lean way.

[dzhiurgis]

How do you fit all the codebase tho? GPT-4 is limited to 32k tokens which is like 500 lines?

[tinco]

I scan through files in chunks, with particular questions for the LLM, building up a context that is eventually used to write the test.

[guy98238710]

> take this codebase and make it look like my code

That's exactly what LLMs don't do. In my experience, there is no way to convince ChatGPT (even v4) to follow any conventions or obey any rules. It might try a bit, but it always ends up writing everything its own way, usually as verbosely as possible.

[throwaway675309]

You are 100% false on this one, especially with GPT-4. I can take entire vanilla JS functions, and ask the language model to rewrite them using typescript, snake case naming conventions, and show it a block of other code that I've written to adhere to the same structure including jsdocs and it nails it nearly every time.

[guy98238710]

In your example, you are asking GPT-4 to rewrite code from one public style to another public style. I was saying that LLMs cannot reflect your personal style, because they know nothing about it. Even when instructed at length and provided with short examples (that fit in the context window), they always gravitate to the public average.

[fswd]

Then you're doing something wrong because it's exactly what it does for me.

> Given this schema ... some create statement for a table > Use this as a template ... some unrelated function > I want you to implement this ... some pseudo code

[arach]

> So swapping languages, yeah maybe, but I expect of more practical use would be the situation where you inherit a legacy codebase in an ancient version of a language or framework that hasn't been loved in a long time. I saw this so many times when doing dev team for hire work.

I'm exploring this problem space - this is a wide spread pain point across almost all companies that are older than 3 years old. I've seen this at tech startups as well as very large companies and anything in between. Dropbox is arguably a top tier engineering organization that probably manages tech debt as responsibly as can be expected and they still had to make major investments to move their codebase forward from various eras of web tech 1) https://dropbox.tech/frontend/the-great-coffeescript-to-type... 2) https://dropbox.tech/frontend/edison-webserver-a-faster-more... Everyone else is much worse off so the investment required to move forward is usually immense. This leads to full rewrites. Which is nice but error prone and sometimes entails huge opportunity costs

> Obviously you'd want to do boat loads of testing and there may well be manual work left to do afterwards, but I think it would be the kind of manual work that felt like you were polishing something new and clean and beautiful rather than trying to apply bits of sticky tape to something unmaintainable.

Agreed. In my opinion, now's a great time to get started with a semi automated approach like this while betting that the program synthesis and code generation capabilities will rapidly improve over the next few years. Larger context windows, solutions for hallucinations / reliability and better training data will help reduce the manual labor required.

> I also wonder about eventually being able to say to an LLM "take this codebase and make it look like my code", or maybe one of your favourite open source developer's code. Maybe everyone could end up with their own code style vector attached to their github profile describing their style. You could find devs with styles close to yours to work on your team, or maybe find devs with styles different to yours so you could go and argue about tabs vs spaces or something.

I've been thinking that personal style / training / fine tuning could become somewhat of an asset. "You are a principal software engineer at Google with particular expertise migrating codebases from {sourcelang} to {targetlang}." works fine but imagining a much richer portable input would possibly be quite valuable.

[tomComb]

I’ve encountered lots of companies that keep paying license fees to oracle for DBs that could just as well be Postgres but for the work of updating and testing the code…