[dheera]

Except for startups when commit messages are more like "asdf", "aoeu", "quick fix", or "demo" because some investor barged in and demanded a demo before they would wire funds.

If ChatGPT could change that to something like "disable current limits" or "disable safety checks" or whatever that might be marginally better.

[TeMPOraL]

A ChatGPT-generated message, pasted without editing, is purely functional transformation of the code, adding zero information. This means I could just as well run it on your diff myself, if I thought it would be useful. More than that, when I do it a year or two after you made your commit, the then-current ChatGPT will likely do a much better job at summarizing the change. So perhaps it's best to leave auto-summarization to (interested) readers, and write a commit message with some actual information instead.

[bee_rider]

If the programmer checks and OKs the message, then it still conveys information that you don’t have a year or two down the line. ChatGPT is guessing what their intent was, it could guess wrong, but if it guesses right and they validate that guess, then their intent has been summarized.

[TeMPOraL]

A programmer checking and OKing the message only tells you they didn't think it's bad enough to expend effort correcting it.

ChatGPT can't correctly guess what the author's intent was, because that information is not contained in the code (exception: if the code includes a comment explaining the intent).

[dheera]

> purely functional transformation of the code, adding zero information

I mean, a human brain is arguably also a purely functional transformation adding zero information.

[TeMPOraL]

It's not. The diff, which is the sole input to GPT-3 here, does not carry the causal context - that is, why the change was made. Nor does it carry the broader understanding necessary to summarize the what succinctly and accurately - things like high-level design terms that mostly exist on diagrams or in issue trackers, but not in the code itself. By adding those details in a commit message, the author can add extra information.

And yes, technically they could do it in comments, which would allow GPT-3 to process it. Except, I think it's highly unlikely for a coder using ChatGPT to write commit messages for them to actually author useful comments on their own. If they write any at all, they'll probably use ChatGPT (or Copilot) for that too.

[welshwelsh]

We could give the model access to JIRA, confluence, meeting transcripts etc. so that it has all the same contextual information as the developers.

[TeMPOraL]

You know what? You do that. I'm gonna start a SaaS business providing AI generators for code, commit messages and documentation. Models that take into account all the context to deliver more accurate, better results. All I need is a live feed from your JIRA, Confluence, Teams/Slack, Zoom, Exchange, SharePoint or whatever alternatives you use. This will guarantee you the biggest bang for your buck!

Elsewhere in this thread someone was wondering if sending a change diff to a fly-by-night third party SaaS could be leaking company IP. They were thinking too small.

But cynicism aside - giving the model access to all that contextual info would definitely increase the chances it would generate useful commit summaries. It would also increase chances it would generate much more convincing bullshit, full of just the right phrases to confuse your programmers, PMs and architects alike.

[hgsgm]

Even better: let GPT write the JIRA tickets and Confluence pages, and speak for me in meetings.

[dheera]

I guess they could feed all of the team's Slack history into GPT as well and it would then have the context?

[funcDropShadow]

It could even continue filling Slack.

[ahartmetz]

Though in this case, it has more and better quality context (i.e. input), like requirements or anything else that isn't in the model's training set.

[eternityforest]

asdf is still better than having lies sprinkled in randomly.

Maybe prefixing them all with gpt: would help

[ketralnis]

> asdf is still better than having lies sprinkled in randomly.

I think this is the core of my argument, yeah. If a _reader_ needs better than they can run GPT themselves. But the _writer_ using it is worse than useless, it's actively harmful.

[llukas]

This indicates commit quality. Why lose this info? If you have only time to put "aoeu" into commit message would you have time to correct ChatGPT output? ;)

[javier2]

this is just normal every day commit messages in most startups I've seen

[TeMPOraL]

Startups tend to be a "do a rush job so the business won't die, worry about fixing it later" kind of a deal. I don't envy those working on original codebase after the startup is no longer racing its own runway.

I've experienced messages barely better than this in products that were under no immediate threat, and let me tell you this: having to figure out why some changes were made, three years earlier, in a bunch of badly-described commits whose author already left for another job, with no old documentation hinting at the purpose of the changes - this is one of the few things in this job that make me want to shout someone's ear off.

[javier2]

yeah, it happens a lot in established projects as well. especially projects with very small teams.

[teddyh]

Haaaaaaaaands

— https://xkcd.com/1296/