[phil917]

I just canceled my Copilot subscription last night. I definitely never saw the productivity boost that I’ve seen so many claim.

I actually feel like the suggestions seemed to get worse during my month of using it for some reason.

Towards the end, it started suggesting these large blocks of code (another issue I had with the interface as well) which were very not relevant to what I was attempting to write.

All in all, I’m very underwhelmed from my first experience with “AI enhanced” coding.

[simonw]

Did you change the way you worked to help facilitate it at all?

I get great results from Copilot, but that's because I do a bunch of things to help it work for me.

One example: I'll often paste in a big chunk of text - the class definition for an ORM model for example - then use Copilot to write code that uses that class, then delete the code I pasted in again later.

Or I'll add a few lines of comments describing what I'm about to do, and let Copilot write that code for me.

Did you try the trick where you paste in a bunch of code and then start writing tests for it and Copilot spits out the test code for you?

A few more notes about how I've used Copilot here:

- https://til.simonwillison.net/gpt3/writing-test-with-copilot

- https://til.simonwillison.net/gpt3/reformatting-text-with-co...

I've also used Copilot to take educated guesses at things - like an inline mini-ChatGPT - with good results:

- https://til.simonwillison.net/gpt3/guessing-amazon-urls

As with so many other AI tools, Copilot is desperately lacking detailed documentation. It's not at all obvious how to get the most out of it.

[ignoramous]

I sometimes prompt Copilot with if statements and var declarations and it actually ends up suggesting missed corner cases and fixing bugs. Definitely, there's an art and form to using it.

I wonder if this quest for the perfect prompt is dumbing me down, though.

[SparkyMcUnicorn]

Using Copilot has taught me how to write better code comments. And the improved comment quality benefits everyone, not just the AI.

Even something as simple as adding a doc block for a method can boost the quality quite a bit.

I could live without it, but I'd also keep paying if the price increased.

[aerio]

At the current price, if it saves me a single hour in the course of a year it pays for itself. And it definitely does.

[leothelion_]

I agree. It has made me better at articulating what I want to do, what a function does, and something as granular as what a particular loop hopes to achieve

[lostmsu]

Just today Copilot saved me close to 1 hour of typing and replaced it essentially with ~1 minute of pressing Tab.

I was hooking IAudioClient COM class to capture and silence arbitrary app's audio, and as soon as I wrote signatures for its members Copilot was able to generate skeletal stub implementation with logging as well as hooking code totaling about 150 lines of Rust.

[DrakeDeaton]

This is an example of why codebases with strict typing & clear conventions will benefit the most from copilot.

[ColonelPhantom]

It would be even better if Copilot was able to make use of the strict typing to prevent generating code that doesn't typecheck in the first place, to be honest. Copilot as-is actually feels more useful for dynamic languages, because static languages get really far without any LLMs and simple static autocomplete gets you really far. So there's more ground to make up.

In addition, Copilot benefits from a large training set, which I guess is best for Python and JavaScript as two very popular languages. Strictly typed languages seem like they are generally less well represented, at least among public GitHub projects.

[gavinray]

To offer a counter opinion: while I can live without ChatGPT and cancelled my OpenAI plus membership, I cannot live without Copilot and would be willing to pay a lot to keep it.

There are a lot of tricks to optimizing output from Copilot ("Copilot-Driven Development"), I wrote a little about what I've discovered here:

https://gavinray97.github.io/blog/a-day-without-a-copilot#co...

[mns]

> Use explicit names, erring on the side of overly-verbose (e.g. ensureUserIsLoggedIn() instead of euil())

I don't care that much if I sound pretentious or too harsh, but how is this not common sense? Maybe the general standards in the industry are way lower than they used to be. This looks like when one of the consultants hired by our company and paid 10k+ per month for a ISO 27001 certification said: you need to make sure your password fields show a * instead of the actual character.

After using copilot for 2 months, I found it somehow useful for very generic repetitive cases, but for complex specific stuff it was very bad. I'd rather spend an hour writing code and understanding the problem I'm trying to solve than spending an hour understanding and adapting some generic code that partially does what it needs to do. And if there are people out there that fully trust it and then deploy code that they don't understand and can't explain themselves, then we're in for a lot of fun in the future.

[endisneigh]

Same. It’s funny reading comments on here. Even gpt4 I think is just ok. Maybe the people who see these huge increases are more junior?

[coolsunglasses]

I've been working as a programmer for almost 15 years and I work on fairly touchy (perf, reliability-wise) networked systems in a programming language with a good type system. I find Copilot to be very valuable even if I have to give it a little bump or edit what it produces sometimes.

[simonw]

I'm definitely not more junior, and I'm seeing massive productivity improvement from Copilot and GPT-4 - but I'm finding it takes a lot of expertise to get the best results, both in my ability as a programmer and in terms of knowing the best ways to use the AI tools.

Learning how to get the best results of them takes a great deal of experimentation.

[lolinder]

This has been my experience as well. I actually avoid recommending Copilot to juniors because I think using it requires a deeper understanding of what you're trying to do than is required to just write the code by hand. When you have that understanding it can be a huge time saver, but it's not something you can just pick up and magically ask to solve your problems for you.

[pjot]

I’m curious what your workflow is like switching between copilot and gpt-4. I typically have an open window for each, though this can feel more cumbersome than necessary at times.

[simonw]

I actually use the ChatGPT UI more than copilot - I write code with it while I'm out walking the dog, by describing what I want, then copy and paste it into my text editor on my laptop when I get home.

I'm increasingly using my LLM CLI tool for quick lookups, and sometimes to make changes to existing code too - see https://github.com/simonw/symbex/releases/tag/1.0

[SparkyMcUnicorn]

Copilot Chat is pretty great. Highly recommend signing up for the waitlist.

https://github.com/github-copilot/chat_waitlist_signup/join

[aerio]

>i usually have an open window for each

Do you not use the direct vscode integration? I don't think I'd use it otherwise. It's been hugely convenient with.

[Tronno]

My experience was very similar. Copilot became a sort of improved autocomplete, that would save copy/pasting repeated variable names into tests, logs, etc, but offered no other significant improvements. Its suggestions were frequently incorrect, often not obviously so.

[benabbottnz]

> that would save copy/pasting repeated variable names into tests, logs, etc

This is all it takes for me to get value out of my subscription. It's saving me time by anticipating my next test / block of code based on what I've previously written, and that saved time really adds up.

Sure it's not "Jesus take the wheel" level AI yet, but time is money, and so is my sanity.

[glaslong]

I find it’s really similar to having a team of junior devs under you. Lots of output, generally good for problems that are googleable, need heavy guidance if the problem is both new and niche, lots of subtle bugs.

The first two points in @gavinray’s methodology being unfortunately critical.

[berkes]

This month, for the third time, I evaluated copilot. It has vastly improved. So far that this time I'll probably pay for it.

I think it's biggest benefit isn't for me, though. Copilot is great at churning out predictable or repetitive stuff, much less in writing a abstractions over or around that repetitive or predictable stuff. E.g. it's great at writing yet another CreditcardPayment ActiveRecord model for Rails. But less so for an obscure business domain model used outside of a framework. And great at writing fifteen getters and setters, less for an abstract base class, DSL or macro that introduces all these getters and setters for me.

It's also bad at certain rewrites. A codebase was slowly rewritten from OldStyle (and old libs) to NewStyle. It stubbornly kept suggesting OldStyle snippets.

And last, I find Copilot has a far higher ROI in some languages and frameworks than others. E.g. dynamic languages (like Ruby) have very poor LSP and intellisense support compared to typed and static (like Rust). So the added benefit differs a lot based on where it's used, is my experience.

I guess esp. the latter is why I too am underwhelmed. But also why I'll keep using it, this time, for when I do the inevitable Rails, JavaScript or Panda's gigs. But less so for my weird eventsourced, hexagonal, Rust project.

[tomashubelbauer]

I also find Copilot to be rather underwhelming and half the time I find myself wishing it didn't go as overboard with the suggestions as it does. Sometimes it suggests exactly what I want to type and then adds a bunch of extra crap towards the end so it is the same amount of effort to type it manually or accept the suggestion and erase what I don't need.

Also its inability to match parens and braces/brackets is legendary. So annoying.

But! It is extremely helpful for filling in error messages. I don't think I wrote a `throw new ` line fully since I started using it. The quality and information density of my error messages has increased considerably.

Not sure if it is worth keeping for just that but it is a nice benefit.

[darkerside]

I found the same, and I was really hoping for a productivity boost. I really wonder how much this correlates to developer speed. I code using Vim and type probably 80-100wpm. Typing is usually not the bottleneck for me. Thinking is. Copilot is good at typing, but not good at thinking. IME. Occasionally it will surprise me with a valid contribution, which I then need to check just as carefully as if I were writing it myself.

[_ea1k]

TBH, using this as a fancy smart autocomplete doesn't seem like the best idea. I can't believe that I'm suggesting something that sounds like Clippy... but...

It really needs to be something where you can rattle off a list of requirements and have it build the code for you. The code context is necessary, but not sufficient.

[bgarbiak]

It’s coming up: https://github.com/features/preview/copilot-x

[nikeee]

I don't get the thing with automated PR and commit messages. They mostly describe _what_ was done and not why.

Also, if you enhance your available information with AI, why not just write no message at all (or a short one as always) and use the AI when _looking_ at the commits/PRs? The AI will certainly be better at this in a later stage because they themselves will get better and they will have more context due to being able to look at the commits that followed the PR/commit.

There is no point in generating the commit message when committing.

When auto-generated messages will become a commodity, there will be issues with it. For example, if the message doesn't fit with the actual commit contents (semantically or syntactically), I have to think hard whether this is an AI-message-generator-bug, or the author missed something or wether I am missing something. This is not cool and makes reading commit messages harder.

[__loam]

I'm excited to continue to be disappointed.

[wouldbecouldbe]

Yeah not worth it at the moment. Often it annoys me be overwriting legitimate suggestions from the editor.

Only reason I still have it is because the beta claims to have GPT-4 integration and that could prove to be a very powerful feature. But I still haven;t gotten an invite to install.

[62951413]

Are there other people who find that IntelliJ's autocomplete suggestions and shortcuts (including live templates&Co) already do 80% of what Copilot is trying?

[SkyPuncher]

Yea, but that 20% is ultra powerful. It handles the fuzzy things really well and it pattern matches even better.

If I write a comment describing what I’m doing, it will generate multiple lines of correct code

[roflyear]

People love to cargo-cult. It's wild. It was okay but I'm happier with ChatGPT for one-off things I don't know how to do "give me a sql query that does this obscure thing" etc..

[lolinder]

It's okay that it didn't fit well into your workflow, but accusing those who use it of cargo culting isn't constructive. Other people don't have the same workflow as you do, and as simonw noted above, some of us actually adjusted our workflow to get the most out of Copilot.

[roflyear]

Not everyone who used it cargo-culted but certainly some did.