As someone who has been using ChatGPT to code recently - code reviews are one of the worst ways to use ChatGPT. Sure, it'll catch some stuff every now and then, but realize that it's still just a language model and things like design or code reviews require real understanding.
Using ChatGPT to code review is very much like posting a picture of your code on StackOverflow or Reddit.
The bot is not meant to completely replace code review but to improve the code to be more production ready and catch any errors that might not be caught by the developers. In my experience more than 40% of code review comments relate to the specific changes and do not need a global context from the project.
It feels weird to see so many companies sending their codebases to an ai company in exchange for feedback on their practices. People used to be more secretive about their tech years ago
I thought of this recently as well. SaaS and the modern "cloud" services really have radically changed the game in this regard. Even as recently as 10 years ago, the idea that you would willingly exfiltrate your most precious data to some company, where they would then essentially "own" it (in the sense that your data is in their database, and you don't have direct database access to it. You can only access it through what they provide an API for), would have been really difficult to fathom. We had nearly everything "on prem."
In many ways I think the change is good. It's certainly more convenient. But the radical shift in mindset is something I deeply regret. The SaaS-ification of everything, where the user has zero control over anything (even installation of updates) has had a majorly negative impact on my life. I loathe the fact that every few days, something important to my workflow/routine is probably going to break in some way from a CI/CD deploy that contained a bug, and I'll be stuck until they patch it. And of course there are times when you are mid-something, and the app starts 500ing and you click refresh, and congratulations! You're the first user of the massive UI overhaul that you didn't want or need, so now you get to re-learn how to use the app!
As a dev I love SaaS. It's so wonderfully convenient for me! As a user, I hate it.
I think it's because we're more used to sending our codebases to Github, Vercel, and other places?
Also, it seems that for most web sites and apps, it's more about the craft and speed of execution than the code itself, so I've stopped being paranoid about this kind of stuff. I don't think deep tech engineering co's should use these things though.
My employer has banned Copilot (and presumably this app as well), for what it's worth. I'm guessing most large employers will end up doing the same if they're at all paranoid about their code, although by that time Copilot may offer an on-site version, or at least a version that can be deployed to an enterprise's cloud instance.
Well, I think there's value to both points - how much of the maps code would you need to be able to get something working, and how much work would it take to implement all the custom internal dependencies that would probably be missing?
I'm sure there are very complex valuable pieces of code within the maps codebase (or any other), but it would be a fairly massive task finding those pieces, extracting them, and getting them to work properly in a different codebase...
The code is already sent to github and hosting provider. We just have to trust that multi billion dollar companies have more to loose by using our code. And most of the codes that people work with is not rocket science, so unless openai directly sells the code to the competitor, the potential losses is likely not big.
Given the recent EU cybersecurity directive requiring code to be audited it's not so weird. However they will partly break this model with their AI regulation directive that is in the works.
It has to be an online service until they get the models more refined and hardware advanced enough that you can run these locally. If you can't run the model locally then it might as well be a service you tie into at the service (eg. github) level. That is, IMO, services are most useful with other services.
Hmm. Not seeing your argument. The company has an API key. Employees query the local service which passes it up to the pay service. What does user executed vs. github commit triggered have to do with anything? What am I missing?
The bot does not work like a linter or testing framework, these are suggestions that are subject to discussions, and the pull request is a great place to get reviews from the team
Agreed. I was expecting the AI to uncover and offer more nuanced suggestions. The examples on the site were highly local issues, with most of them being one-liners our linters are already catching. I was excited to see what the AI can do when it has the context of the entire repo, but I was not wowed.
The examples on the site are meant to showcase the features and chatGPT capabilities being where it shines. Depending on the project's context the bot will make great reviews, chatGPT already does a good job at reviewing code if given enough context and that's what the bot is trying to achieve. In the future the bot can be onboarded to a codebase and the overall project for better review and possible code contribution.
Rather than a natural language code review, having a machine learning model that can spot code that "looks wrong" might be really valuable, especially if it was able to get a "good enough" SNR. Static checkers today can be limited in their applicability. But if you trained a model on { block of code } => "defect class <X>" it could be really powerful. Perhaps seeing examples of applied fixes might be a good way to convince someone whether or not there was really a bug there.
Maybe it could even collaborate with static checkers as a quick screen -- seeing as how some of static checkers today are a compromise between the execution time of the checker and the comprehensiveness of the check.
Great idea, but you stopped too early. By being a bit bolder, you could have solved the problem of open-source funding.
The idea: Train your bot based on the code reviews of prominent open-source contributors. Next, let the user choose the bot they want a review from (e.g. "Dan Abramov Bot", "Linus Torvalds Bot", etc.). Each time the bot does a code review, the open-source contributor gets paid for that. This also solves the legal problem of the licence of training data.
It is, but Github's terms do not allow your code to be shared with others. Although ChatGPT represents that it does not retain information provided in conversations, it does “learn” from every conversation. There is currently little reassurance as to how those "learnings" are leveraged outside of your own usage.
Chatgpt doesn't "learn from every conversation", they trained it once and the output isn't back propagated through the network. Keep in mind that Githubs terms are the same ones they willingly ignored to make copilotm
It's year 0 of this being useful. I would imagine sooner rather than later we'll start seeing it licensed and run internally/easily spun up in your organization's AWS or Azure or what have you.
How does this generally work? I've played with openai's API and I can send it questions and get answers. Is this app just parsing new additions to commits and passing in the whole code block as a prompt, with an additional line saying "please review this code"?
Additionally, how is this app able to handle so many potential requests to openai? Are they already paying for a bunch of tokens in anticipation? Or do you pay for how many tokens used per month? (The openai pricing is a little confusing since there's so much you can do.)
The goal is to provide more context to the model and use the changes with the title and description of the PR to get a review, while asking for improvements and suggestions.
This is harder than it seems. The way we are doing with Robin (Reviewpad's AI Reviewer) is by feeding it context about the PR and results from our static analysers and allowing the developer to prompt directly through the PR comment. For example, https://github.com/marcelosousa/robin-preview/pull/2
Most PR reviews contain comments and suggestion that could easily be automated with ChatGPT, and reduce the time it takes for a PR to be merged, which makes the team more productive.
Surely you would be very comfortable to use this to 'code review' and send all your code, API keys, ENV files and secret urls to a random online service?
I don't think so.
There is a reason why large companies ban the use of unaudited third-party services like this, especially when they aren't compliant with security standards.
The bot is built to act as an extra team member dedicated to code review, these suggestions could then be discussed on the thread within the team and new commits can then be made
Edit: downvote? So, I received a couple PRs, it commented way too much. Sometimes a comment would say "Everything looks good". Needs better "calibrating".
The Github bot is built to review pull requests by analyzing the changes made and providing detailed comments on the changes, by suggesting improvements or highlighting vulnerabilities or bugs.
I don't use them myself but it depends entirely on what language(s) you're using and the rest of your CI flow. Do a quick search and use the one you're using locally. It will most likely use the same config.
But since you're solo I'd recommend discipline before pushing instead of adding another layer on your CI, unless the point is to learn of course then go ahead.
Looking at the product it seems very solid. I don't believe in competition in this case but rather that we're trying to help more and more devs.
If they play their cards right, such as PP, they can add modular verifications of security issues, code change recommendations/optimizations, plagiarism detection, etc.
Using ChatGPT to code review is very much like posting a picture of your code on StackOverflow or Reddit.