[jventura]

CS Professor here: just yesterday I did the discussion of a course projects' (Parallel Computing), and one of the three groups that I did yesterday have clearly gone the ChatGPT way. They couldn't even understand the choices the LLM made regarding the architecture, etc. The way to "catch" these students is similar to what we did in the past when students copied from other students which is "to give them rope to hang" - ask for clarifications until they follow unintended paths that lead nowhere.

To fellow professors, when you're suspicious my suggestion is to appeal to their honesty (like "let's be honest, how much of this code is yours, and how much is ChatGPT's?") and offer some empathy and understanding (like understanding they may had multiple deadlines in the same week, etc.). Nevertheless, don't miss the chance to give them the lesson on how is the correct way of doing things. The way to catch these students is to find the same signs of yesteryear copying from other students (which in essence is what copying from an LLM is, although the number has increased because they found us professors unprepared for the volume).

The other two groups also used LLM but in a high-level and architectural way. They were clearly responsible for the code (even if they didn't wrote it 100% manually) and could explain their reasoning and strategies used to solve the problems.

Me and my colleagues still have a lot of projects to review, and I asked them to keep the score of the number of projects like these, but so far, the score is 1 in 3 (33%).

[rahimnathwani]

So all groups outsourced their thinking to an LLM. Will they have learned anything they can apply to different types of projects in future?

[calepayson]

In your opinion where is the line on where LLMs are useful/harmful to learning?

In my school projects I've found them super useful for working with libraries. In the past it felt like the theory I learned was pretty low impact. 80% of my time was spent learning the quirks of a library. Now it feels like I can take theory and iterate over a ton of different solutions without having to worry about learning whatever library the professor requires. Basically I'm I feel like it lets me spend more time learning the thing I want to learn rather than all the busywork around it. Would be curious to hear your thoughts. Thanks!

[numeri]

I would just warn that you may not be able to recognize what is worth learning at your stage.

Intuition for library design and the architecture of software packages/external APIs is something you can only learn by doing.

[rahimnathwani]

If you don't want to improve your ability to work with new libraries, then that's your decision.

[smcg]

I thought this was one of the most important skills to have early in my career.

[rahimnathwani]

Yup!

[jventura]

Not in the same level, see my other comments..

[rahimnathwani]

I think we're using two standards without naming the difference.

- "Responsible for the code", i.e. understands it, can explain it, can modify it on request, is the right bar for a working engineer using an LLM to go faster. But it's the wrong bar for a student, because in a course the deliverable isn't the point, the practice is.

I presume that as course like 'Parallel Computing' exists to build the habits of reasoning about decomposition, contention, race conditions, strategy tradeoffs. Those habits come from doing the reps, the same way you don't learn to write by reading good prose. Your two "successful" groups outsourced exactly the reps the course is (or should be) there to build.

Being able to follow and tweak a solution proves comprehension, but comprehension is a much lower bar than producing the solution.

Your students may understand this particular solution well enough to explain it. But did they learn enough that they could tackle a different problem with a different shape, without using an LLM?

[jventura]

I'm not a native EN speaker (I'm Portuguese), therefore a lot is lost in my attempt to find the right words. I'm writing these words without any use of AI or even a sentence-checker (only the spell-checker in Firefox).

Of course that the "succesfull" groups implemented their solutions, tweaked them and, tried their best to escape from race conditions and were able to talk about the performance/memory/etc. tradeoffs. We have a grid to grade for those and other items. They did not outsource all of their thinking to an LLM.. That is what I mean as taking responsability for their code, maybe not the correct word, but the one that came to my head when I wrote it..

[rahimnathwani]

Thanks. I'm still curious whether they could apply what they learned to a new problem, without using an LLM.

[kuschku]

> The other two groups also used LLM but in a high-level and architectural way.

Sounds more like the score is 3/3 (100%)

Would you have accepted them cooy-pasting code from libraries together to build their project? If not, why is using LLM generated code different?

[newswangerd]

Using AI to inform architecture doesn’t seem so different from googling architecture in this case. Architectural patterns are mostly well understood and well documented these days and are something that you could piece together via Google search pre AI. The thing that AI brings to the table that wasn’t google able in the past is code generation. Previously you had to understand the architecture patterns to implement them yourself, but now the AI can just do it for you.

[jventura]

> Would you have accepted them cooy-pasting code from libraries together to build their project?

Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc.

In this case, the students had to invent multiple strategies to solve a specific problem. The "successful" groups did a mix of generated and hand-crafted code (don't know percentages), implemented different strategies and knew their plus and minuses, could change the code in a timely manner to accommodate some of my requests, etc. The "unsuccessful" group couldn't do any of that.

I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students.

But in reality I've been slowly transitioning from group projects (for a subset of the grade) to "practical tests", where they must implement a significant subset of a larger project in a 2h class. Still experimenting though.

[cushychicken]

> Yes, if they are "responsible" for the code delivered, where responsible means they understand the code, the architecture, the decisions made, etc.

This is a good principle to maintain, I think.

I'm not a professor, but I manage a team of about a dozen people. The maxim I have is: "You're responsible for anything that hits git."

Don't care if the LLM generated it, or the LLM told you if it's a good idea. If you commit it, you are endorsing it as a good idea - so you're the one I'm going to ask about it. I see the same principle at work in your pedagogy.

> I'm not anti-AI (and really, what could I do if I were?) since I use it myself, I'm just anti-slop, especially from my students.

This hits. Especially this part:

> and really, what could I do if I were?

My completely unsolicited opinion: you're doing a responsible thing by teaching these students how to use AI as a reference, and keeping them honest about not using it as a substitute for their own critical thinking.

[lokar]

I'd say most of my CS classes (in the 90s) had us write out code by hand, on paper, during in-class exams.

It was fine.

[dysoco]

Usually you get both kind of assignments, short exercises in paper without any help whatsoever and more open, larger take-home assignments where you can usually use external help.

[archagon]

> and really, what could I do if I were?

I’m not sure what you mean. Have a no-AI policy and fail those who ignore it. If students lie, discipline them appropriately for cheating.

[jventura]

How could I realistically enforce it? These are projects students do at home..

[archagon]

Same as other instances of cheating? A lot of students copy from each other and don't get caught. If anything, LLM-isms seem a lot more obvious, especially if the students are trying to avoid thinking or doing work.

Anyway, I'm not a CS professor, so I'm just venting.

[yubblegum]

He explains why in his comment. Read it again, carefully. Or ask an LLM for an "explainer"..

[ygouzerh]

How do you see impact of using LLM on the teaching content? (genuine curiosity)

In the industry, almost everyone here is doing the exact same thing, outsourcing more and more of their thinking to LLM: so even if students learned how to manually write code, they will probably loose it later on (happening to us already, mostly can be seen when people are working on new solutions/frameworks... they now have the same issue than students).

[LovelyButterfly]

As a professor, what's you opinion regard the Socratic method with LLMs? Would that be preferable over simply "give me the answer" prompt?

[jventura]

Everything that provides students with a workflow to think and to try to find solutions to a problem is much better than giving the answer directly! Unfortunately there will always be students that prefer to take the shortcut..

How could we "force" the students to use an LLM that confronted their doubts with more questions? We could tell them to start each chat with a specific prompt (to use the socratic method, etc), but they could eventually jail-break it..

But nevertheless, I like your idea! This is something that a document highlighting methodologies for students on how to use LLMs effectively could/should contain..

[LovelyButterfly]

Yeah that makes sense. I'm a Master's student myself, ironically on Artificial Intelligence. Often I feel overwhelmed by the amount of papers and the mysterious math logic I have to understand when presented with a model.

Accidentally in Google Colab, I've found that there is a "Learn mode" and somehow I managed to access the instructions for it which indeed instructed to ask questions, then I researched a bit more and learned about the socratic method.

So far I'm doing the combo of Zettelkasten + Socratic on Claude on papers + slides or video classes. I'm not sure if it's placebo or not, but I feel much more confident after iterating with each paper or topic this way. Asking the answer to the LLM never felt like I was studying. I read, but I never understood, I was never challenged. Perhaps what we need is the Unis around the globe to come up with a standard, a guideline so that the students can maximise the advantages of having the LLM when a professor or assistant is not available.

[e-topy]

I'm curious on your opinion, as a professor, how should universities (not individual professors) adapt to LLMs?

As an undergrad, I hope schools move to educating students to use LLMs in a more responsible way. You can't put the genie back in the bottle, and resisting progress is futile, might as well use the tool we now have to help students learn even faster and better (e.g., making feedback instant and not answers, helping digest or split up material, checking answers).

I know opinions about AI at (not only at) my faculty are very mixed, but I think the answer is going to be in the rational mean, just like how technorealism reacted to the internet[0].

In our last program board sitting, some teachers said that they think programming as a job will be completely irrelevant in two years, while other pushed for more adoption. And meanwhile I know of some students that are basically only passing because of LLMs, and it's bad, like "leaving claude output in markdown files and finished source code on the faculty server in /tmp because opencode did so" bad. And our first year classes completely prohibit even sharing tests or talking about the solutions, which in my opinion a) makes people extremely asocial and atomized b) doesn't prepare students for real life c) promotes dishonesty.

Still, I think our university's thinking is in a stalemate, not wanting pure AI output and useless students, while also wanting to move with the times, and I doubt it's the only one.

[0]: https://web.archive.org/web/20081009111415/https://artefaktu... (absolutely amazing read, recommend it)

[bo1024]

What is the policy and guidance you gave the students regarding LLM use?

[jventura]

This is all so new, and caught us completely unprepared that there's no official university-level policies. Most of us are still navigating the waters and seeing what works and what doesn't work anymore.

I have colleagues that are teaching for more than 30 years, few years away from retirement, who suddenly have been confronted with a new way of doing things. Those are the ones that are still insisting on doing practical projects, etc. I've only been doing this for 20 years, and I'm quite lazy (worked previously as software engineer), so I've moved to those practical tests. I guess that there should probably exist a class or workshop to teach these students how to use LLMs effectively, but as I said, this technology and its implications is quite new.

Personally, what I did was to give them the "lecture" in the line of that they do not understand what the machine has generated, that is not the way a true engineer does, try to do some parallel with things like an LLM designing a bridge and civil engineers building that bridge, and a fatal flaw collapsing the all thing, etc.

In other words, we do not have a formal system in place, it's all talking and convincing them. Obviously it's a big enough problem that should deserve more investment in solutions, but we are all overwhelmed by other tasks. Maybe LLM studios should be held responsible for all these "disruptions" and provide solutions to problems they created! :)

[estetlinus]

I did a course of parallell computing 6 years ago. I still dont understand anything about it. Probably would if I had used an LLM, since I would have used it as a private tutor.