[_8091149529]

I spent a few years at Caltech in an academic role (not as a student). I never understood the rationale behind the unsupervised take-home exams. Could you elaborate what is it that makes it a 'success', in your opinion?

The shortcomings that seem obvious to me are: - Penalizes honesty - Implied notion that Caltech students are "honourable" and honest. How is this achieved in practice?

[beisner]

The idea is that you create questions that are ungoogleable. Most of my upper-level CS courses had take-homes that were open-book, open-internet, and the professor basically said “good luck finding the answer to this problem online; you will not”.

This is harder for professors to do, and maybe fails in intro courses, but generally is quite effective.

[BalinKing]

One of my Ph 1b (freshman special relativity and electrostatics) quizzes guided you through "discovering" a new method of solving that type of problem. I don't remember the specific details, but it's one of the rare tests I've taken that I actually enjoyed!

Although most homeworks and exams aren't nearly that cool IME, they're virtually all about synthesis and fundamental understanding rather than rote memorization. This "first principles" approach to most everything is one of my favorite things about Caltech.

[tzs]

When Carl Sagan was writing "Contact" he attended a conference that was also attended by his friend Kip Thorne. Sagan knew that one of Thorne's pet peeves was science fiction that just hand-waved away the physics of FTL travel.

Sagan told Thorne he was writing a science fiction novel, needed FTL travel, and asked Thorne if he could suggest something that would be reasonable. Thorne agreed to look into it.

After the conference, Thorne spent a while working on it and came up with a wormhole approach and worked out the physics of it.

In addition to giving it to Sagan, Thorne also put it on the Ph 236 (General Relativity) final exam. He didn't tell the students on the exam that it might imply FTL travel. He just set up the conditions and had them work out the physics. Most of his students succeeded in that, but he was a little disappointed that none of them happened to notice that it implied FTL travel.

(I got the above from an unpublished book Thorne was working on in the early '80s. It was a collection of biographies of and interviews with physicists, astronomers, cosmologists, etc., written and conducted by Thorne. He had the draft chapters and the raw interview transcripts in a world readable directory on the physic's departments VAX, where they were widely read by the rest of us with accounts on that machine).

[aschatten]

Do you know if it's possible to get ahold of a copy?

[tzs]

As far as I know, he never finished it, so unless he still has and would give out a copy it is probably unobtainable.

[908B64B197]

This.

Else you might as well call the degree "rote" because that's what you measure.

> maybe fails in intro courses

If someone cheats their way through intro courses they are up for a bad surprise since the upper level courses assumes a mastery of the intro classes!

[_8091149529]

Interesting, thanks for the replies.

I'd still point out that the system is not robust against: - Working in groups. (Bad if tests are supposed to assess individual performance.) - Asking outsiders for help.

Also, your downplaying of "rote" learning feels misguided, no matter how advanced/abstract/high-level the domain in question is. Cue the Bruce Lee quote about 10,000 kicks..

[908B64B197]

> Working in groups.

Students hate carrying dead weight so to speak. Sure they might be tempted to swap favors (A helps B for a certain subject and the reverse is true for an other one). If the exam is properly constructed you might be able to detect plagiarism however.

Rote is absolutely necessary for a well rounded education, but I really feel the need to overcorrect in the opposite direction because testing for rote is the lazy approach. Over time a lot of testing has the tendency to shift to simply measuring rote.

[turbinerneiter]

Just imagine a world were solving problems in groups and asking for outside help is bad.

We are so used to exams being about doing stuff that literally nobody in real life would ever even consider.

[_8091149529]

I was aware of the implications, hence the disclaimer "(Bad if ...)"

However, I'd assume that doing either during a test would be against the honor code.

[maerF0x0]

I had a professor that intentionally asked a question which wikipedia has the wrong answer... Called out and shamed everyone who used wikipedia as a source.

[dghughes]

That reminds me of when I took and distance learning electronics technician course. You could Google (search) all you wanted to but seeing answers didn't help in knowing how to solve the equations.

[jackson1442]

I think the most important part of "proctoring" exams should be that it is individual work. Let's face it- you will almost always have access to the internet. The only thing that should matter is that the assigned student can solve the problem _in some manner._ This may include the internet, notes, or really anything.

The real problem is people paying others to take exams, or simply copying answers from others. I work as an independent tutor, and I get contacted far too much by people looking to have me take their exams for them.

[asdff]

That was certainly a problem in person too. At my undergrad, for larger exams you would just pass your ID down the row to the person in the aisle and a TA would collect them all then check everyone in. Wouldn't be hard to just hand your ringer your ID to pass on up. Not unusual to look nothing like your freshman year ID photo in person.

[impendia]

I was an undergrad at Rice where we had a very strong honor code. Unsupervised exams were common, and to the best of my knowledge cheating was very rare. If you were accused of cheating then you went before the Honor Council, which was run entirely by students.

I considered this an enormous success. It created an atmosphere where we were treated with respect, and where we were expected to treat others in the same way. If someone did cheat, then they would certainly be too ashamed to admit to it. It was a community where I was proud to spend four years.

Yes, I would say that Rice students are honorable -- without the scare quotes.

How is this achieved in practice? Wish I knew. This is the sort of thing that every university, and every organization, wants to achieve. Trust is very difficult to build if it's not already there; and it's much easier to erode.

[RonanTheGrey]

> How is this achieved in practice?

You were accountable to each other, not to an authority. Authority gets you enough to do the bare minimum to not get fired or not get caught.

Being accountable to each other feels very very different. It's tough to explain in an HN post but anyone who has felt the difference knows what I meant. You care what your peers think. You care about that authority only enough for them to leave you alone.

Never let it be said that shame isn't an effective tool.

[smogcutter]

> Never let it be said that shame isn't an effective tool.

This is true, but at the same time I think shame as a motivator is often really unhealthy. People can do all kinds of terrible things to avoid shame: at the extremes, witness things like family annihilators or so-called “honor killings”.

I’ve been reading a book about Midway lately so this is on the brain, but a culture of shame is part of the complex that prevented the Japanese from reeling in insubordinate but “patriotic” junior officers, and from realistically assessing their own and their opponents’ capabilities. Meanwhile the modern process for evaluating air accidents (constantly applauded on HN) explicitly removes the question of blame and shame.

So, I’m not really saying anything about Rice university. I’m sure they don’t have to worry about honor killings on campus. This has just been on my mind lately. Just saying that without social release valves for shame you can get some really wild consequences.

PS for anyone interested in Midway, read Shattered Sword for everything you ever wanted to know about early-war Japanese flight deck operations.

[_8091149529]

> How is this achieved in practice? Wish I knew.

I think my take-home from this discussion is that the honor code can be made to work in the right circumstances that exist, at least, at Caltech, Rice, etc.

At the same time, I believe it is impossible to induce the requisite "cohesion" in other contexts such as 100% remote learning or high-stakes mass testing (entrance exams etc.), even if the student body stayed the same.

[908B64B197]

> high-stakes mass testing (entrance exams etc.)

That's because it's done wrong.

There are countries where they simply sort by descending scores on whatever standard test they came up with to admit. As if the test was perfect and results could be compared at an infinite decimal place (hint, stats and physics disagree with that!).

So the test basically becomes a measure of how someone is good at taking the test and you start to see min-maxing behaviors. It's a contest to see who can pour the most time into maximizing its results. The incentives for cheating are simply so high. The downside of cheating is that you don't really learn, but since the only skill these tests teach you is taking the tests, you are really robbing yourself of a skill that becomes useless 3 minutes after the test.

My advice is to throw these tests in the trash.

[progman32]

Fellow Rice graduate here. Looking back, being treated with trust and respect - after a decade of the opposite treatment at the hands of middle and high school - was a real turning point for me. We were trusted to make our own choices, and expected to reciprocate. In my experience, this leads to fewer cases of anti-social behavior. Which is more trustworthy - a company that simply expects tasks to be completed well and on-time, vs. a company that takes screenshots of everyone's machine and tracks bathroom breaks? I find it's overwhelmingly the former. Granted, the causal arrow may go either way.

The council does have teeth. My friend was sent in and found guilty of cheating on a test. They got kicked out in short order. The straight-A student they'd allegedly copied the answers from got a few months' suspension. I never got an explanation as to why the sentence differed.

No system is perfect.

A professor questioned me as a witness at the "student council" hearing. I expected a student to do the job. The questions were all worded to make my testimony skew towards a guilty verdict, calling my own integrity into question. This is expected if there was a mechanism for an organized defense, but I was never questioned by the "defense attorney". I left the questioning disturbed and frankly afraid for the safety of my own academic record despite not having done anything wrong. Guilt by association. It felt like a witch hunt and a front for implicit faculty power. My friend still denies wrongdoing and had to rebuild their degree from scratch at another institution. They are still proud of the time spent at Rice, even though they feel their degree was taken away unfairly. The only evidence I heard against my friend was that they had made the same mathematical error as the other implicated party.

My first take-away from the situation was that organizations based on trust work well, but it is critical to have robust mechanisms in place for dealing with moments where that trust gets called into question. It is human nature to take trust away faster than it is given, even if it turns out there was no wrongdoing.

My second take-away is that organizations based on trust tend to punish violations disproportionately, especially if the violation reflects badly upon the group.

My third take-away (only realized years after the fact) was that trust-based systems tend to create a tyranny of implicit rules that tend to exclude newcomers (like students) unless specifically addressed.

I'm still very much in favor of a high-trust environment, but it is far from a cure-all. I find it's a prerequisite to a robust system, but must be paired with counterbalances.

For my beef with the Student Council, Rice was an overwhelmingly positive and nurturing experience. If I ever decide to get another degree, that's where I'll go, and I always recommend it to prospective students with an independent streak.

[mmmBacon]

I had these at my university (not Caltech). These take home exams are much, much harder than anything they can give you in a timed in-person exam. It’s not even close.

This is simply because the set of problems a person can solve in 3hrs limits the scope of the questions that could be asked. In fact, my friends and I would make lists of things we were sure wouldn’t be on the exam due to time constraints.

I have to say that nobody cheated. This was partly due to my physics graduating class being 12 students.

[BalinKing]

Regarding "penalizing honesty," I think it actually rewards honestly quite greatly—because of the Honor Code's importance in Caltech culture (i.e. one of the undergrad application essays is specifically about the Honor Code), it means that everyone gets to benefit from take-home exams.

I guess what I'm trying to say is that the "implied notion that Caltech students are... honest" is, in my experience, largely true (possibly because Caltech tries to select people who think the Honor Code is good from the get-go).

[_8091149529]

My counterarguments, in good faith:

Clearly, any society collectively benefits from honesty, whereas an honest act is, at least in a strict game-theoretic view, a loss to the individual in the short term.

The question "Are you honest?" does not necessarily filter out dishonest people. (Incidentally, I always felt like I would not have been admitted to Caltech as a student.)

[an_opabinia]

Insofar as universities produce pedigrees, it doesn’t really matter if Caltech students cheat. I suppose there’s greater justice in a student cheating on an exam and getting good grades while nobody knows - the student already won the lottery, they are getting the degree and a floor of a good lifestyle.

Yet they will be less likely to commit suicide, you know? Or some other act of desperation.

It’s a success because it lets students quietly cheat - if they need to - without reducing the graduation rate.