[Paul-E]

I've met Ross during my time at Princeton and he is a really genuine person, he is not trying to ruin anyone's life. This incident is the result of an uncharacteristic blind spot in empathy: a mistake.

I also have experience with the Princeton IRB on similar topics. The reality is that Princeton's IRB, and IRBs in general, are not equipped to deal with this sort of online research. IRBs were created as a reaction to unethical medical research, in particular the Tuskegee Syphilis Study [1]. My experience has been that the IRB has a greater expertise on medical and sociological studies. This leads the IRB to having a very narrow view of its remit in other domains. Unless humans are in a very literal way "subjects" of the study, then the IRB doesn't see it as human subjects research. In this case the IRB likely saw "Free Radical" and other websites as the subject. In both my studies and those done by my peers, the responses on what is and isn't human subjects research is uneven and you will often get a generic "this study does not constitute human subjects research" response from the IRB. This can be the case even if there possible negative repercussions to the "not subjects" in your research.

For example, say your study involves testing the vulnerability disclosure policies. How well do websites respond to vuln reports? In your study you send out 100 vulnerability disclosures. After you report these vulnerabilities, a human may read your vulnerability report and make a decision based on it. This presents a risk that the individual security team employees involved in your study will be scapegoated and fired when you publish your (potentially damning) results. How do you balance the value this study provides the public against the risk to the individual employees' livelihoods? The IRB isn't going to help you do this balancing, they will just say "this isn't human subjects research".

IRBs quite simply aren't equipped to evaluate this sort of research at the moment. This can be frustrating for a young twenty-something researcher just out of college trying to do the right thing while generating impactful research. You come in thinking that the IRB will be a guiding hand of wisdom and prudence, but you are quickly disabused of that notion after most of your interactions feel like conversations with lawyers in a compliance department. Many researchers in "CS" don't even involve the IRB, because they don't always see the ethical dimension of their work, but the fact that Ross did shows that he was trying to do the right thing here.

[1] https://en.wikipedia.org/wiki/Tuskegee_Syphilis_Study

[kstrauser]

I don’t doubt that Ross is a nice person, and I think he meant well. FWIW, I think this is a great thing to study and in other circumstances I’d be glad he’s doing it. But much as Ross didn’t intend for me to be hyperventilating, heart pounding as I imagine trying to explain to my wife how my little hobby is getting us sued, that’s exactly what happened. That was a whole awful lot of extra stress that I didn’t need.

[michaelhoffman]

Howdy Paul!

I definitely see a problem in that some people think that if the IRB doesn't object to what they're doing, it's OK. But ethics is a responsibility of the entire research team, and the research team is usually far better placed to understand the implications of their research strategy than the IRB.

The following are big problems here:

  - lack of informed consent
  - deception

Researchers should be trained that those are only allowed in exceptional cases where the benefits outweigh the harms.

[gpm]

I feel like "coercion" (legal threats) should probably be a separate bullet point from "deception"?

[michaelhoffman]

Well if you have informed consent, it's not going to be a problem. If you don't, then you need to do a more careful analysis of ill effects might ensue when someone gets the letter (feel distress, spend money on a lawyer).

[zajio1am]

Aren't these issues common in many other societal studies, for example fake resume hiring studies?

[DannyBee]

Yes.

IRB's exist, in part, to weight the cost to the humans/etc vs the possible benefit of the study.

Take: https://www.nber.org/system/files/working_papers/w21560/w215...

Look at footnote 3.

There is often a tendency to dehumanize things when it involves sending stuff to corporations. Even in that footnote, it's not employers processing fictious resumes, it's people.

So it's much more likely you'd get approval to do something "to a corporation" even though 99% of the time, it's really still being done to humans

[cal-throwaway]

I'd like to disagree as someone who knew Ross during my time at Berkeley. He absolutely is intelligent and thoughtful enough to know what he was doing -- including the consequences.

Berkeley's IRB is similarly illed -- resulting, a lot of trust (i.e. empathy) is placed that the lead will not do anything as obviously unethical as this. This is not the mistake that someone as intelligent as Ross makes, this was a conscious decision that backfired.

[geofft]

The fact that Ross didn't mean to do this is all the more reason why someone - maybe an IRB, maybe not (your argument makes sense) - should be assisting 20-something researchers with having a well-informed perspective.

In the absence of an organization that's good at this (which doesn't seem to exist and should), this probably should be the supervising professors.

[michaelhoffman]

As a research group leader, I find it unfortunate that the grad student seems to be the public face of this and is therefore attracting most of the ire. Feels like the student is being thrown under the bus, and responsibility for ensuring the study is conducted ethically should ultimately be that of the principal investigator.

[UncleMeat]

Mayer posted an apology last night taking full responsibility. Hardly throwing his student under the bus.

[Paul-E]

Thank you for making this point. I didn't articulate it, but this is part of why I felt I had to say something.

[TedDoesntTalk]

I hope this fellow Ross does not become suicidal or otherwise depressed when he sees the weight of the internet coming down on him for this faux pas. Ross, none of this wil matter in a year. Or 5 years.

[SilasX]

I think it’s kinda funny how sending vaguely threading emails (suggesting violation of statutes) sailed right through an IRB, but Scott Alexander got the third degree for giving patients a survey. Description:

>> When we got patients, I would give them the bipolar screening exam and record the results. Then Dr. W. would conduct a full clinical interview and formally assess them. We’d compare notes and see how often the screening test results matched Dr. W’s expert diagnosis. We usually got about twenty new patients a week; if half of them were willing and able to join our study, we should be able to gather about a hundred data points over the next three months.

https://slatestarcodex.com/2017/08/29/my-irb-nightmare/

[DannyBee]

UCI, for example, seems to have a very well defined notion of human subjects research, and this would clearly meet it.

Let's look: https://services-web.research.uci.edu/compliance/human-resea...

"Any systematic investigation (including pilot studies, program evaluations, qualitative research), that is designed to develop or contribute to generalizable (scholarly) knowledge, and which uses living humans or identifiable private information about living humans qualifies as human subjects research. See Definition of Human Subjects Research for more information."

Down the rabbit hole to https://services-web.research.uci.edu/compliance/human-resea...

"Research is as a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge. ...

Examples of systematic investigations include:

Surveys and questionnaires

"

So far, we got it in one.

I'll skip the part of whether it's generalizable - it's clearly intended to be here.

"A human subject means a living individual about whom an investigator (whether professional or student) conducting research:

Obtains information or biospecimens through intervention or interaction with the individual, and uses, studies, or analyzes the information or biospecimens; or

<The or is about getting PII in more cases, but this study is not getting PII>

...

Interaction includes communication or interpersonal contact between investigator and subject.

... "

Well, there we go.

Seems a lot more straightforward in various IRBs than you seem to say. As an aside, lots of IRB's also have mass email policies and are required to approve the text.

Now, maybe Princeton's IRB does not have as clear a definition. I can buy it, in fact!

But honestly, it doesn't seem that hard. If you are going to simulate fake emails to humans, for the purpose of gathering their responses, you are in fact, doing human subject research.

It also doesn't seem very hard to draw bright lines:

1. If you are interacting with people to see what their response is, even by email, they need to consent.

2. Do not deliberately deceive humans.

(You can even modify #2 to "do not deliberately deceive humans without an IRB explicitly understanding and weighing the cost/benefit" if you like, but most of the time, you actually do not need to deceive humans)

It's also really really hard to believe someone went to an IRB, and said "i'm going to survey people by sending them emails from fake people that seem mildly threatening, and seeing how they respond.", and an IRB was like "yeah, that seems okay, it's definitely not human subjects research".

It's up to the researchers to explain precisely what thy are doing in an accurate way. Saying you are surveying websites is totally inaccurate and confusing.

If a sociological researcher was like "whoa, i'm not emailing people asking for their family histories", that would be human subject research. Instead, i'm just "retrieving directed graph data from remote email addresses". I don't think that would go over very well.

Finally, as for not seeing the ethical dimension of their work, there is an easy fix for this (IMHO): Make ethics classes required. In fact, in lots of places, IRB's wont' review things if you haven't!

[pseudalopex]

I think their point was IRBs say information isn't about an individual when the individual would say it is. Everything you quoted depends on the word about. And UCI's policy refers to US regulations. Those regulations contain surprisingly broad exemptions.[1]

People talk about emailing web sites any time they don't know if it's a person or a company in my experience. And ethics classes don't give everyone the same understanding of ethics.

[1] https://www.hhs.gov/ohrp/regulations-and-policy/regulations/...

[vessenes]

Quick meta-comment; this is useful and informative information about the original link - please don’t downvote because you disagree with the point of view. If you disagree, please add a comment and make HN a great place for discourse!

[wglb]

Downvotes are ok to use to indicate and it has been so forever. Pg and others have supported this view