Real-world problems are so much messier and more frustrating than the clean abstractions many of us choose to work with. But this is really meaningful work that should have concrete social benefit. We should all hope to do good as you have.
Do you think more user and stakeholder engagement would have helped the public accept your research? I have no idea if BPS has the bandwidth for supporting this type of engagement, but I imagine a lot more fears would be assuaged if parents could have some role in "co-creation" of the algorithm, rather than just accepting results.
A lot of research from the fields of citizen science, public understanding of technology, and user experience show that participatory design techniques can enhance acceptance. E.g., https://www.bmj.com/content/319/7212/774.short
I am sure more engagement is always a good thing! One of the tricky tasks of BPS is to make sure that this engagement is "balanced", i.e., everyone's voice is equally heard. And you can imagine how hard that may be in practice. For example, Boston Public Schools had this survey asking people for their preferences on the start times. But most of the answers from these survey came from schools with less economically disadvantaged families. One way to do it is to call the families directly or have a lot of local meetings. But it is still a complicated problem.
Thanks for the response. I'm in the technology and policy program within IDSS and we have discussions about these "politics of technology" issues a lot. I'm sure you could become a case study in one of our classes if you wanted :)
Did you think the project was useful in the sense of getting the parents to think about possible changes and rejecting them? So they as citizens are clear that the current system is sub-optimal but they are OK with that
That's a great question, that I asked myself a lot.
I think this project was useful because it started this debate and brought more information to the table. For example, I found that a lot of people did not know how much the start times affected the district costs (tens of millions of dollars every year), or how the schools times seemed to be unequitable.
And this Boston Globe article allows even more people to read about this issue and make their own opinion. Ultimately, a lot of people in the US want later start times for high schoolers. This work can provide some information about the ways to implement such a change, as well as the consequences.
Did anyone suggest making the change, but announcing it today and having it come into effect a year (or even two) from now? That would give parents time to plan adjustments to their schedule, and would give a lot of people the chance to move schools if they really wanted to.
> District officials expected some pushback when they released the new school schedule on a Thursday night in December, with plans to implement in the fall of 2018.
The first round of applications for schools in Boston is January, so it's only a few weeks (over Christmas!) for families to figure out if they can make it work, if it's worth moving schools, which school they want to move to, etc. (Your chances of getting into your first choice school decline rapidly after the first round).
Or altering the ideal results to be less drastic at least initially to incrementally work towards the goal. Clearly the problem is the size of the change, not what the change is.
The engagement of the MIT team is interesting, but this reads to me like a classic failure of a "big-bang" rollout, rather than a gradual adoption. Was there any back and forth to create a slower-paced rollout plan?
We talked about it at some point, and it seems like a good idea. There are two main issues with it: first it can be hard to do a step by step change that is (a) fair (how do you choose the first schools to change?) and (b) works for every step (make sure that the transportation costs do not explode in between).
> it can be hard to do a step by step change that is (a) fair (...) and (b) works for every step
Yes. This is what separates the pros from the not-pros in major systems design and operation. The transition plan is the major key to plan, not the algorithm (which plays the role of Shiny Object in this case).
It is lots harder, lots, to get smoothly from A to B than to simply big-bang B. It takes enormous attention to detail and thinking through of every angle, and when done well is a thing of wonder.
I think in this case your team was not well served by how it was tasked.
Thanks again for responding, and for your team's efforts in the design.
One of the changes was moving start times of high schoolers later, which often came at the expense of younger students who would've had to start earlier. In my neighborhood, having to put 5 year olds on the bus at 6am or earlier was a big cause for the uproar from parents.
Can you talk about how you made this tradeoff? There's research that suggests high schoolers do better with later start times. But is there any research that backs up doing it at the expense of younger students? ie, is it possible that all students benefit from a later start time?
> Research findings suggest that changes occur in the “biological clock” during adolescence. As a result, teenagers have a natural tendency to fall asleep later and to wake up later. This is referred to as sleep phase delay.
> A correlation was found between subjects' melatonin secretion and their stage of development. The results indicated that melatonin onset occurs later in adolescents, making it difficult for them to go to sleep earlier at night. At the same time, the hormone “turns off” later in the morning, making it harder for them to wake up early (Carskadon et al., 1998, 1999).
It's like things are happening to the body during puberty.
I am no specialist of children health, but my understanding is the following: I guess starting too early is never good for any child, but there is something specific with teenagers that makes it a lot worse. There is much more medical literature documenting this fact than for the younger kids. So yes, in a way, given the constraints of costs, the algorithm had to choose the lesser of two evils.
Is it not possible to limit your algorithm's maximum time change for each school? Say, capping the change at 1 hour, and finding the best result of that set of times. Even if it is not as good as the 100% optimized solution, it seems like more palatable while also being an improvement.
Presumably, you are already doing that, by limiting the start/end times from 7am-10am, or something similar. In this case, instead of a global time restraint, there would be a local time change restraint +-1 hour for each school.
It seems the amount of change was a large problem. Was it attempted to minimize either number of student start times changed or total delta of minutes changed? Was it discussed how this could be done incrementally to slowly move towards more optimal start times?
Yes incremental changes looks like a good solution. I think it is hard to do it with the delta minutes changed, as you could have dramatic consequences on the costs (the ripples on the transportation systems can be quite surprising, in that changing the time of one school can have a big impact on the number of buses needed). As to the number of student start time, the main question is how do you choose the schools that change first? How do you make sure that your changes are not interrupted in the middle for political reasons, which would be unfair for the first schools and maybe the whole system.
Also, it is an algorithmic challenge to find a "path" of good solutions that go step by step to the good final start times.
I’m curious if an additional constraint like “total (absolute) time shift from this year to next (over all families in the district) <= K” or “max shift in first bell over all schools in district <= MS” was considered, along with the acknowledgement that this is e.g. an N year process to get from the current schedule S_0 to the optimal schedule S^*.
In my mind, this alleviates a lot of the “would have to change jobs this year” concerns, because the people making changes would ALREADY be making bigger changes (starting to take their kids to school), and the most extreme schedule changes would be deferred until the children on the existing schedule had aged out of that grade’s current schedule.
(Done enough optimization modeling to know this isn’t an easy path... but seems like it has some potential.)
Similarly, (maybe this was just the article dumbing down reality a bit, but it sounded like the transit expenditure vs parent happiness units were somewhat fungible. Any possibility of offsetting a bit of upfront expenditure with future savings of the more efficient (say from year N/2 onward) future state?
Thanks for working on the technical side of this social-equity project. Was the possibility of parent offering car pools or "walking groups" for kids included in the algorithm? I'm assuming that if volunteer parents are compensated to give carpool rides or lead walking buddies, then the negative impact would be lessened to parents who has to handle drastically different school start/end times. At the same time, it may be cheaper to compensate volunteer parents than for the district to procure or maintain more buses.
Another way to ask this question is: were there volunteer transit, ride share, or pre/post-school care incentives included in your algorithm, to balance the potential negative impacts to parents?
Yes and no. The algorithm runs with historical student data and therefore only considers students using the bus system in order to compute the effects of school start times on transportation.
On the other hand I think the district had planned to used the saved transportation money (~18 million/year) in order to help the most impacted parents (pre/post school care...)
I guess volunteer transit is always a positive thing for the parents and the districts. The algorithm was just trying to fulfil the objectives given by the district (costs, later students, fairness...). The compensations are what should happen after the algorithm: what do you do with the saved money, how do you help the most impacted parents? These negative effects were expected to disappear with the years as new parents start to choose the schools after the start times have changed (and therefore choose schools that can accomodate their schedules).
Not about your research, but any thoughts on SFUSD's assignment algorithm? I ask because it was also designed in consultation with MIT (in part) and also is an example of a very thoughtfully designed school district algorithm[1] that has been met with widespread displeasure in practice and is beginning to be dismantled.[2]
I think changing the start times is something the district had wanted to do for a long time. Therefore once the research and the algorithm suggested it was a good idea, they quickly jumped on board.
Now I am sure we have done some communications mistakes. For example the algorithm does not "decide" the start times. You just give it a (long) list of preferences and objectives (e.g., less HS students starting early, more fairness across the population...), and then the algorithm generates thousands of solution that try to satisfy these objectives. Solutions are carefully reviewed, and the objective updated if we think we can do better.
So in the end, this research is just a tool, it does not solve all the problems, and in the end people still need to find a way to agree. Hopefully they have a little bit more information to do so!
> start times is something the district had wanted to do for a long time
So, was the real motivation here one of efficiency and cost-savings then?
That is a clear thorn they might have wanted to solve.
Is the added sleep beneficial to students, all other things equal? I'm sure it is, but was the real motivation a cost savings and the sleep deprivation a nice cover for the real motivation?
No, as far as I know having later school start times for high schoolers was the main objective, and that was the strongest one in the solution that was chosen by Boston when they used our algorithm.
There has been a lot of research on high school students lack of sleep, and it seems that the school times do have a important impact on the students :
Did you read the article? It clearly stated that cost-savings was one of the outcomes to be balanced, but not the motivation. There were solutions that saved lots of money that were thrown out because they didn't balance the other outcomes enough.
BPS is pretty forward-thinking, and lots of research has shown that high school students are better off with later start times. Don't dismiss this as the actual motivation.
Did BPS consider applying the changes to only one or
a few high schools and their feeder schools? Perhaps
the realities of distribution of students among schools
in Boston not allow doing such a beta test?
That could be a way to do it, and Boston decided to do it all at once. I think their reasoning was the fairness of the process (it's hard/unfair to use beta testing on public schools). Also, it is such a hard change that maybe it is easier to do it all at once. You may also want to consider the political implications that are very important when making this kind of decisions.
Don't get discouraged, if you were feeling that way. It was important you substantiated the envelope of what kind of improvements are theoretically possible. And although the wealthier residents, who really ought to have the property to absorb inconvenience, may have been self-serving, they might reasonably have a point that sudden upheaval can cause problems for others that cannot attend meetings.
Maybe some will remember this if money for student transportation comes up to ballot.
My company had pitched the students of the A-Lab a couple years ago, but unfortunately we didn't get picked. BPS was one of the others pitching the students at the same event.
We submitted it to a journal that has a fast review process, and we are towards the end of it. We did not think soon enough to put it on arxiv, and we should be able to do it soon (it requires some editing to remove the journal-specific things).
The process went too fast to be entirely transparent. Boston tried to communicate as much as they could, and anything can be requested (Freedom of Information Law). We also have a paper that should not be too far from publication that details every detail of the thing.
I would argue that although it could have been more transparent, it is way more transparent than the status quo, where each individual school can try to pressure the district to get the time that they want, without considering the "big picture" and the impact on other schools/neighborhood/budget.
All decisions were made after a lot of group conversations with parents/staff/principals... But I am sure some things could have been better, in the end we were just the grad students trying to help :)
This can happen and did happen before. Maybe being suspicious of an algorithm is not a bad things, we should have extremely high standards for them ! My hope is that some people will just have a better view of how complicated this problem is, and be able to construct their own opinion from there. And this would already be something I would be proud of!
Real-world problems are so much messier and more frustrating than the clean abstractions many of us choose to work with. But this is really meaningful work that should have concrete social benefit. We should all hope to do good as you have.