[iancmceachern]

I emailed them. I live I'm SF. This is the sake, solved, problem we have in the machine tool world (milling machines, lathes, etc). Similar to old keyboards, you can just buy and use a floppy emulator.

They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.

[jandrese]

This was my thought. Floppy emulators are a dime a dozen thanks to the retrocomputing community. Emulators for the custom boards full of discrete logic, opaque ROM chips, PLAs, and so on from long dead companies are going to be a much bigger challenge.

If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.

Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.

[inferiorhuman]

SelTrac was designed to use commercial off the shelf (COTS) parts so there shouldn't be much in terms of needing custom chips, etc. The problem is that Alcatel (now Thales) basically doesn't support any of this (and hasn't for a long time).

The computers in question are, I believe, just thin clients.

[asdefghyk]

RE "....Still an expensive process, ....." Actually probably NOT , if all the $$ saved over the last 25 years , and also considering the many millions to get a completely new replacement system....

[jandrese]

The expense will be in "certifying" the machines for use with passenger traffic. Nobody wants personal liability, so we wrap ourselves in knots trying to pass responsibility on to someone, anybody, else.

[fdr]

A once a generation refresh seems okay in my book. The Breda trains that came in 1996 were being phased out about twenty years into their tenure, most generally seem to think that is about reasonable a time to do something like that. I don’t see software related regimes as inherently that different.

Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.

[mikewarot]

That ancient code is the crystalization of the knowledge of how to run, at the most basic level, all of the MTA. Emulating hardware is a good way to extend it's life while the decade long task of replacing it is underway.

Until you have directly used "legacy" machine that are mission critical, and understand how even a tiny error could cause the failure of a business and all the jobs that go with it, you can't avoid underestimating the true scale and scope of the problem.

[compuguy]

Here in the Washington DC area, our metro still has two series of rolling stock that date back to 1982 and 1987 (2k and 3k series). At the earliest they won't start retiring them in 2024 and 2025 at the earliest...

[jtbayly]

Wait, you’re suggesting replacing trains because the computers inside them are old?

[asveikau]

I think they're just saying parts should be periodically replaced across the board on that timeframe, and using the fact that the trains are being swapped out as an example. The Breda trains are still running some percentage of the fleet however. I rode in one today.

[fdr]

That’s right. System refreshes are a fact of life, especially if one is parsimonious in using what works until obsolescence. There are downsides of doing things that way, but I don’t see it inherently more defective than other approaches, such as some kind of continuous refresh that has a conceit of keeping a system “modern” in perpetuity.

[thsksbd]

But twenty years for a train seems very low. My daily driver is 12 years old, I thrash it every morning and never gives me problems. I live in the snowbelt.

By comparison a train is vastly over-engineered and has an electric drive train. Sure stuff will need replacing but a train should last 40 years at least

[fdr]

I wouldn't call myself as particular on train maintenance...but consider that these trains are run more or less continuously. You thrash your car in seasonally tough conditions every morning, but these drive all day and most of the night. So, I'm not sure if the comparison is apt on these grounds.

A cursory search suggests that quite a few manufacturers design for light rail car lifetimes of 25-30 years, not forty. These tend to be of European origin, which tracks San Francisco switching from an Italian to a German vendor. I don't see evidence that it's a common practice to significantly extend the tenure of those devices there nor here.

Funnily enough, I see no problems with the Breda trains personally as a passenger. But once they've shaken out the major bugs, the Siemens train reliability is anticipated to be triple or more. It may not make sense to design a traincar for a fifty year term.

[jcgrillo]

https://www.wbur.org/morningedition/2016/03/07/mbta-signal-m...

[yonran]

The real scope of the project is a mind-boggling $600 million to track the light rail trains inside and outside the tunnels and to control the trains inside the tunnels. Divided by the 250 light rail vehicles (apparently does not include historic streetcars), it works out to $2.6 million per car. Plus $36M for consulting (2023-11-07 board presentation from the documents that I linked earlier https://www.sfmta.com/sites/default/files/reports-and-docume...).

[kazinator]

> like this at every layer

It may feel emotionally like this at every layer, but the layers that are not floppy discs are completely different from floppy discs.

Other than electrolytics caps (easy replacement items), old electronics is reliable.

Moving parts need service; that's life.

> needs replacement

Not believable without detailed justification.

In the present article, someone is literally quoted as everything working just fine.

[iancmceachern]

I agree with you.

Good luck getting the city to see it that way. If you'd like help, let me know.

[JohnTHaller]

Same thought I had immediately when I read the story. Worth fixing that failure point at least.

[asdefghyk]

The PC and the floppy disk should be running as a virtual machine. ( The Vm software and host OS and the hardware would all be current supported versions of course )

[nostrademons]

That doesn't result in bigger budgets. If the goal of a manager is to increase their budget and hence scope of responsibility, any solution which doesn't result in a bigger budget isn't really a solution. Same reason it's going to cost a hundred million, yet a single indie game dev can write a train scheduling simulator by themselves.

[andbberger]

the state-of-good-repair grift will continue until the morale improves! SelTrac is fine. and if it ever actually breaks (dubious) they can fall back to block signaling which comfortably handles more trains than muni is capable of running. the MBTA does just fine with it on the green line, and when the MBTA is making you look incompetent you've got to question where it all went wrong.

this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.

[boringg]

Protected from hackers though!

[jmb99]

Unless said hacker has a big magnet.

[SllX]

What are they still running OS/2?

(they probably are…)

[snvzz]

Hopefully once they finally look into replacing the system, seL4 will be selected.

[Hrnrurj]

Here is 2017 article from NY to get an idea.

https://www.businessinsider.com/nyc-mta-subway-delay-2017-6

[akira2501]

> it needs replacing they say.

It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.

They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.

I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.

This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.

[arrosenberg]

> I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.

Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.

The problem is the public and the politicians - transit employees largely do the best they can with the resources and constraints they are given. In order to succeed transit needs municipalities and states to: pay more taxes; accept that transit won't generate a profit, but does generate non-monetary public value; and gather their resolve to deal with NIMBYs.

[skrbjc]

>Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.

I've worked in the public sector for several years, and there are definitely a decently-sized group of people in many different positions that are riding out their time until retirement trying to fly under the radar and do as little as possible. Not to say there aren't people that aren't competent, but as a competent person you have to work around the people that don't care and often actively don't want to see more work come their way to get anything done. It becomes tiring and eventually the private sector is too enticing and the competent people leave.

[arrosenberg]

Eh, plenty of private companies have that issue too. It is a bit harder to fix in the public sector though, I will give you that.

[madaxe_again]

Transit employees are not the same people as, nor have the same motivations as the politicians who manage transport agencies.

Of course the employees work hard. It’s the vocation they’ve chosen. I’m always amazed by the diligence of TFL and rail employees in the U.K., for instance.

No, the guiding principle for the folks at the top is “so, how can I make money for myself out of this before I fail upwards?”.

I’d wager the SF situation is just the same old looting by the Very Important People who run the show, leaving employees and actual managers with a shoestring and two peanuts to make it all work, while the head honchos bathe in gravy.

[arrosenberg]

> No, the guiding principle for the folks at the top is “so, how can I make money for myself out of this before I fail upwards?”.

Who are "the folks at the top"? Transit agency executives in California make less than software engineers (though if you put in your time, you get a nice pension to make up for it) - how are these people getting rich? If you mean the politicians, sure, I've already said they're half the problem.

[bhawks]

They make money by exploiting their position not through their salary.

These people are around in SF - see Mohammed Nuru, Rodrigo Santos and so on.

[iancmceachern]

I live in Victor Makras' Apartment, the one that got him in trouble for the port thing.

It's hard to live in SF for long without inadvertently touching corruption, even if you actively try to avoid it.

[inferiorhuman]

Nah, Jeffrey Tumlin (current SFMTA head) makes about $500k total comp.

[arrosenberg]

So the CEO of one of the 4-5 largest transit agencies in the country makes about what a staff level FAANG engineer makes...

[stackskipton]

There is always stable "Go work as consultant at company that does a ton of business with former agency. This position is in no way related to any agency work they did. :wink:"

[rayiner]

> Of course the employees work hard. It’s the vocation they’ve chosen. I’m always amazed by the diligence of TFL and rail employees in the U.K., for instance.

We are talking about American transit workers. I haven’t been to London, but I’ve been amazed by the diligence and efficiency of Japanese transit workers. But they bear zero resemblance to the folks running the transit system in DC, where I live.

[sidlls]

It wouldn't surprise me. This city is corrupt as hell.

[iancmceachern]

As I learn more about how this city operates, living here longer, I keep finding myself truly amazed at how corrupt this city is. Even if you try, you can't avoid it. It's truly everywhere.

[madaxe_again]

You underestimate. The more places you go, the more you see, the more you do, the more you realise it is truly EVERYWHERE.

I honestly find it incredibly disheartening how rampant and underreported corruption and nepotism are in practically every sector, industry, everything, everywhere, always.

The corrupt are a minority - but they are global, and the impacts of their actions are damaging the entire human race in a very significant fashion, and always have.

Murder, I can forgive.

Corruption, I cannot.

[rayiner]

> Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.

I think American public servants are lazy, overpaid, and unskilled, yes. You don’t need experience running the government to know that the government sucks at its job, just like Windows users could tell Windows 95 sucked without being programmers themselves.

US transit systems almost uniformly suck compared to those in other developed countries, while costing the public just as much money, and in many cases a lot more money: https://www.forbes.com/sites/stephensmith/2011/12/11/surpris.... (“ The Swiss Transit department made a presentation a year or two ago in San Francisco. As far as I could tell, the public transit subsidies for all of Switzerland are in the neighborhood of the subsidies for transit in the Bay Area… He gives total [Swiss] transit subsidies at $1.5 billion.The 2009 MTC report gives Bay Area transit subsidies at $1.3 billion.”).

[dctoedt]

> I think American public servants are lazy, overpaid, and unskilled, yes.

Sweeping generalizations much?

[rayiner]

Your experience as a customer of a transit service is based on the aggregated output of public employees in general, not individuals. I’m sure there are hard working, efficient people at WMATA. But those individuals can’t single handedly keep the system from falling apart when their coworkers are trying to do as little as possible.

[arrosenberg]

Once again, I will reiterate that right wing characterizations of public employees are very silly, not reflective of how most municipal agencies operate, and is intended to erode public institutions. Public employees are not inherently more lazy or less skilled than private sector employees. Their incentives and challenges are different. If the performance is subpar, look in the mirror to find the problem.

Its completely besides my point, but comparing sprawling American metros to compact Swiss cantons is not a useful exercise. If we want to redevelop American cities to be more transit friendly, I am on board!

[rayiner]

I’m very left wing on transit. I rode the Amtrak from Baltimore to DC every day for two years, I commuted from the suburbs to various DC metro stations for years to commute into the city. The right wing characterization of transit proved true based on experience despite my strong desire for it not to be true. Don’t get mad at right wingers for simply pointing out what’s plain to see. There’s a reason Americans are overwhelmingly leaving the places that have good transit for places that have no transit—the American version that’s always slow, late, dirty, and doesn’t go where you’re going is just not an amenity that’s worth it for most people.

The problem is not funding or public policy or geography. The northeast corridor is densely populated; there is no reason in terms of geography why you shouldn’t be able to have efficient transit in Baltimore and DC. These systems are well funded and heavily subsidized by the federal government. The problem is the people, and that’s a problem you can’t fix. You cannot redevelop American cities into anything other than what they are so long as you have Americans running them. It’s a futile exercise.

[akira2501]

> Let me guess, you also think public servants are lazy, overpaid and unskilled?

Do you not see this article as evidence of this fact? If you walked into any other business reliant on "floppy disks" just to take your money, would you praise their practices and efficiency?

> Comments like this belie how many people form opinions of public agencies without having any real experience with government.

I form opinions because I try to use their services and am disappointed and often disgusted. Do you really think "experience with public transport" is particularly hard to come by? That this is all too esoteric for the "common man" to opine on? Please...

> transit employees largely do the best they can with the resources and constraints they are given

Perhaps they should earn those resources competitively instead of being given them? Otherwise, this seems like a perennial chicken and the egg problem that swallows tax dollars with no worthwhile result.

[arrosenberg]

I don't think the article is a reflection on efficacy of the public agency. If their budget for new software is $0, how exactly are they supposed to upgrade it? It's certainly possible that the agency has been utter shit for 25 years, but this article doesn't give the information to conclude that.

> Perhaps they should earn those resources competitively instead of being given them?

How? They're a public agency. There is no profit motive - if the public wants nicer stuff, they need to pay for it. That includes both the software costs and the higher salary costs to hire people capable of those types of projects.

Again, you come off as someone running their mouth with no actual experience interacting with public agencies. It's very frustrating to know how to fix a problem without being given the resources or permission to do so, which is often the case in transit. Add to that most transit agencies have a tiny budget for IT, and I'm not sure how you expect this to get solved without a big public push.

[bongodongobob]

The problem is "lower my taxes" single issue voters. People vote for whomever lowers taxes and then complain about potholes. It's always the same people. GOVERNMENT IS INCOMPETENT BLAH BLAH. Yeah, cause you refuse to fund it and everything goes to shit, then is 10x more expensive to fix it compared to properly funding maintenance. You end up paying more in the end because these people can't see past the couple hundred dollars they saved on their property taxes and wonder why everything is turning to shit.

[aorloff]

You have some evidence that government is not incompetent ?

[afavour]

It’s weird that people have this notion about government and only government. There are things governments do well and things they do badly. You could say the exact same thing about private business: businesses go bankrupt every day, wasting time and resources. But no one argues that the very concept of business is “incompetent”.

[stctw]

What is weird about this?

If a business fails to perform well, its customers can choose another provider, and the business will have to improve, or else it will fail.

If a government fails, what recourse do its "customers" have? Wait a few years for the next election and hope that the truth about the government's poor performance wins out over the propaganda, and hope that the citizens vote in competent, honest officials, who might gradually make improvements, despite having to work through non-elected officials who are difficult to replace for incompetency? How often does that happen? And what if the officials in question are merely appointed by those who are elected, and thereby insulated from accountability by several levels of indirection? What recourse do the "customers" have then? Can they choose another "rail provider"?

The private sector has natural consequences for failure and a natural incentive to excel and be efficient (certain circumstances, such as monopolies, notwithstanding). The public sector is heavily insulated from consequences for failure and inefficiency.

That doesn't mean that everything should be privatized. It means that we shouldn't be surprised when a service provided by the public sector fails to perform well or efficiently, and we should carefully consider what the government ought to be responsible for. Also, we should make it easier to hold government officials accountable and replace them for incompetence. If someone wants the security of a public sector job, they ought to be held to the highest standards.

[inferiorhuman]

  If a business fails to perform well, its customers can choose another
  provider, and the business will have to improve, or else it will fail.

In this case, mayor Willie Brown (who predates the creation of SFMTA) steered the contracts for the trains and train control to Breda and Alcatel/Thales via Booz Allen Hamilton. Essentially he set up a few generations of sole source contacts for train control as you can see when the MTA was brining the Siemens cars and central subway.

https://www.youtube.com/watch?v=U6mQfWC1v98

Thing is, Alcatel did fail. Thales scooped them up. From a customer POV Alcatel/Thales failed too. They sold an end of life product that the MTA had to spend millions on just to add a new route (first part of the central subway). They've largely abandoned support of the inductive loop system.

Sure, the MTA could choose another vendor. If memory serves that's where they're stuck currently. Of course this brings out all the anti-government folks despite the main problem being a private vendor (Thales) and private consultants (BAH).

[samatman]

Yes, that happens: when private ventures go sour, which happens rather frequently, they go bankrupt.

Public ventures are like that, but without the bankruptcy.

[aorloff]

Trust me, it’s not only government.

[asveikau]

The US government created the network you used to write that comment.

[aorloff]

As a byproduct of other efforts, not on purpose.

[whyenot]

It reads like you are conflating newer with better.

It was more than a few years ago, but I remember the posts on Slashdot where tech people were wringing their hands over the fact that election offices were not adopting touchscreen ballots fast enough. Some posters were also promoting things like internet voting. How did that work out?

Sometimes it actually is better to move slowly instead of adopting something new just because it is new. With old equipment and older technology, we know the failure modes, we have the procedures in place for addressing them. Whether it's floppy disks, paper ballots, (or human beings,) older does not necessarily mean worse.

[DowagerDave]

or they're not immune to the same urge all SW devs have to rewrite - it's a lot easier to start from scratch than (a) figure out how an existing system works, (b) figure out how to upgrade or modernize