SFMTA's train system running on floppy disks; city fears 'catastrophic failure'

I emailed them. I live I&#x27;m SF. This is the sake, solved, problem we have in the machine tool world (milling machines, lathes, etc). Similar to old keyboards, you can just buy and use a floppy emulator.<p>They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It&#x27;s really the whole system that&#x27;s like this at every layer, it needs replacing they say.

My company provides new compute modules for another major global city&#x27;s infrastructure that still relies on Intel CPU&#x27;s from the early 80&#x27;s. That is, we are building them brand new boards populated with chips that are over 40 years old.<p>They haven&#x27;t shown any interest in updating the system. It works, they can get service, and get &quot;new&quot; replacements for things that go bad.<p>What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He&#x27;s getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.

&gt; This system was designed to last 20 to 25 years. SFMTA&#x27;s director Jeffrey Tumlin said upgrading the system will take another decade and cost hundreds of millions of dollars.<p>This is the problem, not that they&#x27;re using a floppy. This isn&#x27;t web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that&#x27;s not happening quickly enough here.<p>Edit: It was last updated in 1998, so it&#x27;s due now not a decade from now.

&gt;Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.<p>&gt; &quot;We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn&#x27;t have a hard drive so you have to load the software from floppy disks on to the computer,&quot;<p>In 1998, most personal computers already had hard drives [0]. From Wikipedia &quot;The IBM PC&#x2F;XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers.&quot;<p>The 3.5&quot; floppy is from the mid 80&#x27;s, again from Wiki [1] &quot;In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design...&quot;<p>[0] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Hard_disk_drive" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Hard_disk_drive</a><p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Floppy_disk" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Floppy_disk</a><p>Why do I have to do this research instead of the &quot;journalist&quot;?

Well... I feel slightly better about Boston constantly pushing back being able to use our phones as tickets after seeing the timeline for the transition and how just that they are still doing this.<p>What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn&#x27;t even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it&#x27;s run.<p>It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.<p>I am very curious what other countries are doing that we are not.

I grew up in the south and was fortunate enough to move to the Bay a few years ago.<p>I was expecting it to be some kind of utopia, with futuristic technology on every corner.<p>In reality, it is roughly the same as anywhere else in America. A bit of a let down. The innovation does not take place in the infrastructure.<p>Beautiful place, though!

SF Voters rejected Proposition A in 2022 [1], which would have included funding to upgrade Muni&#x27;s control systems (among many other projects). We&#x27;ll eventually have to find the money somewhere else when the system fails.<p>[1] <a href="https:&#x2F;&#x2F;www.sfchronicle.com&#x2F;sf&#x2F;article&#x2F;S-F-voters-narrowly-reject-Muni-s-400-million-17241868.php" rel="nofollow">https:&#x2F;&#x2F;www.sfchronicle.com&#x2F;sf&#x2F;article&#x2F;S-F-voters-narrowly-r...</a>

Does no one at the organization know what a GoTek FlashFloppy is?<p>Sounds like they are using the floppy as an excuse to push for an upgrade that has nothing to do with the floppy drives.

Can someone just ask Foone or lcamtuf to help them get it running off an internet-connected toothbrush or something? It seems like there are so many people doing complex reverse-engineering of ancient stuff _just for kicks_ that this floppy-disk issue just shouldn&#x27;t require a massive project.<p>yes, I know safety-critical systems are different. I also expect that the floppy-disk issue is just the easiest problem to explain of a long chain of terrible legacy lock-ins. However, if they&#x27;re literally holding their breath every morning when it&#x27;s time to IPL the system off a floppy...that part sounds solvable.

&gt;SFMTA&#x27;s train control system relies every morning on 5 inch floppy disks.<p>That&#x27;s not the 3.5&quot; floppy disk in the video. This is the <i>old</i> floppy disks[1]<p>1. <a href="https:&#x2F;&#x2F;www.digitaltreasures.ca&#x2F;img&#x2F;level2_floppy_525.jpg" rel="nofollow">https:&#x2F;&#x2F;www.digitaltreasures.ca&#x2F;img&#x2F;level2_floppy_525.jpg</a>

&gt; Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.<p>&gt; &quot;We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn&#x27;t have a hard drive so you have to load the software from floppy disks on to the computer,&quot; said Mariana Maguire, SFMTA Train Control Project.<p>&gt; SFMTA&#x27;s train control system relies every morning on 5 inch floppy disks.<p>This doesn’t make any sense. 5.25-inch floppy disks and no hard disks was not “cutting edge technology” in 1998. It arguably wasn’t even “cutting edge technology” in 1988

Frustratingly, though the article goes into how replacing&#x2F;upgrading the whole control system will be an big, expensive project, they don&#x27;t attempt to say why they can&#x27;t just update the &quot;read from floppy&quot; part of the system to read the same info from a modern component.

I dont really see the issue?<p>The technology works, there is a replacement outlined, there is no shortage of floppy disks - even 5 1&#x2F;4 ones.

I believe that the article is about this RFP for “Contract No. SFMTA-2022-40 FTA” to upgrade the Communications-Based Train Control System (CBTC) or Advanced Train Control System (ATCS). The resolution to create the RFP for the supplier was approved 2023-01-17 (<a href="https:&#x2F;&#x2F;www.sfmta.com&#x2F;reports&#x2F;1-17-23-mtab-item-14-communications-based-train-control-system-rfp" rel="nofollow">https:&#x2F;&#x2F;www.sfmta.com&#x2F;reports&#x2F;1-17-23-mtab-item-14-communica...</a>), and a $36M resolution to create an RFP for a consultant was approved 2023-11-07 (<a href="https:&#x2F;&#x2F;www.sfmta.com&#x2F;reports&#x2F;11-7-23-mtab-item-11-train-control-upgrade-project-consultant-rfp" rel="nofollow">https:&#x2F;&#x2F;www.sfmta.com&#x2F;reports&#x2F;11-7-23-mtab-item-11-train-con...</a>). I’m not sure where the actual RFPs are pubished though.

SF is a joke. 20 years of being one of the richest cities in the world (probably in the history of human beings) and it comes out worse than it was before.

Upgrade? More like launch the process of planning for drafting a bid solicitation announcement. If I were 3.5&quot; floppies, I wouldn&#x27;t worry about retirement any time soon.

&gt; &quot;Wow. I mean I thought we were moving on to AI. So why are we doing floppy disk,&quot; said Katie Guillen, SFMTA passenger.<p>This naivety is <i>not</i> Katie&#x27;s fault. We who work in tech are to blame for constantly pushing our half-baked experimental garbage as if it was &quot;engineering&quot; on par with civil or aeronautical systems. We can&#x27;t blame people for occasionally believing the lies.<p>The tech-washed version of this quote might go something like &quot;wow I thought everything was moving to the multi-cloud serverless kooberneetus now, why is it still running on a computer?&quot;<p>&gt; It is easy to run a secure computer system. You merely have to disconnect all dial-up connections and permit only direct-wired terminals, put the machine and its terminals in a shielded room, and post a guard at the door[1].<p>This is the kind of thing I want running the trains. Give it ECC RAM too, please.<p>[1] <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Robert_Morris_(cryptographer)#cite_note-17" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Robert_Morris_(cryptographer...</a>

So, replace the floppies with a high reliability flash device that emulates a floppy drive, should have happened years ago.

&gt; &quot;It&#x27;s like if you lose your memory overnight, and every morning, somebody has to tell you hey &#x27;this is who you are and what your purpose is what you have to do today,&#x27;&quot; said Maguire.<p>Yeah, that&#x27;s called a cold boot. Moving to not-floppies doesn&#x27;t mean you can avoid this. Clearly it&#x27;s off of floppies instead of ROM so you can more easily update the software, but I am wondering how often that ended up happening. Maybe EEPROMs would have been better.<p>&gt; Luz Pena: &quot;How dire is it to change the system to upgrade it from a floppy disk to a wireless system?&quot;<p>I agree that floppies aren&#x27;t the peak of reliability, but &quot;a wireless system&quot; also sounds like a disaster. I don&#x27;t want critical urban infrastructure running on extremely hackable OTA updates. For the love of god, SF, you can avoid pretty much all potential cybersecurity problems by just <i>not putting your trains online</i>.<p>I feel like neither the interviewer nor the interviewee really had the technical expertise to speak to this. This entire piece is just, &quot;oooooo, floppies are old. Old bad! Why not new yet? New good.&quot;

Japan is doing away with 3 1&#x2F;2 floppies . <a href="https:&#x2F;&#x2F;arstechnica.com&#x2F;gadgets&#x2F;2024&#x2F;01&#x2F;floppy-disk-requirements-finally-axed-from-japan-government-regulations&#x2F;" rel="nofollow">https:&#x2F;&#x2F;arstechnica.com&#x2F;gadgets&#x2F;2024&#x2F;01&#x2F;floppy-disk-requirem...</a>

I don&#x27;t understand.<p>The retro community has proven reliably that a simple Raspberry PI can easily bit-bang floppy controllers. We have myriad floppy-to-SD card adapters.<p>Surely a plug-and-play solution that removes the area of most concern (reliance on the media itself) should be easily achievable in a few months?

&gt;&quot;&quot;The system is currently working just fine, but we know that with each increasing year, risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure,&quot; Tumlin told ABC7.&quot;<p>Do they know that floppy can be backed up?

They haven&#x27;t even settled on a contractor yet? Maybe the problem here is that they&#x27;re trying to write one check to fix all their problems at once instead of taking an incremental approach.

The longer a thing has not failed, the less likely it is to fail at any given moment. On the other hand, the older a thing is, the more likely it is to fail at any given moment.

Floppy disks seems common in the us. Just since 2019, they don&#x27;t use 8&quot; floppys for the nuclear rockets.<p><a href="https:&#x2F;&#x2F;www.theverge.com&#x2F;2019&#x2F;10&#x2F;25&#x2F;20931800&#x2F;usa-nuclear-8-inch-floppy-disk-solid-state-transition" rel="nofollow">https:&#x2F;&#x2F;www.theverge.com&#x2F;2019&#x2F;10&#x2F;25&#x2F;20931800&#x2F;usa-nuclear-8-i...</a>

Upgrade to 3.5 inch ones!

Trains are expected to last for decades, so this really isn&#x27;t surprising. Developing a new system with the latest hardware to replace the floppy disk system would probably cost millions and might introduce bugs and security issues.

&quot;If it ain&#x27;t broke, don&#x27;t fix it&quot; definitely has its practical limits

Just a few short hours ago I was hearing some Muni workers at West Portal station talk about this story. One said to the other, &quot;young people don&#x27;t know what a floppy disk is anymore&quot;.

&gt; SFMTA&#x27;s train control system relies every morning on 5 inch floppy disks.<p>And in the video, she says “on 3x 5 inch floppy disks like this one &lt;shows a 3.5 inch floppy&gt;”

The reporter waving around a 3.5&quot; floppy because most viewers wouldn&#x27;t even know what the 5.25&quot; was

If SFMTA was &quot;on the ball&quot; they would have obtained source code from the original company.

Didn&#x27;t it used to be that they broke often when you would keep them in the sun, even for a little?

Probably just a ploy to privatize the whole thing. Nothing to see here, move along.

If it works don’t fix it

Raspberry pi controller $30, couple of people to make it work - let&#x27;s say $500k. Sure... there are loads of things that probably need attention, but maybe you don&#x27;t need to fix them all at once?

Remember when, less than 3 years ago, SF projected a $108M <i>surplus</i> for the next two years?<p>Woulda been a nice time to clean up some of this technical debt!<p>Or how about the SF Emergency Sirens, taken offline in late 2019 for a &quot;2 year&quot; upgrade plan that officeholders implied was already in place?<p>In August 2023, with <i>no</i> progress whatsover, with the Maui fire disaster fresh on their minds, Mayor Breed &amp; Supervisors President Peskin touted they&#x27;d finally funded a plan to return them to service soon: <a href="https:&#x2F;&#x2F;www.sf.gov&#x2F;news&#x2F;mayor-breed-and-board-president-peskin-announce-citywide-outdoor-public-warning-system-upgrade" rel="nofollow">https:&#x2F;&#x2F;www.sf.gov&#x2F;news&#x2F;mayor-breed-and-board-president-pesk...</a><p>In that same August 2023 timeframe, Peskin said the plan would bring this &quot;need to have&quot; system &quot;up and running&quot; &amp; to &quot;state of the art&quot; by end of 2024, for $5.5M: <a href="https:&#x2F;&#x2F;www.cbsnews.com&#x2F;sanfrancisco&#x2F;news&#x2F;san-francisco-city-leaders-look-to-bring-back-emergency-sirens-by-end-of-2024&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.cbsnews.com&#x2F;sanfrancisco&#x2F;news&#x2F;san-francisco-city...</a><p>Of course, this was just more blatant self-exonerating bullshit from our local political machines immune from any real accountability for incompetence in basic public functions.<p>A mere 6 months later in February 2024, nothing&#x27;s been started, Peskin admitted &quot;we don&#x27;t even have a plan&quot;, the department is still waiting until &quot;funding is identified&quot;, and the cost estimate has ballooned to $20.5m: <a href="https:&#x2F;&#x2F;abc7news.com&#x2F;san-francisco-sirens-emergency-911-alert-system&#x2F;14461668&#x2F;" rel="nofollow">https:&#x2F;&#x2F;abc7news.com&#x2F;san-francisco-sirens-emergency-911-aler...</a><p>That works out to $170K+ for each of 119 units – units that each could probably just be a weatherized consumer-grade handheld device with multiple mobile&#x2F;packet&#x2F;sat radios, &amp; a simple authenticated-playback app, mounted on existing poles that presumably already have power and even loudspeakers.

&gt; <i>It&#x27;s a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure.</i><p>The key words in the sentence are: &quot;it&#x27;s working just fine&quot;.<p>Data degradation of floppy discs is easy: just copy them to fresh ones, and verify that you have a good copy. The images should be safely backed up so they can be regenerated. (Plus there are emulators; a topic covered elsewhere under this submission.)<p>I mean, are they really using the same 30 year old floppy discs over and over again until they degrade?

Absurd errors in this article. Starting with:<p>&quot;it was from an era that computers didn&#x27;t have a hard drive&quot;<p>Absolute BS. Pretty much every computer had a hard drive in 1998, and most had CD-ROM.<p>Then they referred to the 3.5&quot; disk as a &quot;5-inch floppy.&quot;<p>&lt;sigh&gt;