Ask HN: What's Happening at Southwest?

76 点作者 JimmyL超过 2 年前

The media narrative is that there's a massive software failure going on in its scheduling system, and they need to stop flying for a day or two to "reset" it. Anyone have first-hand experience on their systems about what that really means?

19 条评论

phil21超过 2 年前

From what I know... Not affiliated at all with the airline industry, but I fly a lot and tend to nerd out on this stuff...Their crew (and flight) scheduling software functions in a manner where it more or less simulates a "perfect" day of operations. Airplanes take off on time, land, and continue on. If anything disrupts this simulation, crew members had to call in and talk to someone to update the computer system to tell it that both the airplane and crew members are not where the system thought they were.Once the call center got overwhelmed it was a cascading failure with Southwest quickly not understanding where most of their flight crew happened to be at any given moment. It appears they feel the only way to solve this is get everything (planes and crews) back into "starting position" to restart the simulation.

评论 #34174014 未加载

评论 #34175597 未加载

mtmail超过 2 年前

Via reddit/r/bestof <a href="https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pilots_hows_it_going/j1tne9z/" rel="nofollow">https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pi...</a>"So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the phone system was updated not too long ago and it was not working well.""I used to work for a large company trying to fill the void [huge gap in the market for good aviation scheduling software], and our software was damn good too. SW was one of the airlines interested, we would demo it exactly like the scenario today, but it was "too expensive" and they stayed on their homebuilt stuff."

评论 #34173875 未加载

matt_s超过 2 年前

Basically what I read was the former CEO was accountant educated and focused on stock price and financials and ignored system upgrades for like a decade. That coupled with their decentralized nature of operating (no hubs or star pattern) the recent storms caused their systems to not “know” where people or planes were even though the people knew where they were. That knowledge is the critical feature because hours of fly-time for pilots and airplanes is crucial for legally mandated rest periods, maintenance and logistics.Having personally experienced the delays and issues with SW earlier this year I can suggest not flying them other than direct flights. Them not having any hubs means if one plane is late or a crew times out because of delays you are stranded. We got stranded at one airport overnight and got routed thru 2 other airports, changing planes at one, before reaching our destination. They have zero capacity of crews/planes on standby at any airport, probably because of the former CEO leadership and pandemic complications.

评论 #34173769 未加载

评论 #34174821 未加载

评论 #34174188 未加载

punkrex超过 2 年前

What's being skipped over in a lot of reporting is that Southwest has serious issues maintaining ground staff staffing, and has been struggling with that the entire pandemic because they refuse to pay market rate. This means that planes can't be unloaded and turned around in any reasonable amount of time. I gave up completely on southwest after I was stuck on a plane for 4 hours this summer, because of lack of ground crew to unload the planes, and thus planes were stuck on the tarmac.Add on to that the latest technology woes, the storm was just the thing that tipped over a company that had cut operational capacity to the bone.

评论 #34175665 未加载

bena超过 2 年前

From what I've gathered, Southwest effectively performs Just-In-Time resource management.They arrange flights and crews so that the right number of planes and people are in the right places at the right times.There's some tolerance in the system. So if the plane from New York to Cincinnati is late, it's ok-ish. The flight from Cincinnati to Dallas should be able to make it in time if things aren't too bad. Then the flight from Dallas to Phoenix should take off when it should. The Phoenix to Las Vegas flight will never know there was a problem.It also matters for crew. Pilots can only fly for so many hours. So if you have someone stuck in a holding pattern, that cuts into times.However, if that plane from New York to Cincinnati shits the bed, it'll fuck over Cincinnati to Dallas, Dallas to Phoenix, and Phoenix to Las Vegas. The failure just cascades. You lose planes, you lose crews, nothing is matching up and everything is fucked.Now imagine this happens a few hundred times. Thousands of flights are affected.Other airlines don't have this problem because they can just not do a flight. They fly people into a hub, then out of a hub. Delta will go from New York to Atlanta, and back again. Cincinnati to Atlanta, and back again. They work more like a busses. Miss a bus, catch the next one. So if you crap out a day's of flights, you can still put those people on planes and get them out. You know they're either at the hub or on their spoke. So if they're not in Atlanta, they're in their city.

kjellsbells超过 2 年前

In HN terms: imagine your assets are encoded in an array or list. Movement is represented by shifting elements right or left.Most airlines are hub and spoke. Elements shift right (planes leave the hub) then shift left (return). Sometimes they shift two places right. If something disrupts this, like, a plane is stuck in [1] when it needs to be in [0], its generally recoverable when the event clears, or, you can use one of those 2 step moves in another flight to pick up stranded passengers on your way back to the starting position [0].Southwest operate a point to point model. Assets start at [0] and have to traverse every point on the list through to [N] to succeed. N can be 3, 4 or higher. You can imagine what happens if a disruption happens in the middle of this list. Everybody upstream of the break is left planeless, everybody downstream who wants to go up cant make it further than the airport before the break, and everybody at the breakpoint is miserable.So Southwest then fallback to a new scheduling model for which their system was not designed. Its like having a graph traversal system and trying to get it to solve matrix equations. Yeah, there are some similarities, but its not really the same.

thomasjudge超过 2 年前

Fun fact I found in an NYT article and LinkedIn: The current CEO Bob Jordan started at Southwest in 1988 as a computer programmer. From 2006-2008 he was EVP of Strategy and Technology

评论 #34174801 未加载

MontyCarloHall超过 2 年前

A related question: can Southwest ever fully recover from this? Their reputation had taken such a hit that many loyal customers will hesitate to fly with them again, and even more fairweather customers will permanently cross Southwest off their list of carriers to consider flying with.Their stock price is only down ~10% since the meltdown, which seems extremely optimistic to me.

评论 #34174502 未加载

评论 #34174922 未加载

评论 #34174847 未加载

评论 #34179329 未加载

评论 #34213082 未加载

FesterCluck超过 2 年前

Everyone here is at least a little misinformed regarding their software and it's simulation abilities. They have control systems with the data they need, and this data is fed into the flight scheduling & monitoring UI. Their tech works, it just gets bashed on because it's different than a lot of other airlines. Used properly it can account for these things.What causes things like this is overbooking. Overbooking happens in more than just butts in seats. If you're not going to have enough ground crew to handle your SLA on flights, flights should be preemptively cancelled to keep them under that SLA. Airlines keep that SLA as low as safety permits. The max is set by the FAA at 3 hours for domestic flights and 4 hours for international flights (both ends).Southwest's software is doing it's job, it has adjustable tolerances. It will even take into account weather conditions reducing ground crew therefore raising SLA. But, the effect of the weather conditions on the ground crew is also adjustable by humans. As a matter of fact, while one would think it would just reference historical data, that's not exactly true. It references predictive models that are adjusted by experienced individuals. Those individuals can be ordered to adjust the parameters outside their honest assessment to allow for steps at the beginning of the process to operate smoothly.Weather prediction has a horizon. Southwest allowed excess bookings beyond that horizon, or they allowed excess last minute bookings. This weather event was massive and one-sided, yes, but it was also completely predictable. A human made the decision to widen the guardrails. There aren't a ton of people allowed to do that, I can think of maybe 3.Disclosure: I wrote software for SWA many years ago. Nothing I've said here is privileged, in fact most airlines operate this exact way.

评论 #34175273 未加载

评论 #34175601 未加载

sjm-lbm超过 2 年前

I don't think the reset is around resetting the software logic exactly, but rather that things have gotten so far outside of what they should be they no longer know where planes are, where crews that can find those planes are, and how many people even are still planning on using Southwest to fly to their intended destination.They need a "reset" to spend a few days inputting as much of that data as possible into their computer system, at which point they can start operating with some level of efficiency and start functioning again.

temptemp888超过 2 年前

Truth is even they won't really know until they have time to do a post mortem.Some observations, though.- Via the departure control, manifests, etc...they DID know where their crews went. Any employee flying non-rev, deadhead, or on duty is recorded by employee number. There was likely a problem with keeping that synced with the crew management platform.- Large parts of this have little to do with technology. Point to point versus hub and spoke means less slack. And recently, SWA has removed even more slack trying to drive revenue with aggressively optimistic schedules, overbooking, tight crew slack, etc. If they didn't pull pack the schedule in small, manageable pieces like the other airlines did, there's no system that would save them. Seems likely they didn't pull back enough before the storm. There's a brink you can't go past, and a rate of cancels you shouldn't exceed. You have to guess right earlier.- Crew unions have historically negotiated out anything that looks like big brother, location tracking for example...even on-demand "I'm here". So lots of things that could have made this better didn't exist , on purpose.- A fair amount of the reset is just practicalities of using planes empty of pax to get rested crews staged in the right cities and bags to the right places.

greggarious超过 2 年前

I basically only used them for the PIT -> LAS direct flight for Defcon, and while I never got enough miles or whatever for any kind of formal loyalty program, I got the impression that a decade and a half of respecting sky law[1] got me some conversations most passengers don't get to have :-)One of their pilots once confided the reason he had to have them plant a drink cart in the aisle while he popped out to use the bathroom was because they're given such little time between connections they can't even stop at a urinal. (<15 minutes)They flew too close to the sun, and they had a cascading failure. Womp womp![1] <a href="https://www.youtube.com/watch?v=vK1HINtjdHY">https://www.youtube.com/watch?v=vK1HINtjdHY</a>

shagie超过 2 年前

A key point is how they do routing - <a href="https://www.ajot.com/news/southwest-air-faces-gridlock-with-over-80-of-flights-scrapped-or-late" rel="nofollow">https://www.ajot.com/news/southwest-air-faces-gridlock-with-...</a>> Unlike competitors that use a so-called hub-and-spoke system to funnel passengers to large airports, Southwest is focused on point-to-point service, flying the same aircraft — Boeing Co. 737s — on trips that may hopscotch around the US.With a hub and spoke system, all the planes go from A to HUB and then from the HUB to somewhere. If the route A-HUB gets saturated, they can put more planes on that, and those planes can always be found at the HUB. This applies to crews too.You'll have something that looks like this: <a href="https://www.airlineroutemaps.com/maps/Delta_Air_Lines/North_America/Detroit" rel="nofollow">https://www.airlineroutemaps.com/maps/Delta_Air_Lines/North_...</a>This comes at the cost of having oversupply at some spots and its harder to offer the "ideal" routes as everyone needs to transfer to another plane with a layover somewhere... and your baggage is more likely to get lost. There's a bit not to like as a passenger on such an airline unless it's a nice one leg route - but then who wants to fly to Detroit?Southwest is different - they go from anywhere they want to anywhere they want with non-stop flights and picking the most lucrative routes they can. This lowers the effective cost per flight and are likely non-stop flights. Everything that a customer wants.Southwest routes from 2001 <a href="https://www.flickr.com/photos/erussell1984/15863298679" rel="nofollow">https://www.flickr.com/photos/erussell1984/15863298679</a> - you can see the lack of hubs there.This is done through some crafty constraint programming to try to make sure that all the capacity is where it needs to be when it needs to be there.However, when the capacity hits "holidays - everything is at max", along with "big storm prevents flights from going to where they need to be for the next leg" this system breaks down and planes and crews are out of position or need to sleep. Their software was able to handle this constraint system when it was a smaller company with fewer routes - there were fewer constraints.The "reset" is not "shut down the computers and start them back up" but rather "let all the crews get their required sleep and then go to the spot where they need to be in order to handle the load - not where they currently are (out of position)".Hypothetically, this is solvable if you have enough compute... but that's a lot of compute that needs to be recomputed each time something changes (weather, crew gets sick, passenger load changes) and that ends up being impractical and expensive.---Related reading:NYT - What Caused the Chaos at Southwest : <a href="https://www.nytimes.com/2022/12/28/travel/southwest-airlines-flight-cancellations.html?unlocked_article_code=6VPiQilkeMys9acoeGC6vfMSS6_v012ZLqICDXYlXpz-XrxSRfoduoUKEGlC9I_Zd5iXUO4XuvVHOkMIkGQRb9_mtIoSx12rjtan4daGFRdI_ClhRDDy9Zuokp0yKBZ0vfl5eW9CQ02yQyqPkzWZzDk9SE5kIeNacg8UCitoZP_cL8Xfz8OM91TQ3udQiVpjObO2fPT9e2P1g7QOo8Nzz6krt4kjfmZ87lko5q7Vqx5dNPe8eNhJ82dV3qPPSU-xdCRb26f_XY05XMJH5SzQdMdEtnRkmYM__Btnkj2rYTePGsh9fD4o-aZZ-GwmEpZqldK-Eub8XqoHhnXw3mtjzyIZhdij_Q5mS3ZDGszHp-o&smid=share-url" rel="nofollow">https://www.nytimes.com/2022/12/28/travel/southwest-airlines...</a>WSJ - How Southwest Airlines Melted Down : <a href="https://www.wsj.com/articles/southwest-airlines-melting-down-flights-cancelled-11672257523?st=y31zwvcsw79hmd9&reflink=desktopwebshare_permalink" rel="nofollow">https://www.wsj.com/articles/southwest-airlines-melting-down...</a> --- <a href="https://news.ycombinator.com/item?id=34165791" rel="nofollow">https://news.ycombinator.com/item?id=34165791</a>

评论 #34173697 未加载

评论 #34174202 未加载

评论 #34174692 未加载

评论 #34173965 未加载

gitfan86超过 2 年前

From what I've read here and other places there was a unique situation of:1. Large holiday volume of flights AND a lot of cancelations due to weather happened at the same time. 2. Their software system was unable to recommend optimal next steps for flight crews because it had not been designed to take this specific situation into account.Other airlines also had problem #1. But since they mostly operate on a hub model it is easy to tell their flight crews where they should go once the weather gets better.

simne超过 2 年前

Disclaimer: I've read comments before write my own.It remembers me long history of failures of post-soviet passenger railroad booking system.They constantly have issues, it is just impossible reliable buy tickets when hot season.Situation becomes much better latest years (to be honest, we just don't see them), just because free market - railroads give up large share of load to other transport - to air, buses, private autos, and also large part of tourists go abroad.

评论 #34180888 未加载

_plg_超过 2 年前

Lorin Hochstein's take on it: <a href="https://surfingcomplexity.blog/2022/12/28/southwest-airlines-a-case-study-in-brittleness/" rel="nofollow">https://surfingcomplexity.blog/2022/12/28/southwest-airlines...</a>

n1b0m超过 2 年前

Unlike other airlines that operate directly to and from hub airports, Southwest tends to operate a point-to-point service, meaning that when one flight is disrupted, there may not be spare aircraft and crews to pick up the route, leading to disruptions through the scheduling chain

Tempest1981超过 2 年前

So how do other airlines track where their planes are? Radio? ADS-B?How about their crews? In-house mobile-phone app?The Southwest telephone-based system sounds super low-tech. AirTags might work better.

评论 #34175222 未加载

mikewarot超过 2 年前

How could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can?The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the fact.[Edit]Software helps optimize things, but as long as the crews and suppliers have sufficient faith that they will be treated fairly, you could run this all with paper and pen for a few days while the software gets straightened out.Decide which hubs are most important, and have someone work out what goes where by hand. Once those routes are up and running, you can work outward to the less trafficked airports. Just work on getting the most people to the places that help the most.Crews know how many hours they've worked, and can track that themselves for the moment, or perhaps the Captain could do that. Everyone has cell phones, and could route around this damage.The consensus here on HN seems to be to give in without trying, which I find disturbing.

评论 #34173813 未加载

评论 #34173969 未加载

评论 #34174384 未加载

评论 #34173651 未加载

评论 #34173574 未加载

评论 #34174374 未加载