Plane Tests Must Use Average Pilots, NTSB Says After 737 Max Crashes

This seems like a good thing, they should be testing with people who don’t know the answers. Though I can’t help but wonder if this is a very subtle re-framing of the Max accidents to shift blame back towards pilots. This approach of a regulatory agency recommending “average pilot” testing makes both the agency and Boeing look good, and tends to downplay the fact that Boeing intentionally avoided training the pilots, and that the system was only recoverable if you had some extremely specific knowledge. This wasn’t an issue of the quality of the pilots, this was an issue of the quality of the training, which we know was kept from all pilots to attempt to save time and money.

Uh oh. What are they going to do when they discover that there aren&#x27;t any average pilots?<p><a href="https:&#x2F;&#x2F;www.thestar.com&#x2F;news&#x2F;insight&#x2F;2016&#x2F;01&#x2F;16&#x2F;when-us-air-force-discovered-the-flaw-of-averages.html" rel="nofollow">https:&#x2F;&#x2F;www.thestar.com&#x2F;news&#x2F;insight&#x2F;2016&#x2F;01&#x2F;16&#x2F;when-us-air-...</a> (HN discussion: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11230287" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11230287</a>)<p><a href="https:&#x2F;&#x2F;www.theatlantic.com&#x2F;business&#x2F;archive&#x2F;2016&#x2F;02&#x2F;the-invention-of-the-normal-person&#x2F;463365&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.theatlantic.com&#x2F;business&#x2F;archive&#x2F;2016&#x2F;02&#x2F;the-inv...</a> (HN discussion: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11155889" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11155889</a>)

Actually they must use the worst pilots. If they use average pilots that means 50% of the pilots might be out of their depth flying that plane. What kind of insanity is this?

The primary design objective of the 737 max was to avoid pilot retraining. Until that stops being a major economic driver in aviation no problem was solved here.<p>Changing MCAS another dozen times and updating the manuals does nothing to prevent this from happening in new future system design.

I&#x27;d say that&#x27;s a clickbait title. The actual report is at: <a href="https:&#x2F;&#x2F;www.ntsb.gov&#x2F;investigations&#x2F;AccidentReports&#x2F;Reports&#x2F;ASR1901.pdf" rel="nofollow">https:&#x2F;&#x2F;www.ntsb.gov&#x2F;investigations&#x2F;AccidentReports&#x2F;Reports&#x2F;...</a>, and framing it as a recommendation to change the test procedure to use &#x27;average&#x27; pilots seems like a big liberty.<p>To me, the thrust of the article is on making sure human factors are incorporated into flight certification process. As they highlight, although uncommanded MCAS input was part of the certification tests, it was only marked as a &#x27;major&#x27; risk rather than more severe classifiers. This wasn&#x27;t because of any phenomenal test pilots, but just because the FAA (rather than Boeing) guidelines behind the certification assume immediate assessment and diagnosis of the issue.<p>The report goes into a lot of depth about other human factors, for example, how transport-category planes should have straightforward highlighting of the primary issue since often the primary cause bubbles through various interconnected systems, and the main pilot response tends to be to identify the root cause under stress and then having to fix it.<p>To the extent I could re-summarize the report, I&#x27;d say &quot;Plane Certification Processes Must Consider Real-World Scenarios And Factor Human Response to In-Flight Failures in Plane Design&quot;

The irony here is that when I wrote &quot;The Tyranny of the Minimum Viable User&quot; I&#x27;d specifically excluded cases such as commercial aviation, where a minimum qualifying standard can be used.<p>By that logic, aircraft certification should be based on the minimum qualifying pilot.<p><a href="https:&#x2F;&#x2F;old.reddit.com&#x2F;r&#x2F;dredmorbius&#x2F;comments&#x2F;69wk8y&#x2F;the_tyranny_of_the_minimum_viable_user&#x2F;" rel="nofollow">https:&#x2F;&#x2F;old.reddit.com&#x2F;r&#x2F;dredmorbius&#x2F;comments&#x2F;69wk8y&#x2F;the_tyr...</a>

Wouldn&#x27;t you need the worse pilot you can get? Average doesn&#x27;t account for the fact that 50% of the sample are WORSE.

I have 15 hours in a Cessna 172, I’d be happy to give it a go!<p>But in all seriousness, in no way shape or form should the testing be informed by the skill assessment of the test operator.<p>The analysis should be looking at the tasks that had to be performed during the test, whether they were properly identified by easily selected checklists, and quantitative measurements of the complexity and physical difficulty of the tasks.<p>If it comes anywhere close to a question of the skill or strength of the specific test-taker then something has gone terribly horribly wrong.<p>In old school UI testing you would do GOMS analysis which allowed calculating the difficulty of specific tasks based on the actual manipulations required to perform them. The skill of the particular operator is not generally a factor in these types of analysis.<p>The subtitle of TFA states;<p>&gt; <i>Safety board says FAA should embrace data-driven approach to assumptions about pilot responses</i><p>This is a much more reasonable statement. For instance, the maximum physical force required to perform the task must be an industry standard that pilots are all certified to be able to perform.

This makes a lot of sense. They need similar requirements for evacuation testing. Regulations say all passengers in a plane need to be able to evacuate within 90 seconds. As Americans get bigger and airplane seats get smaller the airlines &quot;comply&quot; with the 90 second requirement by conducting an evacuation test in completely unrealistic circumstances.[1] The test participants are all handpicked, fit, airline employees who prepare and practice for the drill. The test &quot;starts&quot; with everyone in their seat (seat backs &amp; try tables up) and with their life jacket already on.<p>And in recent years the FAA hasn&#x27;t required these tests as frequently. Why? Because even with all the preparation that goes into it, the mad-rush to the exits in a <i>staged</i> emergency, tends to be quite dangerous for the participants.<p>[1] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=XIaovi1JWyY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=XIaovi1JWyY</a>

I wonder if a bigger factor would be the test pilots knowing in advance what the test was.<p>Eg. being told, &quot;next we are testing what happens when MACS fails&quot;, whereas in a real life situation the pilot has to diagnose what has happened, recall the correct remedial action, then act on it.

Of course it should be. But the precedent was set in the early days of aviation when test pilots were flying experimental aircraft from alpha to v1. You don&#x27;t want to kill rookies who freak out in a flat spin, so you put your coldest mfer in the seat.<p>And then they do barrel rolls during the first public flight of your 707.<p>But now that we&#x27;ve got such accurate virtual cockpits, we don&#x27;t have to worry about killing pilots as much.

If you test for average pilots, then you&#x27;ve still got approx 50% of pilots who aren&#x27;t capable. Whatever happened to testing for the lowest common denominator - especially in life and death situations?

Those can be found in the spherical cow aisle.<p>I&#x27;m sure the intentions are good but shouldn&#x27;t the goal here be to use the <i>least</i> qualified, but still qualified pilots for testing? After all if the average pilots are passing the test but the least qualified, but still qualified pilots do not then it is a matter of time before the next crash. Boeing sells their planes all over the globe and I suspect that the average and the mean could be quite a ways apart and that the least good but still qualified pilot still sits comfortably below either.

Perhaps a portion of testing could be performed in the simulator using a random sample of line pilots?<p>The current advanced qualification program in use at the airlines kinda is set up to test operational policies. All pilots of a fleet (essentially a random sample of abilities) go into the sim once a year and perform a check on what’s considered a ‘normal’ flight in the sim where anomalies are presented without being pre-briefed and the crew must over come them just as if they were out flying the line. The data is aggregated and problematic trends are identified and corrected.<p>Translate the same to new aircraft certification. Get the airplane basically ready and train some pilots in the sim. Just regular line pilots who will be flying the aircraft for their airline. Then spend an appropriate amount of time giving them realistic scenarios (without pre-briefing the anomalies) that test the boundaries of the aircraft performance and see how the human&#x2F;machine interaction occurs and if there are problems that need to be addressed. In the case of the MAX, MCAS was a new system so scenarios to test pilot interaction with it would’ve been a requirement and I’m sure these issues could’ve been identified.<p>This already happens to some degree when airlines take delivery of a new aircraft. They make proving runs to hammer out any small bugs and get good real world data on fuel usage, performance, etc. My proposal is simply to expand the scope of proving runs in the sim with a good dose of abnormal operations using normal every day line pilots.

Sounds like FAA &amp; Boeing are trying to place blame on pilot error for the MAX crashes, to deflect attention from the MCAS issue and the FAA&#x27;s failure to notice it. I agree with the conclusions as the majority of aircraft accidents are pilot error, but in the case of MAX there is nothing the pilots could have done to save those aircraft.

Half of pilots are going to be below average - why not test against their ability?

Why not use a system akin to that of jury duty? If you fly X type of jet you&#x27;re in a pool of &#x27;random&#x27; testers for X jet as opposed to using data driven methods to shake out average pilots.<p>I&#x27;d have to assume the pilots in some fashion would be incentivised to not be chosen to test for fear of simply being an average pilot in the first place.

If Boeing needs a less than average pilot I can surely use the paycheck

&gt;Typically, overseas regulators follow the FAA’s lead. But after the MAX crashes revealed shortcomings in the FAA’s certification process, that’s no longer certain.<p>&gt;Before the MAX is cleared to fly passengers again, both EASA and the FAA will require flight tests of the new updated software. In addition, Ky said, EASA will require Boeing to demonstrate the stability of the jet in flight tests that include high-speed turn and stall maneuvers with MCAS switched off.<p>Source: <a href="https:&#x2F;&#x2F;www.seattletimes.com&#x2F;business&#x2F;boeing-aerospace&#x2F;european-aviation-safety-agency-sets-strict-demands-for-737-max-return-to-flight&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.seattletimes.com&#x2F;business&#x2F;boeing-aerospace&#x2F;europ...</a>

As a developer I never find any bugs when I test my programs

This is probably a good idea, using testers closer to your target audience should improve results a degree. However, I think this is kind of missing the point. Wasn&#x27;t the problem with the MCAS system on the max that the system was added to the airframe without re-qualification required? Shouldn&#x27;t the solution address how Boeing and the FAA qualify aircraft for qualifications, not how the aircraft itself was tested?

Make sure you read the actual report before commenting. The article is misrepresenting the recommendations: <a href="https:&#x2F;&#x2F;www.ntsb.gov&#x2F;investigations&#x2F;AccidentReports&#x2F;Reports&#x2F;.." rel="nofollow">https:&#x2F;&#x2F;www.ntsb.gov&#x2F;investigations&#x2F;AccidentReports&#x2F;Reports&#x2F;...</a>.<p>Also, kudos to the airline industry, their postmortem culture is second to none, even software engineering.

Regardless of the report and the article, I worked with the Boeing pilots for years.<p>On the plus side they always argued that emergency equipment in the cockpit should be able to be operated by almost any idiot off the street to ensure safety.<p>On the negative side they had special expeditions from the FAA that allowed them to skip mandatory training on emergency equipment that airline pilots had to take.

I live on the Kitsap Peninsula across the Puget Sound from Seattle. I enjoy tracking aircraft on a phone app that circle over us fairly low when approaching or taking off from SeaTac, Everett and Boeing Field. I recently spotted a 737Max and assumed it was part of some testing. Curious what phase they are in and how much in-flight testing is going on.

I&#x27;m immediately reminded of the whole thing with Captain Sully, where initial recreations in simulators could safely land the plane on the ground... but only because the pilots already knew exactly what to do, instead of having any realistic delays to evaluate the state of the plane and consider landing options.

That awkward moment when you are called in for a plane test and you realize you are mediocre.

If we assume the distribution of pilot talent for a Max 737 is symmetrical, then mean=median so the tests only require the better half of pilots land the plane.<p>Which is a pretty bad standard.<p>A better standard is that the 20% percentile commerical pilot can land the plane.

I&#x27;m not sure the criteria for determining an &#x27;average&#x27; pilot but if becoming a test pilot is a big incentive for trying to be the best you can wouldn&#x27;t this drag the average pilot quality down?

Seriously, what not below average pilots? Assuming an even distribution (yes, which it might not be) then roughly half the pilots are below average. If the bottom 25% can&#x27;t do it, that&#x27;s a problem, no?

Eek. How are they measuring average? Those with a median amount of hours under their belts? Those with average IQ? The term seems likely to cause a ruckus as who wants to be called average.

They need to establish minimum upper body strength qualifications, since MAX pilots have to be able to manually trim the horizontal stabilizer using the wheel and cable mechanical controls.

That actually strikes me as a really good insight&#x2F;improvement to come out of this mess.<p>Using the top 1% talent or whatever is sure to gloss over flaws that will bite avg pilots in the ass in real life

Shouldn&#x27;t they use pilots with the absolute minimal skills required to pass the ATP test and get into a 737 cockpit?<p>If you use average, that still leaves ~half the pilots to crash...

Good luck finding a pilot who will admit to being average!

Surely they mean “later in development”. Test pilots are chosen for their extreme skill precisely because a new aircraft isn’t proven safe.

The test should use a minimum viable pilot with the minimum required amount of sleep.<p>How many hours are pilots allowed to operate without deep sleep?

Surely there are only two types of pilots?<p>* Pilots<p>* Dead &#x2F; disqualified pilots

If they use average pilots, it still means that reactions of the other half of pilots will be worse than the ones they tested.

Like many others have pointed out, &quot;average&quot; is not bad enough. They need to be trying it with the worst pilots.

I would like even the bottom 0.1% of pilots to be able to recover the plane from a common hardware failure.

Let&#x27;s add to this: A plane should also be recoverable without usage of a manual by a trained pilot.

Test with an average pilot, but have an expert backup pilot on board who can take over, just in case.

No! Plane tests must use bad pilots. They must use tired pilots. They must use pilots under stress.

So test pilot at Boeing would not be something a pilot would want on their C.V in the future?

How about below-average pilots? Just-barely-good-enough-to-get-hired pilots?

Who is going to apply for the job of being an &quot;average&quot; pilot??

Out of all the unfortunate hyperbole from the 737 MAX (which I think fundamentally is a safe aircraft), this change will improve air safety across all airliners. Pilots should be held to the highest of standards, but assessment should be done with average joes.

average can meant many things even in stat. like modal ...<p>But if only 50% ...<p>Should it be &quot;normal&quot;, &quot;common&quot;, ... or most, not just professional test pilot

Shouldn’t tests use <i>below</i> average pilots?

NTSB apparently need better employees for making such a recommendation... (ie: it is not ok for 1&#x2F;2 of the pilots to not be 100% in control of their plane)

Please don’t. I am a multi engine rated, average pilot. Please don’t use someone like me as the basis for your safety.

What about below average pilots? If half of them are still going to crash, shouldn&#x27;t the true idiots in the industry be the test dummies? No pun intended.