> Fortunately, JPL knows exactly what went wrong.<p>> and while the flight uncovered a timing vulnerability that will now have to be addressed, it also confirmed the robustness of the system in multiple ways.<p>This is so important. Imagine if it would have crashed and damaged communications, that would have been the end of it. This extra step in engineering effort really payed off and gave Ingenuity a second life, and NASA a really good opportunity to learn important lessons and do further debugging.<p>Also, the article at mars.nasa.gov is a delight to read, with all the information one would like to know being in there. Not to mention that there is even a video for us to see.
This is the kind of failure mode that I imagine (now with hindsight) should be mitigated by (re)designing the system in such a way that the timestamp desync would always be detected because each part of the system should have its own internal model of the approximate times that it should be receiving from other systems and never blindly trust them.<p>I imagine it is sort of like the organisms early in evolution that simply believed their sensory system when it had in fact been hijacked by something that wanted to eat them. There aren't many sensory systems with those issues these days because anything that doesn't have an internal model that it can use for comparison winds up being a snack or crashing on an alien planet.
I feel this bug in my bones. I work a lot on WAMI/aerial imagery. I've dealt with this exact problem of timestamp-off-by-one-image. I have 9 cameras with different exposure times all triggered by the same hardware pulse. Fortunately my system is data-collect only so the worst outcome is some wrong filenames.<p>I learned only way too late in the project that most GigeVision cameras emit timestamps that can be synched using NTP and/or PTP. The camera specs/manual said nothing about this. Only found it by accident looking through the GeniCam XML API.<p>Moral of the story: ensure your cameras emit timestamps on device. Don't rely on system time.
Look at the quality of those images of the ground in that video. Those blades are whirring around at super-high rates to provide lift in the Martian atmosphere and taking 30 pictures per second, but in each exposure you can clearly see the shadow of the blades with very little blurring. And with ambient solar brightness quite a bit lower than Earth's "full sunlight", too. Those are quality cameras.
I don’t think the article mentions whether or not they can update the code remotely. Can they patch it?<p>Edit: appears so: <a href="https://astronomynow.com/2021/04/11/nasa-delays-mars-helicopter-flight-to-troubleshoot-test-glitch/" rel="nofollow">https://astronomynow.com/2021/04/11/nasa-delays-mars-helicop...</a>
Incredible! on so many levels.<p>My favorite is we got details. Legitimate details that give us a pretty clear picture where it went wrong and how it can be corrected. No calling it "software glitch" and stopping there. Those kinds of articles frustrate me to no end.<p>As a very small portion of the HN crowd, believe we are quite pleased.
Summary of the issue from another page;<p>"For the majority of time airborne, the downward-looking navcams takes 30 pictures a second of the Martian surface and immediately feeds them into the helicopter’s navigation system. Each time an image arrives, the navigation system’s algorithm performs a series of actions: First, it examines the timestamp that it receives together with the image in order to determine when the image was taken. Then, the algorithm makes a prediction about what the camera should have been seeing at that particular point in time, in terms of surface features that it can recognize from previous images taken moments before (typically due to color variations and protuberances like rocks and sand ripples). Finally, the algorithm looks at where those features actually appear in the image. The navigation algorithm uses the difference between the predicted and actual locations of these features to correct its estimates of position, velocity, and attitude.<p>Approximately 54 seconds into the flight, a glitch occurred in the pipeline of images being delivered by the navigation camera. This glitch caused a single image to be lost, but more importantly, it resulted in all later navigation images being delivered with inaccurate timestamps. From this point on, each time the navigation algorithm performed a correction based on a navigation image, it was operating on the basis of incorrect information about when the image was taken. The resulting inconsistencies significantly degraded the information used to fly the helicopter, leading to estimates being constantly “corrected” to account for phantom errors. Large oscillations ensued."
Here's a cool relevant podcast (from before flight 6 but gives some nice insights into what it takes to fly that Helicopter on Mars): [0]<p>Description:<p>Tim Canham, Mars Helicopter Operations Lead at NASA’s JPL joins us again to share technical details you've never heard about the Ingenuity Linux Copter on Mars. And the challenges they had to work around to achieve their five successful flights.<p>[0]: <a href="https://linuxunplugged.com/406" rel="nofollow">https://linuxunplugged.com/406</a>
It's incredible that the thing works at all- but to add the resilience that they seem to have accidentally built in (switching from visual at landing) and it surviving such a wild flight envelope is just incredible.<p>I bought a cheap drone on a common importing website and the thing likes to get lost even though it has GPS and is in the perfect clear. It flew into my truck, uncommanded, at full speed. On Earth. With GPS. So for this thing on another planet to do what it just did? Wow.
The article links to NASA but here it is anyways. Let me skim it and I might add some additional information here but go and check out the first picture there yourself - it is amazing. It could as well be an aerial photograph of a desert area on Earth by the looks of it.<p><a href="https://mars.nasa.gov/technology/helicopter/status/305/surviving-an-in-flight-anomaly-what-happened-on-ingenuitys-sixth-flight/" rel="nofollow">https://mars.nasa.gov/technology/helicopter/status/305/survi...</a><p>OK, so from the NASA article, it seems that they use an inertial measurement unit [0] (think an accelerometer/rotational sensor similiar to what your smartphone uses) that reads out heading and acceleration 500 times per second and sums it up to get the the helicopter's position. This is "dead reckoning" [1]) which unfortunately suffers from accumulating errors. To work around these accumulating errors, they sync their prediction to camera data thirty times per second by predicting how features visible on the surface should have moved in this time. So, sensor fusion [2], in a sense. The images <i>are</i> delivered with timestamps (I would have been surprised if they were not, to be honest) but one of the images went missing and for some reason, its timestamp was not. (Variable not cleared? Are they maybe using parallel queues to match images to timestamps (would be very weird, imo but I am no helicopter scientist?)). Either way, the following timestamps were now attributed to wrong images. I really wonder how that happened and hope they expand on that later. As the system predicts how far a feature in an image should have moved between frames, things were now considerably off (example: from the IMU prediction, a feature should have moved <i>1 unit per second</i> but with a missing frame and the previous timestamp, it now looks as if it moved <i>2 units in a second</i>) and the state gets updated with this wrong information.<p>Here is an interesting bit about safety margins from this article by the way:<p><i>Despite encountering this anomaly, Ingenuity was able to maintain flight and land safely [...]. One reason it was able to do so is the considerable effort that has gone into ensuring that the helicopter’s flight control system has ample “stability margin”: We designed Ingenuity to tolerate significant errors without becoming unstable, including errors in timing. This built-in margin was not fully needed in Ingenuity’s previous flights, because the vehicle’s behavior was in-family with our expectations, but this margin came to the rescue in Flight Six.</i><p>[0] <a href="https://en.wikipedia.org/wiki/Inertial_measurement_unit" rel="nofollow">https://en.wikipedia.org/wiki/Inertial_measurement_unit</a><p>[1] <a href="https://en.wikipedia.org/wiki/Dead_reckoning" rel="nofollow">https://en.wikipedia.org/wiki/Dead_reckoning</a><p>[2] <a href="https://en.wikipedia.org/wiki/Sensor_fusion" rel="nofollow">https://en.wikipedia.org/wiki/Sensor_fusion</a>
Because of using multiple, independent sensors (camera in the air, IMU near the ground) a disaster was averted. This is an important lesson: do not rely totally on one sensing modality. ( I wish Elon would learn this lesson: <a href="https://jalopnik.com/teslas-removing-radar-for-semi-automated-driving-on-mod-1846976679" rel="nofollow">https://jalopnik.com/teslas-removing-radar-for-semi-automate...</a> )
Very interesting problem.<p>I wonder, why not have a real time clock capture time stamps for different inputs for same frame, like one when written to memory, one when the trigger for the frame is done, etc and have a comparative analysis. Discard the process when there is a discrepancy, and re-start the captures from beginning, and go into a default hover state when such a fault is found.
I wonder what the shutter speed for the nav camera used in that GIF. They said the frames themselves are captured at 30Hz, but for the shadow of the rotors to be frozen without blurring, the shutter must be much shorter. Does anyone here know?
If you enjoy this route of analysis you might find NASA's public 'lessons leaned' database interesting.<p><a href="https://llis.nasa.gov/" rel="nofollow">https://llis.nasa.gov/</a>
It looks like the real issue is "we wrote all the camera code ourself rather than using battle tested libraries".<p>The android camera API for example will tag every frame with the timestamps and exposure settings it was captured with. Unit and integration tests can simulate dropped frames and verify the algorithm still runs as it should.<p>I would guess they tried to go for direct camera hardware access and ended up writing their own logic to schedule frames, and it contained a bug.