Reprojecting the Perseverance landing footage onto satellite imagery

An interesting side point is that the graph optimization approach used here is somewhat similar to modern graph-based visual SLAM<p>The graph in the article can be seen as a factor graph. VSLAM systems usually have a (kind of bipartite) factor graph with vertices&#x2F;variables that are either keyframes or ‘good’ features, with edges&#x2F;factors between features and the frames that see them <i>and</i> between adjacent frames; each of these edges are the factors in the graph. This structure results in very large but sparse graphs, and there are factor graph optimization libraries that take advantage of this sparsity (e.g. g2o or GTSAM.) These libraries also use specialized optimization techniques for some of the nonlinear manifolds (e.g. SO(3)) that arise in SLAM problems.

Not only impressively coded but a beautiful result as well fascinating to have that real time, frame by frame comparison. Great job!<p>It only reinforces that I really need to learn my matrix math.

Pix4D made a 3D reconstruction:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=20wCGKTpOJw" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=20wCGKTpOJw</a>

Fascinating and well explained!<p>Reminded me of the video tracking work these folks do: <a href="https:&#x2F;&#x2F;forensic-architecture.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;forensic-architecture.org&#x2F;</a>

It&#x27;d be pretty neat to lift this up into 3d - you could probably reverse the transforms to find the camera pose for each frame, then drop it into a scene alongside the camera frustum and the topography so we can see exactly how much steering the descent stage did to hit its target and how fast it was descending at every stage.

The graph based approach is interesting... But I wonder if better and far simpler results might be had by simply using a few iterations of optical flow to perfect the alignment of each frame starting from the alignment of the previous frame?<p>As a benefit, the transformation could use images after being projected onto a deformable mesh to model the hills etc.

I love that you can see the approach angle in the distortion of the field. It also helps to convey how thin the atmosphere is to see how long it takes for that to square up.

I&#x27;ve done this kind of stuff through a point and click UI in GIS software. It&#x27;s really cool seeing a lot of the underlying math and concepts laid out like this.

Very impressive.<p>A next step could be to leave the already projected images where they are, and only draw over them, while marking the latest frame with a border. Eventually use frame sections which cover multiple frames to perform multi-frame superresolution.

Beautiful!<p>Extra kudos to the author for not calling the work done in Torch &quot;learning&quot;.

Did Scott Manley do something similar with the Change 5 landing footage on the Moon.<p><a href="https:&#x2F;&#x2F;youtu.be&#x2F;lwLPzU8H3HI" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;lwLPzU8H3HI</a>

FYI open cv has gradient based image alignment built in, findTransformECC.<p><a href="https:&#x2F;&#x2F;docs.opencv.org&#x2F;3.4&#x2F;dc&#x2F;d6b&#x2F;group__video__track.html#ga7ded46f9a55c0364c92ccd2019d43e3a" rel="nofollow">https:&#x2F;&#x2F;docs.opencv.org&#x2F;3.4&#x2F;dc&#x2F;d6b&#x2F;group__video__track.html#...</a>

OP should try out SuperGlue for features instead of SIFT

Excellent post! I wonder why SIFT didn&#x27;t find sufficient keypoints early on, it&#x27;s typically a beast of a method for such a task. It looks like there&#x27;s some intensity variation, the satellite image is darker, but I&#x27;m not sure that would explain it all.

I&#x27;ve been having trouble finding the answer to this. How close to it&#x27;s intended target did it land?<p>Thanks

interesting! couldn&#x27;t you do it with blender&#x27;s tracking without python? although it&#x27;s much more impressive with python.