The iPhone 15 Pro’s Depth Maps

Other commenters here are correct that the LIDAR is too low-resolution to be used as the primary source for the depth maps. In fact, iPhones use four-ish methods, that I know of, to capture depth data, depending on the model and camera used. Traditionally these depth maps were only captured for Portrait photos, but apparently recent iPhones capture them for standard photos as well.<p>1. The original method uses two cameras on the back, taking a picture from both simultaneously and using parallax to construct a depth map, similar to human vision. This was introduced on the iPhone 7 Plus, the first iPhone with two rear cameras (a 1x main camera and 2x telephoto camera.) Since the depth map depends on comparing the two images, it will naturally be limited to the field of view of the narrower lens.<p>2. A second method was later used on iPhone XR, which has only a single rear camera, using focus pixels on the sensor to roughly gauge depth. The raw result is low-res and imprecise, so it&#x27;s refined using machine learning. See: <a href="https:&#x2F;&#x2F;www.lux.camera&#x2F;iphone-xr-a-deep-dive-into-depth&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.lux.camera&#x2F;iphone-xr-a-deep-dive-into-depth&#x2F;</a><p>3. An extension of this method was used on an iPhone SE that didn&#x27;t even have focus pixels, producing depth maps purely based on machine learning. As you would expect, such depth maps have the least correlation to reality, and the system could be fooled by taking a picture of a picture. See: <a href="https:&#x2F;&#x2F;www.lux.camera&#x2F;iphone-se-the-one-eyed-king&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.lux.camera&#x2F;iphone-se-the-one-eyed-king&#x2F;</a><p>4. The fourth method is used for selfies on iPhones with FaceID; it uses the TrueDepth camera&#x27;s 3D scanning to produce a depth map. You can see this with the selfie in the article; it has a noticeably fuzzier and low-res look.<p>You can also see some other auxiliary images in the article, which use white to indicate the human subject, glasses, hair, and skin. Apple calls these portrait effects mattes and they are produced using machine learning.<p>I made an app that used the depth maps and portrait effects mattes from Portraits for some creative filters. It was pretty fun, but it&#x27;s no longer available. There are a lot of novel artistic possibilities for depth maps.

Cool article. I assume these depth maps are used for the depth of field background blurring &#x2F; faux bokeh in &quot;Portrait&quot; mode photos. I always thought it was interesting you can change the focal point and control the depth of field via the &quot;aperture&quot; <i>after</i> a photo is taken, though I really don&#x27;t like the look of the fake bokeh. It always looks like a bad photoshop.<p>I think there might be a few typos of the file format?<p>- 14 instances of &quot;HEIC&quot;<p>- 3 instances of &quot;HIEC&quot;

There’s Reality Composer for iOS which has a LIDAR-enabled specific feature allowing you to capture objects. I was bummed to find out that on non-LIDAR equipped Apple devices it does not in fact fall back to photogrammetry.<p>Just in case you were doing 3d modeling work or photogrammetry and wanted to know, like I was.

Yes, those depth maps + semantic maps are pretty fun to look at - and if you load them into a program like TouchDesigner (or Blender or Cinema 4D whatever else you want) you can make some cool little depth effects with your photos. Or you can use them for photographic processing (which is what Apple uses them for, ultimately)<p>As another commenter pointed out, they used to be captured only in Portrait mode, but on recent iPhones they get captured automatically pretty much whenever a subject (human or pet) is detected in the scene.<p>I make photography apps &amp; tools (<a href="https:&#x2F;&#x2F;heliographe.net" rel="nofollow">https:&#x2F;&#x2F;heliographe.net</a>), and one of the tools I built, Matte Viewer, is specifically for viewing &amp; exporting them: <a href="https:&#x2F;&#x2F;apps.apple.com&#x2F;us&#x2F;app&#x2F;matte-viewer&#x2F;id6476831058">https:&#x2F;&#x2F;apps.apple.com&#x2F;us&#x2F;app&#x2F;matte-viewer&#x2F;id6476831058</a>

LIDAR itself has much much lower resolution that the depth maps shown. It has to be synthesized from combined LIDAR and regular camera data.

I might be missing something here but the article spends quite a bit discussing the HDR gain map. Why is this relevant to the depth maps? Can you skip the HDR gain map related processing but retain the depth maps?<p>FWIW I personally hate the display of HDR on iPhones (they make the screen brightness higher than the maximum user-specified brightness) and in my own pictures I try to strip HDR gain maps. I still remember the time when HDR meant taking three photos and then stitching them together while removing all underexposed and overexposed parts; the resulting image doesn&#x27;t carry any information about its HDR-ness.

I am waiting for a day when all phone hardwares defaulting to Gaussian splatting to take 3d images without expensive sensors. It may be computationally expensive but probably cheaper than adding expensive sensors and adding more weight.

Aha! I wonder if Apple uses this for their “create sticker” feature, where you press a subject on an image and can extract it to a sticker, or copy it to another image.

Just wonder if depth maps can be used to generate stereograms or SIRDS. I remember having playing with stereogram generation starting from very similar grey-scaled images.

Every time I glance at the title my brain reads “death maps”

Christ, that liquid cooled system is totally overkill for what he does. I&#x27;m so glad I don&#x27;t bother with this stuff anymore, all to run his preferred operating system in virtualization because Windows uses his aging Nvidia card better<p>Chimera<p>The old gpu is an aberration and odd place to skimp. If he upgraded to a newer nvidia gpu it would have linux driver support and he could ditch windows entirely<p>And if he wasn’t married to arcgis he could just get a mac studio

site does something really strange on iOS chrome - when I scroll down on the page the font size swaps larger, when I scroll up it swaps back smaller. Really disorienting<p>Anyways, never heard of oiiotool before! Super cool

Off the topic at hand but this site is elegantly simple... I wonder what static site generator he uses?

You can make autostereograms from those.

Cool article. I read the title as &#x27;Death Maps&#x27; at first though.

Truedepth from FaceID since iphone 13 got significantly worse - its very bumpy and noisy - we had to do significant denoising and filtering to make it useful again for 3d scanning<p>Lidar is a let down. First I would expect that Lidar would trickle down to non-pro devices. Come on apple FaceID got introduced in iphone X and next year it was in all iphone models. Lidar was introduced in iphone 12 pro and still only pro devices have it. As 3rd party dev it makes me reluctant to make any app using it if it limits my user base by 50%.<p>I&#x27;m also disappointed they didn&#x27;t improve FaceID or Lidar in the last ~5 years (Truedepth still only 30fps, no camera format to mix 30fps depth + 120fps rgb, still big latency, Lidar still low resolution, no improvement to field of view)

anyone combining these with photos for feeding to gpt4o to get more accurate outputs (like for calorie counting as a typical example)?

&gt; *describes a top of the line system<p>&gt; I&#x27;m running Ubuntu 24 LTS via Microsoft&#x27;s Ubuntu for Windows on Windows 11 Pro<p>this is like hearing someone buying yet another automatic super car.