Background Features in Google Meet, Powered by Web ML

Sony released some software a couple of months ago that lets you use most of their DSLRs as webcams with USB. My goodness, paired with a fast lens, what a difference to my MacBook webcam, even with these ml blurred backgrounds!<p>It&#x27;s only 720p and around 15fps but real shallow dof, very little sensor noise, autofocus works. Well worth trying if you have a Sony camera from the last few years.<p>Sensor size and good optics still wins. Having said that,the effort and detail gone into this feature is very impressive, enjoyed the blog post. Also webassembly SIMD looks super cool, looking forward to a new class of webapps using wasm.

Having used both Zoom and Meet extensively now for the past 6 months, my experience is:<p>1&#x2F; Your internet connection, especially upload bandwidth and latency matter a lot.<p>2&#x2F; Zoom&#x27;s desktop app performs very well, but its web version is atrocious. Not just because of the dark patterns they use to force you to install the desktop app, but also its performance is terrible compared to its desktop version, as well as worse than almost everything else. Unfortunately, I don&#x27;t trust them and refuse to use their desktop app on anything but my iPad.<p>3&#x2F; Meet used to be bad like Zoom on web 6 months ago, but has improved a lot and is slowly approaching Zoom desktop in performance. I have noticed that Meet on my work GSuite calls at work perform much better than on my personal account. This might be explained by #1 above I.e. my family has worse internet connections than my coworkers, but I am not sure if all improvements have been rolled out to personal accounts.

The example video clips in the post look nothing like me and my team&#x27;s view when using the new feature. Most of the time half of our hair gets blurred or replaced and hand gestures will cause either our hands or head to disappear.

Aside: Imagine you’re driving down the road and you need to make a right turn. Well, for some reason the steering wheel is stowed away and disappeared! You need to hover your hand around the center console in a specific area to be able to expose it. Out comes the steering wheel and now you can make a right turn.<p>Google UX&#x2F;UI team: Please fucking make the mute&#x2F;unmute button visible at all times.

Happy to see ML become mainstream. In the future, I don&#x27;t think ML will be a separate field of programming. It&#x27;ll just be &quot;programming,&quot; the same way webdev is.<p>There&#x27;s a tendency to think of ML as &quot;not programming,&quot; or something other than just plain programming. But as the tooling matures, that&#x27;ll go away.<p>(Lisp used to be considered &quot;AI programming,&quot; till it became useful in many other contexts.)

Interesting that this post made it to #1. It seems like Google marketing trick.<p>Anyone who uses the blue realizes that it&#x27;s far lacking in quality from other offerings and Google Meet UI is very bad also.<p>Zoom, Teams, even WebEx are superior quality and usability wise.

You all notice that this is a PR piece to get tech people interested in using google meet instead of zoom right.

Too bad it doesn&#x27;t seem to be supported in Firefox.

Why are people replacing their backgrounds?<p>I thought the whole point of having a video call is to see who you are talking to, and their environment to further enhance the effectiveness of the conversation.<p>If you are in your kitchen, or under a tree, I definitely would like to see that because that environment will have an effect on how we communicate.

I don’t understand this part:<p>&gt; In the current version, model inference is executed on the client’s CPU for low power consumption and widest device coverage.<p>Naively I would think model inference done server side would have the lower CPU power (from the client point of view) and widest device coverage (client does nothing more), what am I missing ?

I wish my coworkers would stop using background blur.<p>It sucks and it’s distracting.<p>Your hair and hands pop in and out of blur. Sometimes part of your face will blur.<p>I don’t care if your workspace is messy or your kid walks in the room. I do care that we’re all being distracted by your weirdly blurred hair and hands.

Google finally catching up to where Zoom was two years ago.<p>Can we get a mute button visible at all times before 2024?

They mention SIMD support, but It&#x27;s unclear to me in what capacity the GPU is leveraged. The hair segmentation example on the MediaPipe webpage suggests it&#x27;s evaluating the graph on the GPU though.

It would be nice if there was a webcam on the market that took actual lenses so you could get free, legit depth of field. Paying $700 for a used DSLR that has a clean hdmi out is not appealing, especially when I have a mirrorless from the same company that could probably do the same with a firmware update (that will never come)

Wtf happened to hangouts? How many video products does google have?

Blur is awesome. Way better, less distracting, than backgrounds. Everyone who try it uses it permanently because it works so well.<p>I also think it makes the subject look better for some reason.

The single biggest missing feature compared to Zoom for my team is background noise cancellation. It&#x27;s an unfortunate decision to limit it to Enterprise users.

Not supported in Firefox or Safari.

I guess this is why when I open Google Meet my fan starts spinning and making noise.

I was going to point out that xnnpack was basically created by a single guy who also created qnnpack, and how amazing it is for the work of a single guy to have so much impact, then I realized he posted it! Congratz dude!

As an aside, this example looks faked:<p><a href="https:&#x2F;&#x2F;1.bp.blogspot.com&#x2F;-viEA4OY0sxA&#x2F;X5s7IBwoXOI&#x2F;AAAAAAAAGv0&#x2F;nBYk9Nzxbc4Q_YEU6Ao2noAVCS4Ov9naACLcBGAsYHQ&#x2F;s960&#x2F;image5.gif" rel="nofollow">https:&#x2F;&#x2F;1.bp.blogspot.com&#x2F;-viEA4OY0sxA&#x2F;X5s7IBwoXOI&#x2F;AAAAAAAAG...</a><p>As in, the blurred background looks totally different (light:dark, shapes, etc.) to the unblurred background.<p>(I get that they’d need to do something funky to show blurred and unblurred backgrounds with the same foreground video, and faking it is likely easier than doing it programmatically, but this is just odd&#x2F;sloppy.)

Very awesome!<p>Although there&#x27;s a lot of blurring on the shoulder of the guy at the beach: <a href="https:&#x2F;&#x2F;i.imgur.com&#x2F;D5ueGUh.png" rel="nofollow">https:&#x2F;&#x2F;i.imgur.com&#x2F;D5ueGUh.png</a>

If you have a Windows computer with a RTX graphic card, you can use nvidia broadcast to get similar perks. It creates a virtual camera that you can select in whatever conference apps&#x2F;browsers you are using.<p>There are some works on OBS to get the green screen AI working, so I hope we will get that on GNU&#x2F;Linux one day.

The listed CPU usage &#x2F; elapsed time for the features in this article is obscene. Only 62FPS = maxing out at least one core on a 60hz display, just to replace&#x2F;blur a background. Kiss your laptop&#x27;s battery goodbye. How is this worth it?

Why isn&#x27;t Mediapipe built on gstreamer? Nvidia gets this right. If you&#x27;re slinging frame buffers around, use an API that there is already an ecosystem for.

A few people commented that the foreground&#x2F;background detection cannot keep up with movements fast enough. Here&#x27;s an idea that might help, although I&#x27;m not sure if it can realistically be done:<p>When the video is encoded, the codec does motion estimation (among other things) to reduce the bandwidth required. So why don&#x27;t we use the motion vectors from the video codec to modify the foreground&#x2F;background mask in real time? Obviously this is going to create weird artifacts pretty soon, but it might just be good enough for a few frames before the ML model produces another accurate mask.

I have a different issue with Google Meet.<p>I have observed in the last couple months that whenever I create a Google Calendar invite with others, Google has started <i>inserting</i> a Google Meet conference as the location to meet.<p>It was one thing to ask&#x2F;offer this as an option if you&#x27;d like to use it, but now Google is positioning it as if you had chosen that. So if you left it empty, because you usually use some other understood method with your friends&#x2F;colleagues, now your participants are confused and think you wanted to use Google Meet.<p>I think that&#x27;s going too far to get people to adopt your product.

From the title I thought this was two distinct features running in google&#x27;s background that used Web ML to figure out how to work together.

I wish one of the lovely people in the examples were wearing headphones.

Badly.

Honestly it&#x27;s the worst implementation I&#x27;ve seen of this technology yet. Just absolute and total garbage.

What would be the inference time like on a modern smartphone?

It’s funny how Google pours time into things like this but the last person I know who uses a Google chat product just stopped because it’s less reliable than Zoom. Losing 15 minutes with someone trying to get the sound working counts more than a gimmick many people never notice, not to mention now even normal people don’t want to yet install another app because they expect it to be cancelled soon.