Microsoft patents tech to score meetings with body language, facial expressions

I like how thinly veiled the doublespeak is these days: “Let me be clear: Productivity Score is not a work monitoring tool,” he added. “Productivity Score ... focuses on actionable insights about the ways in which people and teams are using the tools so you can make improvements or provide training to further your digital transformation.”

My fear is that these kinds of tools won’t be used to gain more data and make better decisions, but that these tools will be used to gather all kinds of random data to justify firing decisions.<p>Gather enough types of data, you’ll look bad by one metric, and companies can justify firing anybody by pretending to cite the data.

The first thing that comes to mind is Hirevue[1] rating job candidates based on metrics like eye contact. Generally I see this product as automating the process of discriminating against people with autism and see it as deeply immoral.<p>[1] <a href="https:&#x2F;&#x2F;www.washingtonpost.com&#x2F;technology&#x2F;2019&#x2F;10&#x2F;22&#x2F;ai-hiring-face-scanning-algorithm-increasingly-decides-whether-you-deserve-job&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.washingtonpost.com&#x2F;technology&#x2F;2019&#x2F;10&#x2F;22&#x2F;ai-hiri...</a>

It will be interesting to see in what new ways this disproportionately negatively affects those on the autistic spectrum, or with other similar behavioural traits.

I helped build something kind of like this in college for a research team that was aimed at facilitating in-person arguments between 2+ people. It took speaking tone, body language, and facial expressions as input to judge how productive the conversation was in terms of resolving an argument, and would offer spoken interjections to steer the conversation toward more productive &#x2F; less aggressive paths.<p>It was significantly more effective when participants went in blind and didn&#x27;t know how it &quot;knew&quot; how they were feeling, but usually after 30-45 minutes of observation people would start to pick up on how it worked and often tried to manipulate it instead of actually trying to resolve the conflict.<p>Our application seemed good in theory, but worked horribly in practice. Hopefully Microsoft can figure something out that is net-beneficial in some scenarios without going full-dystopian.

I just finished watching the second season episode of The Good Place &quot;The Trolley Problem&quot; with my daughter, spoilers ahead if you haven&#x27;t watched to the end of season 1, so don&#x27;t go any further if that kind of thing is important to you:<p>.<p>.<p>.<p>.<p>in the show, Michael, a demon, tells Eleanor, a human, that he is a superior being that can see in 9 dimensions and because of his superior senses and understanding of the universe knows that she is about to fart and lie to try to cover it up.<p>so what I&#x27;m saying, I guess, is that Microsoft wants to build the Bad Place, and then try to convince us it&#x27;s the actual Good Place.

This is so stupid. Have these people not heard of Goodhart&#x27;s law?<p>&quot;Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.&quot;<p>Not only is the application of these surveillance technologies morally reprehensible, it&#x27;s also a complete joke from a performance standpoint. The only thing it will do is turn meetings into beauty contests. If you&#x27;ve ever asked yourself why congressional hearings of famous CEOs are such a clowncar, pay attention to the number of cameras in the room.

The one positive version of this I would like is if you could score anonymously. Then it might be nice to have an independent tool tell management certain meetings have been a waste of everyones time and nobody has been paying attention

It is just a matter of time before execs at megacorps use these arbitrary metrics from Microsoft to evaluate knowledge workers. Of course, there will be no differentiation between quality and quantity of employee interactions. And, of course, the execs themselves will not be subject to these metrics.

A meeting room filled with endless cameras and censors to record and judge every facial expression and body movement I make...<p>Do they purposefully make these patent diagrams look as dystopic as possible?

There seems to be a lot of employee surveillance technologies that are being developed and pushed by Microsoft. I recall them advertising what they have in mind for AI in the whole Office ecosystem and they were both depressing to think about from an employee perspective and also unsound from a technical perspective. I think employees should demand more transparency at least and know how their everyday behavior is being monitored and analyzed, especially as it relates to emails and messaging.

I experienced some of their meeting monitoring stuff several years ago on a visit to Microsoft Labs. I noticed the meeting score stuff and made a joke about how the meeting room was watching us. One of the MS people responded with &quot;Yes, it is.&quot; And then they showed us the &quot;telemetry&quot; the room was taking.. IDing us with facial recognition, guessing at our moods, etc. I definitely felt uncomfortable for the rest of the meeting.

It’s an arms race, hopefully a group of smart folks will open source a OBS (or similar) plugin to manipulate or completely mask minor facial expressions.

From the owners of GitHub and LinkedIn.<p>I’ve been wondering how long it will take them to “cross the streams,” then sell the result to employers.

So we have a Voight-Kampff test to find employees who don&#x27;t give a shit (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=25229291" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=25229291</a>)

I think there&#x27;s a valid conversation about how this is just a straight up dystopian bad idea. However, I think one thing also worth mentioning is that essentially everything we do on computers looks like this: We take something we know how to do, and we make the computer do it faster. We don&#x27;t know how to score meetings using body language and facial expressions - or atleast I would be incredibly sceptical of anyone who claimed they could do so without incredibly intimate knowledge of the people involved. Any manager will tell you there are people who are productive when talking, people who are productive when quiet, and people who are unproductive when talking or unproductive when quiet.<p>What seems likely is this system is going to invite extroverts to your meetings. My gregarious bizdev lead is going to be coming to our standups to boost our scores.<p>Let&#x27;s just lay aside the concerns about this being a dystopian nightmare, it&#x27;s also going to be crap.

If my company announced that they were going to use this, that announcement would be followed in fairly short order by a brain dump of all the stuff I&#x27;ve been working on for them and then my resignation effective immediately.

I am reading this after waking up... kinda wishing I did not wake up. What on Earth is going on? We, the people on whom they do this crap cannot (should not) tolerate this. Hopefully someone has ideas about what to do against this kind of shit other than not work for Microsoft or wherever this technology is used, like what is happening to those if this becomes widespread? We cannot all just expect others to not work. We need other solutions, and we kind of need them fast because it is getting out of hand. We frown at China, but this sort of thing is not any different than for what we frown at China, is it?

&quot;Policy&quot;-style debate[1][2]. What happens when metrics must be gamed to win.<p>State as many points as you can while gasping for breath. Coming to coworkers near you.<p>[1] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=0FPsEwWT6K0" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=0FPsEwWT6K0</a><p>[2] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=murvOaHB66A" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=murvOaHB66A</a>

&gt;The system uses cameras, sensors, and software tools to determine, for example, “how much a participant contributes to a meeting vs performing other tasks (e.g., texting, checking email, browsing the Internet).”<p>It implies your company is recording what&#x27;s going on your computer and on your phone, too, not just from body language and facial expressions!<p>So toxic&#x2F;dystopian.

Glad I work for a company that does not require the use of a Webcam, and I never ever turn my on for meetings

Don&#x27;t work at any company that utilizes tech like this. Brain drain.

So if the meeting score was low, does that mean that no useful decisions were made?<p>How is this better than just having a survey after every meeting?

I&#x27;m guessing that this <i>didn&#x27;t</i> come out of Microsoft Research?

As long as I can practice on the system to know what facial expressions to produce.