TL;DR - information workers complete simulated information work tasks much faster and with a higher quality of output when using generative AI based tools, with minimal impacts on quality. Experienced workers are less frustrated by mundane tasks, and inexperienced workers have their productivity accelerated.<p>Learning how to prompt effectively can be challenging. It is important to have strong user experience in the loop, to give users the ability to interrogate outputs and have some insight into uncertainty for outputs. For software engineering, the benefits of LLMs depend quite a bit on the task -- and AI is much more effective when breaking down large, complex tasks into more tractable subtasks.<p>There is a risk when using LLMs for education that students might use them as "steroids" rather than "coaches".<p>Nuggets from the slides:<p>• People took 37% less time on common writing tasks (Noy & Zhang 2023).<p>• BCG consultants produced >40% higher quality on one simulated consulting project (Dell’Acqua et al. 2023).<p>• Users were also 2x faster at solving simulated decision-making problems when using LLM-based search over traditional search (Spatharioti et al. 2023).<p>Study participants with Copilot completed experimenter-designed tasks in 26-73% as much time as those without it.
• A survey of enterprise users with access to Copilot also showed substantial perceived time savings.<p>• 73% agreed that Copilot helped them complete tasks faster, and 85% said it would help them get to a good first draft faster.<p>• Many studies found no statistically significant or meaningful effect on quality.<p>The study of M365 Defender Security Copilot found security novices with Copilot were 44% more accurate in answering questions about the security incidents they examined.<p>• Of enterprise Copilot users, 68% of respondents agreed that Copilot actually improved quality of their work.<p>• Users also reported tasks required less effort with Copilot. In the Teams Meeting Study, participants with access to Copilot found the task to be 58% less draining than participants without access.<p>• Among enterprise Copilot users, 72% agreed that Copilot helped them spend less mental effort on mundane or repetitive tasks.<p>• In studying the staggered rollout of a generative AI-based conversational assistant, Brynjolfsson et al. (2023) found that the tool helped novice and low-skilled workers the most. They found suggestive evidence that the tool helped disseminate tacit knowledge that the experienced and high-skilled workers already had.<p>As AI is applied to more generative tasks, human work is shifting to “critical integration” of AI output, requiring expertise and judgement (Sarkar 2023). Moving beyond just error correction, AI provocateurs would challenge assumptions, encourage evaluation, and offer counterarguments.<p>The concept of “microproductivity”, in which complex tasks are decomposed into smaller subtasks and performed in “micromoments” by the person most skilled to do so, can be enhanced through automation (Teevan 2016). For example, Kokkalis et al. (2013) demonstrated that high level tasks broken into multistep action plans through crowdsourcing result in people completing significantly more tasks (47.1% task completion) compared to the control condition of no plans (37.8%). These benefits were scaled by applying NLP algorithms to automatically create action plans for a larger variety of tasks based on a training set of similar tasks, and the plans were further refined through human intervention.