This report presents the initial findings of Microsoft’s research initiative on “AI and Productivity”, which seeks to measure and accelerate the productivity gains created by LLM-powered productivity tools like Microsoft’s Copilot. The many studies summarized in this report, the initiative’s first, focus on common enterprise information worker tasks for which LLMs are most likely to provide significant value. Results from the studies support the hypothesis that the first versions of Copilot tools substantially increase productivity on these tasks. This productivity boost usually appeared in the studies as a meaningful increase in speed of execution without a significant decrease in quality. Furthermore, we observed that the willingness-to-pay for LLM-based tools is higher for people who have used the tools than those who have not, suggesting that the tools provide value above initial expectations. The report also highlights future directions for the AI and Productivity initiative, including an emphasis on approaches that capture a wider range of tasks and roles.

Studies I led that are included within this report:

• Measuring the Impact of AI on Information Worker Productivity
• Sound Like Me: Findings from a Randomized Experiment
• Randomized Controlled Trial for Microsoft Security Copilot

We conducted randomized controlled trials (RCTs) to measure the efficiency gains from using Security Copilot, including speed and quality improvements. External experimental subjects logged into a M365 Defender instance created for this experiment and performed four tasks: Incident Summarization, Script Analyzer, Incident Report, and Guided Response. We found that Security Copilot delivered large improvements on both speed and accuracy. Copilot brought improvements for both novices and security professionals.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “Role-specific pain points and opportunities: Security.” Also summarized in AI and Productivity Report at “M365 Defender Security Copilot study.”)

A new version of Copilot for Microsoft 365 includes a feature to let Outlook draft messages that “Sound Like Me” (SLM) based on training from messages in a user’s Sent Items folder. We sought to evaluate whether SLM lives up to its name. We find that it does, and more. Users widely and systematically praise SLM-generated messages as being more clear, more concise, and more “couldn’t have said it better myself”. When presented with a human-written message versus a SLM rewrite, users say they’d rather receive the SLM rewrite. All these findings are statistically significant. Furthermore, when presented with human and SLM messages, users struggle to tell the difference, in one specification doing worse than random.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “Email effectiveness.” Also summarized in AI and Productivity Report at “Outlook Email Study.”)

This paper reports the results of two randomized controlled trials evaluating the performance and user satisfaction of a new AI product in the context of common information worker tasks. We designed workplace scenarios to test common information worker tasks: retrieving information from files, emails, and calendar; catching up after a missed online meeting; and drafting prose. We assigned these tasks to 310 subjects tasked to find relevant information, answer multiple choice questions about what they found, and write marketing content. In both studies, users with the AI tool were statistically significantly faster, a difference that holds both on its own and when controlling for accuracy/quality. Furthermore, users who tried the AI tool reported higher willingness to pay relative to users who merely heard about it but didn’t get to try it, indicating that the product exceeded expectations.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “A day in the life” and “The strain of searching.” Also summarized in AI and Productivity Report at “Copilot Common Tasks Study” and “Copilot Information Retrieval Study.”)

My promotion to tenure was derailed by improper disciplinary proceedings — a kangaroo court that ran roughshod over the governing rules. For example, the rules require the disciplinary committee to share with me (and readers of its report) “the evidence gathered.” Far from providing evidence, the 2017 report attached zero emails, zero transcripts of interviews, and zero other documents. Instead of providing evidence, the committee offered mere summaries of twelve anonymous interview remarks, with both names and contexts intentionally removed, in brazen violation of the requirement to provide “the evidence gathered.” That’s not justice, and it’s clearly not permitted under the applicable rules.

I’m suing Harvard to insist that these proceedings be corrected, in conformance with the rules. I’m not perfect — who is? — but if the proceedings follow the rules, I will clear my name of the incorrect charges, and my candidacy can then be evaluated on its merits.

Case web site including complaint and other documents

Status: Pending.

Summary: Federal regulation favors private resolution of disputes between passengers and airlines. But Southwest’s “class waiver” disallows passengers from gathering together with a single set of lawyers and experts for efficient, cost-effective group resolution of their complaints.

Status: Pending.

Summary: Governing regulation requires an airline to provide the exact price for a passenger’s first and second checked bag within the text of an eticket confirmation email, but Alaska did not do so. Furthermore, the regulation requires bag allowance and price information in a booking summary page, but again Alaska did not. Meanwhile Alaska’s Manage Trip page provided an incorrect statement of baggage benefits and fees.

Status: Pending.

Summary: The American Airlines Business Extra site misrepresented carrier surcharges as “tax” in violation of governing regulation and prior DOT consent decrees. Furthermore, the site listed “approx” charges rather than the exact amount to be paid. And contrary to governing regulation, the site entirely omitted carrier surcharges from initial fare quotes.

Status: Pending.

Summary: The American Airlines Vacations site misrepresented carrier surcharges as “tax” in violation of governing regulation and prior DOT consent decrees.

Status: Pending

Summary: Governing regulation requires an airline to refund a passenger’s TSA fee if the passenger does not travel, but Delta refused to do so.