Category: Microsoft

Microsoft

Posted on

Early LLM-based Tools for Enterprise Information Workers Likely Provide Meaningful Boosts to Productivity

Early LLM-based Tools for Enterprise Information Workers Likely Provide Meaningful Boosts to Productivity. Microsoft Research Report – AI and Productivity Team. With Alexia Cambon, Brent Hecht, Donald Ngwe, Sonia Jaffe, Amy Heger, Mihaela Vorvoreanu, Sida Peng, Jake Hofman, Alex Farach, Margarita Bermejo-Cano, Eric Knudsen, James Bono, Hardik Sanghavi, Sofia Spatharioti, David Rothschild, Daniel G. Goldstein, Eirini Kalliamvakou, Peter Cihon, Mert Demirer, Michael Schwarz, and Jaime Teevan.

This report presents the initial findings of Microsoft’s research initiative on “AI and Productivity”, which seeks to measure and accelerate the productivity gains created by LLM-powered productivity tools like Microsoft’s Copilot. The many studies summarized in this report, the initiative’s first, focus on common enterprise information worker tasks for which LLMs are most likely to provide significant value. Results from the studies support the hypothesis that the first versions of Copilot tools substantially increase productivity on these tasks. This productivity boost usually appeared in the studies as a meaningful increase in speed of execution without a significant decrease in quality. Furthermore, we observed that the willingness-to-pay for LLM-based tools is higher for people who have used the tools than those who have not, suggesting that the tools provide value above initial expectations. The report also highlights future directions for the AI and Productivity initiative, including an emphasis on approaches that capture a wider range of tasks and roles.

Studies I led that are included within this report:

Posted on

Randomized Controlled Trials for Microsoft Copilot for Security updated March 29, 2024

Randomized Controlled Trials for Microsoft Copilot for Security. SSRN Working Paper 4648700. With James Bono, Sida Peng, Roberto Rodriguez, and Sandra Ho.

We conducted randomized controlled trials (RCTs) to measure the efficiency gains from using Security Copilot, including speed and quality improvements. External experimental subjects logged into a M365 Defender instance created for this experiment and performed four tasks: Incident Summarization, Script Analyzer, Incident Report, and Guided Response. We found that Security Copilot delivered large improvements on both speed and accuracy. Copilot brought improvements for both novices and security professionals.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “Role-specific pain points and opportunities: Security.” Also summarized in AI and Productivity Report at “M365 Defender Security Copilot study.”)

Posted on

Sound Like Me: Findings from a Randomized Experiment

Sound Like Me: Findings from a Randomized Experiment. SSRN Working Paper 4648689. With Donald Ngwe.

A new version of Copilot for Microsoft 365 includes a feature to let Outlook draft messages that “Sound Like Me” (SLM) based on training from messages in a user’s Sent Items folder. We sought to evaluate whether SLM lives up to its name. We find that it does, and more. Users widely and systematically praise SLM-generated messages as being more clear, more concise, and more “couldn’t have said it better myself”. When presented with a human-written message versus a SLM rewrite, users say they’d rather receive the SLM rewrite. All these findings are statistically significant. Furthermore, when presented with human and SLM messages, users struggle to tell the difference, in one specification doing worse than random.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “Email effectiveness.” Also summarized in AI and Productivity Report at “Outlook Email Study.”)

Posted on

Measuring the Impact of AI on Information Worker Productivity

Measuring the Impact of AI on Information Worker Productivity. SSRN Working Paper 4648686. With Donald Ngwe and Sida Peng.

This paper reports the results of two randomized controlled trials evaluating the performance and user satisfaction of a new AI product in the context of common information worker tasks. We designed workplace scenarios to test common information worker tasks: retrieving information from files, emails, and calendar; catching up after a missed online meeting; and drafting prose. We assigned these tasks to 310 subjects tasked to find relevant information, answer multiple choice questions about what they found, and write marketing content. In both studies, users with the AI tool were statistically significantly faster, a difference that holds both on its own and when controlling for accuracy/quality. Furthermore, users who tried the AI tool reported higher willingness to pay relative to users who merely heard about it but didn’t get to try it, indicating that the product exceeded expectations.

(Also summarized in What Can Copilot’s Earliest Users Teach Us About Generative AI at Work? at “A day in the life” and “The strain of searching.” Also summarized in AI and Productivity Report at “Copilot Common Tasks Study” and “Copilot Information Retrieval Study.”)

Posted on

An Introduction to the Competition Law and Economics of “Free” with Damien Geradin

Benjamin Edelman and Damien Geradin. An Introduction to the Competition Law and Economics of ‘Free’.  Antitrust Chronicle, Competition Policy International.  August 2018.

Many of the largest and most successful businesses today rely on providing services at no charge to at least a portion of their users. Consider companies as diverse as Dropbox, Facebook, Google, LinkedIn, The Guardian, Wikipedia, and the Yellow Pages.

For consumers, it is easy to celebrate free service. At least in the short term, free services are often high quality, and users find a zero price virtually irresistible.

But long-term assessments could differ, particularly if the free service reduces quality and consumer choice. In this short paper, we examine these concerns.  Some highlights:

First, “free” service tends to be free only in terms of currency.  Consumers typically pay in other ways, such as seeing advertising and providing data, though these payments tend to be more difficult to measure.

Second, free service sometimes exacerbates market concentration.  Most notably, free service impedes a natural strategy for entrants: offer a similar product or service at a lower price.  Entrants usually can’t pay users to accept their service.  (That would tend to attract undesirable users who might even discard the product without trying it.)  As a result, prices are stuck at zero, entry may be more difficult, effectively shielding incumbents from entry.

In this short paper, we examine the competition economics of “free” — how competition works in affected markets, what role competition policy might have and what approach it should take, and finally how competitors and prospective competitors can compete with “free.” Our bottom line: While free service has undeniable appeal for consumers, it can also impede competition, and especially entry. Competition authorities should be correspondingly attuned to allegations arising out of “free” service and should, at least, enforce existing doctrines strictly in affected markets.

Posted on

In Accusing Microsoft, Google Doth Protest Too Much

In Accusing Microsoft, Google Doth Protest Too Much. HBR Online. February 3, 2011.

Google this week sparked a media uproar by alleging that Microsoft Bing “copies” Google results. But is that actually the best characterization of what happened? In fact Google’s engineers intentionally clicked bogus listings they had previously inserted into Google’s results, and they did this on computers where they had specifically authorized Microsoft to examine their browsing in order to improve Bing.

Strikingly, Google’s own Matt Cutts previously endorsed the use of Toolbar and similar data to improve search results — calling this approach “a good idea.” And Google’s own Toolbar Privacy Policy allows Google to perform the same analysis Bing used. So I don’t have much sympathy for Google’s allegations of impropriety. Quite the contrary: With Bing’s small market share, this data is important in improving Bing search results and building a viable competitor to Google’s dominant search offering.

Details, including what exactly happened, Google’s prior statements, and Google’s widespread use of others’ intellectual property:

In Accusing Microsoft, Google Doth Protest Too Much

Posted on

Windows Vista (teaching materials)

Edelman, Benjamin. “Windows Vista.” Harvard Business School Case 909-038, February 2009. (Revised December 2010.) (educator access at HBP. request a courtesy copy.)

Microsoft designs, modifies, publicizes, and distributes Windows Vista—against a backdrop of consumers already largely satisfied with their existing Windows XP systems. Microsoft must decide what features to include and what to drop, how to compete with its own installed base, and how to mobilize partners to offer Vista-compatible systems.

Posted on

Microsoft adCenter (teaching materials) with Peter Coles

Coles, Peter, and Benjamin Edelman. “Microsoft adCenter.” Harvard Business School Case 908-049, January 2008. (Revised February 2010.) (educator access at HBP. request a courtesy copy.)

Microsoft considers alternatives to expand its presence in online advertising, especially text-based pay-per-click advertising. Google dominates, and it is unclear how Microsoft can grow, despite considerable technical and financial resources. Microsoft considers a set of alternatives, each with clear benefits but also serious challenges.

Teaching Materials:

Microsoft adCenter (Teaching Note) – HBP 908062

Posted on

Microsoft to Buy Claria? updated July 12, 2005

Today’s New York Times reports Microsoft “in talks” to buy Claria. Leading commentators think it’s a bad idea (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11). I agree.

I first heard this rumor several weeks back, but I laughed it off as too crazy to be taken seriously. What could Claria offer Microsoft? Most obvious is Claria’s large installed base — reportedly some 40-million PCs. But Claria’s installation practices are troubled — tricking users with ads that look like Windows dialog boxes, on kids sites, touting features Claria knows users don’t need (like clock-synchronizers already built into current versions of Windows). And in Claria’s oft-installed bundle with Kazaa, Claria’s long license lacks section headings, making it exceptionally hard for users to figure out what Claria does or to reasonably assess Claria’s terms. (These problems remain, seven months after I first reported them.) Microsoft wouldn’t want installations obtained through such poor practices.

Claria could also offer Microsoft substantial data about users’ surfing habits. A November 2003 eWeek article reported that Claria’s then-12.1 terabyte database was already the seventh largest in the world — bigger than Federal Express, and rivaling Amazon and Kmart. Claria recently told Release 1.0 its database is now 120 terabytes, the fifth-largest commercial Oracle database in the world. All very interesting, and perhaps troubling to those who worry about illicit use of such detailed data. But why would Microsoft invite this unnecessary privacy firestorm?

Claria could offer Microsoft its experience at advertisement targeting. But Claria’s targeting seems surprisingly simple: If a user goes to one car rental site, show an ad for another, whether in a pop-up, a delayed pop-under, or perhaps some subsequent banner ad placed via Claria’s new BehaviorLink program. Microsoft could design a similar system of its own in a matter of months, for far less than the $500 million it would reportedly cost to buy Claria.

Claria does have some interesting patents, a few making surprisingly broad claims as to software and advertisement delivery. But I’m not sure these patents are actually valid. If Microsoft wanted to implement client-side advertisement targeting, the more natural approach would be a design-around that didn’t infringe Claria’s design. Building it themselves avoids taint from Claria’s bad name, bad history, and bad installation practices.

Microsoft’s role as an operating system vendor and anti-spyware developer raises additional worries in buying Claria. Programs like Claria’s damage the Windows experience — bombarding users with annoying pop-ups, not to mention slowing boot time, adding complexity, and risking extra crashes. If Microsoft buys Claria, it would face practical difficulty in continuing to criticize, detect, and remove similar programs from others.

The Times says Microsoft’s Ballmer wants to be “more aggressive” in pursuing Google. But an aggressive strategy need not ignore business ethics — even if Google’s current distributors and partners are less than praiseworthy (1, 2). So I’m surprised that Ballmer reportedly personally approved negotiations with Claria. That said, others within Microsoft apparently oppose the acquisition, and negotiations are reportedly “on the verge” of breaking off. Cooler heads prevail, or so it seems.

It’s worth noting that no one from Microsoft or Claria has officially confirmed the negotiations. Techdirt and SiliconBeat claim this is all just a rumor. I have somewhat more faith in the Times’ reporting procedures; I’d like to think their editors wouldn’t run the story without confirmation from reasonable sources. Alex Eckelberry of Sunbelt offers what seems to me the most natural explanation: Microsoft leaked this story on purpose, as a “trial balloon” to test public response.

Microsoft AntiSpyware now recommends that users "ignore" Claria's presence on their PCs.Update (July 1): A Dozleng.com post reports that Microsoft’s AntiSpyware Beta now recommends that users “ignore” Claria. To confirm this result, I downloaded Claria’s DashBar and Precision Time products, then installed MSAS, all on a fresh virtual PC that hadn’t previously run any of these programs. MSAS’s recommendation and default action was “Ignore.” (See screenshot at right.) In contrast, when last I ran MSAS on a PC with Claria software installed, MSAS recommended removing these same programs. This is exactly the kind of conflict of interest I worried about three paragraphs above — but I didn’t anticipate how quickly this problem would come into effect.

Update (July 8): Apparently Microsoft’s “Ignore” recommendation doesn’t reflect special treatment for Claria in anticipation of an acquisition. Instead, Microsoft recommends “Ignore” for a variety of dubious “adware” programs. Sunbelt reports that Microsoft downgraded Claria to “Ignore” on March 31 — far before acquisition talks reportedly began. A comment from Webroot’s Richard Stiennon claims that Microsoft recently recommended ignoring 180solutions, and Sunbelt adds that Microsoft also recommends ignoring WebHancer and Ezula. My subsequent testing indicates that there are plenty of other “Ignore” programs still to be uncovered. (More on this in the future.)

These odd recommendations demonstrate the misguidedness of Microsoft’s “Ignore” classification. I know of no PC technician who advises users to ignore infection with any of these programs, which give users extra ads without anything offering substantial in return. If Bill Gates sought to clean up a friend’s PC, I bet he’d want all these programs gone. Competing anti-spyware programs all recommend removal. Yet somehow Microsoft’s AntiSpyware app sees no problem.

Has Microsoft given in to vendors’ threats? Or forgotten how badly “adware” damages the Windows experience (ultimately encouraging users to switch to other platforms)? I’ve previously been impressed with Microsoft’s AntiSpyware offering; I’ve often used it and often recommended it to others. But screw-ups like this call Microsoft’s judgment into question. During this sensitive period, with Microsoft unwilling to deny the continued Claria acquisition rumors, Microsoft should be especially careful to put users’ interests first. Instead, Microsoft’s recommendations cater to the interests of the advertising industry. I’m not impressed.

Microsoft’s recently-published response to questions about Claria defends Microsoft’s treatment as the result of ordinary application of Microsoft’s usual criteria, without any special exceptiosn. Perhaps. But if this Microsoft’s criteria say to ignore a program known to be installed through fake-user interface ads on kids sites, showing a EULA only after installation, with a broken uninstaller, then Microsoft’s criteria leave a lot to be desired.

Update (July 12): ClickZ reports that Microsoft has ended acquisition talks with Claria.

Posted on

Media Files that Spread Spyware updated January 3, 2005

Users have a lot to worry about when downloading and playing media files. Are the files legal? Can their computers play the required file formats? Now there’s yet another problem to add to the list: Will a media file try to install spyware?

When Windows Media Player encounters a file with certain “rights management” features enabled, it opens the web page specified by the file’s creator. This page is intended to help a content providers promote its products — perhaps other music by the same artist or label. However, the specified web page can show deceptive messages, including pop-ups that try to install software on users’ PCs. User with all the latest updates (Windows XP Service Pack 2 plus Windows Media Player 10) won’t get these popups. But with older software, confusing and misleading messages can trick users into installing software they don’t want and don’t need — potentially so many programs that otherwise-satisfactory computers become slow and unreliable.

Screen-shot of the initial on-screen display. If users press Yes, scores of unwanted programs are installed onto their PCs. Click to enlarge.Screen-shot of the initial on-screen display. If users press Yes, scores of unwanted programs are installed onto their PCs.

I recently tested a Windows Media video file, reportedly circulating through P2P networks, that displays a misleading pop-up which in turn attempts to install unwanted software onto users’ computers. I consider the installation misleading for at least three reasons.

  1. The pop-up fails to name the software to be installed or the company providing the software, and it fails to give even a general description of the function of the software.
  2. The pop-up claims “You must agree to our terms and conditions” — falsely suggesting that accepting the installation is necessary to view the requested Windows Media video. (It’s not.)
  3. Even when a user specifically requests more information about the program to be installed, the pop-up does not provide the requested information — not even in euphemisms or in provisions hidden mid-way through a long license. Clicking the pop-up’s hyperlink opens SpiderSearch’s Terms and Conditions — a page that mentions “receiving ads of adult nature” and that disclaims warranty over any third-party software “accessed in conjunction with or through” SpiderSearch, but that does not disclose installation of any third-party software.

Screen-shot of my Program Files folder, showing some of the programs installed on my test computer.Screen-shot of my Program Files folder, showing some of the programs installed on my test computer.

On a fresh test computer, I pressed Yes once to allow the installation. My computer quickly became contaminated with the most spyware programs I have ever received in a single sitting, including at least the following 31 programs: 180solutions, Addictive Technologies, AdMilli, BargainBuddy, begin2search, BookedSpace, BullsEye, CoolWebSearch, DealHelper, DyFuca, EliteBar, Elitum, Ezula, Favoriteman, HotSearchBar, I-Lookup, Instafin, Internet Optimizer, ISTbar, Megasearch, PowerScan, ShopAtHome Select, SearchRelevancy, SideFind, TargetSavers, TrafficHog, TV Media, WebRebates, WindUpdates, Winpup32, and VX2 (Direct Revenue). (Most product names are as detected by Lavasoft Ad-Aware.) All told, the infection added 58 folders, 786 files, and an incredible 11,915 registry entries to my test computer. Not one of these programs had showed me any license agreement, nor had I consented to their installation on my computer.

I retained video, packet log, registry, and file system logs of what occurred. As in my prior video of spyware installing through security holes, my records make it possible to track down who’s behind the installations — just follow the money trail, as captured by the “partner IDs” within the various software installation procedures. When one program installs another, the second generally pays the first a commission, using a partner ID number to track who to pay. These numbers make it possible to figure out who’s profiting from the unwanted installations and, ultimately, where the money is going.

Figuring Out Who’s Responsible

Most directly responsible for this mess is ProtectedMedia — the company that caused my computer to display the initial misleading pop-up shown above. ProtectedMedia invited the installation of some unwanted programs, which in turn installed others, but ProtectedMedia could readily stop these behaviors, e.g. by disabling its misleading pop-up installation attempts.

Screen-shot of the icons added to my test computer's desktop. Note a new link to Dell -- an affiliate link such that Dell pays commissions when users make purchases after clicking through this link.Screen-shot of the icons added to my test computer’s desktop. Note a new link to Dell — an affiliate link such that Dell pays commissions when users make purchases after clicking through this link.

But who pays ProtectedMedia? As I started to follow the money trail, I was surprised to see that some of the unrequested programs receive funds from respected online merchants. Several of the spyware installations added new toolbars to my computer’s browser and new icons to my desktop. If users click through these links, then make purchases from the specified merchants, the merchants pay commission to the affiliates who placed these toolbars and icons on users’ PCs. Even large, otherwise-reputable companies pay commissions through these systems, thereby funding those who install unwanted software on users’ computers. In my testing, I received affiliate links to Amazon, Dell, Hotwire, Match.com, Travelocity, and others. Many of these links pass through affiliate tracking networks LinkShare and Commission Junction.

Of course, these merchants may not have intended to support spyware developers. For example, merchants may have approved the affiliates without taking time to investigate the affiliates’ practices, or the affiliates’ actions may be unauthorized by the merchants. (That’s what Dell said when I previously found Dell ads running on Claria.) In future work, I’ll look in greater detail at which merchants pay affiliate commissions to which spyware programs, and I’ll also further document which merchants purchase advertising from companies whose software sneaks onto users’ computers.

Other companies partially responsible for these practices are the providers of the unwanted software — companies that pay commissions to distributors foisting their software onto users’ computers. In general there’s no reason to expect honorable behavior by providers of unwanted software. But some of the programs I received come from big companies with major investment backing: 180solutions received $40 million from Spectrum Equity Investors; Direct Revenue received $20 million from Insight Venture Partners; and eXact Advertising (makers of BargainBuddy and BullsEye) received $15 million from Technology Investment Capital Corp. With so much cash on hand, these companies are far from judgment-proof. Why are they paying distributors to install their software on users’ computers without notice and consent?

The problematic installations ultimately result from the “feature” of Windows Media Player that lets media files open web pages. But most users will only receive the contaminated files if they download files from P2P filesharing networks. Of course, rogue media files are but one way that P2P networks spread spyware. For example, users requesting Kazaa receive a large bundle of software (including Claria’s GAIN), after poor disclosures that bury key terms within lengthy licenses, without even section headers to help readers find what’s where. Users requesting Grokster receive unwanted software even if they press Cancel to decline Grokster’s installation (details).

Ed Bott offers an interesting, if slightly different, interpretation of these installations. Ed rightly notes that users with all the latest software — not just Windows XP Service Pack 2, but also Windows Media Player 10 — won’t get the tricky pop-ups described above. Ed also points out that Windows Media Player displays of ActiveX installation prompt pop-ups are similar to deceptive methods users have seen before, i.e. when web sites try to trick users into installing software. True. But I think Ed gives too little weight to the especially deceptive circumstances of a software installation prompt shown when users try to watch a video. For one, legitimate media players actually do use these prompts to install necessary updates (i.e. the latest version of Macromedia Flash), and Windows Media Player often shows similar prompts when it needs new codecs or other upgrades. In addition, the unusually misleading (purported) product name and company name make it particularly easy to be led astray here. Users deserve better.