The Pathologies of Online Display Advertising Marketplaces

Edelman, Benjamin. “The Pathologies of Online Display Advertising Marketplaces.” Art. 2. SIGecom Exchanges (June 2010).

Display advertising marketplaces place “banner” ads on all manner of popular sites. While these services are widely used, they suffer significant challenges, including weak user response and low accountability for both advertisers and web site publishers. I survey a few major challenges, flagging possible areas for future research.

Facebook Leaks Usernames, User IDs, and Personal Details to Advertisers updated May 26, 2010

Browse Facebook, and you wouldn’t expect Facebook’s advertisers to learn who you are. After all, Facebook’s privacy policy and blog posts promise not to share user data with advertisers except when users grant specific permission. For example, on April 6, 2010 Facebook’s Barry Schnitt promised: “We don’t share your information with advertisers unless you tell us to (e.g. to get a sample, hear more, or enter a contest). Any assertion to the contrary is false. Period.”

My findings are exactly the contrary: Merely clicking an advertiser’s ad reveals to the advertiser the user’s Facebook username or user ID. With default privacy settings, the advertiser can then see almost all of a user’s activity on Facebook, including name, photos, friends, and more.

In this article, I show examples of Facebook’s data leaks. I compare these leaks to Facebook’s privacy promises, and I point out that Facebook has been on notice of this problem for at least eight months. I conclude with specific suggestions for Facebook to fix this problem and prevent its reoccurrence.

Details of the Data Leak

Facebook’s data leak is straightforward: Consider a user who clicks a Facebook advertisement while viewing her own Facebook profile, or while viewing a page linked from her profile (e.g. a friend’s profile or a photo). Upon such a click, Facebook provides the advertiser with the user’s Facebook username or user ID.

Facebook leaks usernames and user IDs to advertisers because Facebook embeds usernames and user IDs in URLs which are passed to advertisers through the HTTP Referer header. For example, my Facebook profile URL is http://www.facebook.com/bedelman. Notice my username (yellow).

Of course, it would be incorrect to assume that a person looking at a given profile is in fact the owner of that profile. A request for a given profile might reflect that user looking at her own profile, but it might instead be some other user looking at the user’s profile. However, when a user views her own profile page, Facebook automatically embeds a “profile” tag (green) in the URL:

http://www.facebook.com/bedelman?ref=profile

Furthermore, when a user clicks from her profile page to another page, the resulting URL still bears the user’s own user ID or username, along with the details of the later-requested page. For example, when I view a friend’s profile, the resulting URL is as shown below. Notice the continued reference to my username (yellow) and the fact that this is indeed my profile (green), along with an appendage naming the user whose page I am now viewing (blue).

http://www.facebook.com/bedelman?ref=profile#!/pacoles

Each of these URLs is passed to advertisers whenever a user clicks an ad on Facebook. For example, when I clicked a Livingsocial ad on my own profile page, Facebook redirected me to the advertiser, yielding the following traffic to the advertiser’s server. Notice the transmission in the Referer header (red) of my username (yellow) and the fact that I was viewing my own profile page (green).

GET /deals/socialads_reflector?do_not_redirect=1&preferred_city=152&ref=AUTO_LOWE_Deals_ 1273608790_uniq_bt1_b100_oci123_gM_a21-99 HTTP/1.1
Accept: */*
Referer: http://www.facebook.com/bedelman?ref=profile

Host: livingsocial.com

The same transmission occurs when a user clicks from her profile page to a friend’s page. For example, I clicked through to a friend’s profile, http://www.facebook.com/bedelman?ref=profile#!/pacoles, where I clicked another Livingsocial ad. Again, Facebook’s redirect caused my browser to transmit in its Referer header (red) my username (yellow), the fact that that username reflects my personal profile (green). Interestingly, my friend’s username was omitted from the transmission because it occurred after a pound sign, causing it to be automatically removed from Referer transmission.

GET /deals/socialads_reflector?do_not_redirect=1&preferred_city=152&ref=AUTO_LOWE_Deals_ 1273608790_uniq_bt1_b100_oci123_gM_a21-99 HTTP/1.1
Accept: */*
Referer: http://www.facebook.com/bedelman?ref=profile

Host: livingsocial.com

In further testing, I confirmed that the same transmission occurs when a user clicks from her profile page to a photo page, or to any of various other pages linked form a user’s profile.

With a Facebook member’s username or user ID, current Facebook defaults allow an advertiser (and anyone else) to obtain a user’s name, gender, other profile data, picture, friends, networks, wall posts, photos, and likes. Furthermore, the advertiser already knows the user’s basic demographics, since the advertiser knows the user fits the profile the advertiser had requested from Facebook. For example, in grey highlighting above, the advertiser learned from Facebook my age, gender, and geographic location.

Facebook’s Contrary Statements about User Privacy vis-a-vis Advertisers

Facebook has made specific promises as to what information it will share with advertisers. For one, Facebook’s privacy policy promises “we do not share your information with advertisers without your consent” (section 5). Then, in section 7, Facebook lists eleven specific circumstances in which it may share information with others — but none of these circumstances applies to the transmission detailed above.

Facebook’s recent blog postings also deny that Facebook shares users’ identities with advertisers. In an April 6, 2010 post, Facebook promised: “We don’t share your information with advertisers unless you tell us to (e.g. to get a sample, hear more, or enter a contest). Any assertion to the contrary is false. Period.” Facebook’s prior postings were similar. July 1, 2009: “Facebook does not share personal information with advertisers except under the direction and control of a user. … You can feel confident that Facebook will not share your personal information with advertisers unless and until you want to share that information.” December 9, 2009: “Facebook never shares personal information with advertisers except under your direction and control.” As to all these claims, I disagree. Sharing a username or user ID upon a single click, without any disclosure or indication that such information will be shared, is not at a user’s direction and control.

Facebook Has Been on Notice of This Problem for Eight Months

AT&T Labs researcher Balachander Krishnamurthy and Worcester Polytechnic Instituteprofessor Craig Wills previously identified the general problem of social networks leaking user information to advertisers, including leakage through the Referer headers detailed above. In August 2009, their On the Leakage of Personally Identifiable Information Via Online Social Networks was posted to the web and presented at the Workshop on Online Social Networks (WOSN).

Through Krishnamurthy and Wills’ research, Facebook eight months ago received actual notice of the data leakage at issue. A September 2009 MediaPost article confirms Facebook’s knowledge through it spokesperson’s response. However, Facebook spokesperson Simon Axten severely understated the severity of the data leak: Axten commented “The average Facebook user views a number of different profile pages over the course of a session …. It’s thus difficult for a tracking website to know whether the identifier belongs to the person being tracked, or whether it instead belongs to a friend or someone else whose profile that person is viewing.” I emphatically disagree. As shown above, when a user views her own profile, or a page linked from her own profile, the “?ref=profile” tag is added to the URL — exactly confirming the identity of the profile owner.

What Facebook Should Do

Since receiving actual notice of these data leaks, Facebook has implemented scores of new features for advertising, monetization, information-sharing, and reorganization. Inexplicably, Facebook has failed to address leakage of user information to advertisers. That’s ill-advised and short-sighted: Users don’t expect ad clicks to reveal their names and details, and Facebook’s privacy policy and blog posts promise to honor that expectation. So Facebook needs to adjust its actual practices to meet its promises.

Preventing advertisers from receiving usernames and user IDs is strikingly straightforward: A modified redirect can mask referring URLs. Currently, Facebook uses a simple HTTP 301 redirect, which preserves referring URLs — exactly creating the problem detailed above. But a FORM POST redirect, META REFRESH redirect, or JavaScript redirect could conceal referring URLs — preventing advertisers from receiving username or user ID information.

Instead, Facebook has partially implemented the pound sign method described above — putting some, but not all, sensitive information after a pound sign, with the result that sometimes this information is not transmitted as a Referer. If fully implemented across the Facebook site, this approach might prevent the data leakage I uncovered. However, in my testing, numerous within-Facebook links bypass the pound sign masking. In any event, an improved redirect would be much simpler to implement — requiring only a single adjustment to the ad click-redirect script, rather than requiring changes to URL formats across the Facebook site.

Finally, Facebook should inform users of what has occurred. Facebook should apologize to users, explain why it didn’t live up to its explicit privacy commitments, and establish procedures — at least robust testing, if not full external review — to assure that users’ privacy is correctly protected in the future.

Update – May 26, 2010

On May 20, 2010, the Wall Street Journal reported the problem detailed above. On or about that same day, Facebook removed the ref=profile tags that were the crux of the data leak.

I yesterday spoke with Arturo Bejar, a Facebook engineer who investigated this problem. Arturo told me that after Krishnamurthy and Wills’ article, he reviewed relevant Facebook systems in search of leakage of user information. At that time, he found none, in that Facebook revealed the URLs users were browsing when they clicked ads, but did not indicate whether the user clicking a given ad was in fact the owner of the profile showing that ad. However, in a subsequent Facebook redesign, beginning in February 2010, Facebook user home pages received a new “profile” button which carried the ref=profile URL tags I analyze above. Because this tag was added without a further privacy review, Arturo tells me that he and others at Facebook did not consider the interaction between this tag and the problem I describe above. Arturo says that’s why this problem occurred despite the prior Krishnamurthy and Wills article.

Arturo also pointed out that the problem I describe did not affect advertisers whose landing pages were pages on Facebook (rather than advertisers’ own external sites).

Meanwhile, Facebook’s May 24 “Protecting Privacy with Referrers” presents Facebook’s view of the problem in greater detail. Facebook’s posting offers a fine analysis of the various methods of redirects and Facebook’s choice among them. It’s worth a read.

After discussing the problem with Arturo and reading Facebook’s new post, I reached a more favorable impression of Facebook’s response. But my view is tempered by Facebook’s ill-advised attempts to downplay the breach.

  • Rather than affirmatively describing the specific design flaw, Facebook’s post describes what “could” “potentially” occur. Facebook’s post never gives a clear affirmative statement of the problem.
  • Facebook says advertisers would need to “infer” a user’s username/ID. But usernames and IDs are sent directly, in clear and unambiguous URLs, hardly requiring complex analysis
  • Facebook claims that the breach affected only “one case … if a user takes a specific route on the site” (WSJ quote). Facebook also calls the problem “a rarely occurring case” (posting). I dispute these characterizations. It is hardly “rare” for a user to view her own profile. To view her own profile and click an ad? There’s no reason to think that’s any less frequent than clicking an ad elsewhere. To view her own profile, click through to another page, and then click an ad? That’s perfectly standard. Furthermore, although Facebook told the Journal there is “one case” in which data is leaked improperly, in fact I’ve found many such cases including clicking from profile to ad, from profile to friend’s page to ad, and from profile to photo page to ad, to name three.
  • Through transmission in HTTP Referer headers, usernames and IDs appears reach advertisers’ web servers in a manner such that default server log files would store this data indefinitely, and default analytics would tabulate it accordingly. Facebook says it has “no reason to believe that any advertisers were exploiting” the data breach I reported, but the fact is, this data ends up in a place where advertisers could (and, as to historic data, still can) access it easily, using standard tools, and at their convenience.
  • Although Facebook’s post says the problem is “potential,” I found that a user’s username/ID is sent with each and every click in the affected circumstances.

So the problem was substantial, real, and immediate. Facebook errs in suggesting the contrary.

Measuring the Perpetrators and Funders of Typosquatting

Moore, Tyler, and Benjamin Edelman. “Measuring the Perpetrators and Funders of Typosquatting.” Lecture Notes in Computer Science. Springer-Verlag. Financial Cryptography and Data Security: Proceedings of the International Conference 6052 (2010). (Introduction, Web appendix.)

We describe a method for identifying “typosquatting”, the intentional registration of misspellings of popular website addresses. We estimate that at least 938,000 typosquatting domains target the top 3,264 .com sites, and we crawl more than 285,000 of these domains to analyze their revenue sources. We find that 80% are supported by pay-per-click ads, often advertising the correctly spelled domain and its competitors. Another 20% include static redirection to other sites. We present an automated technique that uncovered 75 otherwise legitimate websites which benefited from direct links from thousands of misspellings of competing websites. Using regression analysis, we find that websites in categories with higher pay-per-click ad prices face more typosquatting registrations, indicating that ad platforms such as Google AdWords exacerbate typosquatting. However, our investigations also confirm the feasibility of significantly reducing typosquatting. We find that typosquatting is highly concentrated: of typo domains showing Google ads, 63% use one of five advertising IDs, and some large name servers host typosquatting domains as much as four times as often as the web as a whole.

Who Owns Metrics?: Building a Bill of Rights for Online Advertisers

Edelman, Benjamin. “Who Owns Metrics?: Building a Bill of Rights for Online Advertisers.” Journal of Advertising Research 49, no. 4 (December 2009). (Adapted from Towards a Bill of Rights for Online Advertisers.)

I offer five rights to protect advertisers from increasingly powerful ad networks-avoiding fraudulent charges for services not rendered, guaranteeing data portability so advertisers get the best possible value, and assuring price transparency so advertisers know what they’re buying. I explain the need for these rights by presenting specific practices causing particular concern.

The Dark Underbelly of Online Advertising

Edelman, Benjamin. “The Dark Underbelly of Online Advertising.” HBR Now. (November 17, 2009).

The Internet is sold to advertisers as a highly measurable medium that is the most efficient way to target exactly the right customers. But online advertising is also easily subverted–letting fraudsters claim advertising fees for work they did not actually do. The trickiest frauds deceive advertisers so effectively that measurements of ad effectiveness report the fraudsters as exceptionally productive and high quality, rather than revealing that their traffic was actually worthless. This is a quiet scandal. In a time of tightening ad budgets, losses to advertising fraud come straight from the bottom line–but savings can be equally dramatic. Here’s a look behind the veil–an explanation of ad practices that have cheated even the Web’s largest advertisers. Advertising scams take plenty of victims, both witting and not, but I offer strategies to help determined marketers protect themselves.

Deception in Post-Transaction Marketing Offers

Edelman, Benjamin. “Deception in Post-Transaction Marketing Offers.” U.S. Senate, Committee on Commerce, Science, & Transportation, November 2009.

I examine the consumer protection issues raised by post-transaction marketing offers. My key concerns:

  • Post‐transaction marketing offers systematically reach consumers in a time when consumers are particularly vulnerable. Post‐transaction offers feature deceptive designs that invite consumers to conclude, mistakenly, that the offers comes from the companies the consumers have chosen to frequent, and that the offers are a required part of the checkout process.
  • The automatic transfer of consumers’ payment information from a merchant to a post ‐ transaction marketer runs contrary to consumer expectations, and creates a heightened risk that consumers will “accept” financial obligations they did not intend to incur.
  • Disclosures fail to cure the deception created by post-transaction offers, their timing and formatting, and their automatic transfer of consumers’ payment information.
  • Straightforward remedies could protect consumers who have suffered unwanted charges, and could prevent further consumers from incurring similar charges.

eBay Partner Network (teaching materials) with Ian Larkin

Edelman, Benjamin, and Ian Larkin. “eBay Partner Network (A).” Harvard Business School Case 910-008, September 2009. (Revised March 2015.) (educator access at HBP. request a courtesy copy.)

eBay considers adjustments to the structure and rules of its affiliate marketing program, eBay Partner Network (ePN). In particular, eBay reevaluates affiliate compensation structure, the role of bonuses for especially productive affiliates, and the overall rationale for outsourcing online marketing efforts to independent affiliates. The case presents the history and development of ePN, ePN’s importance to eBay, and the mechanics of online affiliate marketing.

Supplements:

eBay Partner Network (B) – Supplement (HBP 910009)

eBay Partner Network (C) – Supplement (HBP 910012)

eBay Partner Network (D) – Supplement (HBP 914016)

Teaching Materials:

eBay Partner Network (A), (B), (C), and (D) – Teaching Note (HBP 910025)

eBay Partner Network – slide supplement (HBP 911039)

eBay Partner Network – slide supplement (widescreen) (HBP 914040)

Towards a Bill of Rights for Online Advertisers

Edelman, Benjamin. “Towards a Bill of Rights for Online Advertisers.” Advertising Week (September 21, 2009).

Online advertising presents remarkable efficiencies–better targeting, improved measurement and greater return on investment. Yet there are challenges, particularly when networks of intermediaries place ads through convoluted relationships, and all the more so when small advertisers cannot effectively negotiate terms dictated by advertising powerhouses. The result is a troubling mess of ads gone wrong–advertisers charged in ways they didn’t fairly agree to, and on terms they didn’t meaningfully accept. But online advertising doesn’t have to be a wild west. I propose five specific rights advertisers should demand as they buy online placements.

Advertising Week abridgement extracted from full original article:

Towards a Bill of Rights for Online Advertisers

How Google and Its Partners Inflate Measured Conversion Rates and Inflate Advertisers’ Costs

When advertisers measure the effectiveness of their pay-per-click ad campaigns, advertisers systematically assume additionality, i.e. that the sales that follow a paid click are sales that would not have happened without the ad platform’s assistance. This assumption offers intuitive appeal: If a user clicked an ad and then bought the advertised product, by all indications the ad platform should be thanked for finding and sending an interested customer. Or should it?

As it turns out, Google and its partners systematically inflate advertisers’ conversion rates by interceding in transactions advertisers would otherwise have received for free. This conversion-inflation syndication fraud overstates the true effectiveness of the ads Google delivers — leading advertisers to pay more than they should.

In this piece, I offer four examples of Google and its partners inflating conversions to claim credit for traffic advertisers would otherwise have received for free. In each example, an advertiser intensely measuring its conversion rate would mismeasure the true effectiveness of its ads, and would end up overpaying for traffic that is far less valuable than reporting systems suggest.

Traffic source How users are found What would have happened had Google and its partners not interceded
WhenU – adware User requests advertiser’s site. Adware covers advertiser’s site with pay-per-click listings. Advertiser’s site displays as usual, with no covering popup. User stays at advertiser’s site, and advertiser pays no PPC fee.
SmileyCentral – toolbar Reconfigured browser tricks user into running a “search” for a site’s domain name. User’s browser retains its ordinary configuration. User runs a direct navigation, and advertiser pays no PPC fee.
Typosquatting User misspells advertiser’s domain name. User’s browser shows a list of alternatives, and user selects one — reaching advertiser’s site at no charge. Or, user sees an error page, notices the misspelling, and corrects the spelling to reach the advertiser’s site without a PPC fee.
Chrome – browser suggestions User typing a web address is encouraged to run a search instead. User finishes typing the site’s web address and reaches the advertiser’s site without a PPC fee.

WhenU Covers Advertisers’ Sites with Advertisers’ Own Google Ads

WhenU covers RCN with its own Google ads -- charging ad fees for traffic RCN would otherwise have received for free.
WhenU covers RCN with its own Google ads — chargingad fees for traffic RCN would otherwise have received for free.


PPC advertisers (e.g. RCN)
money viewers
   Google   
money viewers
InfoSpace
money viewers
Nbcsearch
money viewers
LocalPages
money viewers
WhenU

The money trail – how funds flow from advertisers
to Google to WhenU.

Through popups shown to users already at advertisers’ sites, WhenU (and its partners) charge advertisers for traffic they would have otherwise received for free. Google passes along these clicks through its advertising platform, and Google charges advertisers as if these were genuine leads, even though in fact these users were already on advertisers’ sites.

In testing of May 9, 2009, I browsed the web site RCN.com. Advertising software (generously, “adware”) from WhenU popped open the window shown at right — covering more than 80% of the RCN site with a list of pay-per-click ads, among them a prominent ad for RCN. I clicked the RCN ad, and I was taken back to RCN. See screenshots detailing the full sequence, or a screen-capture video.

As a provider of high-speed Internet access (with attendant concerns for user security, PC reliability, and tech support expense), RCN is unlikely to advertise with a notorious adware program like WhenU. Nor is RCN likely to want to show its ads to users already at rcn.com — users who RCN has already managed to reach. So how did RCN’s ads end up in this unfortunate placement? Using a network monitor, I confirmed the full sequence of intermediaries: WhenU and its MediaTraffic ad system sent traffic to LocalPages, which redirected to Nbcsearch, which forwarded the traffic to Infospace, which finally passed the traffic to Google. Google in turn paid Infospace, which paid Nbcsearch, which paid LocalPages, which paid WhenU. See the diagram at right and the full packet log.

For RCN, this placement is a rotten deal. RCN has already paid to get a user to its site — perhaps via a postcard, TV advertisement, display ad, or other paid search activity. But then WhenU intercedes and puts a roadblock in front of that user, in the form of the popup at right. A typical user presented with that popup will click the RCN entry to get back to RCN and continue the signup process. But then RCN pays twice to reach a single user.

Meanwhile, if RCN is tracking conversion rates, it will notice that this WhenU/Google placement seems to have a high conversion rate — reflecting that this ad was shown to users already on the verge of signing up with RCN. So in all likelihood, RCN will increase its Google bid to attempt to obtain more traffic of similar (apparent) quality. But the supposed high conversion rate is misleading at best: Since these are users who were already at the RCN site, without any intervention by Google or WhenU, it is nonsense to credit Google or WhenU for resulting sales.

This ad placement also lacks the labeling required by FTC policy and precedent. For one, all sponsored search ads are to be labeled as such, pursuant to the FTC ‘s 2002 instructions. But look closely at the popup screenshot. On my ordinary 800×600 screen, no such label appears. Furthermore, the FTC’s Direct Revenue and Zango complaints, decisions, and orders reveal an additional duty that adware vendors specifically and clearly label each popup with, among other information, a statement that the popup is an advertisement. (See Direct Revenue Decision and Order, provision VI.(1), and Zango Decision and Order, provision VI.(1).) Here, the popup bears the WhenU icon and even a phone number, but it lacks any disclosure that the window’s listings are advertisements. I gather the required label would ordinarily appear on a sufficiently large screen, but the FTC’s policies make no exceptions for users with small screens. Indeed, as netbooks gain popularity, small screens are increasingly common.

These ad placements are particularly troubling because they have continued for months on end. I first observed substantially similar placements on February 7, 2009, though I have reason to think the placements began well before then. Furthermore, Google has been on actual notice of these practices for 3+ months: I first reported these placements to the public in my lunch keynote at the INTA Trademark Law and the Internet conference on February 10, 2009. I posted my slides that very day, and slides 23-24 show substantially this same set of relationships. Importantly, Google’s Rose Hagan, Senior Trademark Counsel, was present in the audience, and she responded to this portion of the talk by promising to investigate. But despite her presence, my clear posting of the parties responsible, and a three month opportunity to act, Google has nonetheless failed to sever these relationships.

Plenty more could be said about WhenU. I have personally observed a wide variety of deceptive WhenU installations, including even WhenU installations via security exploits. (I never had occasion to post those videos to my public site, but I have them on file.) The Google-funded StopBadware declared multiple WhenU-bundlers to be “badware” (1, 2, 3) based on WhenU’s automatic startup, disruptive pop-ups, and other “bad or undisclosed” behaviors. But I doubt Google will defend its WhenU placements, so I’ll save full critique of WhenU for another day.

IAC’s SmileyCentral Grabs Advertisers’ Organic Traffic to Show Google Ads

Standard web browsers place a search bar at top-left. Click that box, type an address, and press enter to reach a site.
Standard web browsers place a search bar at top-left. Click that box, type an address, and press enter to reach a site.

With SmileyCentral installed, a 'direct navigation' request for www.verizon.com yields search results, not the Verizon site. With SmileyCentral installed, a “direct navigation” request for www.verizon.com yields search results, not the Verizon site.


PPC advertisers (e.g. Verizon)
money viewers
   Google   
money viewers
IAC/SmileyCentral

The money trail – how funds flow from advertisers
to Google to IAC/SmileyCentral.

By reconfiguring users’ web browsers, IAC’s SmileyCentral toolbars charge advertisers for traffic they would otherwise have received for free. Google passes along these clicks through its advertising platform, and Google charges advertisers as if these were genuine leads, even though in fact these are users who were specifically and unambiguously trying to reach advertisers’ sites directly.

Since the earliest web browsers, a user seeking a particular web site clicks in the browser’s upper-left text box, types the desired address, and presses Enter. See the first inset image at right — standard Internet Explorer 6, offering exactly this layout to request a site.

But suppose a user installs the “SmileyCentral” toolbar from IAC (the advertising powerhouse that also owns Ask.com, Match.com, and more). SmileyCentral modifies longstanding browser layout by pushing a user’s Address Bar to the right, and inserting at left a search box where users naturally expect the Address Bar to appear. Then, when a user goes to the top-left box and enters a domain name, seeking a direct navigation to the specified site, SmileyCentral runs a search instead. See the second inset image at right — the result of typing www.verizon.com, then pressing Enter, into the top-left box. Notice that the user receives a list of search ads for Verizon, even though the user asked for Verizon by its correct web address.

IAC/SmileyCentral’s search results increase advertisers’ costs. Consider user behavior upon reaching the listings shown at right. In all likelihood, a user facing these listings will click one of the top links — perhaps the first listed (to verizonwireless.com) or the second (which specifically indicates it is the “Verizon Official Site”). But these are sponsored links. Every time a user clicks one of these links, Verizon pays a fee to Google, which in turn pays IAC/SmileyCentral. (To confirm Google’s role, see the packet log.)

As in the WhenU example above, this placement is a bad deal for advertisers. Had it not been for IAC/SmileyCentral’s tricky toolbar, these users would have directly reached the sites they requested. Instead, IAC/SmileyCentral grabs the users and sends them to lists of ads — saddling advertisers with unnecessary advertising expense.

Here too, advertisers are likely to be tricked if they attempt to assess the value of this traffic based on its conversion rate. The conversion rate may well be high — for these users asked for advertisers by name, suggesting that the users are nearly ready to make purchases. But the traffic is still a bad deal for advertisers: By all rights, this is still traffic advertisers should have gotten for free.

This placement is all the worse for advertisers because IAC makes ad clicks excessively easy. Even a click 230 pixels to the right of a Verizon Wireless ad would be treated as a paid click “on” that ad. See video at 0:23, and accompanying screenshot, showing that the hyperlink area extends far beyond the text of the ad.

IAC may claim that users understand that, with MyWebSearch installed, the top-left box no longer allows direct navigations by domain name or URL. But IAC often sneaks its toolbars onto users’ computers in ways that don’t obtain meaningful consent: I previously demonstrated nonconsensual installations through security exploits and undisclosed bundles. IAC’s installations often target kids (as I have repeatedly documented). And even a direct installation from Smileycentral.com places a picture of the post-installation browser layout in a bottom-of-page image, below the fold on 800×600 screens and easily overlooked on larger screens. IAC’s use of the generic “My Web Search” label (just as in “My Documents”, “My Computer”, etc.) further compounds users’ sense that this box is part of Internet Explorer. Combining IAC’s user focus with its choice of labeling and positioning, IAC creates conditions in which users are bound to be confused.

Typosquatting: Cmcast.com, MediaLogik, and Thousands More Intercept Users’ Misspellings to Show Google Ads

Requesting Cmcast.com (s.i.c.) yields a page of Google ads.
Requesting cmcast.com (s.i.c.) yields a page of Google ads.

If no typosquatter had registered Cmcast.com, a user would have reached this page and reached Comcast without charge. If no typosquatter had registered cmcast.com, a user would have reached this page and found Comcast without charge.


PPC advertisers (e.g. Comcast)
money viewers
   Google   
money viewers
Typosquatters & brokers (e.g. MediaLogik)

The money trail – how funds flow from advertisers
to Google to typosquatters.

By paying partners to register typosquatting sites and to send the resulting clicks to Google’s ad platform, Google charges advertisers for traffic they would otherwise have received for free. Google charges advertisers as if these were genuine leads. Yet had typosquatters not registered these domains, ordinary web browsers would have assisted users in reaching advertisers’ sites without charge.

A staggering number of typosquatting web sites target users who misspell the domain names of famous web sites. For example, in my INTA presentation this spring, I showed 200+ typosquatting sites all targeting variations of “cartoonnetwork.com.” I’ll have further thoughts on typosquatting in a future article. But for present purposes, two facts are most important: Typosquatting sites are overwhelmingly funded through pay-per-click ads, and these days Google is the advertising provider of choice for most typosquatters. (My preliminary analysis indicates that Google funds more than 75% of those typosquatting sites that show Google ads.)

Consider a user who misspells comcast.com, instead typing cmcast.com (s.i.c.). Such a user will receive the first screen shown at right, presenting a series of pay-per-click links for Comcast. In all likelihood, such a user will then click a paid link to Comcast — meaning Comcast has to pay Google an advertising fee. Google in turn pays the typosquatter which had the foresight to register a domain. See the packet log.

(In some instances, traffic flows from a typosquatter to one or more brokers or intermediaries, then to Google and finally to the advertiser. As to cmcast.com: The packet log confirms that Google pays MediaLogik of Hong Kong. I cannot readily determine whether MediaLogik registered this domain for its own account, or whether MediaLogik brokers ads for some other registrant. Based on the domain’s hosting in the Bahamas, its monetization through a Hong Kong service provider, and its display of Google PPC ads, it is highly unlikely that this domain was authorized by Comcast.)

Although these typosquatting sites ultimately direct users where the users were trying to go, typosquatters leave targeted advertisers importantly worse-off. Had it not been for the typosquatter, the user would have received a standard browser page identifying the typo and, in general, referring the user to the requested site without charge. See the second screenshot at right, showing a request for cmcast.com in a standard installation of Internet Explorer 6. So, even if no typosquatter had registered cmcast.com, the user would still reach Comcast with equal ease; the typosquatter does not actually make it easier for the user to reach Comcast. But notice that a standard IE6 error page shows few ads — here, just one small ad, at far right — whereas the MediaLogik page showed all ads, front and center. So passing the user’s misspelled request to a standard IE landing page, rather than to a typosquatter, would spare Comcast an expensive Google pay-per-click fee.

Notably, typosquatting exactly fits the traffic pattern detailed in preceding examples. First, Google and its partners identify users who are already trying to reach a given advertiser. Then Google and its partners intercede — here, by registering a typosquatting domain. The user thus stumbles into Google’s ad listings and clicks through to the advertiser’s site — letting Google charge the advertiser a fee, even though the user would have reached the advertiser even without Google’s assistance.

Google Chrome Suggestions Divert Users from Direct Navigation to Search

Typing 'expedia' yields a suggestion that users search Google (simply by pressing 'Enter') rather than visiting Expedia.com directly (third entry -- requiring a mouse-click or multiple keystrokes). Meanwhile, the second link ('expedia/') yields an error -- discouraging future exploration of green direct-navigation listings.
Typing “expedia” yields a suggestion that users search Google (just press “Enter”) rather than visit Expedia.com directly (third entry — requiring a mouse-click or multiple keystrokes). The second link (“expedia/”) yields an error — discouraging future exploration of green direct-navigation listings.

As users type web addresses into Google’s Chrome web browser, Chrome’s “Omnibox” address bar suggests that users run searches instead of direct navigation. See the screenshot at right, showing the Omnibox’s suggestion after I typed “Expedia” and before I typed the final “.com”.

If a user accepts Chrome’s suggestion — by clicking the “Search for …” drop-down, or by merely pressing Enter — the user is taken to a page of Google search results for the specified term. See screenshot and screen-capture video. As usual, Google’s most prominent search result is an advertisement. If the user clicks the ad, the advertiser pays a pay-per-click fee — even though the user was nearly at the advertiser’s site, for free, before Chrome interceded with its “Search for…” suggestion.

To complete a direct navigation to the Expedia site, without passing through Google search results, a user must ignore Google’s suggestion and continue typing (“.com”), click the “expedia.com” entry (third on Google’s autocomplete list), or use the keyboard (down-down-enter) to navigate manually. In principle these steps are straightforward — just a few extra seconds. But by pushing default behavior from direct navigation to search, Google makes searches that much more frequent — yielding that many more ad-clicks, that much more revenue to Google, and that much more expense for advertisers.

Chrome further encourages searching by mixing useful autocomplete direct navigation listings (e.g. “www.expedia.com/”) with nonfunctional listings. Notice that the second entry on Chrome’s autocomplete drop-down is “expedia/” (s.i.c.) — not a valid domain name. If a user clicks that entry, the user suffers a delay followed by a DNS error — hardly a good user experience. (video) By placing this nonfunctional result prominently in the autocomplete drop-down, above the one working direct link to Expedia, and in the same distinctive green font as the working direct link, Chrome discourages users from exploring direct links. After all, if the prominently-listed “expedia/” link did not work, users are less likely to try the similar-looking link Google ranked lower.

Although competing browsers also perform searches if users enter malformed text into the Address Bar, competing browsers are less pushy in suggesting and encouraging searches in lieu of direct navigation. Chrome thus extends browser norms through its increasingly forceful suggestion of search, not direct navigation, as the default way to reach a site.

Beyond its effects on advertisers’ costs, Chrome’s Omnibox raises other serious concerns too. For example, Chrome transmits users’ every navigation keystroke to Google — including what users search for, what addresses users request by name, and what addresses users begin to request but ultimately decline to visit. CNET reports EFF and EPIC staff concerned about Omnibox’s privacy implications, and I share their discomfort.

Fair Play and a Way Forward

Notice similarities across all four examples:

  • In all four examples, Google and its partners intercede to divert traffic that, but for their intervention, would reach advertisers’ sites directly — without advertisers incurring any advertising expense.
  • In all four examples, Google and its partners ultimately pass the traffic back to the advertisers users were trying to reach — but only after collecting pay-per-click advertising fees.
  • In all four examples, if an advertiser attempts to measure its conversion rate, it will conclude that it receives traffic at attractive prices — at effective cost-per-acquisition consistent with its objectives. Standard conversion rate measurements will overlook the fact that, were it not for the intervention of Google and its partners, the advertiser would have received this traffic for free.

Not all advertisers measure conversion rates. For one, some advertisers may take Google at its word, rather than attempting detailed and rigorous monitoring of Google’s effectiveness. Furthermore, for some advertisers, conversion rates are unmeasurable. (Advertisers may have offline sales process, rather than online sales. Certain advertisers may seek brand awareness rather than immediate purchases.) But for advertisers that measure conversion rates and adjust bids accordingly, inflated conversion rates are of grave concern. In particular, inflated conversion rates yield records that seem to indicate that campaigns are working well and delivering fair value, when the truth might be exactly the opposite.

Aggrieved advertisers could sue Google over these placements. But Google imposes terms and conditions that discourage advertisers from pursuing such claims. Google’s Advertising Program Terms purport to “disclaim[] all warranties … [and] guarantees regarding positioning, levels, quality, or timing of costs per click, … clicks, conversions, or any other results” (internal numbering omitted). Google further claims that “Any refunds for suspected invalid impressions or clicks are within Google’s sole discretion.” Google also insists that advertisers’ sole remedy is advertising credits (never refunds), and Google says advertisers must complain within 60 days of a disputed charge (even if advertisers did not know about the improper charges because the charges were concealed through, e.g., inflated conversion rates). I doubt whether all these harsh provisions are enforceable. But if these provisions are enforceable, they would tightly limit advertisers’ ability to obtain redress from Google. Even if these provisions are unenforceable, their mere presence serves to discourage advertisers from filing complaints, whether informal (through Google’s customer service staff) or in court.

In a recent presentation defending its competitive posture and overall approach, Google claimed “advertisers pay what a click is worth to them” (slides 17-21). But does Google have the right to charge such a fee for all traffic, even the traffic advertisers receive without any Google assistance whatsoever? Few advertisers would tolerate such charges, if presented explicitly. Nor does Google ever seek permission to charge advertisers for their existing traffic. Yet in the future Google seems to envision, all manner of ruses intercede to claim Google delivered a customer, when in fact the customer reached the advertiser’s site without bona fide assistance from Google.

More generally, when Google and its partners overstate the effectiveness of the ads they deliver, advertisers cannot make well-informed decisions about which ads to buy or how much to be willing to pay. Danny Sullivan calls AdWords pricing a “black box,” and I certainly share that sentiment. But here, Google’s actions are even worse than opacity: By claiming to have delivered traffic advertisers would have received anyway, Google tricks advertisers into paying for that traffic — and even tricks advertisers into concluding, mistakenly, that the traffic is a good deal.

It’s also hard to reconcile Google’s WhenU and IAC/SmileyCentral placements with Google’s “Software Principles” requirements. With bundled installations disclosing WhenU’s presence only midway through installation, WhenU is anything but “upfront” (contrary to Google’s “upfront disclosure” section heading). Showing the generic “My Web Search” toolbar label after users requested the distinctively-named SmileyCentral, Ask’s offering similarly flunks Google’s call for “clear behavior.” I could continue. But even if these programs satisfied Google’s criteria, they still fall short of advertisers’ reasonable expectations — for they still intercede to claim traffic advertisers they did nothing to deliver.

Looking forward, I see two key principles. First, ad platforms must deliver genuine, incremental traffic — not just intercept and repackage traffic advertisers were already going to receive for free. Second, ad platforms need to stand behind their products — abandoning one-sided attempts to disclaim responsibility, and instead offering meaningful commitments to provide what they promise. In both these respects, Google currently falls short.