A new report shows that more than half of Internet traffic comes from bots — software applications that perform automated tasks for almost anything you can think of on the internet.

Bots are so prevalent in the digital media ecosystem that even the most sophisticated publishers can't avoid them. The good news is that the increase in bot traffic last year was mostly due to the addition of good bots in the ecosystem, while bad bot traffic remained nearly the same.

(function () { var attempt = 0, init = function(){ if (window.pym) { var pymParent = new pym.Parent("2017-02-04-internet-robots", "https://graphics.axios.com/2017-02-04-internet-robots/internet-robots-chart.html", {}); } else if (attempt++ < 40) { setTimeout(init, 50); } }; init(); })();

Data: Imperva Incapsula Bot Traffic Report. Based on 16.7+ billion visits to 100,000 randomly-selected domains on the Incapsula network; Chart: Lazaro Gamio / Axios

Good bots: Almost a quarter of internet traffic is good bots, which monitor or enhance the web. Common examples of good bots are "spiders," which Google uses to map the internet to expand their search results, and "feed fetchers,"which Facebook uses to refresh your newsfeed on mobile. Together, Google search "spiders" and Facebook "feed fetchers" make up over 8% of all web traffic.

Bad bots: Bad bots are typically used to steal content from websites, mimic websites to capture ad dollars, create spam pages to collect user data, or shut down websites by overwhelming them with traffic. According to the same study, commissioned by cybersecurity firm Imperva Inc., bad bots make up nearly 30 percent of all internet traffic, and the majority of them impersonate real websites or people to bypass security measures. In December, White Ops, a cybersecurity firm focused on malware and ad fraud, uncovered the largest bad bot criminal ring in history, called the "Methbot." Methbot stole millions of advertising dollars, mostly from ad networks, by recreating common websites and stealing their ad traffic and revenue. White Ops Senior Director Mark Schlosser tells Axios that cybercriminals in the ad space have become so advanced that the traffic they steal "is able to fake all key advertising metrics including clicks, viewability and even simple email signups."

Why it matters: The Association of National Advertisers and White Ops estimate that bots cost advertisers more than $7 billion in revenue annually. Most ad fraud occurs in the programmatic space, where ads are purchased through an automated bidding system. Sophisticated ad agencies will place trackers on ads that enable them to only bid on quality inventory, which saves their clients money and makes for more effective ad campaigns. One of Washington's largest advertising agencies tells Axios that they have dozens of team members working to ensure advertising is seen by real people, not bots. "The investment we have made in technology to ensure we aren't buying fraudulent traffic is extensive," says Ben Coffey Clark, a partner at Bully Pulpit Interactive. "It's a trust we take very seriously."

Go deeper

3 hours ago - Sports

Pac-12 will play football this fall, reversing course

A view of Levi's Stadium during the 2019 Pac-12 Championship football game. Photo: Alika Jenner/Getty Images

The Pac-12, which includes universities in Arizona, California, Colorado, Oregon, Utah and Washington state, will play football starting Nov. 6, reversing its earlier decision to postpone the season because of the coronavirus pandemic.

Why it matters: The conference's about-face follows a similar move by the Big Ten last week and comes as President Trump has publicly pressured sports to resume despite the ongoing pandemic. The Pac-12 will play a seven-game conference football season, according to ESPN.

Dave Lawler, author of World
3 hours ago - World

Global coronavirus vaccine initiative launches without U.S. or China

Data: Gavi, The Vaccine Alliance; Map: Naema Ahmed/Axios

A global initiative to ensure equitable distribution of coronavirus vaccines now includes most of the world — but not the U.S., China or Russia.

Why it matters: Assuming one or more vaccines ultimately gain approval, there will be a period of months or even years in which supply lags far behind global demand. The COVAX initiative is an attempt to ensure doses go where they're most needed, rather than simply to countries that can produce or buy them at scale.

Updated 4 hours ago - Politics & Policy

Coronavirus dashboard

Illustration: Sarah Grillo/Axios

  1. Global: Total confirmed cases as of 6:15 p.m. EST: 32,062,182 — Total deaths: 979,701 — Total recoveries: 22,057,268Map.
  2. U.S.: Total confirmed cases as of 6:15 p.m EST: 6,967,103 — Total deaths: 202,558 — Total recoveries: 2,670,256 — Total tests: 97,459,742Map.
  3. Health: Cases are surging again in 22 states — New York will conduct its own review of coronavirus vaccine.
  4. Business: America is closing out its strongest quarter of economic growth.
  5. Technology: 2020 tech solutions may be sapping our resolve to beat the pandemic.
  6. Sports: Here's what college basketball will look like this season.
  7. Science: During COVID-19 shutdown, a common sparrow changed its song.

Get Axios AM in your inbox

Catch up on coronavirus stories and special reports, curated by Mike Allen everyday

Please enter a valid email.

Subscription failed
Thank you for subscribing!