A new report shows that more than half of Internet traffic comes from bots — software applications that perform automated tasks for almost anything you can think of on the internet.

Bots are so prevalent in the digital media ecosystem that even the most sophisticated publishers can't avoid them. The good news is that the increase in bot traffic last year was mostly due to the addition of good bots in the ecosystem, while bad bot traffic remained nearly the same.

(function () { var attempt = 0, init = function(){ if (window.pym) { var pymParent = new pym.Parent("2017-02-04-internet-robots", "https://graphics.axios.com/2017-02-04-internet-robots/internet-robots-chart.html", {}); } else if (attempt++ < 40) { setTimeout(init, 50); } }; init(); })();

Data: Imperva Incapsula Bot Traffic Report. Based on 16.7+ billion visits to 100,000 randomly-selected domains on the Incapsula network; Chart: Lazaro Gamio / Axios

Good bots: Almost a quarter of internet traffic is good bots, which monitor or enhance the web. Common examples of good bots are "spiders," which Google uses to map the internet to expand their search results, and "feed fetchers,"which Facebook uses to refresh your newsfeed on mobile. Together, Google search "spiders" and Facebook "feed fetchers" make up over 8% of all web traffic.

Bad bots: Bad bots are typically used to steal content from websites, mimic websites to capture ad dollars, create spam pages to collect user data, or shut down websites by overwhelming them with traffic. According to the same study, commissioned by cybersecurity firm Imperva Inc., bad bots make up nearly 30 percent of all internet traffic, and the majority of them impersonate real websites or people to bypass security measures. In December, White Ops, a cybersecurity firm focused on malware and ad fraud, uncovered the largest bad bot criminal ring in history, called the "Methbot." Methbot stole millions of advertising dollars, mostly from ad networks, by recreating common websites and stealing their ad traffic and revenue. White Ops Senior Director Mark Schlosser tells Axios that cybercriminals in the ad space have become so advanced that the traffic they steal "is able to fake all key advertising metrics including clicks, viewability and even simple email signups."

Why it matters: The Association of National Advertisers and White Ops estimate that bots cost advertisers more than $7 billion in revenue annually. Most ad fraud occurs in the programmatic space, where ads are purchased through an automated bidding system. Sophisticated ad agencies will place trackers on ads that enable them to only bid on quality inventory, which saves their clients money and makes for more effective ad campaigns. One of Washington's largest advertising agencies tells Axios that they have dozens of team members working to ensure advertising is seen by real people, not bots. "The investment we have made in technology to ensure we aren't buying fraudulent traffic is extensive," says Ben Coffey Clark, a partner at Bully Pulpit Interactive. "It's a trust we take very seriously."

Go deeper

Updated 9 mins ago - Politics & Policy

Coronavirus dashboard

Illustration: Sarah Grillo/Axios

  1. Politics: Ex-FDA chief: Pence campaigning after COVID exposure puts others at risk — Mark Meadows: "We are not going to control the pandemic"— COVID-19 looms over White House Halloween celebrations.
  2. Health: 13 states set single-day coronavirus case records last week — Fauci says maybe we should mandate masks if people don't wear them — America was sick well before it ever got COVID-19.
  3. World: Italy tightens restrictions Spain declares new state of emergency.
Updated 15 mins ago - Health

13 states set single-day coronavirus case records last week

Data: Compiled by Axios; Map: Danielle Alberti/Axios

13 states set new highs last week for coronavirus infections recorded in a single day, according to the COVID Tracking Project (CTP) and state health departments. Kansas, Montana, North Dakota and Wyoming surpassed records from the previous week.

The big picture: The pandemic is getting worse again across the country, and daily coronavirus cases have risen in the U.S. for six straight weeks, according to a seven-day average tracked by Axios. The U.S. reported over 80,000 new cases on both Friday and Saturday.

Senate advances Amy Coney Barrett nomination, setting up final confirmation vote

Photo: Xinhua/Ting Shen via Getty Images

The Senate voted 51-48 on Sunday to advance the Supreme Court nomination of Judge Amy Coney Barrett, setting up a final confirmation vote for Monday.

Why it matters: It's now virtually inevitable that the Senate will vote to confirm President Trump's third Supreme Court nominee before the election, which is just nine days away.

Get Axios AM in your inbox

Catch up on coronavirus stories and special reports, curated by Mike Allen everyday

Please enter a valid email.

Subscription failed
Thank you for subscribing!