Most Internet traffic comes from bots, not people

A new report shows that more than half of Internet traffic comes from bots — software applications that perform automated tasks for almost anything you can think of on the internet.

Bots are so prevalent in the digital media ecosystem that even the most sophisticated publishers can't avoid them. The good news is that the increase in bot traffic last year was mostly due to the addition of good bots in the ecosystem, while bad bot traffic remained nearly the same.

(function () { var attempt = 0, init = function(){ if (window.pym) { var pymParent = new pym.Parent("2017-02-04-internet-robots", "https://graphics.axios.com/2017-02-04-internet-robots/internet-robots-chart.html", {}); } else if (attempt++ < 40) { setTimeout(init, 50); } }; init(); })();

Data: Imperva Incapsula Bot Traffic Report. Based on 16.7+ billion visits to 100,000 randomly-selected domains on the Incapsula network; Chart: Lazaro Gamio / Axios

Good bots: Almost a quarter of internet traffic is good bots, which monitor or enhance the web. Common examples of good bots are "spiders," which Google uses to map the internet to expand their search results, and "feed fetchers,"which Facebook uses to refresh your newsfeed on mobile. Together, Google search "spiders" and Facebook "feed fetchers" make up over 8% of all web traffic.

Bad bots: Bad bots are typically used to steal content from websites, mimic websites to capture ad dollars, create spam pages to collect user data, or shut down websites by overwhelming them with traffic. According to the same study, commissioned by cybersecurity firm Imperva Inc., bad bots make up nearly 30 percent of all internet traffic, and the majority of them impersonate real websites or people to bypass security measures. In December, White Ops, a cybersecurity firm focused on malware and ad fraud, uncovered the largest bad bot criminal ring in history, called the "Methbot." Methbot stole millions of advertising dollars, mostly from ad networks, by recreating common websites and stealing their ad traffic and revenue. White Ops Senior Director Mark Schlosser tells Axios that cybercriminals in the ad space have become so advanced that the traffic they steal "is able to fake all key advertising metrics including clicks, viewability and even simple email signups."

Why it matters: The Association of National Advertisers and White Ops estimate that bots cost advertisers more than $7 billion in revenue annually. Most ad fraud occurs in the programmatic space, where ads are purchased through an automated bidding system. Sophisticated ad agencies will place trackers on ads that enable them to only bid on quality inventory, which saves their clients money and makes for more effective ad campaigns. One of Washington's largest advertising agencies tells Axios that they have dozens of team members working to ensure advertising is seen by real people, not bots. "The investment we have made in technology to ensure we aren't buying fraudulent traffic is extensive," says Ben Coffey Clark, a partner at Bully Pulpit Interactive. "It's a trust we take very seriously."