Feb 3, 2017

Most Internet traffic comes from bots, not people

A new report shows that more than half of Internet traffic comes from bots — software applications that perform automated tasks for almost anything you can think of on the internet.

Bots are so prevalent in the digital media ecosystem that even the most sophisticated publishers can't avoid them. The good news is that the increase in bot traffic last year was mostly due to the addition of good bots in the ecosystem, while bad bot traffic remained nearly the same.

(function () { var attempt = 0, init = function(){ if (window.pym) { var pymParent = new pym.Parent("2017-02-04-internet-robots", "https://graphics.axios.com/2017-02-04-internet-robots/internet-robots-chart.html", {}); } else if (attempt++ < 40) { setTimeout(init, 50); } }; init(); })();

Data: Imperva Incapsula Bot Traffic Report. Based on 16.7+ billion visits to 100,000 randomly-selected domains on the Incapsula network; Chart: Lazaro Gamio / Axios

Good bots: Almost a quarter of internet traffic is good bots, which monitor or enhance the web. Common examples of good bots are "spiders," which Google uses to map the internet to expand their search results, and "feed fetchers,"which Facebook uses to refresh your newsfeed on mobile. Together, Google search "spiders" and Facebook "feed fetchers" make up over 8% of all web traffic.

Bad bots: Bad bots are typically used to steal content from websites, mimic websites to capture ad dollars, create spam pages to collect user data, or shut down websites by overwhelming them with traffic. According to the same study, commissioned by cybersecurity firm Imperva Inc., bad bots make up nearly 30 percent of all internet traffic, and the majority of them impersonate real websites or people to bypass security measures. In December, White Ops, a cybersecurity firm focused on malware and ad fraud, uncovered the largest bad bot criminal ring in history, called the "Methbot." Methbot stole millions of advertising dollars, mostly from ad networks, by recreating common websites and stealing their ad traffic and revenue. White Ops Senior Director Mark Schlosser tells Axios that cybercriminals in the ad space have become so advanced that the traffic they steal "is able to fake all key advertising metrics including clicks, viewability and even simple email signups."

Why it matters: The Association of National Advertisers and White Ops estimate that bots cost advertisers more than $7 billion in revenue annually. Most ad fraud occurs in the programmatic space, where ads are purchased through an automated bidding system. Sophisticated ad agencies will place trackers on ads that enable them to only bid on quality inventory, which saves their clients money and makes for more effective ad campaigns. One of Washington's largest advertising agencies tells Axios that they have dozens of team members working to ensure advertising is seen by real people, not bots. "The investment we have made in technology to ensure we aren't buying fraudulent traffic is extensive," says Ben Coffey Clark, a partner at Bully Pulpit Interactive. "It's a trust we take very seriously."

Go deeper

Primary elections test impact of protests, coronavirus on voting

Election official at a polling place at McKinley Technology High School in Washington, D.C. Photo: Drew Angerer/Getty Images

In the midst of a global pandemic and national protests over the death of George Floyd, eight states and the District of Columbia held primary elections on Tuesday.

Why it matters: Joe Biden, the presumptive Democratic nominee, needs to win 425 of the 479 delegates up for grabs in order to officially clinch the nomination. There are a number of key down-ballot races throughout the country as well, including a primary in Iowa that could determine the fate of Rep. Steve King (R-Iowa).

Iowa Rep. Steve King defeated in GOP primary

Rep. Steve King. Photo: Alex Wroblewski/Getty Images

State Sen. Randy Feenstra defeated incumbent Rep. Steve King in Tuesday's Republican primary for Iowa's 4th congressional district, according to the Cook Political Report.

Why it matters: King's history of racist remarks has made him one of the most controversial politicians in the country and a pariah within the Republican Party.

Updates: George Floyd protests continue past curfews

Police officers wearing riot gear push back demonstrators outside of the White House on Monday. Photo: Jose Luis Magana/AFP via Getty Images

Protests over the death of George Floyd and other police-related killings of black people continued Tuesday across the U.S. for the eighth consecutive day — prompting a federal response from the National Guard, Immigration and Customs Enforcement and Customs and Border Protection.

The latest: Protesters were still out en masse even as curfews set in in New York City and Washington, D.C. Large crowds kneeled at Arizona's state capitol nearly an hour before the statewide 8 p.m. curfew, and a peaceful march dispersed in Chicago ahead of the city's 9 p.m. curfew.