Mock Chinese attacks test security tools

When it comes to cybersecurity research, the not-for-profit lab MITRE has traditionally maintained neutrality toward commercial products. But last week, it released its first security product evaluations. Here's why and how MITRE made the leap into what might at first sound like Yelp territory (but really isn't).

Why it matters: MITRE is best known for its role in assisting the government in public/private partnerships. In cybersecurity, until now, a lot of its high-profile work was more as an archivist than an active defender.

• That's not meant as a dig. One of their key projects, the ATT&CK framework, is an important database of prominent hacking groups' techniques across a broad spectrum of categories.

What MITRE released last week were the results of simulated attacks from the believed-to-be-Chinese espionage group known as Gothic Panda or APT3 using the information collected for that ATT&CK framework. MITRE plans this release of product evaluations to be the first of many, with other tests gauging products against other attackers.

Yes, but: "We're not Consumer Reports," said Frank Duff, lead engineer for the evaluations program.

• MITRE isn't providing head-to-head comparisons of products or ranked lists. There are no best buys.
• The primary goal, at least to MITRE, is to encourage product improvement.
• But MITRE also expects potential buyers will use the results and vendors will view the process as, in part, a marketing exercise.
• "It can serve all of those purposes," said Duff.

Techniques, not malware: Before the MITRE tests were announced, there were already a lot of places for antivirus companies to test whether they could detect malicious programs that hackers installed on a system. But as CrowdStrike's Scott Taschler, director of product marketing, noted, "When it comes to advanced, targeted attacks, malware is only a part of the problem." A hacker might not use any malware, and security products still need to be tested on how they respond to those attacks.

Vendors paid for their tests, with the first cohort including Carbon Black, CounterTack, CrowdStrike, Endgame, Microsoft, RSA and SentinelOne.

• No product weeds out the simulated attacker at every technique and tactic — and that's largely by design. Products focus on specific subsections of security, and most companies being targeted by spies have more than one product running at a time.
• But the testing does show specifically where products could adapt to discover an attacker at different points. And that's important: As hackers upgrade their tools or learn new techniques, they can evade some previously effective defenses.

The bottom line: Scott Lundgren, chief technology officer at Carbon Black, said, "If the community rises up and documents and positions their security posture with ATT&CK in mind, we are all raising the bar and making it more expensive for adversaries to operate."