Decoding the Implications: A Guide to Making Sense of the 2023 MITRE ATT&CK Evaluation Results

MITRE Engenuity ATT&CK Evaluation: An In-Depth Analysis of Cybersecurity Vendors

Thorough and independent testing is a crucial aspect when evaluating a provider’s capabilities to protect an organization against increasingly sophisticated cyber threats. Among the trusted assessments, the annual MITRE Engenuity ATT&CK Evaluation stands out as a widely recognized and respected benchmark. This evaluation plays a vital role in assessing cybersecurity vendors, as it is nearly impossible to rely solely on their own claims of performance. It complements other evaluation methods such as vendor reference checks and proof of value (POV) evaluations by providing objective and holistic input. In this article, we will delve into MITRE’s methodology, explore the interpretation of the 2023 MITRE ATT&CK Evaluation results by Cynet, and highlight key takeaways from the evaluation.

MITRE’s Methodology for Testing Security Vendors

The MITRE ATT&CK Evaluation, conducted by MITRE Engenuity, involves testing endpoint protection products against simulated attack sequences inspired by real-life tactics used by well-known advanced persistent threat (APT) groups. In the 2023 evaluation, 31 vendor solutions were tested against the attack sequences employed by Turla, a sophisticated threat group based in Russia that has targeted victims in over 45 countries. It is important to note that MITRE does not rank or score vendor results; instead, it publishes the raw test data and provides some basic online comparison tools for buyers to assess vendors based on their organization’s unique priorities and needs. The interpretations of the results by the participating vendors are subjective and should be treated as such.

Interpreting the MITRE ATT&CK Results

The presentation of the MITRE ATT&CK Evaluation results may be unfamiliar to many, making it challenging for organizations to determine the best vendor fit. While other independent researchers may declare “winners” to simplify the decision-making process, MITRE takes a subjective approach. To navigate this complexity, it is essential to review the results and compare how participating vendors performed against Turla.

Key Measurements for Interpreting Results

When analyzing the results of participating vendors, two key measurements stand out: overall visibility and detection quality. These metrics provide valuable insights into a solution’s ability to accurately detect and effectively respond to threats.

Threat Visibility

Endpoint protection solutions are primarily evaluated based on their ability to detect threats. Detecting each step of an attack sequence, as defined by the MITRE ATT&CK model, is critical for effective protection. In the case of the Turla attack sequence, which consisted of 19 steps with 143 substeps, visibility is measured in terms of the number or fraction of detections out of the total possible chances. It is worth noting that MITRE allows vendors to reconfigure their systems to improve threat detection after a miss, but this modifier does not reflect real-world scenarios. Therefore, it is more meaningful to prioritize detections without configuration changes when reviewing the MITRE ATT&CK Evaluation outcomes.

Analytic Detections

Analytic detections provide additional context by identifying the tactics and techniques associated with each detection. This information is invaluable for security analysts when investigating an alert and helps distinguish real threats from false positives. Vendors may not have provided analytic information for every step in the Turla attack sequence. Therefore, it is advisable to consider analytic information before any configuration changes were made.

Charting Visibility and Detection Quality

A comprehensive analysis of the solutions’ performance in detecting threats and providing actionable context can be represented through a chart. It highlights how well each vendor scored in these two critical areas. Missed detections pose a significant risk, potentially leading to breaches, while poor-quality detections can overwhelm security analysts or be disregarded, creating vulnerabilities. Therefore, it is important to consider both visibility and detection quality when evaluating vendors.

Expert Advice for Decoding the MITRE ATT&CK Results

As the MITRE ATT&CK Evaluation results can be complex to interpret, Cynet, a cybersecurity solutions provider, is hosting a webinar on September 20th to review the newly released results. The webinar aims to provide expert guidance for cybersecurity leaders to understand and leverage the results to select the most suitable vendor for their organization’s specific needs. Cynet CTO Aviad Hasnis and ISMG Senior Vice President, Editorial, Tom Field will share further details on the MITRE ATT&CK tests and outcomes during the webinar. Additionally, organizations can access Cynet’s full analysis of the 2023 MITRE ATT&CK Evaluation results.

About the Author

George Tubin, the author, is the Director of Product Strategy at Cynet. He is a recognized expert in cybercrime prevention, digital banking, and payments security. With prior experience as the Vice President of Marketing at Socure and as a Senior Research Director at TowerGroup (acquired by Gartner), George Tubin brings a wealth of knowledge on business strategies, technologies, cybersecurity, and identity and fraud management to the table.

”’

Conclusion: The Importance of Independent Evaluations

Thorough and independent evaluations such as the MITRE ATT&CK Evaluation play a crucial role in assessing the capabilities of cybersecurity vendors. Relying solely on vendors’ claims can be misleading, and evaluating their performance against simulated real-world attack sequences provides a more accurate perspective.

While the MITRE ATT&CK Evaluation results may initially seem complex, understanding key measurements such as threat visibility and detection quality can help organizations make informed decisions. By analyzing the outcomes and comparing vendors’ performances, organizations can identify the best cybersecurity solutions that align with their unique needs and priorities to enhance their security posture.

It is essential for organizations to leverage expert guidance, such as the upcoming webinar hosted by Cynet, to decode and interpret the MITRE ATT&CK Evaluation results effectively. By staying informed and using reliable and comprehensive analysis, organizations can make informed decisions in selecting the most appropriate cybersecurity vendor for their specific requirements.