The Need for Test Data

Last week at the RSA Conference, I spoke to several vendors about their challenges offering products and services in the security arena. One mentioned a problem I had not heard before, but which made sense to me. The same topic will likely resonate with security researchers, academics, and developers.

The vendor said that his company needed access to large amounts of realistic computing evidence to test and refine their product and service. For example, if a vendor develops software that inspects network traffic, it's important to have realistic network traffic on hand. The same is true of software that works on the endpoint, or on application logs.

Nothing in the lab is quite the same as what one finds in the wild. If vendors create products that work well in the lab but fail in production, no one wins. The same is true for those who conduct research, either as coders or academics.

When I asked vendors about their challenges, I was looking for issues that might meet the criteria of Allan Friedman's new project, as reported in the Federal Register: Stakeholder Engagement on Cybersecurity in the Digital Ecosystem. Allan's work at the Department of Commerce seeks "substantive cybersecurity issues that affect the digital ecosystem and digital economic growth where broad consensus, coordinated action, and the development of best practices could substantially improve security for organizations and consumers."

I don't know if "realistic computing evidence" counts, but perhaps others have ideas that are helpful?

Comments

Unknown said…
Richard, you're absolutely correct that this is a challenge. Lack of good, labeled data, in useful quantities and in realistic formats and proportions, is a huge challenge in evaluating products, developing security analytics, etc. For instance, when it comes to testing network sensors, there's too much temptation to use traffic generators that don't represent application layer traffic very well. For evaluations, I tend to favor replay of existing, real-world traffic captured from the network location similar or identical to where the proposed solution would be placed. However, that tends to come with its own baggage, like the sensitivity of the data, such that the vendor can't get a copy to troubleshoot, and buying or building good, high-speed replay devices can be a huge undertaking in and of itself. The freely available data sets, such as SANS holiday challenges or Digital Corpora's, tend to be too small and too narrow in use case to be generally useful except to kick the tires. Every once in awhile, I still catch people trying to use ye olde "DARPA data" set from MIT Lincoln Labs in 1999, which is ancient and completely useless. I wish I had a good solution.
8:34 PM
Anonymous said…
PREDICT
The Protected Repository for the Defense of Infrastructure Against Cyber Threats

Synthetic and Real World Data sets
(Some may require user agreements)

https://predict.org
4:44 AM
Post a Comment

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technical or pol
Read more