Securitymetrics.org

Measuring security effectiveness.

welcome

Welcome to securitymetrics.org, a community website for security practitioners. Securitymetrics.org offers a community blog (this website) and a members-only mailing list.

topics

aggregation · benchmarking · catalog project · definitions · empirical studies · metricon · modeling · ROI · visualization

contributing

Review the proceedings from the Metricon 8 conference, which was held on March 1st, 2013 at the RSA Conference in San Francisco.

Join the mailing list.

Announcing Metricon 1.0

- - posted in metricon | Comments

At this year’s RSA show, a decent portion of the securitymetrics mailing list (about 30 people) convened for lunch. I enjoyed meeting my colleagues immensely, and I received good feedback from others who attended.

One thing everyone agreed on is there is enough activity in the security metrics area to merit convening the group a bit more formally. Thus, I am pleased to announce Metricon 1.0, the first-ever convention devoted exclusively to security metrics.

Metricon 1.0 will be held in Vancouver on August 1, 2006. The program chair is Pete Lindstrom. The program committee includes me and Dan Geer, who managed to persuade the USENIX folks to allow us to attach Metricon 1.0 to their own gathering.

We will publish more details shortly, along with a Call for Participation.

Mark your calendars!

Aggregation

- - posted in aggregation | Comments

The following provides information on techniques for aggregating, refining, and reporting security metrics. Fair game for this page includes technologies for gathering and transforming raw data, generating reports, and creating security dashboards.

  • Security Metrics Guide for Information Technology Systems, Swanson M, Bartol N, Sabato J, Hash J, & Graffo L, Security Metrics Guide for Information Technology Systems, NIST Special Publication 800-55 (Washington, DC: National Institutes of Standards and Technology, 2003), 99 pp, PDF.

    This document provides guidance on how an organization, through the use of metrics, identifies the adequacy of in-place security controls, policies, and procedures. It provides an approach to help management decide where to invest in additional security protection resources or identify and evaluate nonproductive controls. It explains the metric development and implementation process and how it can also be used to adequately justify security control investments. The results of an effective metric program can provide useful data for directing the allocation of information security resources and should simplify the preparation of performance-related reports.

  • Implementing a Network Security Metrics Program, Lowans PW, Global Information Assurance Certification Thesis, March 2004, 10pp, MS Word.

    You need to measure something before you can manage it. Metrics are the only way you can measure the quality of your network and its security. It is the only way you can tell if the improvements to your security are working. You need to be able to report this quality to your management and they mainly understand numbers, percentages, graphs and charts. They need to know the threats to their network and the amount risk in not taking action to correct them. Metrics can help you quantify this information. This paper will provide you with information on how to implement a security metrics program that is based in part on the already extensive amount of information on software metric programs

  • Metrics: You Are What You Measure!, Hauser JR & Katz GM, European Management Journal, October 1998, 28pp, PDF.

    This paper focuses on the selection of good metrics. There is no magic bullet. Many metrics seem right and are easy to measure, but have subtle, counter-productive consequences. Other metrics are more difficult to measure, but focus the enterprise on those decisions and actions that are critical to success. We suggest how to identify metrics that achieve balance in these effects and enhance long-term profitability. To gain an understanding of the properties of good metrics we begin with a summary of how metrics fail. These seven pitfalls provide examples of where metrics have produced counter-productive results. We then suggest a seven step system to design effective, “lean” metrics.

  • The Goal: A Process of Ongoing Improvement, Eliyahu Goldratt and Jeff Cox, North River Press, copyright 1984, 3rd edition, 2004, 386 pp, paperback.

    This “business novel” launched a thousand careers in process improvement through metrics. It focuses on critical thinking skills and especially on defining the right questions to answer. A solid foundation for practical efforts at metrics.

  • A Guide to Security Metrics, Payne SC, SANS Security Essentials GSEC Practical Assignment Version 1.2e, July 11, 2001, 7 pp, PDF.

    Security managers will, more than ever before, be held accountable for demonstrating the effectiveness of their security programs and the value of those programs to the organization. Some experts believe that a key aspect of this accountability will be security metrics. This guide provides a definition of security metrics, explains their value, discusses the difficulties in generating them, and suggests a methodology for building a security metrics program.

  • Metrics of Network Integrity, Soo Hoo KJ, Sygate Technologies White Paper, July 2004, 6 pp, PDF.

    This paper demonstrates the metrics development process by defining a goal of network integrity and building specific metrics to support the decisions of each principal security constituent.

Metrics Definitions

- - posted in definitions | Comments

No consensus exists on what security metrics should be used for measuring security effectiveness. This page documents commentary on metrics definitions from external sources.

Metrics Definitions

The Robert Frances Group recently reported in CSO magazine that the companies it surveyed used these metrics definitions:

Metric% using
Viruses detected in user files92.3%
Viruses detected in e-mail messages92.3%
Invalid logins (failed password)84.6%
Intrusion attempts84.6%
Spam detected/filtered76.9%
Unauthorized website access (content filering)69.2%
Invalid logins (failed username)69.2%
Viruses detected on websites61.5%
Unauthorized access attempts (internal)61.5%
Admin violations (unauthorized changes)61.5%
Intrusion successes53.8%
Unauthorized information disclosures38.5%
Spam not detected (missed)38.5%
Spam false positives30.8%
Other23.1%

Click on the link above to see the full article.

Security Modeling

- - posted in modeling | Comments

This page includes links to research that attempts to model security phenonema mathematically.

  • Cascade control in complex networks, Motter AE, Max Planck Institute, report #mpi-pks/0312006, 4pp, 10 July 2004, PDF.

    Complex networks with a skewed distribution of loads may undergo a global cascade of overload failures when key elements of the network are attacked or removed. Since a small shock has potential to trigger a global cascade, a fundamental question regards the possible mechanisms of defense. Here we show that a selective further removal of network elements can be used to prevent the cascade from propagating through the entire network, substantially reducing the damage caused by the attack or failure.

  • Technological Networks and the Spread of Computer Viruses, Balthrop J, Forrest S, Newman MEJ, & Williamson MM, Science, v304 p527-529, 23 April 2004, PDF.

    Targeted vaccination strategies for the control of computer viruses are unlikely to be generally effective because the networks over which viruses spread are not sufficiently dominated by highly connected nodes, and because network topology can be influenced strongly by the way in
    which a virus is written. Throttling provides a promising alternative
    strategy that works with any network topology and can greatly reduce
    viruses’ impact by slowing their spread to the point where
    they can be treated by conventional means.

  • Epidemic Spreading in Scale-Free Networks, Pastor-Satorras R & Vespignani A, Physical Review Letters, v86 n14 p3200-3203, 2 April 2001, PDF.

    The Internet has a very complex connectivity recently modeled by the class of scale-free networks. This feature, which appears to be very efficient for a communications network, favors at the same time the spreading of computer viruses. We analyze real data from computer virus infections and find the average lifetime and persistence of viral strains on the Internet. We define a dynamical model for the spreading of infections on scale-free networks, finding the absence of an epidemic threshold and its associated critical behavior. This new epidemiological framework rationalizes data of computer viruses and could help in the understanding of other spreading phenomena on communication and social networks.

Empirical Studies

- - posted in empirical | Comments

This section includes links to studies and research measuring current security practices. The topics may (or may not) be related to security economics. Fair game includes end-user policies and practice, password effectiveness, patch management, and other subjects that lend themselves to controlled studies.

End-User Security

  • The Memorability and Security of Passwords — Some Empirical Results, Yan J, Blackwell A, Anderson R, & Grant A, June 2004.

    Ross Anderson and colleagues from Cambridge University have released an empiricial study of password effectiveness measuring the relative effectiveness of simple, random, and mnemonic passwords. The principal headline: “passwords based on mnemonic phrases are just as hard to crack as random passwords yet just as easy to remember as naive user selections.” Highly recommended.

Network Security

  • AusCERT Presentation on MS Security Bulletins, Cooper R, May 2004

    Russ Cooper gave a presentation at AusCert 2004 stating that patching does not reduce insecurity unless it can be done 100% effectively (which is impossible). His analysis was widely reported in the press. The link above is Russ’ summary, rather than the presentation itself. (Ed. – if you discover link to it, edit this entry)

  • The Laws of Vulnerabilities, Eschelbeck G, March 2004

    In this presentation given by the CTO of Qualys, the author highlights some findings based on network sensor vulnerability data aggregated across customers. The headlines: new vulnerabilities tend to have a half-life of 30 days, and 80% of vulnerabilty exploits (attack scripts) are available within 60 days of disclosure of the vulnerability. Insightful, although we wish the raw data were available for review.

Applications

  • The Security of Applications: Not All Are Created Equal, Andrew Jaquith, @stake, Inc., 2002.

    Companies increasingly require ways of prioritizing security initiatives. We have found that the best-designed e-business applications have one-quarter as many security defects as the worst. By making the right investments in application security, companies can out-perform their peers and reduce risk by eighty percent.