Measuring security effectiveness.


Welcome to, a community website for security practitioners. offers a community blog (this website) and a members-only mailing list.


aggregation · benchmarking · catalog project · definitions · empirical studies · metricon · modeling · ROI · visualization


Review the proceedings from the Metricon 8 conference, which was held on March 1st, 2013 at the RSA Conference in San Francisco.

Join the mailing list.


- - posted in visualization | Comments

This page catalogs techniques for representing security data visually. Clear, cogent, meaningful visual displays of information enable the audience to rapidy grasp the essence of security issues and trends. Below are some examplars, many of which come from outside the world of information security. ( Wikipedia definition: Information Visualization )

Charts and Graphs

  • Summarizing Clinical Psychiatric Data (November 1997) – Edward Tufte popularized a highly efficient charting technique called “small multiples.” The technique essentially graphs multiple items together, by compressing identically-scaled and labeled graphs onto a single chart. Tufte’s article on visualizing clinical patient data shows the small-multiple technique in action. With a little imagination it is easy to see how this can be applied to security. As an example, see Jaquith’s Application Security: Not All Are Created Equal paper.

Graphs and Network Visualization

Security Dashboards

Small Multiples

  • The New York Times Election Graphics. InfoWorld columnist and blogger Jon Udell scanned in a stunning chart that displays the “small multiple” technique plus some extremely creative “geographic” visualization. This appeared recently in the NYT’s print edition as part of the 2004 US election coverage. Jon believes his scan falls into the realm of fair use. We hope so too.

Pattern Visualization

Rendering Hierarchical Data

  • Wijk, J.J. van, F. van Ham, H.M.M. van de Wetering. Dr. van Wijk’s “squarification” algorithm (as used by Newsmap, above) is already the de facto standard for treemaps. What do do for an encore? In this short ACM paper, he and colleagues examine strategies for visualizing large, tree-like structures: treemaps, beamtrees, and “botanical” graphs.
  • Map of the Market, Java-applet-based treemap of stock market activity.
  • Freshcookies Treemap Library, Andrew Jaquith. As part of the research effort for his book on security metrics, Mr. Jaquith has created an open-source treemap library and a sample file-parsing application that reads tab-delimited text files. The library was used to produce the treemap graphics attached to this page. It is easy to use, well documented and free (as in beer).

Three-Dimensional Visualization

  • The Spinning Cube of Potential Doom, Stephen Lau, Lawrence Berkeley National Labs. The author’s custom three dimensional visualizer charts intrusion activity, based on output from the Bro network intrusion detection system. The related presentation explains the rationale. Yes, yes, one does appreciate the irony of linking to a PowerPoint presentation on the same page as a Tufte article.
  • Visitorville Weblog Analysis, As covered in Slashdot, weblog analysis meets SimCity. This product aggregates web log information and displays it as a three-dimensional cityscape. Very interesting; its potential utility to security metrics seems pretty clear. The ensuing Slashdot discussion thread was entertaining also.

Metricon 1 — the Inaugural Event

- - posted in metricon | Comments

Metricon 1.0 was held 1 August 2006 in Vancouver, British Columbia, Canada, coincident and co-located with the 15th USENIX Security Symposium. This page has the final agenda, copies of all presentation materials, and a digest summary of the meeting itself. (As is both typical and appropriate, let me hasten to say as the scribe for the affair that all errors are mine.)


- - posted in benchmarking | Comments

Benchmarking generally refers to the process of ranking or scoring security against an established standard measure. Benchmarks can be absolute or cross-sectional.

Comparative Application Security

  • The Security of Applications: Not All Are Created Equal (February 2002), Andrew Jaquith. This study examples the security practices of 45 web applications, and finds that the most secure e-business applications have one-quarter as many security defects as the worst — and eighty percent less risk.

Benchmarking Goodness Criteria

Established by the DBench Project.

Representativenesshow well inputs like workloads corresponds to real system characteristics
Repeatabilitystatistically equivalent results when run multilple times in the same environment
Reproducabilitydegree to which another party obtains statistically equivalent results when the benchmark is implemented from the same specifications
Portabilityrange of target systems to which benchmark specification applies to allow comparision
Non-Intrusivenessrequires minimum changes to target system and does not affect results
Scalabilityability to evaluate systems of different sizes
Timetime required to obtain the result
Costcost required to obtain result compared to value

Contributed by Sami Saydjari

Return on Investment

- - posted in ROI | Comments

Data Breaches

  • Lost Customer Information: What Does a Data Breach Cost Companies?, Ponemon Institute Survey sponsored by PGP Corporation, PDF. The Ponemon Institute’s benchmark study, sponsored by PGP Corporation, examines the costs incurred by 14 companies that experienced a data breach. Results were not hypothetical responses to possible situations; they represent cost estimates for activities resulting from data loss incidents.

Application Quality

  • Tangible ROI Through Secure Software Engineering, Soo Hoo K, Sudbury AW, & Jaquith AR, Secure Business Quarterly, 5 pp, Q2 2001, PDF. Securely engineering software to proactively fix problems has a concrete value. In this study of investments in security made during the design phase the authors show that ROI can be up to 21 percent.
  • The Economic Impact of Cyber Attacks, Cashell B, Jackson WD, Jickling M, & Baird W, Government and Finance Division, Congressional Research Service, The Library of Congress, document RL32331, 45 pp, 1 April 2004, PDF. This report surveys the state of knowledge on the cost of cyber-attacks and the economics of information security.
    • First, we summarize several studies that use stock market capitalization as a measure of the cost of cyber-attacks to victim firms.
    • Second, we present summaries of the existing empirical data on costs attributable to cyber-crime and computer worms and viruses.
    • Third, we analyze the reasons for the lack of statistical data.
    • Fourth, we examine the efforts of the insurance industry to develop policies that cover cyber-risk.
    • Finally, we consider cyber-attacks as macroeconomic events.