Overview

Security metrics -- an idea whose time has come. No matter whether you read the technical or the business press, there is a desire for converting security from a world of adjectives to a world of numbers. The question is, of course, how exactly to do that. The advantage of starting early is, as ever, harder problems but a clearer field though it is very nearly too late to start early. MetriCon is where hard progress is made and harder problems brought forward.

The MetriCon Workshops offer lively, practical discussion in the area of security metrics. It is a, if not the, forum for quantifiable approaches and results to problems afflicting information security today, with a bias towards practical, specific implementations. Topics and presentations will be selected for their potential to stimulate discussion in the Workshop. Past events are detailed here and here; see, especially, the meeting Digests on those pages.

MetriCon 3.0 will be a one-day event, Tuesday, July 29, 2008, in San Jose, California, USA. The Workshop begins first thing in the morning, meals are taken in the meeting room, and work/discussion extends into the evening. As this is a workshop, attendance is by invitation (and limited to 60 participants). Participants are expected to "come with findings," to "come with problems," or, better still, both. Participants should be willing to discuss what they have and need, i.e., to address the group in some fashion, formally or not. Preference will naturally be given to the authors of position papers/presentations who have actual work in progress.

Presenters will each have a short 10-15 minutes to present his or her idea, followed by a another 10-15 minutes of discussion. If you would like to propose a panel or a group of related presentations on different approaches to the same problem, then please do so. Also consistent with a Workshop format, the Program Committee will be steered by what sorts of proposals come in response to this Call.

Goals and Topics

Our goal is to stimulate discussion of, and thinking about, security metrics and to do so in ways that lead to realistic, early results of lasting value. Potential attendees are invited to submit position papers to be shared with all, with or without discussion on the day of the Workshop. Such position papers are expected to address security metrics in one of the following categories:

Benchmarking of security technologies
Empirical studies in specific subject matter areas
Financial planning
Long-term trend analysis and forecasts
Metrics definitions that can be operationalized
Security and risk modeling including calibrations
Tools, technologies, tips, and tricks
Visualization methods both for insight and lay audiences
Data and analyses emerging from ongoing metrics efforts
Other novel areas where security metrics may apply

Practical implementations, real world case studies, and detailed models will be preferred over broader models or general ideas.

How to Participate

Submit a short position paper or description of work done or ongoing. Your submission must be brief -- no longer than five (5) paragraphs or presentation slides. Author names and affiliations should appear first in or on the submission. Submissions may be in PDF, PowerPoint, HTML, or plaintext email and must be submitted to metricon3 AT securitymetrics.org. These requests to participate are due no later than noon GMT, Monday, May 12, 2008 (a hard deadline).

The Program Committee will invite both attendees and presenters. Participants of either sort will be notified of acceptance quickly -- by June 2, 2008. Presenters who want hardcopy materials to be distributed at the Workshop must provide originals of those materials to the Program Committee by July 21, 2008. All slides, position papers, and what-not will be made available to all participants at the Workshop. No formal academic proceedings are intended, but a digest of the meeting will be prepared and distributed to participants and the general public. (Digests for previous MetriCon meetings are on the past event pages mentioned above.) Plagiarism is dishonest, and the organizers of this Workshop will take appropriate action if dishonesty of this sort is found. Submission of recent, previously published work as well as simultaneous submissions to multiple venues is entirely acceptable, but only if you disclose this in your proposal.

Location

MetriCon 3.0 will be co-located with the 17th USENIX Security Symposium at the Fairmont Hotel in San Jose, California.

Cost

$225 all-inclusive of meeting space, materials preparation, and meals for the day.

Important Dates

Requests to participate: by May 12, 2008
Notification of acceptance: by June 2, 2008
Materials for distribution: by July 21, 2008

Workshop Organizers

Dan Geer, Geer Risk Services, Chair
Bob Blakley, The Burton Group
Fred Cohen, Fred Cohen & Associates & California Sciences Institute
Dan Conway, Indiana University
Lloyd Ellam, Iceberg Networks
Andrew Jaquith, The Yankee Group
Elizabeth Nichols, PlexLogic
Gunnar Peterson, Arctec Group
Bryan Ware, Digital Sandbox
Christine Whalley, Pfizer

The queue of people who want to get in is -- and has been for a while -- about 175 deep. The queue is not getting smaller. This is officially a "success disaster." The situation isn't going to improve soon, either, until I get some more help with the mailing list.

In the meantime, you can significantly increase your chances of being approved quickly by doing the following:

Use your work e-mail address

Do not submit more than one request

When all else fails, send me an e-mail to my work address directly

Sorry for appearing to be cranky. It's not so much crankiness as exasperation. This list has been wildly successful, and that's a tribute to the membership. But the process for adding new members isn't scaling well, and I hope you will bear with me for a little while longer.

1. How many online accounts do you manage, in total? How many "sensitive" accounts do you maintain?

By "account" I mean a public or private website, server or network that you log in to, for which you maintain a password or other credential. For example, a password or application entry in an OS X Keychain could be considered an account.

For purposes of this question, "sensitive accounts" means ones that you would consider problematic if they were compromised. Typically, these could be accounts that keep credit card information, manage your 401k details, or contain employment details.

Results (n=51):

Metric	All accounts	Sensitive accounts
Mean	60.7 accounts	20.6 accounts
Standard deviation	55.0	29.7
Min	3	0
First quartile	23.5	6
Median	40	15
Second quartile	72.5	25
Max	207	207
Mode	40	20

Comments: I draw 3 conclusions from these figures.

• First, people have lots of accounts to keep track of -- on average.
• That said, the quartiles and median show that respondents skew towards the "conservative case" -- that is, they most don't tend to maintain too many accounts. A few crazy outliers (like me) are pushing the average number up.
• Third, the ratio of sensitive-to-non-sensitive accounts stays fairly constant across quartiles, ranging from 26-38%. In other words: of all of the account passwords people maintain, it's a fair bet that about a third of them will be "sensitive."

I'd also note that the survey base is self-selected -- in the sense that it's the members of this list. Most of us are professional paranoids, right? Not sure if that means that the average user is worse off than the respondent base (more passwords to keep track of) or better off. Regardless, I'd say it does confirm what I already knew: we're drowning in passwords. Further insights or armchair-psychology comments welcome.

2. What is your primary coping strategy for managing your online accounts?

• I keep all of my passwords the same: 10%
• I write everything down on paper: 12%
• I use a form-filler product, like Apple's Keychain, and use random passwords 12%
• No particular strategy: 20%
• Other: 47%

Comments: I can't draw too many conclusions from the responses to this question, because I asked it badly. Considering that my day job is as an analyst, you'd think I would've asked this question in a way that got better answers. :)

3. Do you like the idea of surveying securitymetrics.org members about security practices?

• Yes: This is a good idea: 92%
• No: I've got enough spam as it is: 8%

Comments: Everyone seems to like the idea of surveying the membership more often. Cool! I've asked mailing list members to suggest ideas for future surveys.

Note: I've proposed that we spend some time on the subject of community-building at this year's Mini-Metricon at RSA. More on this later... Betsy Nichols is going to put up a blog entry about Mini-Metricon on the website later today.

The conversation they're having puzzles the man a bit. One of comics at the table yells out, "12!" and everybody just dies laughing. Then another one says, "44!" and a three of them laugh so hard they roll straight out of their chairs and onto the floor.

When a lull in the conversation comes, the new guy introduces himself, and asks, "Hey, what's going on? What's so funny about yelling out numbers?"

One of the comics says, "Oh, you're the new kid on the block, eh? Here's what's going on. We've all been retired for many years. We've been telling and re-telling the same old jokes for so long, we've assigned them all numbers. To save time, instead of telling the joke again, we just say the number!"

"Wow," says the new guy. "I've never seen that before. That's pretty cool. Mind if I join you?"

"Sure," the other comic says, and beckons him to sit down.

The new guy is eager to fit in. So five minutes later, he yells out, "28!" NOBODY laughs -- you could've heard a pin drop.

His voice qwavering, the new guy asks, "What's wrong? Isn't number 28 a good joke too?"

"Sure it is," pipes in the other comic. "But it's all about the delivery!"

I mention this because I can't stand Jeff Jones' quarterly festivals of FUD. Rather than complain yet again, and in detail, about how dumb vulnerability-counting is, why the methodology is flawed, why it has limited bearing on security, how the system is easily gamed, why it's colored by Jeff's obvious agenda, and why it's a tragedy that Microsoft does not do what it should, namely mine the world's most complete bug databases and code repositories for truly compelling information about code quality and application security metrics.

But I won't do that again. I'm just going to, like these comics, just yell out the shorthand.

"Jeff Jones."

Note that I'm not laughing.

But now that the report is in the can, I almost wish I'd waited a few months before writing it. That's because perhaps one of the most elegant solutions for banks, stock trading accounts and credit unions seeking to combat identity theft might be one of the simplest. In essence, instead of having banks worry about whether the user's general-purpose browser is secure, why not require the user to run a dedicated browser that won't allow access to websites other than those its creator intended?

Indeed, if I had to make a prediction, I'd say that the future of Internet banking might look a lot like Todd Ditchendorf's Fluid site-specific browser.

For a well-written overview of what's going on with SSBs, see Chris Messina's article, Fluid, Prism, Mozpad and site-specific browsers.

I've got some e-mails in to the Mozilla and Webkit teams, to prepare for the research note. More details as they emerge.

But in the meantime, I expect that we'll all start hearing "site-specific browsers" and "security" in the same sentence, a lot, in 2008. Remember, you read it here first.

This is a fact-filled but eminently readable paper about 3,290 IRC-based botnet command and control networks in China from June 2006 to June 2007. In addition to doing the normal things you'd expect to see in a botnet analysis, the researchers analyzed the extent of malware samples circulated within the botnets. They also attempted to determine the effectiveness of nine anti-virus engines in detecting the samples in circulation.

If you don't want to read the whole thing, I've put together the Cliff's Notes, at least from the perspective of a data junkie like me. Here are some of the more interesting metrics from the report. Some of these are from the report itself, and I've derived others. Editorial comments are in italics.

• Number of infected bot nodes: 1.5 million over the reporting period, for those 1904 botnets they could analyze in detail. Average botnet size over the period was about 800 nodes. Biggest was 50,000 nodes
• Botnet nodes are strongly diurnal, caused by infected machines being powered off at night
• Average lifetime of a botnet C&C server was 54 days
• Only 8.8% of the IP addresses of bots corresponded to blacklisted IPs on Spamhaus. Not sure whether this means Spamhaus was ineffective, or just that bots have been getting sneakier. It does suggest that "reputation services" won't save us...
• Of the activities seen on the botnet channels, "spreading" commands (finding new victims) were 28% of the commands executed. DDOS attacks were 25%; information theft, 9.8%; self-update activities, 14%. This last figure is interesting; it tells me that bot executables on infected nodes are continually updating themselves to evade detection
• Botnet spreading methods used exploits for the ASN1, DCOM, LSASS vulns in 50% of the spread commands executed. "Weak password" spreading methods were used, by contrast, only about 6% of the time.
• Very few bot commands (low hundreds) were executed for visiting websites (which you'd expect to see for creating fraudulent search result clicks). That shoots my pet theory that bots are ideal "for distributed click fraud"... damn.
• Within the botnets, 2,000-4,000 samples of malware were collected every day, with peaks at 7,000 per day
• 90,000 unique samples collected overall (average 250 new/day)
• For unique samples seen for the first time (within 1 hour of collection) by an AV engine, the malware detection rate was 70% or higher for only 4 of the 9 AV engines used. The four were Kaspersky (92%), BitDefender (86%), Rising (79%) and Trend Micro (78%). The report did not disclose what the other five engines were, but they all came in at between 50.2% and 70% detection for new malware.
• Even when samples were 30 days old, none of the top four AV engines topped 94% detection. It's unclear how many engines would need to have been used together in order to catch "everything."

The report offers plenty of conclusions. My own meta-conclusions are these:

• AV is missing a lot of malware
• Unpatched systems seem to be a key spreading vector... still
• Bot infections are becoming self-modifying to ensure that they evade detection
• Reputation services don't seem to help much
• Distributed click fraud is not yet a popular money-making technique

My last conclusion is perhaps the least intuitive: automated honeypot systems are cool. So cool, in fact, that I'm surprised that the big AV companies aren't selling them yet as a standard detection tool. But, of course, that would tend to undermine the public position that anti-virus products offer "total protection" (to quote a McAfee product name) or "mega detection" (to quote Panda).

On a side note, for my employer Yankee Group I'm currently finishing up a report on the future of the anti-virus industry. Reports like these reinforce my view that security vendors will be forced to strengthen the detection and recovery parts of their product portfolios — and cool the silver-bullet rhetoric about perfect protection. But, that's thinking like a CISSP (the prevent-detect-recover triad), rather than like a desktop software vendor (software is installed, and problems are solved in one step).

Needless to say, it beats the pulp out of any of the other Internet security trend reports I've seen all year. Stupendous. I doff my feathered foofy cap in your general direction, sir Dan.

However, he also states that I got some things wrong. If you read his critique, he faults me for not pointing out that there's not much more broken in Web 2.0 that wasn't already broken. He is right in the sense that the problems are rooted in well-known anti-patterns — notably, ignorance of good security design. That's true of "1.0" apps too (and, I point this out).

What is different is that the Web 2.0 architectural style makes it easier and faster to hose yourself than ever before due to the fact that JavaScript is pretty much essential for any significant application.

I am reminded of the Simpsons episode where Homer decides to legally change his name to accelerate his career prospects. He settles on the name "Max Power" because it was on his hairdryer. At the dinner table that night, he lectures Bart:

"Boy, if there's one thing you should know, it's this. There's the right way, the wrong way, and the Max Power way."

"Uh Dad, isn't that the wrong way?"

"Yeah son, but FASTER."

From a security design standpoint, "Web 2.0" is the wrong way, but faster.

Today, Computerworld reports that CVSS version2 is now out. That's great news; congratulations to Gavin and the rest of the team. I hope Microsoft and other vendors actually start using it.

One thing about that Computerworld story that annoyed me, however, was Robert Beggs' comment that enterprises shouldn't use CVSS to "manage by the numbers." Specific critiques of CVSS aside, why shouldn't we do that? Isn't that the point of measuring things? I guess we should manage by voodoo instead.

Honestly, I find comments like this exasperating. On the other hand, you never know what a reporter is going to pick up on and write in a column. I've said some damned silly things, as throwaways, that were printed. (My comment to InformationWeek's Marty Garvey, calling Mozilla's tabbed browsing feature "the best thing since sliced bread," is one such stinker that got printed.)

Sophos released its list of Dirty Dozen spam-relaying countries. The avoweded goal of the report is to "name and shame" the countries whose servers are apparently the biggest spammers, and by implication, the most sloppily managed and secured.

Symantec's semi-annual Internet Security Threat Report, an otherwise fairly interesting read most of the time, always devotes about three pages to documenting the "top attacking countries," a subset of whose citizens have been determined to be involved in a variety of detectable online hijinks.

Now, I don't want to get off on a rant here, but I have three problems with these sorts of country metrics: accuracy, lack of throats to choke, and general pretentiousness of the whole exercise.

Accuracy

Lack of throats to choke

• Consistently measured,without subjective criteria
• Cheap to gather, preferably in an automated way
• Expressed as a cardinal number or percentage,not with qualitative labels like “high,” “medium,”and “low”
• Expressed using at least one unit of measure, such as “defects,”“hours,”or “dollars”
• Contextually specific: relevant enough to decision-makers so that they can take action

Exactly who is going to benefit from the knowledge that, say, "the US" (note the scare-quotes) is the most aggressive spammer? Who will take action? Will it be...

• The president, George W. Bush? Will he direct the SEC, GAO, FCC and the Treasury to declare a Global War on Spam Relays? Certainly not.
• The captains of industry, such as the member companies that comprise the Dow Jones industrial average? Do you think this information would cause the respective CEOs to call their CISOs on the carpet and get them inspect and correct all of their security systems so that the US, as a whole, would rank better in next month's report? Nope.
• Foreign multinationals? Will they suddenly start curtailing their e-mail and web traffic to US companies, for fear of catching cooties? Meh.
• Consumers? Do you think Johnny is going to pack up his tent and move to Lower Slobovia because the US is now far too dangerous a place to own a computer in, according to something he reads in The Register? Probably not, unless he wants to evade Bermudan gambling controls.

These are admittedly silly examples, but the point I am making is more serious. Namely, it is that no single decision-maker gains anything useful from country-by-country metrics. There is nothing here that a CISO, security director or individual consumer could use to make smarter decisions, allocate their dollars more wisely, or change behaviors for the better. Which brings me to objection three, which is...

The pretentiousness of the whole exercise

The more I think about it, the more irritated I get. Creating geographic charts with impressive numbers on them, knowing full well that nobody can use the information on them to make better decisions, is a really nice, neat way to have one's cake and eat it too. Symantec, Sophos and the like can marshal impressive statistics about particular countries, but they can't be used by anybody for any purpose. Because nobody can gain any benefit from them, nobody can possibly be offended, either. Thus: country-by-country metrics are a safe way to display apparent expertise without rocking the boat.

These reports might make for good PR. But where's the courage in them?

J'accuse!

In other words, security metrics produced by parties who are willing to stand up and say, "J'accuse!" would be useful to those responsible parties who can actually do something with the information.

Here are two example of real courage:

• Spamhaus. They have the de rigeur country-by-country report, of course, but they also report by ISP. Now that's more like it — somebodies we can finger!
• Support Intelligence. Their Month of 0wned Corporations blog initiative was a brilliant public relations move, and it got them written up in the New York Times. How much would you like to bet that most of these companies have found and eliminated most of the botnets that Support Intelligence documented?

I know that this post won't affect the prevailing sentiments or practices of the most aggressive marketeers in the security industry. We will keep seeing more useless country metrics. But I thought I'd mention it...

• What is the operating system and e-mail client you use at work?
• What is the operating system and e-mail client you use at home (or for personal activities)?

I've compiled some preliminary statistics for your reading pleasure. Thanks to the 27 people who responded out of a total membership of about 300. That's nearly a 10% response rate in less than a day — not bad at all!

Objectives and Methodology

I've compiled operating system and e-mail statistics from three related sources:

• Responses to my previous e-mail (27 replies) — what is your operating system and e-mail client at work and at home?
• Analysis of e-mail "X-Mailer" and related headers from the securitymetrics.org mailing list (20 June 2006 to present)
• Analysis of same from metricon@securitymetrics.org traffic (i.e., paper submissions) (31 March 2006 to present)

In total, I identified 170 people who have contributed to this mailing list or sent submissions to Metricon 1.0 and 2.0. Of those, 27 provided OS/email information to me directly; I relied on header analysis for the remaining 143.

In total, I was able to identify a "preferred" operating system (either the one specified as the 'home' OS in a direct e-mail to me, or the one identified in the header) for 93 people. I identified e-mail programs for 131 people.

Operating Systems

For home (n=28), the results are quite different:

Of the 27 respondents, 14 (55%) reported using a different OS at home compared to work. After taking into account X-Mailer headers, I've concluded that for members of this list ("security conscious people"), we can conclude that when they have a choice, our members slightly prefer Macs. Amazingly enough, this suggests that Windows is a minority operating system, at least on this list. Results (n=92):

E-Mail Clients

For home (n=28), the results are, once again, quite different — and quite diverse:

Of the 28 respondents, nearly 2/3 (17 or 63%) specified a different home e-mail client compared to the one they used at work. After analysis of X-Mailer headers is taken into account (n=131), I conclude that our members prefer webmail overall, and prefer free (and non-Microsoft) native clients.

Interesting, no? Statistically relevant — maybe not! Let the debates begin in earnest!