- - posted in benchmarking | Comments

Benchmarking generally refers to the process of ranking or scoring security against an established standard measure. Benchmarks can be absolute or cross-sectional.

Comparative Application Security

  • The Security of Applications: Not All Are Created Equal (February 2002), Andrew Jaquith. This study examples the security practices of 45 web applications, and finds that the most secure e-business applications have one-quarter as many security defects as the worst — and eighty percent less risk.

Benchmarking Goodness Criteria

Established by the DBench Project.

Representativenesshow well inputs like workloads corresponds to real system characteristics
Repeatabilitystatistically equivalent results when run multilple times in the same environment
Reproducabilitydegree to which another party obtains statistically equivalent results when the benchmark is implemented from the same specifications
Portabilityrange of target systems to which benchmark specification applies to allow comparision
Non-Intrusivenessrequires minimum changes to target system and does not affect results
Scalabilityability to evaluate systems of different sizes
Timetime required to obtain the result
Costcost required to obtain result compared to value

Contributed by Sami Saydjari