Security Awareness Blog

Security Awareness Metrics - Part 4

In our recent series on security awareness metrics we discussed what makes good awareness metrics, how they can be implemented, and how they can demonstrate value of security awareness. One of the key metrics we focused on was phishing, re-creating the very same email attacks employees and organizations receive every day. I wanted to share with you some studies and success stories of organizations that have used these very same metrics.

  • Carnegie Mellon University published a study in December, 2008 called Lessons From a Real World Evaluation of Anti-Phishing Training. This was an academic study of the effectiveness of awareness training, and using phishing emails in the training (called PhishGuru). In addition they used phishing emails to track the effectiveness of their training. Over a ten day period they sent out four such assessments, if people fell victim to the assessment and clicked on any links they were immediately sent to a site that explained how they failed a phishing test and what phishing is. Over that period CMU saw an original failure rate of 42% drop to less then 15%. The training had an impact.
  • West Point ran several tests and published a study in 2006 called Phishing For User Security Awareness. One of the interesting findings from this report is the longer the student had been in the school, and the longer they had received awareness training, the more secure they were. Freshman had the highest failure rate of over 50%. There is a steady drop for each following grade level, with seniors having less then %20 failure rate.
  • In 2006 NY state did a phishing exercise against 10,000 government employees. In their original email assessment they had a %17 failure rate. They followed this with a month of education, then did another assessment against the same group of people. There was a %7 failure rate.

Keep in mind this type of research is still in its infancy, I'm sure we will see more research in this area. However even in this very initial research we can see every organization was able to use education to cut failure rates in half if not more. Awareness works and we are just scratching the surface on how to effectively use it. In addition, you may have noticed that all three organizations had very different failure rates, even though they had the relatively same drop in failures. The more targeted the emails are in an assessment, the more people will fall victim. NY State used very generic emails that replicated common attacks while West Point used far more targeted attacks replicating spear phishing. This is very important to keep in mind when doing your own assessment, how targeted do you want your emails to be. In our next post we will cover just that.