Creating a strong metrics framework for your security awareness program can help companies measure the overall impact and demonstrate value to a company’s leadership if it’s aligned with the company’s strategic priorities.
Defining security awareness
A security awareness program is a structured approach to managing an organization’s human risk. You can gauge and measure the maturity of an awareness program by using the security awareness maturity model. This articles assumes a mature program is in place (at least stage three of the maturity model) and are actively partnered with, or are a part of the security team. Mature awareness programs manage human risk by answering three key questions in this order.
- Human risks: What are my top human risks? All human risk cannot be managed, which means companies must identify and prioritize the top human risks. This should be a data-driven process in partnership with key groups within security such as the incident response, security operations, cyber threat intelligence or risk management teams.
- Behaviors: What are the key behaviors that most effectively manage those risks? Once again, we need to prioritize behaviors, the fewer behaviors we focus on the more likely people will change those behaviors, and at a lower cost to the organization.
- Change: How do we motivate and enable people to change those behaviors?
Over time, technology, threats, and business requirements change. As such, the organization’s human risks, in coordination with the security team, should be reviewed and updated at least annually.
Analytics to measure
Once companies look at security awareness and managing human risk through this lens, It becomes much easier to identify what metrics companies should focus on. One thing companies should decide beforehand is if they want to measure and track behavior by individual or by role, department or business unit. If tracking at the individual level, be sure to take measures to protect the information and privacy of every individual. Depending on the size of the organization and the amount of data being collected, companies also may need to partner with someone in the organization who specializes in data analytics/business intelligence to help normalize/analyze findings.
Phishing has been the number one driver of breaches at a global level (2021 Verizon DBIR Report – p15). No matter the number of technical controls we throw at this problem, cyber attackers simply adapt and bypass them. As such we need to teach people how to identify and report these attacks. So, what do we measure? After people have been trained, measure their susceptibility to phishing attacks. Of our top human risks this one is the simplest to measure and why it is such a common metric.
- Click rates: Measure the overall click rate of the organization. When you first roll out phishing training this number will drop fast, perhaps from a 20% click rate to less than 2% click rate for more basic phishing templates. Once you are at around 2 to 3% click rate you may need to start using more difficult / targeted phishing templates. Most phishing vendors support a tiered approach enabling you to use different categories of phishing difficulty. Remember, the goal is not a 0% click rate, as once you hit 2% or less click rate with basic, beginner level phishing lures, first-time clickers are primarily new hires and this is a training event for them.
- Repeat click rates: For many organizations this is their most valuable phishing metric as this measures repeat clickers – the people who are not changing behavior and represent a far greater risk to the organization.
- Reporting rates: If you are training and enabling your workforce to report suspected phishing emails, this helps develop a human sensor network. For this, it’s not so much the number of people that report is key, but how fast the security team gets the first reports. The sooner people report a suspected incident, the faster the security team can respond and manage potential incidents. People who report represent the most resilient of the workforce, as they are not only identifying attacks, but enabling the security team to respond and secure the entire organization more proactively.
For several years now, passwords continue to also be a primary driver of breaches. Cyber attackers have changed their tactics, techniques and procedures (TTPs), moving from gaining access or lateral movement by continually hacking into and infecting systems to using legitimate accounts to more easily pivot and traverse through a victim organization while avoiding detection. As such, both strong passwords and the secure use of those passwords have become key.
- Strong passwords: Ensure people are adapting and using strong passwords. Length is the new entropy; passphrases are now highly encouraged. This can be tested by running brute force/cracking solutions against password databases.
- Password manager adoption: We in many ways have made passwords difficult, confusing, and even intimidating for people with various rules and policies. As such, organizations are starting to adopt password managers to make passwords simpler for their workforce. If the organization is/has deployed password managers, measure the Password Manager adoption, and use rate. What percentage of the workforce is using password managers? Companies should be able to pull this data from which ever department is deploying/managing password managers.
- Multi-factor authentication (MFA) adoption: MFA is especially important for critical or sensitive accounts. Once again, this information should be accessible from whomever is responsible for deploying the MFA solution, responsible for the logging of authentication systems, leads Identity and Access Management, or part of Operations or Security.
- Password reuse/password sharing: Are people reusing the same password across different work accounts (or even worse reusing work and personal accounts)? Or are people sharing their passwords with fellow co-workers? While this behavior sounds difficult to measure you can effectively measure both behaviors with a security behavior/culture survey. The key is using a scientific approach to how you write and measure the survey results. For example, one way to measure password sharing would be to ask the workforce: On a scale of 1 to 5, how likely would one of your co-workers share their password with a fellow employee.
We want to ensure the computers and devices people are using, and the applications and apps installed on them, are updated and current. For some organizations this is not an issue as people do not have admin rights or control over work issued devices, instead their devices are actively patched by IT. However, for many organizations this is an issue as so many people are now working remotely from home and are often using personal devices or home networks for work access. There are several ways to measure this.
- For any devices the organization issues, the operations, IT, or perhaps even vulnerability management teams should be able to remotely track the update status of those devices. In some cases, solutions such as mobile device management (MDM) may be installed on personal devices which can also track updating status.
- The learning management system (LMS) or phishing platform may be able to automatically track the device, operating system and browser version of any device that connects to them.
- Assess and survey your workforce to determine if they understand the importance of updating and are actively updating their personal devices, to include enabling automatic updating.
Strategic metrics to consider
Once you start collecting metrics on people’s behaviors, you can use this data to better understand and manage your overall human risk. Three key uses include:
- Identify what regions, departments, or business units have the fewest secure behaviors and represent the greatest risk to the organization.
- Identify what regions, departments, or business units are most successfully changing behavior and why. Use lessons learned to apply to your less secure departments or regions.
- When an incident does happen, understand whether that individual was trained. Was the department they were in one of the most secure or least secure departments or business units?
You can also demonstrate the strategic value of your program to leadership by aligning behavior with what leadership really cares about.
- Number of incidents: As people change behavior, the overall number of incidents should go down, such as number of infected devices due to people falling victim to phishing attacks or account take-overs due to bad passwords.
- Attacker dwell time: The time it takes to detect a successful cyber attacker in your organization should decrease as you develop a Human Sensor network. The less time an attacker is on your network (dwell time) the less damage they can do.
- Cost of incidents: By reducing the number of incidents, and the dwell time of successful attackers, we can reduce overall costs.
- Policy and audit violations: As behaviors change we should see a reduction in the number (or severity) of policy and audit violations.
This list is neither exhaustive nor perfect, but it’s a starting point. There are a huge number of other metrics you can measure, and sources of data for those metrics. The key is not to measure everything; companies are better off measuring their most useful metrics. To do that, companies first need to know what the top human risks are and the behaviors that manage those risks.