Edition 8: To train or not to train
Edition 8: To train or not to train
Training is easy to get started, but hard to scale. Its also hard to measure outcomes from it. In this post, we explore alternates to training that can attain similar objectives.
Everyone agrees AppSec training is important. Most developers didn't learn to write code in a secure manner. Most architects never considered security while designing their application. They need to be taught to do this. Security and Engineering teams are convinced that we need to spend time and money to get this done.
To dig deeper on how to get this done. We need to answer a few key questions:
Whom to train?
What topics should we train them on?
What type of content works best
Is the training working?
Is this all worth it? In other words, what's the opportunity cost of training?
Are there other ways to solve the same problem?
Enthusiastic Security leaders would want to train all of Engineering[1], with a wide range of topics [2], with slick , on-demand content [3]. Doing all this is really expensive. Even if you have the budget, it is very hard to measure if any of these trainings are actually working (correlation is possible, causation is harder). This essentially leads to most organizations settling with OWASP Top 10 training for everyone (with some cool hacking thrown in to wow audience). This approach has 3 benefits:
Ticks the box on "AppSec training"
Provides good 0 to 1 training for junior engineers (especially ones with no exposure to AppSec)
Does not ruffle too many feathers
Engineering leaders don’t have to allocate time for AppSec training
Does not cost too much money
“Everybody does it” is a good way to convince stakeholders
It has one big drawback though: There is no way to know if this reduces AppSec risk. In fact, most Security folks are convinced that AppSec training does not really reduce AppSec risk. How do we solve this? I don't have a solution, but I do have a somewhat radical hypothesis:
Traditional AppSec training benefits 0 to 1 learning (i.e. introducing AppSec to folks who know nothing about it) and role specific training should be used to on-board engineers to AppSec. Beyond this, AppSec training should be replaced by measurable, enablement initiatives which have a fighting chance in reducing AppSec risk.
Build measurable, enablement initiatives
Let’s break this down. What do we mean by building measurable, enablement initiatives?
Measurable
We should be able to measure if this initiative is reducing AppSec risk. Risk reduces when:
We identify defects in our applications
We find a way to fix detected defects
We find ways to eliminate risk categories even before they are introduced.
Note: Most AppSec folks will tell you *this* is what training does. Train engineers before they can make mistakes. The problem is, the retention from most training activities is minimal and it is very hard to measure if the training actually had an impact.
Enablement
Broadly, Security teams manage risk through initiatives that use one of two methods:
Enforcing rules against risky behaviour - In the context of AppSec, think Penetration Tests with SLAs for remediation, security gates before go-live and so on
Enabling teams to reduce introducing risk - Providing teams the tools and techniques needed to not introduct risk in the first place and if that does not work, provide means to quickly reduce risk.
Most practitioners agree that Security teams need to do a bit of both. My hypothesis is we should look at all of "training" through the lens of enablement. How do we enable our engineering teams to not engage in risky behaviour?
Here are a few examples of measurable initiatives which will enable your engineers to reduce AppSec risk:
Build secure defaults
Secure defaults are what happens when AppSec standards grow up. Before we get into what I mean, I know there is a lot of jargon around what is a standard versus a guideline versus a policy. Here's my definition:
Standards - These are mandatory "rules" Engineering teams have to follow.
Guidelines - Recommendations on how to implement standards. Not mandatory, but not following guidelines means the burden of proof of adherence is on the developers
Now that's out of the way, let's address the elephant in the room. Nobody reads standards. Nobody reads the guidelines. You could hide a dead body on page 17 of your 48 page AppSec Standards document and no one will ever find it there. Not even your auditors.
How does this enable engineers?
OK, that got a little dramatic. The point I want to make is this: You still need standards and guidelines, but they need to be in a manner which makes it easy for developers to consume them (i.e. enable them to use it). A good option would be to convert these guidelines to libraries they can use. Build an encryption library that has functions/routines symmetric and asymmetric encryption and let developers consume it, build a logging library which makes it easy to mask sensitive information being logged and so on.
How do you measure progress?
Measure that number of times security controls are implemented through these libraries versus through bespoke solutions. That will tell you if the libraries are useful. You will need to evangelize these too, but that's a story for another day.
Use SAST as an enablement tool
We all know off and use static analysis as a defect discovery tool. But when applied closer to the developers IDE, static analysis can do a pretty good job at teaching developers secure coding. Many years ago, my former employer built Secureassist, a tool which acted as a spell-check on code. You just wrote a string concatenated SQL query? A red swiggly line would tell you there's a problem. While that idea never quite took off, there are enough IDE tools out there that help you now. In addition, pre-commit hooks and commit time checks which perform basic checks and automated scans on PRs (with helpful remediation guidelines) can go a long way in influencing behaviour.
How to use SAST as an enablement tool?
Focus on hygiene, not likelihood and impact. Take guidelines defined above and convert them to custom rules. Create alerts when guidelines are not followed, irrespective of possible impact. This can incentivise good behaviour
Don't make anything mandatory. Nobody likes being looked over the shoulder during training. When using SAST as a training tool, make sure developers are allowed to ignore some findings, not fix others and even turn them off it they so wish. If they see value, they will turn it back on.
This may require you to run 2 separate SAST programs. One as enablement, the other as enforcement. A lot of the infrastructure can be re-used, but governance needs to be well-defined
How to measure progress?
While we shouldn't make things mandatory, collecting data for some post game analysis can tell us how things are going. Some valuable metrics include:
Percentage of alerts dismissed. Tells you if the rules are well-written and if they guidelines are helpful.
Reduction in defect density over time (measurement at a team/individual level, not org level) — excellent metric to measure risk reduction
Build a security community withing your organization
Build a security community where non-security teams exchange ideas This ones a little fuzzier than the others, but if (like me), you believe that networking and interaction can sometimes spark unlikely conversations, then building an internal community can have outsized results
How does this enable engineers?
Carefully curating a community of AppSec stakeholders allows them to help each other. Security should take a back seat and play the role of organizer and match-maker only. For instance: Bringing together devs from various divisions can help them exchange ideas on how they solved a pesky problem (e.g.: convincing their team to make threat modeling a part of sprint). The chances of learning from each other dramatically increases in a well run community.
How do we measure progress?
Measuring this can be hard and fuzzy. Nonetheless, there are a few input metrics that can be used to know if this is just another "mandatory fun" activity or if people are actually interested
Number of initiatives which have moved from an division of your company to another
Make the events non-mandatory and measure attendance over time
A note on 0 to 1 training
As discussed earlier, With all these initiatives, there is still a need to learn certain topics which where knowledge is non existent. For such topics/areas, Security should focus on content curation. Some of the content can be built in-house too, but the focus should be on ensuring the latest and best content is available for use.
While it will be hard to measure risk reduction with such trainings (and hence, the limitation for 0 to 1 topics), there are a few things we can do to make them better:
Use gamification. This makes the training and gives you some metrics to measure engagement. Companies like Secure Code Warrior do a good job here. If you have the bandwidth, you could also consider open source solutions such as Secure Coding Dojo.
Host all trainings (or links to external training) in one single location. Making it easy to discover will improve adoption. Consider adding all on-demand training to the company LMS portal.
Set aside budget for paid conference trainings: Some of the best AppSec training happens in Security conferences. Set aside some budget for pointed training courses, which meet a specific need. If your company is moving towards Micrsoservices and you have no one in the Security team who is an expert (yet), an external course on Securing Microservices can be really helpful
Identify early adopters: In most in-person training classes I have conducted, there are always ~5% of students who are really engaged. Those students often make for good early adopters for security experiments or as nominees for your Security Champions program. This is another reason I recommend that at least some of the 0 to 1 training be led by your Security team. It gives you an opportunity to meet people and identify early adopters and security champions.
That’s it for today. Are you more optimistic about training than I am? Are there other ways to enable engineers without training them? Let me know! You can drop me a line on twitter, LinkedIn or email. If you find this newsletter useful, do share it with a friend, colleague or on your social media feed.