Edition 7: Using force multipliers to scale AppSec programs

AppSec programs are hard to scale. What works for a portfolio of 10 applications don't work for 1000 apps. Piggybacking off existing organizational programs can super charge your AppSec journey.

Sep 12, 2021

One way to scale AppSec programs is to have a large, central team. That approach has some obvious pitfalls:

Successful AppSec program are in lock step with business. It’s hard to evaluate business risk if you don’t understand the business. This problem is exacerbated when the AppSec team in centralized and many hops away from business. Centralization also plays a role in creating a Security culture that only applies to the Security team.
It’s expensive to build. AppSec talent does not come cheap (this is true for most geographical markets right now) and given Security teams are essentially a cost center*, having a small and nimble team helps. Especially when companies go through a tough time financially.
*cost center - I know Security teams are sensitive about this. I am not using “cost center” in a pejorative way as a proxy for “dispensable”. I mean it in the sense that it's harder to justify budgets in most orgs. In any case, being thrifty is a positive in my book :-)

OK, if we all agree that a "large" AppSec team is bad, the next logical question is What is "large"? How do we quantify it?

There are some metrics which tell you what ratio is common among companies. The latest BSIMM study pegs it around 2% of dev strength (this number also passes the smell test when I look around at companies I know). This is really small. Let's say you have 200 devs with a total portfolio of 100 apps (SaaS+in-house+COTS). An AppSec team of 4 (2% of 200) will now need to figure out tooling, manual assessments, vulnerability management and much more for a hundered apps. Thats spending an average of 2 weeks per app. This leaves time for nothing other than penetration testing and fire fighting. This number gets even scarier if you are an ops-first company where most of your software is bought and hence have a smaller developer count.

If a large team is not an option, how do we scale? Here's my hypothesis:

AppSec teams need to start using force multipliers to scale security. Automation is the obvious one, but there's another one: Piggybacking off existing org-wide initiatives. Many AppSec activities are not specific to Security and similar problems may have been already solved in the rest of your org. Leverage them instead of reinventing the wheel. Avoid as many "security" specific events, tools, dashboards as possible. Use whatever the company already uses, just make security a part of it.

Piggyback off org-wide initiatives

Many AppSec actiivties overlap with other functions in the org.

The best AppSec teams are great at building systems (e.g.: Vulnerability management, assessment tooling), breaking systems (e.g.: PenTesting), evangelization ("sell" security to execs, train developers), vendor management and much more. This requires a varied skillset (tech, sales, marketing, program management etc.), which is expensive to hire. Most AppSec folks started out as people who like to break things and realized very quickly that hacking is such a small part of the job (probably why we see so much turnover in AppSec, but that's for another edition). It is really hard to build a cross-functional team and likely falls into the same pitfalls mentioned earlier (expensive and hard to integrate well with business). Like good hackers, we should find a hacky way around it.

Let's take training as an example. Here is a breakdown of what a successful in-person dev training class needs to achieve:

Build curriculum - Ensure it is technically sound, engaging and relevant to what the company does (not your canned soup version of OWASP Top 10)
Finalize logistics - Should this be in-person or on Zoom? One instructor or two? Four one-hour sessions or a single half-day class? Should lunch be served?
Recruit attendees - Ensure all eligible employees know about the training. Make it sounds exciting. Make it easy to sign-up. Ensure pre-requisites are delivered to them
Figure out incentives - Will there be prizes? What prizes work best? What's our budget? Should we hand out certificates?
Deliver the training

You will notice that 1 & 5 needs traditional AppSec skills. The rest don't. Instead of looking for AppSec folks with those skills, leverage your organization's L&D team (if one exists). Don't build a new Security training platform. Using the existing LMS to host the content. Ask them for guidance on logistics. Talk to HR (or Engineering leadership) on what incentives work better for your target audience. Some companies are suckers for laptop stickers, others just want Amazon gift cards. If someone else has figured out, why reinvent the wheel?

This isn't restricted to training. Anytime an AppSec team decides to do anything outside Security Assessments, we should ask ourselves this: Does anyone else in the company already do something similar? If yes, what stops us from leveraging it?

Security champions program

Another example of leveraging existing systems is to build a Champions program. This isn't new advice, but it's good advice. In my experience (especially during my consulting days), most Champions program feel “forced” and don't work. But the ones that do work appear to make a massive difference in scaling AppSec programs.

The first time I heard about such a program was when I spoke to someone from Yahoo's Paranoid (that's what their InfoSec team is called). They had a hub and spoke structure, where the hub was the infoSec team and the "spokes" were integrated with the dev teams. Netflix's Security Partnership program seems to do something similar. This blog on scaling AppSec from Netflix is pure gold (If someone at Netflix is reading this, here's a request to please write more about it :) ).

Given the high rate of failure, I think laying out a clear set of rules for champions program is important. If wea are unable to enforce them, maybe things need to change. Here are my thoughts on what some good rules are (you should build your own):

Participation should be optional and voluntary. While nudges and incentives can be used to drive participation, mandating it will drive a compliance mindset and we all know how excited people get about "complying" to things :P. This is true for choosing champions (not everyone can be a security champion, but "management nominated" ones are the likeliest to fail), keeping them engaged and growing the program.
Champions should report to the dev teams. Having them report to security will lead to all the pitfalls of bloated AppSec team. Their primary loyalty should be to their business units. Security should be a side show. An important side show that adds value, but a side show nonetheless.
AppSec teams should build a community for Champions . Many Champions will face similar issues. How do we convince our team to add that really important security task to the sprint? How do we ensure that defect we just fixed never shows up again? How do we manage all the Security initiatives without it being a time sink? The reality is, AppSec teams don't have answers for these questions. We are better off creating a community where information sharing is easy and let the Champions talk to each other. We should merely be organizers and note takers :-)

There is value in double-clicking (as the kids say these days) on each of the above points. In the interest of keeping this article at a reasonable length, I will attempt to do that in a different edition.

Program management

Another AppSec activity that isn't unique is program management. Like training, many of the challenges we face are similar to what other teams do, but Security teams are notorious for using special systems that don't play well with tools used by the rest of the org (yes, PDF reports, I am looking at you).

Wherever possible, use terms and systems which are part of the company vocabulary. A few examples:

Don't call them "vulnerabilities", call them "defects" or "bugs".
Use the same defect tracking software as the rest of the org. Vulnerability management tools are great for Security teams, but devs should consume defects on Jira (or whatever defect tracking systems they use). No one wants access to your instance of Defect Dojo or ThreadFix
Use the same visualization and Sprint planning tools your org uses
Deliver Security requirements in the same format as other software requirements are (e.g.: If all requirements are submitted to engineering teams before a quarter by the Product team, work with the Product team t.o bake in Security requirements as well)

And so on. You get the drift.

When not to leverage existing systems?

This suggestion cannot be applied broadly without exceptions. I can think of a few scenarios where this may not apply. YMMV. :

Existing systems are poor or immature: If a large part of your company is unhappy with the current L&D system, there is no value in leveraging them. You are responsible for the success of your training. If the current systems don't work, there is no obligation to use it.
Unique problems: Some AppSec problems are definitely unique to Security teams. SAST may just be another kind of code review, but finding defects through SAST, reducing false positives and triaging them is a hard problem. There is a reason why QA tools have failed miserably at also being SAST tools. It is hard to just use the current peer code review process as a starter kit for Security code review.
Another example is dealing with external bug bounty hunters. Your support/PR teams may have experience in dealing with pissed off customers, but leveraging them to respond to Security researchers (loaded term, I know) can lead to disasters like this:
Source: @Scott_Helme on Twitter

That's it for today. Are there other systems AppSec teams should leverage? Does leveraging fail in other scenarios too? Let me know! You can drop me a line on twitter, LinkedIn or email. If you find this newsletter useful, do share it with a friend, colleague or on your social media feed.