Edition 28: ADR v/s Shift-left should be looked at as a "Stock" v/s "Flow" problem
And like most "Stock" v "Flow" discussions, you need a bit of both.
If you are one of those people who follow AppSec “trends,” you will notice a lot of chatter around ADR (Application detection and response) v/s shift-left in the last year or so. The argument in support of ADR is that it deals with real-time info (production data) and hence has fewer false positives. It also allows you to focus on what’s exploitable, as opposed to focusing on what can possibly go wrong. The argument in support of shift-left is well-known: The earlier you find defects, the easier (and in some cases cheaper) it is to fix them. In other words, given a vulnerability (say you forgot to turn on AuthN on one of your API endpoints), it’s cheaper to find and fix it in the SDLC (design or coding stage) than as a production alert many months later.
Stock v/s Flow
ADR v/s shift-left is a flawed argument because they deal with different things. Shift-left manages change (“flow”), whereas ADR’s focus is on managing what’s already in production (“stock”). Obviously, the answer is, “You gotta do both”. This leads to the “well, we cannot afford to do both” retort, which then leads to the “you gotta prioritize based on what your company does” (feels like I just had an entire dialogue in my head :D).
Before I belabor this analogy further, let me define Stock v/s Flow in AppSec:
“Flow” security controls are the ones that determine security issues as you introduce changes to your software (e.g., In the design or coding stage). This could be new code, a new integration, a migration from one cloud to another, and so on. Think of them as a valve in a water pipe that you can use to regulate the speed of flow or a sensor that checks for leaks. A SAST tool running in your CI tool is a good example. It looks at new code and stops it from progressing (“flowing”) if it finds a bug.
“Stock” security controls are the ones where you manage the code already deployed in production. Open defects in “stock” come from 2 sources.
Leaky “flow” controls. This one is obvious. Imagine your developer did not fix a SAST bug and pushed it to prod anyway. This will increase your “stock” of defects.
The truth changes. In some ways, this is unique to Security. Sometimes, you found all the defects, fixed all the bugs, and made sure all your open-source packages are secure before you push code to production (so your “flow’ is good), but someone discovers a new defect in an open-source package, and now your stock of defects grow. Or, a third party you trusted (they filled out your sweet TPRM forms) just had a breach, and now you cannot trust the data sent to them anymore.
The “ADR is amazing” hypothesis is this: Your Flow controls will always be leaky, and the truth can always change. So, why invest in Flow at all? Instead, let’s make Stock controls Great Again (sorry) and go all-in on ADR. This argument holds reasonably well when you are not building a ton of new software. If you are an engineer at Uber, I am guessing the amount of new software your Android App team is building is minimal. In that case, going all-in on “Stock” controls may make sense. But if you are the team building ChatGPT, your code base is changing every day, and focusing only on “Stock” can become expensive quickly.
Managing “Stock” tools v/s building “Flow” tools
If you are an AppSec leader, there are a few different ways to think about these tools:
“Flow” tools work best in nudge mode. They need to be light touch (“run fast”) and provide enough context for developers (or other stakeholders like Product or DevOps) to make a decision on fix v/s ignore
Results from “Stock” tools are hard to attribute. If you find a software bug in prod (say an OSS library has a new CVE), it can be hard to know which team to assign it to. This can only be fixed if there’s a mapping of production components to engineering teams. Some companies do this well out of the box, but most suck at it. Depending on what your environment looks like, prepare to spend an unreasonable amount of time assigning defects to the right team
Defects from “Flow” and “Stock” tools must be tracked differently. Results from “Flow” tools can become bugs in your Developer’s Jira pipeline. You should allow them some freedom in choosing how to prioritize. However, every “Stock” defect should be treated as an incident. The proper prioritization is critical and can determine if you need to wake the CTO up or if things can wait a few days.
Eventually, you will have to come to a conclusion on whether your program should be “Stock heavy” or “Flow heavy.”
Building “Stock” tools v/s building “Flow” tools
Like me, if you are in the business of building AppSec products, it’s important to know that while both kinds of tools “find bugs and help fix them,” there are a few key differences:
“Flow” tools slow developers down. This is almost always a bad thing. Unless implemented with developer experience in mind, expect them to find ways to circumvent it. If you are building “flow” tools, remember that you have 2 critical stakeholders: AppSec & Developers.
“Flow” tools lack business context. To ensure you don’t slow developers down, these tools often only analyze the change being made (e.g., a SAST tool scanning a new PR). This means a lot of context around the application where the change is made is unavailable. This leads to false positives. If you don’t like this, you could tune your tools only to find high-confidence defects, leading to false negatives. A trade-off must be made, and you (or your user) must pick a lane.
Depending on where in the “Flow” your control is, assumptions made by the tool can change. If you are building a Design Review product, it’s important to remember that the assumptions made during design (“this product is for internal usage only”) can change later. And when it does, your tool’s output will look quite crappy. This should be factored into the UX you build. You cannot promise “exploitability” in a threat modeling app and may need to reword your findings to reflect this reality (you can call them “security requirements”).
“Stock” tools play with live ammo. They often have access to production data and can degrade production. Architecting your tool to reduce the odds of availability loss is critical. Having said that, the risk levels vary depending on what kind of ADR tool you are building (e.g., one that observes a copy of the traffic v/s blocks traffic).
“Stock” tools are monitoring tools that look for only one type of anomaly: vulnerabilities. This means your product design should be inspired by monitoring tools and not (say) a DAST tool.
A few other thoughts…
Is SCA “Stock” or “Flow”? Well, this is tricky. The way SCA tools are used today (integrated into the CI pipeline) can be considered a “Flow” tool. But almost always, the most annoying part about SCA is what happens once the software is deployed. This is precisely why SCA tools have such terrible user experience: They are built and used as a “Flow” tool, but the most critical use cases are as a “Stock” tool.
Is SAST “Stock” or “Flow”? SAST is “Flow”. Even if you scan your “production code” once a month, SAST tools don’t understand deployment config and suffer from a lack of context. It may be tempting to think of them as “Stock” tools in some cases — e.g., Legacy code bases that haven’t changed in decades — but those are exceptions, not the rule.
Why am I thinking about this? At Seezo, we have built a robust Security Design Review product (obviously a “Flow” product). When I started thinking about what we should build next, it wasn’t clear we had a good framework. Should we build a SAST tool (also called a “Semgrep fork” :P) or go all in on threat modeling? Should we help enforce the security requirements we generate or use the tech we have to build threat models on systems in production? Having a framework as a starting point is helpful when faced with such questions. “Stock v/s flow” is the framework we chose. Our current thinking is that we are a “Flow” company and should continue to build out in that space. I will reserve the details on what that may be for the company blog :)
Thanks to Prahathess Rengasamy for reviewing the draft and the researchers at Backslash Security, whose work on “PR scanning” inspired this post
That’s it for today, and thanks for following along so far! You can drop me a message on Twitter (or whatever it is called these days), LinkedIn, or email. If you find this newsletter useful, share it with a friend or colleague or on social media.