Matthew Boston

Security Adoption Is a Developer Experience Problem

April 5, 2024

You may not be writing insecure code, but you are almost certainly running some. Every dependency is a layer of indirection that hides the code your application actually executes, and a modern application has hundreds of them. Security tooling can find the problems buried in that code, but only if developers turn it on and keep it on. They do that when the tool costs them almost nothing, so the developer experience has to come first.

The code you didn’t write

An import line is a pointer to somebody else’s code. So is every entry in Gemfile.lock or package-lock.json. Run npm ls --all on a mid-sized JavaScript app and watch it scroll. Most of those packages were never chosen by anyone on the team. They came along because something you did choose depends on them.

That indirection is the whole point of a package manager, and it’s also the problem. When you read your own code, you’re reading a small fraction of what ships. The rest you take on trust.

Log4Shell made this painfully concrete in December 2021. Plenty of teams spent that week finding out they ran Log4j at all, because it sat several layers down inside some other library. The vulnerable code was never in their repo. It was in their build.

The cheapest dependency to secure is one you never install, which is part of why I think trivial packages should be written instead of imported. Nobody is going to write their own logging framework or TLS stack, though. The real dependencies are staying, so the work is learning to live with them safely.

Dependencies are a community problem

Last week the open source world got another reminder of how fragile this is. A backdoor turned up in xz Utils, a compression library that sits underneath a lot of Linux systems. It was planted by a contributor who had spent years earning enough trust to cut releases, and it was caught because one engineer noticed SSH logins were taking longer than they should.

No scanner was going to save everyone there. Most open source projects are maintained by a handful of people, often volunteers, and the companies building on top of them are the ones with the budget. Looking after dependencies is a technical job and a community one. Report what you find. Send the fix upstream instead of patching a private fork. Pay for the packages your business runs on. Keep your own versions current, so the fixes maintainers ship actually reach your users.

GitHub’s recent post on this topic has the right word in its title: securing the world’s open source together. Every consumer of open source shares responsibility for keeping it secure and working.

Friction decides adoption

Look at how security checks have traditionally reached developers. A review gate right before release. A scanner that runs nightly and files 400 findings into a tracker owned by another team. A pen test report that arrives as a PDF after the code it describes has been rewritten. Each one asks the developer to stop, switch context, and deal with something unfamiliar on someone else’s schedule.

Developers route around friction. Nobody does it out of malice. They have a deadline, and the security check is the thing standing between them and it. A noisy scanner gets ignored. A slow gate gets fewer, bigger changes pushed through it. A finding that lives in a tool nobody opens might as well not exist.

So move the checks to where the developer already is. That’s the idea behind shift left, and it applies to product owners as much as engineers. Findings show up in the pull request, on the changed lines. Vulnerable dependencies arrive as a PR that bumps the version. Secret scanning blocks the push before the key leaves the laptop. If a feature accepts file uploads, the conversation about validating them happens during planning. Three weeks after launch, it’s an incident.

Fixes that teach

GitHub’s code scanning autofix pairs CodeQL with Copilot to suggest a fix for an alert right in the pull request. The fix is nice. What I like more is that it explains why the original code was vulnerable.

Take a query like this:

ruby User.where("email = '#{params[:email]}'")

The scanner flags SQL injection and suggests:

ruby User.where(email: params[:email])

A developer who only gets the patch merges it and writes the same interpolated query next week in a different file. A developer who also learns that interpolation hands the user control of the SQL, and that the hash form lets ActiveRecord bind the value as a parameter, usually stops writing it. That moves security even further left: past the pull request, to the moment the developer types the query.

Measure adoption

If a security program is judged by the number of findings, the incentive is to add more scanners. Judge it by adoption instead. How many repos have code scanning turned on? How long do findings sit open? How many alerts get dismissed as noise? Those numbers tell you whether developers trust the tools.

This is the same job as building a platform that raises the floor. Make secret scanning, dependency updates, and code scanning the default for every new repo, so a team without a security specialist still ships with them on.

The lower the friction, the more teams turn the tools on and leave them on. A scanner that’s on and trusted protects more code than a stricter one everybody works around.