Does your biosecurity screening work?

So, you've decided you don't want to help randos get ahold of smallpox, ebola, and ricin. That's a good start! So, you've decided to check if the DNA, RNA, or protein sequences that people are asking you to build or work with are dangerous, and you've either built your own biosecurity screening system or you've obtained a biosecurity screening system from us or one of the other screening tool providers. Great!

Does it work?

How would you even know?

That's the question that our new paper, Progress and Prospects for a Nucleic Acid Screening Test Set, is beginning to answer. 

We organized a collaboration of four tool providers and two synthesis companies to start with the simplest question of all: on what sequences do we agree if they're dangerous? Even that question wasn't straightforward, because all of the tools have qualitatively different ways of reporting their analysis and some approach the question of sequence danger as "innocent until proven guilty" while others are instead "guilty until proven innocent."

Still, we were able to come up with a way of making their results all sufficiently comparable, and we did a first test on three controlled threat organism groups - one viral, one bacterial, and one fungal:

• The good news is that we all basically agreed on the viral sequences.
• The bad news is that we couldn't agree on about 10% of the bacterial sequences.
• The challenging news is that there wasn't enough information to decide one way or another for more than 30% of the bacterial sequences and more than 80% of the fungal sequences.

That last bit, about not being able to decide, is not actually as bad as it might sound: if nobody knows enough about a sequence to decide whether it's dangerous, then it's also not likely that anybody knows enough to actually do something bad with it. And being able to agree large numbers of sequences is a very good thing if you want to be able to do basic "competence tests" to make sure that a tool isn't making bad decisions.

This might sound very down in the weeds, but being able to answer this question is actually a big deal, and an important part of making the biosecurity screening framework just put out by the White House actually work. So now over the next few months we're scaling up our "bronze standard" test set effort to cover all of the regulated pathogens and toxins out there, and collaborating with EBRC and NIST to make sure what we're doing can be used to benefit the whole community.

So much of civilization depends on little details of measurement and standards... I just hope that we can work quickly and effectively enough to help ward off the threats that are coming over the next few years.