In one of the hardest-hitting GAO reports I’ve ever read, Congress’s auditing organization has, in effect, said that the TSA’s Screening of Passengers by Observation Techniques (SPOT ) program does not work and should be defunded. Members of Congress asked GAO to answer two questions:
1. To what extent does available evidence support use of behavioral indicators to identify aviation security threats?
2. To what extent does TSA have data necessary to assess the effectiveness of the SPOT program in identifying threats to aviation security?
The answer to the first is that there is no such evidence, and to the second is that TSA does not have such data. This is laid out in 55 pages of text plus seven appendices. (“TSA Should Limit Future Funding for Behavior Detection Activities,” GAO-14-159, November 2012)
TSA says the purpose of the program is to identify high-risk passengers based on behavioral indicators that indicate “mal-intent.” Accordingly, its cadre of Behavior Detection Officers (BDOs) are trained to size up passengers as they await screening using a memorized checklist of behaviors indicative of stress, fear, or deception. Passengers with a sufficiently high point score are taken aside for an interview, a pat-down, and a search of their belongings. Assuming nothing bad is found, and the person’s behavior does not “escalate,” that’s the end of the process and the passenger gets back in line. But if the behavior reaches a pre-defined threshold, a law enforcement officer (LEO) is summoned to further question the passenger and decide if an arrest is warranted. The initial (pre-LEO) encounter takes an average of 13 minutes. The program started in 2007 and has grown to about 3,000 BDOs working at 176 airports, at a current annual cost of around $200 million.
The GAO team reviewed two TSA studies of the SPOT program and found both to be non-rigorous, with considerable flaws in their methodology. It then carried out a literature review and a meta-analysis of research studies on “whether nonverbal behavioral indicators can be used to reliably identify deception.” And the answer is that “research from more than 400 separate studies on detecting deceptive behavior based on behavioral cues or indicators found that the ability of human observers to accurately identify behavior based on behavioral cues or indicators is the same as or slightly better than chance.” GAO provides excerpts from several of these studies, by entities such as RAND Corporation, DOD’s JASON program, and MITRE Corporation. Another section of the report documents the wide variation in referral rates by BDOs at various airports, as well as presenting evidence on the subjective nature of some of the behavioral indicators BDOs are taught to look for.
But the most damning information of all is who actually gets identified as “high risk” and referred to a LEO. Not a single potential terrorist was identified by the BDOs. Those who ended up arrested were for such matters as possessing fraudulent documents, possessing prohibited or illegal items, having outstanding warrants, being intoxicated in public, being in the country illegally, or disorderly conduct. While all those things may be law violations, not a single one is, per se, a threat to aviation security. And yet the only measure TSA has for the alleged effectiveness of the program is the referrals to law enforcement.
Nonetheless, TSA has recently conducted a “return-on-investment analysis” which it claims justified the SPOT program. Despite zero evidence that the program can detect or deter aviation-oriented terrorists, the analysis assumes that the BDO “layer of security” prevents a catastrophic (9/11-type) attack. GAO dryly notes that “the analysis relied on assumptions regarding the effectiveness of BDOs and other countermeasures that were based on questionable information.”
In response to previous GAO and Inspector General criticism, TSA is developing a new set of metrics about SPOT, but says it will require at least an additional three years and additional resources to report on the program’s performance and security effectiveness. Meanwhile, it is asking for a budget increase to add 584 more BDOs so the program can expand to smaller airports. In other words, to paraphrase a familiar line about the recent federal health-care law, we have to keep running and expanding the program to see if it works.
GAO sums up this comprehensive assessment as follows: “10 years after the development of the SPOT program, TSA cannot demonstrate the effectiveness of its behavior detection activities. Until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation, the agency risks funding activities that have not been determined to be effective.”
This article also appears in Robert Poole’s Airport Policy and Security News #96.