Cal OES, contractor failed to thoroughly test Next Gen 911 network before deploying to handle real calls, sources say

Two high-level project insiders have come forward with allegations the company NGA and the California Office of Emergency Services (Cal OES) failed to thoroughly test the state’s new digital 911 network end-to-end before deploying the system to handle real 911 calls.

One source, who NBC Bay Area is identifying as ‘NGA insider,’ has direct knowledge of executive decisions at LA-based company NGA. The other independent source we are identifying as ‘project insider’ is actively working on California’s new digital 911 network called Next Generation 911.

Next Gen 911, as the technology is also known, is supposed to upgrade our outdated landline-designed 911 system for the Internet age. Currently, 23 out of about 440 dispatch centers in California are using some version of Cal OES’ Next Generation 911 system.

In a recent interview with NBC Bay Area’s Investigative Unit, the NGA insider said watching the Los Angeles fires destroy homes compelled him to reach out to our investigative news team.

“It made me realize that God help everyone in LA if NGA was actually, you know, turned on live at all the locations,” he said.

NGA is one of four contractors Cal OES hired in 2019 to create Next Generation 911 for Californians. NGA was awarded contracts for the Los Angeles and central regions of California.

At Desert Hot Springs Police Department in the Coachella Valley, dispatch records reveal frequent 911 call problems resulting in delays by emergency responders.

Records from the Department, including communication from dispatchers, show frequent problems these past two years as NBC Bay Area previously reported.

In one case last year, a woman needed an ambulance for her boyfriend and the dispatcher said the call transfer feature was not working.

“It was always somebody’s else’s fault. That was always the word from the top [at NGA]. It's always someone else's fault,” the NGA insider told the Investigative Unit.

When our news organization asked about the problems, NGA said the transfer issues at Desert Hot Springs Police Department stemmed from the agency’s old technology, not their Next Generation 911 system. However, emails from the Department’s 911 manager said the problems occurred only after it went live with NGA’s system – a system the manager called “inconsistent at best.”

NGA’s Next Gen 911 testing in Cal OES lab

NGA’s new Next Gen 911 network was supposed to be tested end-to-end in Cal OES’ new state of the art lab.

The NGA insider and second independent project insider said NGA’s system passed testing procedures in pieces but never end-to-end. And yet, they said, the California Office of Emergency Services still deployed the network to handle real life 911 calls.

“In theory, it should be you’re testing all the functionality at the same time. In reality, they only tested one function at a time,” the NGA insider said.

In reality, they only tested one function at a time.

NGA 911 insider

For example, he described NGA testing 911 call transfers and then separately testing answering calls that were on hold.

“That's an issue because in real life you don't get to test it just function by function,” he said.

When we asked if NGA or Cal OES ever tested the system as a whole, end-to-end, and if the systems ever passed, the NGA insider said, “As a whole? No.”

When NBC Bay Area reached out, NGA’s attorney sent a written statement saying, “NGA does extensive end-to-end testing with all deployment.” However, the NGA insider said that’s not the case, and Cal OES still installed the system into Desert Hot Springs Police Department.  

“They were the guinea pig,” the NGA insider said. “There’s no other way to say it. We knew putting it in that it was not fully baked.”

They were the guinea pig…We knew putting [the system] in, it was not fully baked.

NGA 911 insider

But did Cal OES know? Was the agency’s visionary behind the project at the time, Budge Currier, aware of these testing concerns before deployment?

“Yeah, they knew,” the NGA insider said. “[I know that through] conversations, listening in mostly. It also got to the point that to some degree we were instructed [by NGA] not to put everything in writing.”

NGA says it passed all testing

NBC Bay Area’s Investigative Unit reached out to NGA, Cal OES and Budge Currier. In separate responses, both NGA’s CEO Don Ferguson and Currier said the claims are alarming and inaccurate. When we asked for an explanation, Currier did not respond.

In addition to stating NGA does extensive end-to-end testing, NGA’s attorney told NBC Bay Area there is documentation of the test results and certification showing the company completed testing requirements. However, he declined to provide the records when we requested copies. He said they belong to Cal OES. When NBC Bay Area asked Cal OES for the documentation and filed public records requests, the agency said it has located records and is reviewing them to potentially redact confidential information before deciding to release them. 

Cal OES has not responded to our news organization’s specific questions for this news report. In a written statement, the agency said, “While we continue to actively support the [911 centers] already operating the NG 9-1-1 technology, we have paused the rollout of new technology to additional [911 centers]. This pause gives us the opportunity to review the NG 9-1-1 system and determine whether any adjustments are needed. This review will consider the technology and network requirements and the experience of the 911 call centers. This will inform the go forward plan for the project. Implementation of the NG 9-1-1 system statewide is a priority for Cal OES."

“I’m sorry that people made wrong decisions, and people didn’t hold other people accountable. I think that’s the most disturbing part about it,” the NGA insider said adding that he’s still haunted by what could have happened in Los Angeles with a problematic 911 system.

Candice Nguyen is the investigative reporter on this story. If you have a question or comment, email her at candice.nguyen@nbcuni.com.