An FCC report released yesterday has revealed that a misconfigured “network element” on the AT&T Mobility network caused a nationwide service outage on February 22. The outage, which lasted several hours, affected both FirstNet users and AT&T commercial customers. According to the report, the misconfiguration occurred because the network element was implemented without proper review.

“When you sign up for wireless service, you expect it will be available when you need it—especially for emergencies,” FCC Chairwoman Jessica Rosenworcel stated. “This ‘sunny day’ outage prevented consumers across the country from communicating—including blocking 911 calls—and stopped public safety personnel from using FirstNet. We take this incident seriously and are working to provide accountability for this lapse in service and to prevent similar outages in the future.”

The FCC’s Public Safety and Homeland Security Bureau (PSHSB) conducted an investigation that began immediately after the outage. The findings showed that AT&T, now in the eighth year of a 25-year contract with the FirstNet Authority, took nearly three hours to restore FirstNet service and up to 12 hours to fully restore services for all commercial customers.

The report highlighted that the outage, starting at 2:45 AM Central time on February 22, occurred shortly after a new network element was introduced during routine night maintenance. This misconfiguration triggered AT&T Mobility’s network to enter “protect mode,” disconnecting all devices and causing a loss of voice and 5G data services nationwide. Over 125 million devices were affected, blocking more than 92 million voice calls and preventing over 25,000 calls from reaching 911 centers.

AT&T prioritized restoring services for FirstNet users but failed to notify them about the outage until nearly an hour after their service was restored. The FCC report noted that it took close to two hours to roll back the network change, but full service restoration took much longer due to the high volume of re-registration requests overwhelming AT&T Mobility’s device registration systems.

The report emphasized that if AT&T had followed its internal protocols, the network element would not have been implemented. The configuration error propagated further into the network, causing an automated shutdown to prevent further impact.

“This outage illustrates the need for mobile wireless carriers to adhere to best practices, implement adequate controls in their networks to mitigate risks, and be capable of responding quickly to restore service when an outage occurs,” the report concluded.

AT&T implemented additional technical controls within 48 hours of the outage to prevent future incidents, including scanning the network for misconfigurations and enhancing procedures for peer review and maintenance work.

Public-safety subscribers access FirstNet via AT&T’s radio access network, but one of FirstNet’s key features is its physically separate network core dedicated to first-responder communications.

In response to the report, AT&T stated, “We have implemented changes to prevent what happened in February from occurring again. We fell short of the standards that we hold ourselves to, and we regret that we failed to meet the expectations of our customers and the public-safety community. FirstNet was restored ahead of the AT&T commercial network due to the unique features of the dedicated FirstNet network core and our commitment to serving our nation’s first responders. We continue to work closely with the First Responder Network Authority to provide public safety with the dedicated connectivity they require.”

The FirstNet Authority also commented, “The FirstNet Authority is reviewing the FCC’s report on the nationwide AT&T outage on February 22, 2024. We thank the FCC for their thorough review and are focused on continuing to improve the FirstNet network for public safety.”