INOC & AT&T Business — A Case Study | INOC Network Operations Center

The Challenge

AT&T Business manages a customer base far too large and complex for it to support on its own. The company relies on a cadre of third-party service providers to handle various areas of technical support across its edge-to-edge service offerings.

When AT&T Business first hired INOC to provide 24x7 managed support for its VA hospital guest Wi-Fi program, only a handful of hospitals required external technical support.

As the program grew somewhat rapidly, however, and more VA hospitals required installation and ongoing support, AT&T Business selected two vendors to handle cabling and back-end engineering for installs, and a third, INOC, to provide its NOC services and technical back-end service desk support via its outsourced 24x7 NOC.

When Colleen Jacobs, AT&T Business’s current program manager for the company’s Veterans Affairs program first came on board in the midst of this expansion in 2013, AT&T Business, in her words, simply “didn’t have enough people or bandwidth to support the growth that was happening.” The company’s VA program was taking on more hospitals than it was prepared to support without a significant change to its service levels.

The program at that point had expanded to a total of 20 hospital sites—all still supported by a ticketing system and runbooks designed for far fewer. Both ticketing and the runbooks proved to be operationally immature and underpowered given the higher workloads and greater complexity across the additional sites.

When a ticket was issued at a hospital with no available IT staff, Jacobs remembers getting calls almost every hour, with tickets clogging her inbox. In short, the program’s ITSM services needed to catch up with the business.

Over the next several years, AT&T would work with INOC to scale and enhance this support to keep up with the growth of the program—refining their processes each step of the way.

The Solution

Following an in-depth assessment of the larger VA hospital infrastructure that AT&T Business was providing guest Wi-Fi for, INOC tailored an expanded service package.

This new service package would support on-premise cabling and engineering teams installing and configuring new Wi-Fi services and expand the standard 24x7 service desk support and higher tier technical support through the NOC.

These services included:

24x7 monitoring
Tiers 1-3 support
Assistance with firewall maintenance
User help desk support
Hardware help desk support
Hardware maintenance

As part of this enhanced service package, INOC helped refine runbooks to provide clearer, standardized instructions for new support staff, VA translators, and hospital guests interacting directly with the NOC and the service desk. INOC also refined its system for generating, working, escalating, and resolving tickets to account for more sites and devices and a higher volume of subsequent support activities without loosening service levels.

In addition, INOC created a custom solution for logging security issues, which, at the time, was a new requirement of the OSI security group in the network's contract. INOC’s solution enabled its technical engineers to identify individuals misusing the hospital’s networks—specifically those violating the law. This log could then be referenced by law enforcement or hospital security officials to pinpoint the origin of suspicious behavior on the institution’s guest Wi-Fi networks.

The Results

Today, the guest Wi-Fi program is in place at 86 VA hospitals, clinics, and mental health institutions around the world.

The security logging program is in place at about 40 sites (where once it was four). In the span of roughly two years, INOC’s service enhancements have reduced ticket resolution times by about 35% across the entire program. Other significant outcomes include:

Reduction in NOC support turn-up times from one month to one week
Restructuring of site designs for improved 24x7 supportability and scalability
Ability for AT&T to respond to security incidents rapidly

Looking ahead, INOC is working with AT&T on alarm enrichment and advanced correlation to reduce service restoration times even further.

Jacobs describes INOC as AT&T’s Business’s “sidekick” in its VA hospital program. Since first partnering with INOC, both companies have grown and synergized to form a strong relationship that continues to support further service expansions.