Case Study How INOC Partnered With SHI to Help a Large Financial Company Reduce Average MTTR by 900% Within One Year

Today’s financial services companies rely on their IT infrastructure to keep their employees, advisors, and customers up and running at peak performance 24/7. Especially in the financial services industry, even seconds of downtime can translate into significant direct and indirect costs. 

This was the challenge that brought one of the country’s leading financial services companies together with SHI International Corp and INOC together as partners and has kept them tightly integrated ever since.

This case study briefly summarizes the challenges, solutions, and results of INOC’s partnership with SHI to provide NOC operations consulting and outsourced NOC support to a 12,000-employee financial services company that provides IT services and operational support for tens of thousands of independent financial advisors—and handles roughly $50 trillion in assets—across the United States.

The Challenge

As the IT partner for thousands of independent financial advisors across the country, network uptime is absolutely crucial for the client. Simply put, any time one of its financial advisors can’t access a system or make a trade, everyone loses money.

The company’s expanding IT infrastructure meant higher support volume. Prior to engagement, an outsourced overseas NOC services provider had been responsible for handling Tier 1 and Tier 2 support. But due to a poorly-defined service contract, the service provider was exploiting the vagueness of its Tier classifications to label almost all incoming support activities as Tier 3. It would then route these incidents and events back to the client’s internal engineering team for resolution.

Making matters worse, the existing support provider routinely reported glowing metrics despite obvious unhappiness among internal engineers and end-users.

With virtually all support activities being routed back to the internal IT team as “Tier 3” incidents, support tickets quickly piled up, forcing internal engineers to devote time and attention to break/fix work while putting strategic projects on hold. Over time, morale across the team degraded. Stress turned into burnout, and burnout turned into attrition. The company’s churn rate among its IT staff rose to roughly 70%.

The company realized it was losing crucial engineering talent while paying a support provider millions of dollars each year only to have most of its support workload returned. Making matters worse still, the runbooks those internal engineers relied on to troubleshoot support activities didn’t provide instruction for common and recurrent incidents being routed to them by the support provider.

To fix this problem, the company engaged its managed services provider of over ten years, SHI, to find a competent NOC service provider that could advise on the optimal support strategy. Unsure of whether it was smarter to invest in building an internal NOC or continue outsourcing it, the company wanted to open a consulting engagement with a team of NOC experts to answer this, and other key questions based on its business goals, budget, lead-time requirements, and other factors.

The Solution

SHI enlisted the help of its long-time NOC services partner, INOC, to serve as expert consultants with the requisite technical and operational background to understand—in extreme detail—the client’s requirements, constraints, and opportunities—and then recommend the optimal support model.

The decision to partner with INOC boiled down to the rare operational expertise INOC brought to the table, as well as the flexibility it was able to offer service, and the willingness to customize the engagement to fit the unique needs of the client.

In the Fall of 2019, the client entered into a NOC operations consulting engagement with INOC, which lasted several weeks and culminated in a comprehensive report. This report clearly laid out a series of key considerations and recommendations.

Taken together, outsourcing the NOC function emerged as a clearly optimal option—one the client understood. In March of 2020, at the request of the client’s SVP of Networking Operations, INOC turned those recommendations into a formal service proposal to support the client from its US-based 24/7 NOC.

This new engagement prompted all three parties—the client, SHI, and INOC—to step back and resolve some of the fundamental challenges that impacted the support workflow overall.

For example:

• INOC, SHI, and the client team worked together to radically improve the troubleshooting and resolution workflows for common incidents and rebuild the runbook to standardize those processes across teams.
• All three parties also worked to reclassify incidents and events at their proper Tier levels, so genuine Tier 1 and 2 incidents would be categorized as such and handled by INOC’s frontline. This freed the client team from needless basic support tasks.
• INOC conducted a rigorous analysis of the company’s support activity history to determine that roughly 70% of incoming incidents should have been classified as Tier 1 or Tier 2 rather than Tier 3. This revealed just how massive an unlock accurate classification would be since such a large proportion of support activities could be handled by INOC’s lower Tier resources rather than the company’s internal engineers.
• By implementing properly-written runbooks and INOC’s operational framework, INOC projected it could reduce the client’s advanced support workload by 50% within the first year.

The central solution INOC provided was an operational framework that enabled the NOC to scale right along with the business and the flexibility to customize that framework to be a genuine solution in the unique areas the client needed.

Given the need for immediate performance improvement as well as more foundational enhancements to processes and workflows, INOC’s long-term improvement project ran concurrently with an immediate support initiative. INOC analyzed the client’s support data to identify the most frequent issues and rapidly developed new troubleshooting, response, and resolution processes that could be routed and worked from its NOC for these issues, specifically.

This rapid improvement program immediately reduced the support burden on the internal team while INOC developed its alarm-to-action guides, compiled them into runbooks, and implemented a sophisticated operational framework that brought more foundational improvements across the shared support operation.

The Results

Since the start of its partnership with INOC roughly two years ago, the client can point to measurable improvements across key performance metrics and qualitative improvements across their IT function. We identified a few of these results below.

• 10x (900%) decrease of average MTTR in H2 vs. H1 2021

• 50% decrease in Time-to-Alarm (TTA) across all platforms in H2 vs. H1 2021

• 20% of incidents auto-resolved on average throughout 2021

• 3x (200%) reduction in Time-to-Auto-Resolve in H2 vs. H1 2021

• 70% average incident resolution from the NOC across all assigned incidents in 2021 (no escalation)

Perhaps most significantly, INOC’s support resulted in a 10x (or 900%) decrease in MTTR on average from the first half of 2021 to the second half of the year. The chart on the right breaks the improvements down by platform for a more granular analysis.

Throughout the second year implementing Tier 1 and 2 NOC services and the operational framework that underpins them, INOC has reduced the incident resolution workload on the client’s team by an average of 70%.

Over the course of this engagement, the implementation of alarm-to-action guides—and better runbooks in general—enabled a steady migration of more support activities from internal engineers to INOC’s dedicated NOC team while enabling those internal engineers to better respond to—and resolve—the far fewer advanced issues that land in front of them.

In addition to significantly offloading break/fix NOC activities from the client’s engineering team, INOC has also provided a level of data visibility that simply didn’t exist prior to the engagement. Regular business reviews surface deep, granular insights into where support activities are coming from and how much time and attention is being devoted to each.