What is a NOC?

A NOC is a centralized location where an organization monitors and manages support for its internal and external IT services, network, cloud instances, servers, storage, applications, and other components that comprise its infrastructure. The NOC may reside in an organization’s facility on-site or at an external location and operate at certain periods or continuously 24x7x365.

What is the purpose of a NOC?

While the NOC’s primary job is to maximize performance and uptime, it plays another crucial role that few organizations take full advantage of: minimizing the productivity loss to the business when IT assets go down for any reason.

A high-functioning NOC does more than just keep your network, infrastructure, and applications up and running. It quickly informs impacted users of a problem, enabling them to shift their attention to other tasks without skipping a beat.

The faster users realize a billing application is down, for example, the faster they can shift to email or other tasks. Across the organization, these downstream actions add up to immense savings in time and attention that would otherwise be lost to miscommunication and confusion. This productivity advantage is the number one reason smart companies approach their NOC service as a business investment.

In short, the value that a NOC returns to the business should be far greater than the investment necessary to make it effective.

You know a NOC is working well—and to its full potential—when it’s delivering measurable ROI back to the business it serves.

In addition to detecting issues, the NOC oversees corrective actions to restore functionality after an outage. It’s here that the NOC offers a golden opportunity to keep the business moving during downtime by improving communication between key stakeholders—both those whose work is impacted and those fixing the problem.

Unlocking this capability within your NOC requires a centralized operational framework to deliver information and take action at lightning speed—shortening response and resolution times while giving staff the information they need to adjust to the outage quickly.

To see more clearly how a NOC can save your business the high costs of low productivity, let’s briefly revisit its basic functions.

What does a NOC do?

At its core, a successful NOC performs three functions:

1. Event Monitoring and Management: The NOC team responds to inputs from event monitoring and management systems as well as calls, emails, and other sources.
2. Ticketing: From the event monitoring sources, people, and tools in the NOC create incident, change, and service request records.
3. Troubleshooting and Resolution: Processes and procedures guide the NOC to address each task and work to resolve issues.

Beyond these basic functions, a NOC team’s key responsibilities encompass a wider series of specific tasks presented in the ITIL* framework. While ITIL includes other processes, in our experience these five are the most important to address. Let’s briefly look at them through a practical lens.

• Event Monitoring and Management allows the NOC to monitor, detect, and process events and faults related to the organization’s infrastructure and systems. Events can consist of alarms from systems, calls from internal staff (or customers), and email or chat. The NOC team uses a single or (as is common), multiple tools, including Network Management Systems (NMSs), Element Management System (EMSs), Application Performance Management (APM) tools, and others. These platforms receive and filter messages from devices, servers, cloud instances, applications, and other infrastructure using protocols such as SNMP, TL1, WMI, and, more recently, gRPC and gNMI, among others. Once an event is detected, it’s evaluated, correlated, and acknowledged, and if further management is needed, it’s logged into an incident or ticket.
• Incident Management is, in one respect, the core process of a successful NOC. Using the NOC’s IT service management platform or ticketing system, this process provides support when a network, system, or application event requires action. The event is recorded in a ticket with information in different fields. Tickets are handled by NOC engineers and also sent to other personnel as needed in the form of an email, a call, or a message requesting action to address an issue. These communications also include periodic updates and notifications until the incident has been closed. Incident tickets collectively act as a record of all work efforts in the NOC and allow for reporting that can help manage NOC workflow and resources.
• Problem Management encompasses all the activities needed to diagnose the root cause of incidents and request changes to resolve those problems. Problem Management differs from Incident Management as the focus is to investigate and identify the root cause of an incident rather than its effect. Typically, Problem Management requires greater engineering skills to review the trends leading up to an incident, scour logs for indications that point to possible causes of the failure, and formulate plans to prevent future incidents. The Problem Management service also maintains information about problems and workarounds for use by Incident Management personnel.
• Capacity Management oversees the performance, utilization, and capacity of infrastructure components to ensure that the client’s service level targets are achieved. Capacity Management should ensure that business capacity, service capacity, and component capacity needs all continue to be met. Senior engineers’ regular reviews of reports and alarm thresholds, taking into account the desired business outcomes and the impact of utilization on business operations, will ensure that evolving capacity needs are addressed in a timely manner.
• Change Management reduces risk when changes are made to the supported infrastructure environment. This function includes identifying the types of changes the organization anticipates and establishing how each change should be handled to reduce the impact on the organization. Processes and controls are generally oriented around three types of change:
• Standard changes are routine and low-impact, like resetting passwords. 
• Emergency changes must be addressed promptly, such as by rerouting network traffic when the primary WAN uplink at a regional office is unstable. 
• Normal changes are planned in advance and might include upgrading the operating system on a server cluster, for example. This type of change would be managed through a review process to ensure proper planning.

A Change Advisory Board should review and set policies for all these changes. This group helps mitigate risk by ensuring all possible impacts of a change have been taken into consideration and a proper plan with a recovery process is in place.

Despite being such a foundational component of the technical support operation that keeps organizations running, many NOCs, in both the service provider and enterprise markets, fail to deliver the desired service levels while consuming significant management and financial resources.

By far the biggest cause of this failure is the lack of any authoritative blueprint for the NOC to follow—a documented framework that establishes clear, standardized rules that govern how the NOC team should operate.

Since no two organizations share the same business strategies, technology infrastructures, tools, or service requirements, the factors that make a NOC successful or unsuccessful are different for each organization. Therefore, each NOC deserves its own operational blueprint that takes into account the specific conditions and requirements that pertain to it.

Although the lack of this operational blueprint is the most common root cause of most NOC problems, it’s certainly not the only one. We briefly summarize the ten NOC challenges we see most often below. 

📄 Grab our free white paper to learn more about each of them along with expert solutions to each: Top 10 Challenges to Running a Successful NOC—and How to Overcome Them.

Challenge #1: Overutilized technology staff and exploding support costs due to a lack of a tiered organizational structure to manage workflows

This is the operational blueprint we just mentioned. Failing to organize NOC activities and subsequent workflows by technology and skill level is one of the biggest hurdles in building a successful NOC. When a NOC can’t manage its workflow, it often finds itself overwhelmed by the “wall of red.”

A tiered operational support structure enables managers to leverage the lower-cost first-level or Tier 1 team to perform routine activities and free up higher-level or Tier 2 and 3 technical teams to focus on more advanced support issues.

The figure below lays out the basics of tiered NOC support structure. Central to this structure is the Tier 1 team that uses monitoring tools and interacts with end-user help desks, Tier 2 and 3 engineers, and third parties. Information flows between the various entities within a well-defined process framework.

Issues coming into the NOC should also be prioritized and organized into a set of queues, each of which can be handled by the appropriate group. These can be organized by important variables such as service level agreements (SLAs), technologies, and technician skill levels.

The figure below shows how a set of issues can be broken up into queues and assigned to groups based on skillset.

These visuals are intended to be instructive in building a framework, but also in realizing the distance between what a NOC should have, and what it actually might have. Especially in NOCs supporting enterprises or communication service providers, the further operations are from a structure like this, the more value it can expect to gain from implementing one.

Challenge #2: Blindness to issues and opportunities due to insufficient operational metrics

Anyone working in a NOC is likely to hear statements like these on a routine basis:

• “Why are we always busy?”
• “I feel like we can never catch up,” and
• “My coworkers are not pulling their weight.”

These sentiments are understandable given the fast-paced environment of a NOC and the constant multitasking that is required. Meaningful operational metrics are vital not only in running a successful NOC but also in keeping staff morale high.

In many NOCs, however, not only are important metrics not being measured—the ones that are being measured aren’t being evaluated on a daily, weekly, and monthly basis. In the case of either or both problems, the early indicators of potential issues will almost certainly go ignored and allowed to evolve into more resource-intensive problems. 

For a quick self-evaluation on this point, consider whether you’re tracking first-call resolution, percentage of abandoned calls, mean time to restore, and number of tickets and calls handled. If you have blindspots in any of these areas, we can almost guarantee there’s an operational vulnerability affecting the NOC’s efficiency or effectiveness. Be aware, however, that this shortlist is by no means exhaustive. Even a brief consult with our Solutions Engineering Team typically reveals a number of metric gaps companies can put on their radar.

Challenge #3: High turnover, low morale, and difficulty in hiring, training, and retaining staff due to a lack of a staffing strategy

Great NOCs are a function of great people. But very often, the absence of both a support structure and a skills-based structure can handicap a company’s ability to attract and retain great talent.

Consider the overall activity of your NOC, including the volume of calls, emails, and alarms handled by hour-of-day, day-of-week, and type of support engineer, as well as the duration of incidents. This data should be translated into a working schedule for each type of support engineer needed to satisfy the staffing requirements of your NOC. In addition to using your utilization metrics, benefits, training, and employee growth plans should all be in place.

Read more  about the costs, challenges, and key considerations of staffing a 24x7 NOC.

Challenge #4: Inconsistent responsiveness to issues or difficulty troubleshooting due to poor/unstandardized process frameworks

A lack of consistency is one of the main reasons NOCs don’t perform at optimal levels.

The best way to achieve consistency is through a standardized process framework. Such a framework provides a NOC with a set of specific procedures for handling various support situations. There are several process and management frameworks to choose from, including MOF, FCAPS, and ITIL.

Process frameworks can be overwhelming when considered in their entirety, so it’s best to tackle areas that challenge the organization the most first. Usually, that’s incident management, problem management, and service desk.

Challenge #5: A constant state of vulnerability due to a lack of a business continuity plan

Many NOCs simply don’t have a documented plan that outlines the functions of the business, identifies the critical systems that enable the organization to run, and prescribes specific actions to maintain these systems during a disruption. Others have a plan—perhaps limited to disaster recovery only—but don’t adequately protect against all potential disasters and disruptions.

For a quick gap assessment, consider the following essentials for a NOC business continuity plan against your own:

• Infrastructure redundancy: Regardless of whether your NOC’s data centers are physical or virtual, the disaster recovery plan should include at least two identical data centers running identical software with fully synchronized databases.
• Operational redundancy: An effective disaster recovery plan should also ensure that the NOC can continue to operate in the event of a site failure.
• Technical redundancy: Your NOC’s disaster recovery plan should also prepare for a major technical outage that poses a disaster-level threat to NOC service despite affecting only a portion of your facility. Consider scenarios like loss of a single server or network element, loss of much or all of the data center, and loss of a network link.
  
  

Challenge #6: Recurring problems and an inability to emerge out of a reactive state due to a lack of quality management

Without continuous quality assurance, NOCs risk hurting customer satisfaction and compromising their reputation. There are two components of quality management in this case: quality assurance and quality control.

A good quality control program monitors and measures primary aspects of the NOC service—the key performance indicators referenced earlier. These KPIs provide much-needed visibility into NOC support activity, responsiveness, and effectiveness. NOC management can use this information to ensure, for instance, that stated objectives for event-to-action times and first-level incident resolution are being met for each customer.

A good quality or service assurance program allows the NOC to identify and resolve problems before they impact customers or the business in a significant way. A quality assurance review begins when a customer reports dissatisfaction with any aspect of the NOC service. NOC management follows up with an internal review of the service—responsiveness metrics, adherence to runbook procedures, customer interaction, and technical troubleshooting, to name a few.

Such quantitative and qualitative measures and the resulting feedback lower the probability of the same problem recurring. Monthly and quarterly reviews of the service with stakeholders ensure that customer expectations continue to be met.

Challenge #7: Lots of data, but little actionable insight due to disparate tools and platforms

Especially among enterprises and communication service providers, the NOC has to be able to receive and process alarm or event information from multiple sources and present it in a single, consolidated view for staff to act on—a “single pane of glass.”

Without integration between these tools and platforms, NOC personnel are faced with tracking and managing multiple screens for event information; manually collecting information from multiple sources for the purposes of documentation, notification, and escalation; and then attempting to manage workflow toward service restoration. This makes it nearly impossible to monitor and report on SLA metrics, let alone optimize performance. The results inevitably include operational inefficiencies, missed SLAs, and undue stress on staff.

Challenge #8: Persistent operational problems due to out-of-date documentation and runbooks

Failure to build runbooks, document workflow processes, create structured databases for storage and retrieval of information, and record business results for later analysis and optimization will severely impede the ability of a NOC to function well over the long term. Too often, services are added and changes are made without proper documentation. This limits the ability of the NOC to resolve an issue when it arises.

Poor documentation often stems from a lack of resources and the expertise required to map out processes and create work instructions and documents. Instead, key people simply “know what to do” and new staff learn by “seeing and doing” alongside an experienced mentor.

Challenge #9: Business growth stymied due to a rigid, unscalable NOC

Many NOCs aren’t designed to be scalable; that is, able to handle a growing amount of work as the company grows without compromising the level of service.

Typically, business plans include initial funding, sales and marketing, system build-out, operations support, and the business guidance needed to meet the projected growth. What business plans sometimes don’t take into consideration are predictable growth and process planning. The ability to grow or absorb expansion requires careful consideration of staffing, systems and network, tools, process standardization, and training.

Challenge #10: Unreasonably high operational costs

There are several components that make up the cost of running a 24x7 NOC. Take staff for example. The staff required to support a 24x7 NOC include not only front-line technicians and engineers but also back-end support groups such as systems and network engineering, service transition, human resources, and customer advocacy. 

Resources also need to be allocated for training NOC staff when they are initially hired, as well as when onboarding new customers, and whenever changes are made to existing support or new technologies are introduced. Systems, network connectivity, and security controls need to be deployed in either a data center or the cloud to house the various tools and applications required by the NOC to operate. Resources for ongoing support need to be included.

All of these components present a formidable operating expense but have to be considered in building a successful NOC. Too often, NOCs are built considering only a subset of the above components, and as a result, they struggle to scale and deliver on the required service and financial objectives of the organization.

Organizations considering outsourcing their NOC support often wonder what the process of transitioning or turning up service looks like and what to expect before and after. While this process may look different from one organization to another, in general, a capable NOC support provider will first conduct an assessment to learn precisely how the service needs to be established and operationalized. Then they prepare services according to the needs they identified and work with the client to turn up services after a diligent testing phase. 

Once service is operational, the provider should shift to optimization mode. This means employing quality control and assurance, along with other customer experience management activities, to constantly re-examine what’s working, what’s not, and make ongoing, incremental improvements to their processes while keeping service aligned to the business’s changing needs.

To unpack the onboarding process further, we’ve broken down the general expectations at the “before,” “during,” and “after” stages.

Before onboarding: Assessment and service preparation

When outsourcing NOC support, the first step should always be the same: stepping back to analyze the business, your customers, and the support requirements ahead of you. This initial analysis enables your service provider to gather all the information needed to design the services in a way that strikes the perfect balance of capability and efficiency. The findings of the questions you can expect to be asked should directly inform the NOC’s organization and operationalization across all three essential elements: people, process, and platform. 

This assessment, or business analysis, has four main components:

1. Gathering your support requirements (and/or those of your customers): It’s critical to start by understanding who and what will be supported in as much detail as possible. This first step of gathering support requirements helps characterize the state of your existing support function to identify gaps in capability or problems related to performance and sustainability. Existing issues might be well-known, such as staff fatigue or technology gaps. Other issues may be off the radar entirely, like the critical but generally under-appreciated area of business continuity.
2. Determining necessary or desired service levels: Once your service provider knows exactly what they’ll be supporting, they should work closely with you to understand the service levels that govern that support. Here at INOC, for example, we employ a process for analyzing service level management (SLM) to determine proper staffing levels, what tools will need to comprise the NOC platform, and how success should be measured. An SLM analysis is essentially a performance review of a service as it exists now to determine what it needs to include going forward.
3. Determining the metrics the NOC will need to measure: This component of the analysis examines what’s currently being measured and how that reporting is contributing to the quality of support being provided. The findings should reveal what’s working, what’s not, and what changes should be made to ensure that the right metrics are tracked and reported. A capable NOC service provider should ask you how NOC effectiveness and customer satisfaction are currently being measured, and what those metrics—or lack thereof—say about the state of support being provided in order to retain current strengths and seize opportunities to improve.
4. Determining the total cost of ownership: In the context of new NOC service, the two most prominent figures factoring into the TCO are establishing and maintaining the platform and labor cost to support it. 
   
   

During onboarding: turning up service

Reaping the benefits that come with a highly capable, operationally-mature NOC without having to endure the typical onboarding headaches requires a well-defined process and a commitment to adhere to it. It also means committing to diligent communication at each stage, ensuring both sides are clear on what’s happening and when.

Here at INOC, we’ve invested a lot of time and energy into developing and refining an onboarding process that ensures a successful transition to outsourced NOC service without creating headaches and needlessly time-intensive work. The process graphic below offers a look at the nine steps we take to onboard new NOC clients.

Read our post, Onboarding Outsourced NOC Support: 9 Steps to Success, for a deeper dive into what you can expect as a client going through this process and tips you can use to prepare accordingly.

Since onboarding is such a critical step in transitioning to third-party NOC support, be sure the service provider you trust with protecting and optimizing your infrastructure offers a dedicated service turnup team that walks you through our process with helpful tools to keep everyone aligned. 

In terms of a timeline to service transition, we typically get clients up and running on our award-winning NOC in four to eight weeks. Once you’re up, you should also be given access to a client portal to see what the NOC is tracking in real time. Access to actionable metrics helps both teams benchmark performance and determine the root cause of ongoing issues for more effective resolution and prevention.

After onboarding: Change Management, Continual Service Improvement, and Customer Experience Management

Once NOC service is officially turned up and running optimally, your NOC service provider should transition you from a focus on onboarding to Change Management and Continual Service Improvement as part of its broader Customer Experience Management program.

Its Change Management specialists should have robust processes to both proactively check in on and reactively field notices of changes in your environment so they can be effectively managed in the NOC. Your NOC provider should also put ITIL’s Continual Service Improvement principles into practice—constantly looking for ways to:

• Assess the NOC’s ongoing success in delivering effective ITSM
• Ensure that the IT service catalog is continuously aligned with current and future business needs
• Identify and act on opportunities to improve each service, thereby optimizing speed and effectiveness while boosting process maturity

Your service provider should be able to demonstrate quality control and assurance measures that work proactively to keep most concerns off your radar entirely. Here at INOC, for example, we pull large samples of both specific and random tickets to understand how they were created, worked, and closed to learn from successes, identify opportunities for improvement, and understand any potential impact on service.

Read our post, ITIL CSI: A Guide and Checklist for IT Support and the NOC, for a deeper dive into why CSI is critical in the NOC and what to look for in a prospective service provider.