How to Build and Manage an Effective NOC Team

The teams that serve in the modern Network Operations Center (NOC) are often under-celebrated despite their critical importance to businesses that rely on their networks and technology to function. When this critical function is neglected, the organization quickly realizes the impact through costly disruptions and downtime.

Investing in your NOC team maximizes its power as a business driver. A well-designed NOC operation, staffed with the right people and tools, will increase the uptime of your infrastructure and applications by reducing incident response time, identifying problems, ensuring network and infrastructure capacity, and reducing the impact of changes.

While it’s easy to talk about the importance of a well-managed NOC team, it’s another thing entirely to build an effective one yourself—especially in today’s competitive labor market. Success requires careful planning and care. In this guide, we explore some of the key points of building and managing a successful NOC team.

This is by no means an exhaustive guide to everything you’ll need to know at the outset of a NOC build or optimization project. But it should provide a helpful starting point to orient yourself and reveal some of the most important considerations that often fly under the radar.

Note: This guide is a companion piece to another one of our guides: Staffing a 24x7 NOC: Costs, Challenges, and Key Considerations. Also, be sure to grab our free white paper, which explains how IT Support Managers can cost-effectively address their IT support needs by leveraging lower-cost first-level (Tier 1) support resources: Empowering the IT Support Manager.

Developing an Operational and Organizational Structure

Effectively building a NOC team starts with a well-organized operational structure. Before you can put a traditional team-based org structure together, you have to create an operational framework that dictates how that team will work. This should directly inform who you need in the NOC. 

📄 Read our other guide, Staffing a 24x7 NOC: Costs, Challenges, and Key Considerations, for a summary of how to build out an operational structure. Download our white paper, Top 10 Challenges to Running a Successful NOC — And How to Solve Them, for an even deeper dive into NOC operational structures.

Once you have an operational framework, you can draw out an org structure for the NOC.

"Figure 3" below (excerpted from the white paper linked above) shows an example of a skills-based NOC structure that can be adapted not only to support your NOC requirements but also to provide a growth plan for employees, which, as we’ll get to later in this guide, is essential for retention. 

Keep in mind this is just one approach to structuring a NOC team. Yours may look different depending on the size, scope, and purpose of your NOC.

“If I were to build my own NOC from scratch, I'd have to put together job descriptions, figure out an org structure, and career path to develop the employees. I’d also need to lay out the workflows, processes, and tools for them to be successful. All of that I’d need to continue to improve over time, which requires setting up and managing certain metrics and reporting on them so, I know which changes I need to make as time goes on.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

Since we’re focusing on the personnel elements of building a NOC specifically here, we won’t dive into all the processes a company would then need to develop for their NOC to actually operate efficiently and consistently. But here’s a short list of what you should expect to need to plan for if you’re planning to stand up an operation yourself:

• Escalation
• QA/QC
• Disaster recovery/continuity
• Onboarding
• Training
• Continuous development
• Workflow management
• Incident management
• Problem management
• Knowledge management
• Access/credential management

📄 Download our white paper, A Practical Guide to Running an Effective NOC, for a detailed breakdown of the processes important for the modern NOC.

Defining Roles and Responsibilities—and What Success Looks Like

When you have an operational framework for how the NOC will be organized and operationalized, you can begin hiring. Your org chart should already include those roles. The next step is to describe what, specifically, each role will be responsible for.

Think about the end-users or customers the team will be supporting. If you're building a NOC to support internal services, what applications, network components, or other systems and services will end-users require? The same question can be asked if the NOC supports external customers. What is the full scope of support for each component and service the NOC will monitor?

Depending on the nature of your business, things that factor into the NOC team’s responsibilities may be:

• The regions of support (if the team will be supporting a distributed workforce or customer base)
• The hours of support
• The levels (Tier 1 through 3) of support and what responsibilities exist within each tier

Having built many NOC teams, INOC’s Brandon Atkins suggests companies standing up a NOC team use a RACI chart to identify the roles’ specific responsibilities against required tasks.

“All these initial questions build towards an exercise known as a RACI chart. By doing this, you can define all the responsibilities, as well as determine what role would handle those particular responsibilities—who is accountable, who will be informed, and who will consult. RACI is an industry standard.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

How to create a RACI chart

RACI stands for “responsible,” “accountable,” “consulted,” and “informed.” By creating a RACI chart, managers map tasks, deliverables, and responsibilities to each role. The “responsible” person actions a task or deliverable.

The “accountable” person is responsible for them getting done. The “consulted” person is a subject matter expert that provides information useful to complete the task or deliverable. And the “informed” person is kept up to date on the task or deliverable.

Here are the six basic steps of creating a RACI chart:

1. Identify the roles. Make a list of the roles within the team.

2. Identify the tasks and deliverables. Clearly list the tasks and deliverables the team will be responsible for.

3. Assign the RACI to each role and task. For each task, think about which role should be responsible, accountable, consulted, and informed. Every task or deliverable should have at least one person responsible and accountable for it.

4. Get your team aligned. Host a meeting with those who can help you challenge and correct any assumptions you might have made and get cross-functional buy-in. Given that a NOC team’s work will impact many teams, don’t plan it in a silo.

5. Finalize the RACI chart. Polish the chart and put it to others again. Get buy-in from everyone.

6. Make it useful. Don’t let the chart gather dust; use it! Constantly refer back to it to align on who is responsible for what. Keep it updated as things evolve.

📋 Stanford University IT offers a free RACI template you can download here (.xls).

Prior to developing processes for your NOC, you will have to select the tools your team will use once you have identified the team's core responsibilities.

The MoSCoW prioritization framework, which was originally developed for software development at Oracle, can be helpful in helping you determine which capabilities you “must-have,” “should-have,” could-have,” and “won’t-have,” or “will not have right now” from your tools to guide your licensing and procurement.

The MoSCoW prioritization method is explained in full in the Dynamic System Development Method (DSDM) handbook.

It’s also important to define what success will look like for the NOC in terms of its performance and efficiency. What are the quantitative measures of success you’ll express as service level targets? Read our other guide for a deep-dive here: NOC Service Level Agreements: A Guide to Service Level Management.

What KPIs and other metrics will you track toward those objectives—and how will you report on them? Read our other guide for more on that: NOC Performance Metrics: How to Measure and Optimize Your Operation.

Writing Specialized Job Descriptions

It's important to step back and consider your organization's core values before converting those roles and responsibilities into actual job descriptions. You should reflect these in your job descriptions. Ignoring these values can sabotage your hiring effort from the start.

“When you're building out your job descriptions, they should always reflect the core values you're looking for. If those aren't directly aligned, you’re bound to have problems down the road.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

With these values in mind, ask yourself three questions when putting pen to paper (or fingers to keyboard) when writing the job descriptions:

• What current and future business needs would this role directly address?
• Where else in the organization can this role have an impact?
• What core competencies will make this individual (and, thereby, your company) successful?

Given the fast-moving nature of ITSM, challenge yourself to think about the NOC roles that may be impacted by modernization and automation. How can you make room in the role for flexibility and growth in the way you craft these roles on paper? Consider including sub-sections of the job description, like:

• “Where we see this role going,”
• “Hats you may wear,” and
• “Real-world competencies and examples.”

Despite the complexity of NOC roles, we find that the soft skills that are most critical are the ones that are harder to train on than technical skills in a new hire. Among these skills are communication (verbal and written), urgency, accountability, growth, empathy, and customer service.

INOC’s Brandon Atkins adds some specific points to keep in mind here based on the Tier level:

“When hiring at Tier 1, it's very important candidates have experience quickly reading and interpreting documentation and following written procedures. The higher you get in that Tier structure, the more that people need to be skilled in troubleshooting, investigating, and diagnosing issues.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

Of course, technical skills are obviously critical, too. The modern NOC engineer needs a variety of skills to keep your network, infrastructure, and applications up and running.

Diverse technical knowledge of various network technologies, cloud environments, server operating systems, virtualization, storage systems, and applications, is becoming more and more essential—and harder to find—each day. The technical skills your team will need should be a direct reflection of the technology environment they’ll be stepping into.

📄 Download our free white paper for more information on ensuring the right skills in the NOC: A Practical Guide to Running an Effective NOC

It’s also important to consider how that environment will likely evolve over time as specific technologies like cloud computing and SD-WAN continue to find their way into the modern network infrastructure:

“I think if you look at where technology is going, the common trends companies should be paying attention to are: cloud-based technologies, automation, scripting, and programming languages (even though those last two aren’t typically associated with a network role). Most of what we do in networks today, and how they actually function, like SD-WAN, for example, it’s really automation and scripting being the back-end drivers. So, in order to support it, you need somebody that understands it.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

Having hired into hundreds of NOC and “NOC-adjacent” roles over the more than 20 years we’ve been in business, here are a few observations that might be helpful in honing in on key candidates for NOC roles.

• Look for a passion for technology. People who are professionally or personally passionate about tech tend to be self-motivated to stay on top of innovation and trends and have baseline conceptual knowledge that plays to their advantage in learning ITSM skills.
• Keep an eye out for former military personnel. Military experience naturally lends itself to working in stressful, high-paced environments that involve a lot of multitasking.
• Customer service experience is huge. Virtually any extensive background that involves customer service can signal a strong candidate—especially for frontline roles. Call center experience, retail experience, and service industry experience can all be transposed to the NOC with the right training.
• Former ISP professionals can be strong NOC assets. Frontline ISP experience—both with industry heavyweights or mom-and-pop shops—can signal experience with what happens on the back end of the technologies the NOC supports. Many times, these candidates know you don’t communicate with a carrier the same way you do with, say, a bank.

Onboarding and Training a NOC Team

While onboarding and training will naturally look different from one team to the next, INOC’s Brandon Atkins suggests starting with a strong foundation and building the technical components on top of it:

“With a new hire or team, it's really important to lay a foundation where the person or team familiarizes themselves with the company, its purpose, and its mission. If you toss someone into the deep end without that foundation, it’s easy for a lot of people to get overwhelmed and spend the next two or three weeks training them when one foot is already out the door. The other big piece of that foundation is the company culture. You really want to make sure that everyone has a clear expectation of what that culture is and what they can expect from it. From there, you can comfortably introduce a basic introduction of tools—showing them the ticketing system, making sure they understand how to operate it, work within it, get comfortable with it. Then you start to build into troubleshooting and expanding that knowledge to different platforms.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

A monthly or quarterly training session should be scheduled to keep engineers' skills fresh and update the support team on new services, customer requirements, and equipment changes.

We facilitate trainings at INOC, for instance, by using an LMS system where engineers review a variety of materials. This asynchronous training is supplemented with live NOC meetings where we discuss ways to handle different support scenarios and utilize our tools creatively.

Retaining Staff

Factors that affect retention rates include company culture as well as NOC organization (i.e., whether there’s a clear path for employee growth from one level to the next or to other departments within the organization).

Tier 1 staff are always seeking opportunities to develop their skills and grow professionally. There may be some who want to become managers. Others may want to specialize in a certain technical area. For leaders, it's about creating pathways to pursue professional development. At INOC, for example, a portion of shift work is dedicated to professional development. Staff has this time to work on themselves during the workday. This type of program fosters retention and demonstrates our commitment to the core values of the organization.

“It's important for leaders to understand what their employees want to do. Do they want to become software developers? Network engineers? Supervisors? The bigger the team, the more variance you’ll find among goals and interests. Many companies probably have database administrators. They might have software developers and project managers. There are a ton of different roles. Realize that most people don’t want to be NOC engineers forever, but it’s a great first step into many different disciplines. Lay those paths and actively help to support them. Promote people out of the frontline. Make an example of exemplary talent and show that that path is there.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

Here are a few other ideas that can build into a strong NOC retention program:

• Hybrid roles — Hybridizing a NOC role can be a creative and business-savvy way to attract and retain motivated talent while covering multiple ITOps needs both inside and outside the NOC. In these roles, someone performs NOC duties for part of their schedule, while spending the rest of their time on more interesting projects. INOC's hybrid NOC roles combine traditional NOC engineer duties with roles aimed at improving our support platform and services in general—whether it's scripting, automation, process improvements, and quality assurance—thereby preventing burnout among staff. Consider what kind of hybridized roles can fulfill your NOC needs while adding value to related ITSM functions.
• Recognition — Recognizing success isn't new—it's just easy to forget. You don't need to overcomplicate it, just find a way to appreciate great work and help staff feel appreciated. A kind word or promotion is much more meaningful than a gift, in our experience.

“It's not just about recognizing great performance, but also about recognizing all dimensions of quality. It's not just a matter of meeting or failing. We also recognize those who go above and beyond, as well as make use of these opportunities to showcase exemplary work to the rest of the team. When you're always pointing out your shortcomings, it's easy to overlook the positives, but pointing out the good things is equally important.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

• Stay interviews — Especially during the “great resignation” it’s important to find the people who are committed and understand what loyal employees value most about their role. These are incredibly valuable insights that can serve as inputs back into the hiring and retention program.

Managing Staff

There are many dimensions to effectively managing a NOC team—more than we can cover exhaustively in a blog post. Rather than attempt to cover management in-depth, here are a few key points that speak to some of the common challenges of managing a NOC:

• Manager-to-employee ratio — One of the challenges we’ve worked hard to overcome at INOC is rightsizing the ratio of managers to staff. In the event that a team outgrows its current management, there is no convenient warning light that goes off. In most cases, it happens slowly and quietly. Management, however, may find themselves without the time to help each individual grow. The team should be aware of this problem if and when it grows.
• Culture and its impact on performance — Company culture often has an underappreciated impact on the day-to-day management of a NOC. Getting culture right can eliminate many acute management problems from the start, which is why it’s so critical to be thoughtful about from the start. Here’s INOC’s Brandon Atkins on its impact:

“Leaders cannot really manage in a sustainable way if people are working ticket-to-ticket, trying to close tickets as quickly as possible. They will also be trapped in this anxious environment. In order for NOCs to perform better and enjoy their work, they must shift from a ticketing culture to a customer-driven culture. If you have a large team, shifting that culture will be more difficult. It requires a significant amount of time and effort. A leader should communicate clearly to the entire team, multiple times, the reason for a change in direction. Since culture shifts won't happen overnight, they require constant reinforcement. There will be mistakes. Coaching mistakes is important, but also recognizing and rewarding the right behavior. It’s one of the biggest challenges of management, so I’d advise any team looking to build out a NOC to clearly set that tone and vision from the start.”

— Brandon Atkins, Senior Director of NOC Operations, INOC

• A standardized process framework — A lack of consistency is one of the main reasons NOCs don’t perform at optimal levels and become unwieldy to manage. The best way to create consistency is through a standardized process framework. Such a framework provides a NOC with a set of specific procedures for handling various support situations. There are several process and management frameworks to choose from, including MOF, FCAPS, and ITIL*. ITIL (IT infrastructure library) is a widely used framework that is useful in achieving the ISO 27001 certification. It provides best practices to follow when delivering technology support services as well as the flexibility to include your organization’s custom procedures under its umbrella of life cycle stages. Learn more about applying ITIL to the NOC here: ITIL Service Operation and the NOC: A Quick-Guide.

Building a NOC Team vs. Outsourcing

A 24x7 NOC is often an unnecessary expenditure compared to strategically outsourcing support as a predictable operating expense. Given the payroll and overhead costs of building a NOC in-house, many companies that outsource with us cut their total cost of ownership in half.

Rather than having a staff of 12 or more full-time employees necessary to provide 24x7 coverage, an outsourced service solution that takes full advantage of an economy of scale could provide better service at a much lower cost. Along with staffing costs, acquiring, implementing, and integrating all the NOC tools further points in favor of outsourcing much of the time.

Perhaps the most apparent difference between homegrown and outsourced NOCs are the capabilities that come with an already-mature operation and access to niche NOC expertise. Between planning a NOC build, hiring a team, training that team, and aligning over the operational plan, in-house NOCs can expect 16 to 24 weeks minimum before all the parts are even in place. It can then take months or years to gain confident control over the system and bring it into a state of real operational maturity.

Turning up support with an outsourced NOC condenses all that time and effort into a number of weeks instead of years—often far more cost-effectively.

Through the NOC as a service model, an outsourced support provider helps you take control of your infrastructure through a suite of NOC solutions designed to meet the specific needs of your technology environment and operational workflow—all while enabling you to focus internal resources on the projects that move the business forward.

Want to learn more about our approach to outsourced NOC support? Contact us to see how we can help you improve your IT service strategy and NOC support, schedule a free NOC consultation with our Solutions Engineers, or download our free white paper below.

FREE WHITE PAPER

A Practical Guide to Running an Effective NOC

Download our free white paper and learn how to build, optimize, and manage your NOC to maximize performance and uptime.

Download →

*Originally developed by the UK government’s Office of Government Commerce (OGC) - now known as the Cabinet Office - and currently managed and developed by AXELOS, ITIL is a framework of best practices for delivering efficient and effective support services.