The Anatomy of an Effective NOC Runbook

NOC Runbooks
Eric Idler

By Eric Idler

Director of Shared NOC, INOCEric has worked at INOC for six years, starting with the Service Desk team and progressing into various leadership roles, including Manager of Advanced Incident Management and Senior NOC Manager. With hands-on experience in Operations and Service Desk Management, he currently manages the Shared NOC. He ensures that incidents are handled efficiently and effectively through their life cycle to support INOC clients. Eric has a BS in Economics and Computer Science from the University of Wisconsin-Madison.

Poor documentation is the root of many problems in IT operations. Without formal processes and procedures that are well-documented and accessible, even highly-skilled professionals can struggle to achieve consistent desired results.

This guide takes you inside an effective runbook in the network operations center (NOC) to see how high-performing support teams document their processes. These lessons apply to runbooks used by just about any team responsible for IT service management, whether it’s a formalized NOC or not.

Having spent the last 20+ years crafting runbooks for use in hundreds of support environments, we’ll deconstruct the “anatomy” of our own runbooks to provide a model to assess against your own. Use this guide to articulate your need for better runbooks, create your own, or refine your needs to bring to an external service provider who can develop effective runbooks for you.

We’ll cover:

Need expert-driven runbook development? We can help. Learn more about our runbook services and our broader NOC operations consulting services, and get in touch with us to schedule a free NOC consultation to explore possible solutions.

Before we jump in, let’s level-set on the basics.

What is a NOC Runbook?

A NOC runbook is a set of standardized documents, references, and procedures used to describe common ITSM tasks carried out in the NOC. A runbook walks staff through the steps necessary to accomplish a specific task or troubleshoot a particular issue.

Runbooks are useful both for seasoned professionals and novice engineers. They can refresh one’s memory for a task they haven’t encountered in a while or provide the critical step-by-step guidance a new engineer needs to execute processes they’re not familiar with.

At INOC, we view the runbook as the single source of truth for everyone inside and outside the NOC. It provides clarity into processes and galvanizes teams to coordinate around the same sets of instructions so actions are consistent no matter who is executing them.

What Makes a NOC Runbook Effective?

Runbooks are tools, and to be effective, they have to do a few things well:

  • They have to lay out the critical inputs that drive the NOC service and provide step-by-step procedures for handling them, whether it’s a phone call, email, or event-based notification. Runbooks are essentially "playbooks" for NOC engineers filled with Standard Operations Procedures (SOPs).
  • They have to describe the outcomes of these actions—both successful and unsuccessful—with clear escalation paths to other levels of support. These paths can direct action internally or to external third parties.
  • They have to make it easy for engineers to find what they need quickly and definitively. A runbook’s effectiveness isn’t just a function of its contents, but how those contents are laid out and organized. Each needless scroll or click impacts the speed of service. Poorly organized runbooks quietly exact a cost on efficiency that can compound into a massive drag on performance.

In short, a good runbook ensures everything is fully documented and presented for clear, consistent, quick action.

What Should a NOC Runbook Include?

Before getting into the contents of a NOC runbook, we should acknowledge that every runbook is different. The best practices we prescribe here won’t apply in every situation. Approach this guide as a generic framework to mold around your specific needs.

Broadly speaking, an effective NOC runbook typically has four key parts:

  • Infrastructure Documentation — This includes all diagrams, supporting circuits and applications, server information, and other infrastructure details laid out to show interrelations between infrastructure components and visualize support flows. This documentation should also describe the connections between key alarms and the infrastructure to know when critical services may be impacted.
  • Process Documentation — The runbook should exhaustively lay out all procedural steps teams will take, whether it's answering the phone, collecting and documenting the right information about reported issues and incidents, or any other interaction or activity that will take place as part of the process. Nothing should be left to assumption or interpretation. The runbook should answer questions without letting new ones arise.
  • Links to Tools/Contacts/Data — The runbook should document and link to all external tools and data that the NOC may need as part of issue resolution, as well as all internal and third-party contacts. It should be an interactive tool as much as it is a reference.
  • Alarm-to-Action Guide — The runbook should group together the top incident-generating alarms and ensure the process for resolution is clear. (Here at INOC, we often prioritize the top 90% within the volume of alarms as most of these issues can be handled at Tier 1, lightening the load on advanced engineers.)

 

A few critical pages within the runbook

A NOC runbook typically contains multiple pages, each having its own purpose as a reference. Again, any specific runbook might contain different pages based on what’s needed, but generally speaking, most of our runbooks include the following pages.


  • Client Information — If the NOC is serving multiple clients as ours does, it’s important to have a page dedicated to information about that client an engineer may need to reference. This should include a brief description of the organization, a table of relevant technical information, and anything else that’s important to know about the client itself.
  • Escalations — We often title this page “Next Level of Support.” Very simply, it contains all the escalation contacts, their position in the escalation chain relative to one another, and how to reach them. We present it as a table ordered by escalation level, with contacts and their information listed in columns: Escalation level, contact, title, email, phone, notes, and availability.
  • Device Access — This page provides clear processes for accessing the various NMS (and other) systems in the client’s network. For example, there may be processes for “Accessing the Jumpbox,” “Accessing SolarWinds,” “Accessing Nokia AMS,” etc.
  • Maintenance Support — This page provides instructions for handling new maintenance requests that arrive via emails, phone calls, or portal requests. The process explained here should instruct engineers on how to collect the necessary information and create a ticket. Fields here might include things like Maintenance Summary, Party Conducting Work, Vendor Ticket Number, Service Affecting (yes/no), Impact Duration, Reason for Work, and Maintenance Window.

“You almost always need to have a page for escalation contacts in one spot. And then either one or multiple pages that provide clear instructions for the kinds of alarms the NOC will receive. The key questions the runbook should answer with respect to troubleshooting are, What are we getting? How are we reacting to it? And what does the NOC need to do with it?

— Skylar Carlino, INOC

Here's a simplified example runbook page for alarm response that visualizes the points we've been covering with some key call-outs:ino-RunbookExample-02

Maintaining a NOC Runbook

Change is a constant in almost every IT environment, so processes have to change, too.

Because the NOC engineers are the ones using these processes every day, they’re usually the first to notice something is out of date, broken, or changed. In that way, runbook maintenance is always reactive to some degree, so an open line of communication to make sure changes are made is key.

NOC engineers should be invited to pass observed changes off to those responsible for runbook management and have an easy way to do so.

Expiration dates and regularly scheduled reviews can go a long way in catching these changes proactively. Here at INOC, we establish annual expiration dates on each knowledge article we develop with a report that notifies us when something is set to expire a month ahead of time. This way, we have a rolling review program across all our runbooks.

Tips for Writing Excellent Runbooks

Here are a few additional pieces of advice from our runbook team.

1. Anticipate possible outcomes and write them into your processes.

One of the hallmarks of a truly effective runbook is the absence of any “dead-ends” in the processes. A process anticipates every possible outcome of a given action or status and provides instructions accordingly.

“If the node is up, do this; if the node is down, do this.

Every reasonable possibility is accounted for so engineers don’t get stuck without instruction—possibly extending an outage.
Eric Idler Photo-modified

“It can be a challenge to know what you’re going to see from a network at the outset of establishing support for it. That's where we draw on our existing knowledge to start mapping what might happen in that network based on our clients’ devices and what they want the NOC to do.

We often work closely with our Advanced Technology Services department. This team will actually comb through the network and determine what we need to be seeking to accomplish what we need to deliver. That includes cataloging the important alarm information and filtering out truly unactionable noise.”

— Eric Idler, Director of Shared NOC, INOC

2. Carefully consider format and platform.

Today’s digital knowledge base platforms are a far cry from the static, “unsophisticated” documentation systems from years ago. Of course, if a legacy system is working well, there may be no need to update it. But having seen countless IT teams dragged down by systems that get in the way of efficiency rather than enabling it, it’s important to realize that modern platforms solve many of the problems teams used to more or less be stuck with.

Here at INOC, for example, we’ve embraced the concept of the modern knowledge base and applied it to the way we structure, write, and manage our runbooks. Rather than forcing staff to frantically search long, unwieldy documents, our runbooks are shorter, more skimmable separate pages that use a simple one-click linking structure to make it fast and easy to navigate between them.

In addition to paginating our runbooks within a knowledge base system, we’ve also refined the format into the example shown above in this guide.

One of the most important elements of our runbook template is the “boilerplate” info boxes that appear at the top of each page within a given runbook. These are the same from page to page and present information we’ve found engineers routinely need at their fingertips no matter where they are in the runbook. Rather than constantly going back to a separate “info” page, that information is present on every page.

3. Run the process as you document it—and pressure-test it afterward.

If possible, give your process writer access to whatever they need to actually step through the process themselves so critical details can be captured that may not be otherwise. Tools and platforms are rife with quirks that need to be guided around—and there’s no better way of addressing these details than running the process.

Also, after a process is documented, don’t assume it can’t be immediately refined. Hand it off to someone else that would be responsible for actually executing it and have them pressure-test it. Use their feedback as input for improvement before it ships to the team. 

INOC’s Skylar Carlino explains:

“If I'm doing an alarm response article, I want to be able to execute the process as I'm writing it to capture everything in detail. And when I think I'm done with an article, I'm running it by someone in the NOC to make sure that they can run it. I've worked in the NOC before, which gives me an advantage as a process writer, but processes should still be validated by those doing the work.”

— Skylar Carlino, INOC

4. Take the opportunity to eliminate “assumed knowledge.”

Whether you call it “tribal” knowledge, “assumed” knowledge, or any other term, we’re talking about one of the worst problems in IT operations: letting important knowledge live only inside someone’s head. 

Some teams operate almost exclusively on assumed knowledge. There are no runbooks. Other teams document their processes but bake some assumed knowledge into them. They skip writing down the steps that are “obvious.” 

This works—until it doesn’t. Whether it’s an employee leaving the team and taking that knowledge with them, or it’s time to outsource or augment some of the work to an external team, everything that wasn’t written down can very quickly become “missing information” and generate headaches for all involved. Simply put, don’t leave anything to assumption.

5. Deal with length thoughtfully.

While some runbooks are too “light” on info, others go overboard—including unactionable details in an effort to be exhaustive. A runbook that’s too long can get in its own way. Typically, runbooks that are too long contain peripheral information that someone thought might be useful in some cases—such as the history of a device—but most of the time, won’t be needed.

Rather than obstruct the process with that detail, stick it on the reference sheet so it’s there if and when it’s needed, but not imposed on everyone trying to execute a process.

Outsourcing Runbook Development to Third-Party NOC Experts

Here at INOC, ​​we routinely deliver expert-driven runbook development as a professional service component of our NOC operations consulting services. We work closely with teams looking to radically improve their support workflows by understanding and documenting their processes into a single source of truth for everyone inside and outside of the NOC.

Clients turned up on our 24x7 NOC support service also receive detailed runbooks as part of that service—documenting all the work steps our NOC will execute for managing operations, troubleshooting, and escalations. (While primarily an internal-facing document, our NOC clients get full visibility into our processes and how our team and tools will interface with theirs.)

If you’re looking for expert-driven runbook development, here are a few tips that can make an engagement go smoothly:

  • Know what you want in a runbook in as much detail as possible ahead of time — Simply put, the less a team knows what they want in a runbook, the harder it is to write one that satisfies the desires of every stakeholder. The more detail that's communicated up front about what everyone wants the NOC to do and how they want that done, the less a runbook writer has to re-engage stakeholders to get those critical answers.
  • Designate a contact that can be reached for questions — Runbook development is a collaborative process by nature, so identifying a clear point of contact to field those inevitable questions can help the process go much faster. This person doesn’t need to be the subject matter expert in everything—simply someone who can articulate what’s wanted in a given process and possibly connect the writer to SMEs.
  • Prepare to provide access to tools and systems — Catalog the platforms a runbook writer will need to access and make the preparations necessary so they can write their processes from within those tools, logs, etc.
  • Gather relevant network documentation — This is a big one. What does your network look like? Writing clear and complete processes is much easier when a map of that network can be referenced. Collect any potentially useful documentation that could be helpful to a runbook writer ahead of time, or highlight your gaps in this area so the service provider can get to work filling them.

 

Final Thoughts and Next Steps

The effectiveness of a NOC runbook boils down to a few key qualities:

  • Key contents — An effective NOC runbook includes Infrastructure Documentation, Process Documentation, Links to Tools/Contacts/Data, and Alarm-to-Action Guides. This information is expressed across a few key pages, including Client Information, Escalations, Device Access, Maintenance Support, and Response.
  • Intentional management and maintenance — Those responsible for keeping runbooks current should invite process changes from NOC engineers noticing changes happening across the supported environment and also work proactively to keep processes fresh through expiration dates and regularly scheduled reviews.
  • Format and structure — The runbook platform and its layout should enable rather than inhibit UX.

Want to learn more about our NOC runbook services and how we can help you achieve peak performance while saving your team valuable time and resources? Contact us or use our consultation request form to tell us a little about yourself, your infrastructure, and your challenges. We'll follow up within one business day by phone or email to schedule a time to learn more and explore solutions.

Want a handy guide to solving the top challenges NOCs face today? Download our free white paper below.

Top 10 Challenges to Running a Successful NOC

FREE WHITE PAPER

Top 10 Challenges to Running a Successful NOC — and How to Solve Them

Download our free white paper and learn how to overcome the top challenges in running a successful NOC.

Download

Eric Idler

Author Bio

Eric Idler

Director of Shared NOC, INOCEric has worked at INOC for six years, starting with the Service Desk team and progressing into various leadership roles, including Manager of Advanced Incident Management and Senior NOC Manager. With hands-on experience in Operations and Service Desk Management, he currently manages the Shared NOC. He ensures that incidents are handled efficiently and effectively through their life cycle to support INOC clients. Eric has a BS in Economics and Computer Science from the University of Wisconsin-Madison.

Let’s Talk NOC

Use the form below to drop us a line. We'll follow up within one business day.

men shaking hands after making a deal