NOC Automation: A Guide to Current and Future Capabilities

Right now, troves of raw data sit in digital warehouses begging to be analyzed and distilled into clear instructions—instructions that would be more accurate and complete than a human engineer could ever be expected to create consistently. Instructions that would enable those humans to devote their attention to more important projects while helping to better prevent issues that threaten IT infrastructures and the businesses that rely on them.

AIOps—Artificial Intelligence for IT Operations—is already hard at work in the NOC doing just that. This very second, machine learning systems are plucking critical data points from the massive volumes of data generated across a typical IT environment and then marrying that data with automation to act on it—performing basic tasks that, until recently, only human engineers could do themselves.

For a few years now, top-tier NOCs have started to apply automation to take on the repetitive, low-risk tasks that pull technical specialists away from more important (and frankly more exciting) work. Only more recently, however, have NOCs started arming themselves with vastly better data processing and machine learning power to augment and replace more—and more complex—manual tasks traditionally handled by humans.

Perhaps the most impactful recent advancement is AI-driven event correlation. NOCs can now let machines correlate event data much faster than humans ever could and identify the subtle indicators of approaching issues within a torrent of otherwise noisy data. The outcome can be measured in significantly faster and more proactive response rates—and thus, happier customers and end-users.

This combination of automation and machine learning brings the power and promise to genuinely transform how IT operations teams organize and operate. And as time goes on, automation will steadily continue to replace even more manual activities better suited for machines.

To assuage the lingering and understandable anxiety of human replacement, let’s be clear: the humans won’t be gone. They’ll just be freed from support tasks they’d likely rather not do anyway to focus on solving bigger problems and work on revenue-generating projects.

As a NOC service provider using these tools ourselves, we’re often asked whether it’s possible yet to build a fully autonomous or “dark” NOC. The answer today is still no. But the capabilities available today—the products of progress that have been made in pursuit of a more autonomous future- are already powerful enough to dissolve many of the human and technical limitations underpinning long-standing challenges to running a high-performing NOC operation. Short of a fully autonomous NOC, the value AIOps brings right now is immense and growing rapidly.

Here, we explain exactly how AIOps is helping NOC teams overcome some of the biggest operational challenges standing in their way right now. Rather than repeating the typical hype and hyperbole about AI, we bring the discussion down to earth and into the modern NOC, giving you a clear and concise explanation of AIOps in the NOC support context and what your organization likely stands to gain from applying its power to its support function.

📄 If you'd like a deeper dive into automation/AIOps in the NOC, we invite you to download our free white paper on the subject below or request a NOC consultation with our Solutions Engineering team.

What is NOC Automation, Exactly?

First, a brief primer on the concept of automation in this context:

NOC automation involves implementing tools and processes that shift repetitive day-to-day tasks from humans to machines. This simplifies operations and makes the NOC more efficient and effective as a monitoring and support engine. 

Machine learning enables us to expand what can be automated beyond the realm of the basic and repetitive. With its incredible processing power and the ability to “learn” as AI can, it automates—either fully or partially—more “knowledge-based” activities, such as correlating alarm data and generating tickets in addition to notifications and escalations.

Thinking Clearly About the Autonomous NOC

The promise of what AIOps might (and likely will) be able to do in the future can blur the expectations of what’s actually possible today. 

Let’s be clear about what’s currently possible and what has yet to come:

While AIOps holds the power to deliver many new capabilities, the immediate applicability in NOC support takes the form of augmenting support processes by automating low-risk tasks and improving the accuracy of others.

Just how big of a role these tools can play is also an important point around which to set reasonable expectations. It might be tempting, for instance, to expect AIOps to run support operations autonomously. A full NOC operation—24x7 staff interacting with multiple internal teams, each offering a variety of skills and customer knowledge, and a network of third parties (cloud and SaaS providers, data centers, circuit providers, field support)—still plays a substantial role in maintaining infrastructure availability and performance to ensure customer satisfaction. 

If a fully autonomous future is even possible, it’s far enough in the distance to disregard for now. Yet as machine learning tools consume and process more data, they will inevitably become smarter and more capable, allowing for more in-depth analysis while pinpointing issues more quickly and predicting problems sooner.

New Solutions to Old Problems

A 2018 survey by 451 Research found that 64% of IT infrastructure teams saw their workloads increase from the previous year. Studies like this and our observations from speaking with IT professionals each day make it clear that as environments get larger and more complex, workloads continue to grow without a corresponding increase in the resources needed to manage them. 

Part of the excitement surrounding AIOps isn’t just that machines can now take on this growing workload, it’s that the machines can do some of that work significantly better and faster than humans ever could, thereby solving some long-standing problems that have plagued NOCs for years.

• One of those legacy problems is alarm noise. AIOps can analyze and correlate millions of data points—many more than any human team ever could—to dramatically improve the signal-to-noise ratio, reveal real problems, and intelligently group events to isolate the root causes.
• Another legacy problem AIOps helps solve is outage prediction and prevention. With the ability to analyze so much data, AIOps can identify subtle anomalies and recognize patterns that would otherwise pass right through the fingers of human engineers. With their incredible processing speed, these tools can pick out subtle indicators of impending problems to help predict and prevent them before they occur. It’s a genuinely transformational capability that can save untold amounts of money in prevented outages.

Taken together, these capabilities remove the hurdles standing in the way of the NOC doing its primary job: improving uptime and performance, accelerating incident response, and preventing outages—all while simplifying the NOC operation itself.

How AIOps is Enhancing Core NOC Processes Today

Nearly every imaginable operational issue within the NOC is rooted in at least one of its foundational processes:

• Event Monitoring and Management;
• Incident Management;
• Problem Management, and
• Change Management.

A problem in any of these processes can trigger a cascade of issues that impede the NOC from doing its job in others.

For example, if the NOC can’t operate efficiently, Incident Management processes can’t be executed quickly to detect and fix issues. Connecting event data with past configuration changes becomes impossible. Ultimately, valuable opportunities to improve infrastructure and application availability, and support operations are missed. The resulting costs are high, both in financial expense, customer reputation, and team morale.

By tapping into analysis capabilities that far exceed what even the best human experts can achieve, and automating certain tasks, AIOps eliminates the underlying inefficiencies that cause so many operational problems while bringing a whole new level of data analysis capacities. The patterns it reveals within torrents of data across an entire IT environment aren’t vanity metrics; they provide clear, actionable intelligence to inform NOC support decisions.

Now, let’s unpack the value of AIOps by explaining what advantages it can bring to each of those core processes.

Event Monitoring and Management

AIOps can aggregate data from multiple data sources and multiple technology areas across the entire enterprise and provide a central data collection point. It can then analyze this data quickly and accurately to determine when multiple signals across multiple areas indicate a single issue.

The resulting reduction in alert noise brings into focus those alerts that require action and helps reduce Time to Impact Analysis and thus reduce Mean Time to Repair.

Events can also be correlated with past configuration changes, allowing for faster, more reliable root cause determination.

Incident Management

AIOps can feed analysis into the Incident Management process by autonomously surfacing the probable cause and allowing the NOC engineer to confirm that the analysis and data are sound before implementing a plan for resolution.

The result? Faster incident analysis. AIOps can also automate responses and substantially reduce reduction times when implemented cautiously and thoughtfully.

Once a root cause is determined with high confidence, a solution can be automatically implemented if it’s available. Low-risk automation for routine alerts in non-business-critical workloads is a smart starting point, freeing NOC engineers to focus on complex infrastructure support issues.

Lastly, AIOps can issue predictive alerts when it correlates real-time event and performance data with past event data that resulted in outages to identify developing problems before they require a reactive response.

Problem Management

While the goal of Incident Management is to restore service quickly, Problem Management determines the root cause and finds a permanent solution to avoid the same incident in the future.

Root cause determination is typically resource-intensive, requiring hours of event and log data analysis. With access to multiple sources and massive amounts of data, AIOps can radically improve post-event root cause analysis.

AIOps can provide intelligent analysis—ranking events by their relationship to the original alert, noting anomalies, and suggesting possible causes—to streamline the Problem Management process. This capability allows the NOC engineer to more easily and confidently confirm the analysis, verify the data behind it, and then develop a solution.

Change Management

Maintenance events are common in the NOC. An effective AIOps implementation allows automatic suppression of alarms when an infrastructure or application maintenance event is recorded. Automation will then only create tickets if appropriate after a maintenance window has been completed.

Alarms can be associated with the change by tying in the configuration item; this helps correlate future events with configuration changes. AIOps can also provide for deeper impact analysis by using relationship and topology data from multiple sources, such as the CMDB and monitoring tools, to help IT teams understand how a change on one node may propagate to other nodes, leading to a potentially undesired impact.

Lastly, by applying AIOps to historical change data, IT teams can get insight into the likely consequences before implementing a change. Changes can be given risk scores (such as low, medium, or high) to help quantify acceptable risk and inform the decision to deploy a change.

Overcoming Key Operational Challenges

In addition to looking at AIOps through the lens of what the NOC does, we can also understand its utility through the challenges it can help NOCs overcome. 

Let’s explore five specific hurdles many modern organizations stumble over and how AIOps is poised to help with each.

1. Navigating hybrid, interrelated, and dynamic infrastructures

In today’s application-rich environment, systems have become more interdependent on one another. The result is a highly complex infrastructure ecosystem: a combination of traditional and cloud applications and infrastructure, along with technologies such as containerization, serverless computing, microservices, and orchestration tools.

Add in network technologies such as Software-Defined Networking and Network Functions Virtualization, and you’ll quickly find yourself in an environment with multiple devices and systems and high levels of interdependence to support various organizational services and applications. In these interrelated and interconnected environments, a single outage can reduce organizational knowledge and prevent engineers from accessing needed systems. 

Without good visibility, pinpointing the root cause of incidents—let alone understanding the overall impact of an outage that spans multiple environments—becomes impossible.

How AIOps can help

AIOps solutions can provide visibility into this complex interconnected environment by pulling together network, server, cloud, and application data into a single platform and analyzing topology, metrics, and traces for dependencies and correlations.

In addition, AIOps solutions can enable incident managers and NOC engineers to add operational and technical knowledge to the platform. When this is done using well-structured NOC processes and procedures, such platforms become truly useful in enhancing NOC support over time.

2. Reducing alert noise 

NOC engineers typically field—and can often be overwhelmed by—both genuine alerts and many false positives. High alert noise distracts support teams from focusing on real issues. Over time, alert fatigue sets in. In the best case, the team takes a long time to find the actual issue. More than likely, however, too many alerts have the same effect as no alerts and are simply ignored.

Also, most threshold alerts are static. For example, an alert may be created if CPU usage is above 90% for five minutes. A lack of contextual awareness (batch job execution, for example) for threshold alerts leads to unnecessary incidents being created, adding to the NOC staff workload without actually improving infrastructure availability.

High alert volumes ultimately result in the same situation: higher MTTR and a poor reputation for the NOC as the critical alerts are not acted on quickly.

How AIOps can help

Reduction in alert noise sharpens the alerts that require action, enabling the NOC team to focus on the right issue instead of spending time on false positives. 

AIOps helps find the relevant correlated data in real time from this large volume of alerts, allowing the NOC engineer to focus on the right alerts at the right time. This allows the NOC to resolve the incident within the service level agreement (SLA)/service level objective (SLO) windows. Continued data collection allows for thorough analysis and Problem Management support later.

3. De-siloing systems and teams

The typical enterprise (or IT-complex) organization uses multiple tools to monitor and manage their complex environments, each collecting IT operations data and retaining that data in silos. (This may happen because DevOps teams use tools specific to their environment or because new tools are added when new technologies are adopted.)

With these separate data environments, it becomes very difficult to understand underlying infrastructure or application issues in current technology stacks with high levels of interdependence; siloed systems make centralized insight impossible. 

Organizations expend significant resources to maintain these disparate systems yet continue to lack a single-pane view for monitoring and managing incidents. Typically, each support team within the organization will investigate issues independently. The NOC then combines these teams’ tools and processes, eventually coalescing the data and analysis to determine the root cause. This leads to significant delays in incident resolution. Senior management escalation and involvement in troubleshooting is a common occurrence in these siloed environments, causing significant lost productivity for the organization.

How AIOps can help

AIOps typically integrates with existing tools in the market—allowing the organization to use best-of-breed solutions for various technologies—and pulls together disparate information from these tools. 

By capturing data from these tools and integrating them into a central database, AIOps makes it easier for machine learning algorithms to do a deeper analysis across the hybrid and dynamic infrastructure and can provide crucial insights that help identify root causes in real time and detect chronic infrastructure issues and other problems.

4. Breaking through the limitations of traditional ITOM systems

Traditional ITOM systems depend on an accurately populated CMDB with clearly defined relationships and dependencies between parent and child configuration items. 

However, in most support organizations, the CMDB quickly becomes out of date, given the usually incomplete implementation of reliable change management processes and procedures. Thus, depending on the CMDB when executing NOC processes such as Event Monitoring and Management and Incident Management is fraught with inaccuracies.

How AIOps can help

AIOps allows organizations to preserve existing investments in ITOM tools and bring together data from diverse sources for processing and correlation. It can further enrich the information on alerts with data from NOC runbooks.

Final Thoughts and Next Steps

Effective use of automation and machine learning technologies depends on understanding where to apply them within the multiple processes that a NOC supports. 

• Event Monitoring and Management is the obvious starting point, with AIOps helping reduce alert noise at the event analysis stage.
• Incident Management is enhanced when AIOps suggest the probable cause. 
• With the ability to analyze historical event, incident and performance data, AIOps can identify possible root causes in support of Problem Management.

A successful AIOps initiative provides customers with a much-improved experience, but implementation requires a strong foundation in NOC best practices and a well-developed organizational structure. This includes good knowledge management and training practices.

For AIOps solutions to become more intelligent and truly useful over time, resulting in a more automated NOC, domain expertise—operational and technical—needs to be “encoded” into the tools; logic needs to be reviewed, tested against results, and refined constantly. A continual service improvement program that includes quality control and assurance, and detailed operational and technical reporting insight, is key to getting good value from the AIOps solution.

A centralized monitoring and incident response team can serve as experts and offer high-quality support to dynamic organizations. Such a team can provide centralized management of these tools, developing standard responses—including automation—to incidents using AIOps where possible. Proper application of AIOps in the NOC can reduce event noise, help identify and resolve incidents quickly, and prevent problems from impacting customers

Wondering how your organization can benefit from AIOps? Download our free white paper to learn more about what we cover here and get a worksheet you can use to contextualize the advantages. When you’re ready to take the next step, reach out to schedule a time to talk through these insights and opportunities.

FREE WHITE PAPER

The Role of AIOps in Enhancing NOC Support

Download our free white paper and learn how your NOC support stands to gain from AIOps by overcoming operational challenges and delivering outstanding service. Use the free included worksheet to contextualize the value of AIOps for your organization.

Download →