In case your time is short
- Partial Monitoring Challenges: During network expansions, managing "partial monitoring" becomes complex as not all parts of the network are operational. Solutions like INOC's Ops 3.0 platform address this by enabling detailed tracking and customized alert management for operational segments amidst construction.
- Scalability Under Strain: Rapid network expansion can overwhelm internal resources, causing potential lapses in service and monitoring. Outsourcing NOC support allows companies to focus technical teams on expansion and growth while maintaining daily operational quality.
- Asset and Configuration Management Complexity: Efficiently managing a vast array of infrastructure assets and configurations poses significant challenges. Utilizing a dynamic CMDB that captures detailed network data facilitates better management and quicker resolution of network issues.
- Automation with AIOps: Manual management of growing optical networks is impractical. Implementing AIOps for automation ensures real-time response to alerts, increased accuracy, and efficient handling of complex network operations.
- Strategic NOC Outsourcing Benefits: Strategically outsourcing or augmenting NOC functions to experienced service providers can significantly reduce costs and complexity, offering enhanced support and allowing fiber operators to concentrate on network expansion and improvement.
As demand for reliable and efficient connectivity grows, private fiber providers are rapidly expanding their networks to meet it.
But with expansion comes operational challenges. One particularly pressing issue is these networks' cost-effective monitoring and management, not just during setup and configuration but crucially, throughout their Day 2 operations.
Many providers resort to third-party OEM vendors who offer some level of monitoring and management service—typically for a high cost with questionable quality and capability.
Others discover they can get greater control over network operations by building their own Network Operations Center (NOC) or accessing NOC support through a service provider. We’ve observed firsthand that fiber operators who strategically outsource or augment their NOC function to experienced service providers can attain enhanced support, reducing costs and complexity.
This guide dives into four specific fiber support challenges we see and solve through high-quality NOC support. Get in touch to learn more about how we help fiber companies get the NOC support they need.
1. Partial Monitoring
The most significant challenge we address in fiber networks, particularly during greenfield deployments or expansions, is the issue of "partial monitoring."
As companies build out fiber networks, they often need to turn up monitoring for multiple sites and many more devices, even if the entire network isn't fully operational yet. This situation presents inherent complications. By integrating more devices into our monitoring system, we often outpace these companies’ deployment rates. As a result, some devices may only be in a semi-active state, triggering potential alarms. While these devices might not be fully operational due to ongoing expansions, they may still support active services that require monitoring.
Our own Austin Kelly, Director of Advanced Technical Services, elaborates:
“The challenge we’re seeing now on the fiber side is this concept of partial monitoring. Many fiber companies are deploying maybe 120 sites and 300 devices, and they’d like everything turned up in monitoring as they move along. This presents some challenges because, inherently as we take on additional devices into our monitoring, we will sort of outpace them, meaning some of the devices we take on will only be partially turned up. So, they’ll be in an alarm state; there will be things we can't support or things that we shouldn't be alarming on because we're just pending a build out there."
— Austin Kelly, Director of Advanced Technical Services, INOC, An ITsavvy Company
Historically, the approach to this challenge was binary: either monitor the entire network once it's fully functional or don't monitor it at all until it's complete.
How INOC addresses partial monitoring
Today, we address the nuances of partial monitoring from a NOC perspective to ensure that operational segments receive proper monitoring and support, even while parts of the network are under construction.
Having encountered this challenge with numerous fiber clients, we've incorporated partial monitoring capabilities into our INOC Ops 3.0 platform, the most recent evolution of our NOC service delivery platform.
- Firstly, we often finalize build-outs ahead of a client's official launch. This forward-thinking strategy ensures that any device is meticulously documented once networked, powered, and configured to any degree. We actively record all its services, interfaces, links, circuits, and more, irrespective of their current operational state. Such detailed recording enables us to pinpoint the status of each Configuration Item (CI), whether it's fully functional, awaiting activation, or undergoing repairs. Gathering this information early, we can quickly update operational statuses as needed. This guarantees prompt responses to alerts, with all necessary CIs already logged, enhancing NOC support efficiency.
- Secondly, while older iterations of our Ops platform required any partial monitoring or filtering to be executed at the trap handler level—a tedious task limited to a few experts—Ops 3.0 offers a more flexible solution. The data from our monitoring platform is relayed to AIOps, granting us expanded customization possibilities. This increased flexibility ensures we aren't restricted or dependent on a single expert for modifications.
- Thirdly, we can label components as "in service" or "out of service." This allows us to mute alerts for components intentionally offline, negating the creation and retention of superfluous tickets.
INOC’s Ops 3.0 Platform at a glance
INOC's Ops Platform 3.0, our latest version, is the integration of our tooling—merging technology, operations, and service delivery. It processes alarm feeds, correlates them into unified tickets, and displays them for efficient management.
The Ops 3.0 Platform harnesses machine learning and automation (AIOps) to automate tasks and make data actionable for engineers. It analyzes and correlates alarm data, generating incident tickets linked with Configuration Items (CI) from our CMDB.
Tickets come pre-loaded with essential knowledge articles, streamlining the resolution process. Our support team collaborates with various stakeholders to ensure incidents are resolved efficiently.
Key features include:
- Automated Alarm Correlation: Uses machine learning to swiftly transition from alarm to ticket creation, ensuring no event is overlooked.
- Incident Automation: Processes alarms through our AIOps engine, correlating them and creating a single incident ticket. It also attaches relevant knowledge articles for resolution.
- Client Portal Self-Service: Allows direct interaction with our platform, including updates and maintenance requests.
- Auto-Resolution of Short Duration Incidents: Resolves minor disruptions automatically, focusing on more critical issues.
Platform components include:
- AIOps: Central to our alarm management, it uses machine learning to automate tasks and provide actionable insights. It can integrate with your existing monitoring system, streamlining alarm management.
- NMS: Allows alarm and event data ingestion from your (or our) NMS infrastructure, ensuring seamless alarm management.
- ITSM: Primarily ServiceNow, it enhances automation by linking CIs and records from the CMDB to AIOps-created tickets.
- TicketViewer: A queue management tool for NOC engineers, ensuring effective resource allocation and SLA management.
- Integrations: Our platform integrates with your ITSM tools and communication platforms for a cohesive support experience.
- Data Platform and Portal: Processes data from various sources, displaying it in a client portal with actionable insights.
- Runbooks, Knowledgebase, and CMDB: Provides step-by-step procedures for handling incidents with clear escalation paths. A robust CMDB aids in quick incident resolution.
We’ve also developed strategies to suppress or filter out alarms for components that are still under installation, ensuring that only relevant alarms from operational parts are attended to. Even as parts of the network are being set up, the operational segments remain under vigilant monitoring.
"Historically what we've done is an all-or-nothing deal. You finish the network, and then we bring it on, monitor it, and support it. But now we can bring a device into monitoring and suppress or filter the things that are not actively turned off that don't need our eyes on it—all while still providing support to the services turned up that are important to a customer.”
— Austin Kelly, Director of Advanced Technical Services, INOC, An ITsavvy Company
Clear communication is crucial throughout this process. By maintaining an ongoing dialogue with clients, we stay informed about their plans, enabling us to tailor our monitoring to their changing needs. This flexibility ensures prompt reactions aligned with the client's dynamic setup.
💡 INOC's Best Practices
- Granular Monitoring Configuration: Instead of a one-size-fits-all approach, monitoring systems should be fine-tuned to offer detailed oversight. This ensures individual segments or components of the network are monitored separately, raising only pertinent alerts during construction phases.
- Dynamic Alert Suppression: It's essential for your NOC to have the capability to adaptively mute alerts for components that are deliberately offline or under setup. This prevents alarm flooding and focuses attention on genuine issues.
- Phased Integration: For the rollout of new sites or devices, adopt a step-by-step integration method. Start by overseeing vital components or services and progressively broaden the monitoring range as more infrastructure elements become functional.
- Documentation of Partial Deployments: From the NOC's viewpoint, having precise and thorough data is the key to overseeing expansive deployments. Monitoring active components, sidelining those pending activation, and guaranteeing total data access is vital. Comprehensive documentation, encompassing network Visio diagrams, BOM files, and signal flow charts, should be made available beforehand. If a firm anticipates real-time, anticipatory monitoring from its NOC provider, it's crucial to furnish them with all essential details.
2. Scalability
The rapid growth of fiber networks presents a double-edged sword for providers. On one hand: progress and revenue. On the other, operational challenges that strain a company's technical resources when installs outpace support capabilities. This mismatch can lead to lapses in service and monitoring.
Network expansions often monopolize internal resources, especially for smaller to medium-sized fiber companies with limited technical teams. The very teams focused on prompt and effective project finalization are often the ones handling support. Juggling these dual responsibilities can be daunting, if not unfeasible at times.
"Many companies with limited technical resources are responsible for expansion and development like this. You're effectively losing those resources to your expansion efforts. Typically, the people that would otherwise support this are the ones working on building it—and you can't have it do both."
— Austin Kelly, Director of Advanced Technical Services, INOC, An ITsavvy Company
How INOC addresses scalability
This is where a NOC plays a pivotal role. Companies can redirect their technical specialists toward growth initiatives by delegating daily operational duties to the NOC. The NOC oversees and addresses problems and disruptions and liaises with carriers, delivering a comprehensive service.
"The NOC takes on those alarms; we're handling the customer notifications for their production environment. We're removing a lot of responsibility from those clients' network team."
— Austin Kelly, Director of Advanced Technical Services, INOC, An ITsavvy Company
In our experience supporting fiber companies, many are in the midst of expansion. We've assisted several clients through these growth phases. Our NOC handles alarms and customer notifications and alleviates the burden on the client's network team.
From liaising with carriers for fiber repairs to notifying customers about prolonged outages or collaborating with tech support for hardware issues, the NOC manages these tasks end-to-end. This approach minimizes the need for high-level resources to be consistently involved, allowing them to concentrate on other critical areas.
💡 INOC's Best Practices
- Resource Segmentation: If possible, allocate specific teams for expansion and daily operations. Avoid overburdening technical teams with both roles. This ensures that expansion efforts don't compromise daily operational quality.
- Documentation Priority: Prioritize comprehensive documentation for both expansion efforts and daily operations. This ensures that the NOC has all the information they need to manage daily tasks, and the expansion team has a clear record of what has been done.
- Scalable Tools: Invest in tools and platforms—and/or a NOC service provider—that can scale with your expansion efforts. As the network grows, your tools and services should be able to handle the increased load without compromising performance.
3. Asset and Configuration Management
Given the many infrastructure assets, communication components, and non-asset information, fiber companies often find they don’t have a CMDB capability to capture, store, and manage all this data efficiently.
How services are set up and represented often varies by device. For instance, while a fiber strand connects to an endpoint, the journey from that endpoint to the next defines a larger part of a circuit. Each segment in between can also be viewed as individual circuits. These endpoints might connect to a building in the vicinity, adding layers of complexity to the CMDB representation. However, a CMDB can associate customer services with specific dark fiber spans or multiple pathways. The challenge lies in gathering this intricate information and integrating it seamlessly.
"You're talking about a network that's not just a couple of routers and switches. You're talking about fiber strands, you're talking about endpoints, cross-connects. And it's not just about knowing what you have, but understanding how everything is interconnected."
— Tom Elliot, Change Configuration Specialist, INOC, An ITsavvy Company
Fiber operators differ significantly in their approach to asset and configuration management spending on their level of operational maturity. Some global providers still rely on rudimentary tools like Excel spreadsheets for inventory tracking. In contrast, a sophisticated CMDB can account for assets, serial numbers, contracts, etc. The primary aim is to reduce fault identification time and expedite resolution.
How INOC addresses asset and configuration management
Our CMDB is not just a static repository of assets; it's a dynamic tool that provides a comprehensive view of the entire network, capturing every relevant detail. It’s equipped to represent the multifaceted nature of optical networks—cataloging everything from individual fiber strands to complete circuits and even customer services.
Automation plays a crucial role in our CMDB, presenting all necessary data to the NOC technician, eliminating the need to sift through multiple sources. Centralized information storage is the essence, offering a unified view of all pertinent data.
"With a CMDB, you're not just seeing assets; you're visualizing your entire network ecosystem."
— Tom Elliot, Change Configuration Specialist, INOC, An ITsavvy Company
Unlike other NOC service providers who only monitor critical alerts, we monitor a spectrum ranging from critical to informational. This granular approach ensures cleaner networks with actionable plans for every alarm. Over time, we’ve discovered that understanding system behavior and baselining is paramount. As fiber evolves, systems have become smarter. Any deviation from the baseline, like a change in span loss, triggers an alarm.
Post-repair, it's essential to re-baseline systems to current conditions. Many fiber operators overlook this, leading to threshold alarms and unnecessary complications. Re-baselining and adjusting to evolving conditions is crucial for optimal operations.
INOC’s Gold-Standard CMDB
The INOC CMDB is central to our NOC support, facilitating operational tasks and automations. Beyond just device data, our CMDB encompasses all vital details for AIOps to correlate events, generate tickets, evaluate impact, and aid NOC engineers in problem-solving. We emphasize its upkeep throughout the service lifecycle.
We integrate our CMDB seamlessly with client configurations, associating relevant meta information to every alarm and ticket. This empowers our NOC engineers to make informed choices. We also use our expertise to enhance clients' existing CMDB structures, boosting efficiency.
Here’s a high-level look at the eight key components of our CMDB:
- Knowledge articles: These provide NOC engineers with current information, speeding up service delivery and building customer trust through standardized processes.
- Third-party contact data: Organized contact details for vendors and stakeholders in the CMDB enhance communication, leading to quicker issue resolution and better customer satisfaction.
- Circuit data: Detailed circuit data in the CMDB lets NOC engineers and AIOps swiftly tackle network problems, reducing downtime.
- Customer data: Precise customer information ensures effective communication during disruptions, leading to quicker issue resolution and improved customer satisfaction.
- Alarm data: Managed alarm data supports automation processes and ensures smooth incident management with accurate ticketing and timely customer alerts.
- Location data: Location data helps engineers quickly identify and address issues by pinpointing the physical location of IT assets. This information also assists in determining the impact of potential problems on the network and planning for maintenance or upgrades.
- Asset data: Comprehensive and up-to-date asset data helps NOC engineers understand the network's structure and connections between various components. This gives engineers a more comprehensive understanding of the network and its elements by connecting asset data to other types of data in the CMDB, such as circuits, services, and interfaces.
- Service data: Well-organized service data enables NOC engineers to understand the impact of alarms on customer-facing services and prioritize incident resolution accordingly. This distinction between physical connectivity and customer offerings is significant in cloud services. A structured and integrated approach to service data in the CMDB can help assess and address customer-related incidents more effectively, leading to higher customer satisfaction.
💡 INOC's Best Practices
- Comprehensive CMDB Utilization: Instead of relying on rudimentary tools like Excel spreadsheets, invest in a robust CMDB or NOC service provider that can capture, store, and manage intricate data. This centralized information storage can significantly reduce fault identification time and streamline resolution processes.
- Regular Re-baselining: Following any repair or significant change, it's essential to re-baseline systems to the current conditions. This practice helps avoid unnecessary threshold alarms and ensures the system operates optimally.
- Proactive Monitoring Spectrum: Rather than just focusing on critical alerts, companies should adopt a broader monitoring approach that encompasses everything from critical to informational alerts. This ensures a cleaner network and allows for actionable plans for every potential issue.
4. Machine Learning and Automation (AIOps)
The exponential growth of optical networks brings with it an avalanche of information. Every new asset, connection, or configuration introduces additional data that needs to be managed and monitored. In such a dynamic environment, relying solely on manual processes is inefficient and risky.
The more your network grows, the more data there is to handle. Manual processes just can't keep up. Manual interventions can lead to delays in response times, and the human element introduces the potential for errors, which can have cascading effects on the network's performance and reliability.
"In a world driven by data, relying on manual processes is like trying to catch a river with a bucket."
— Tom Elliot, Change Configuration Specialist, INOC, An ITsavvy Company
The future of efficient optical network support lies in automation. Recognizing the limitations of manual processes, modern NOCs, including us, are turning to automation to streamline their operations. By leveraging API-level integrations, NOCs can automate labor-intensive tasks, such as updating the CMDB. This ensures that the database remains current, reflecting the real-time status of the network.
"Automation is not just about speed; it's about precision. It's about ensuring that every piece of data is accurate, every time."
— Tom Elliot, Change Configuration Specialist, INOC, An ITsavvy Company
With automation, we're responding to alerts and incidents in real time, drastically reducing response times. Moreover, the automated processes ensure a higher degree of accuracy, minimizing the risk of errors. In essence, automation empowers NOCs to manage the complexities of modern optical networks with unparalleled efficiency and precision, ensuring they are always one step ahead of potential challenges.
💡 INOC's AIOps Capabilities
Rather than wasting time gathering information across a set of siloed and fragmented tools, we’re leading the industry in utilizing (and expanding) a suite of AIOps tools at strategic points in the NOC operational workflow.
Here’s a high-level look at how we’re using AIOps to deliver next-level NOC support:
- Event Monitoring and Management: We're the first, and so far the only, NOC support provider using AIOps to consolidate and process alarm and event data from all sources and help the NOC understand the significance of an alarm or event in the proper context, as well as its possible impact on infrastructure services and application availability.
- Incident Management: As AIOps continues to evolve in NOC operations, we’re using it to feed analysis into the Incident Management process by autonomously surfacing the probable cause and allowing NOC engineers to confirm the analysis and the data behind it. This further enables engineers to follow through on the response and beat expectations for resolution times.
- Problem Management: We’re leveraging AIOps’s immense processing power to analyze a trove of historical information to identify root causes.
- Change Management: Managing changes is time-consuming and fraught with risk. We’re using AIOps to reduce time and risk by communicating notifications for upcoming maintenances automatically and analyzing incidents in the context of previous Change Management activities.
📄 Read more: NOC Automation: A Guide to Current and Future Capabilities
Summary and Next Steps
Here’s a recap of the key takeaways—at a glance:
- Fiber operators are experiencing growing pains. As private fiber providers expand their networks to meet increasing demand, they face cost-effective monitoring and management challenges. Many providers either rely on third-party vendors or consider building their own Network Operations Center (NOC).
- Strategically outsourcing NOC support offers a better support model. Fiber operators can reduce costs and complexity by strategically outsourcing or augmenting their NOC function to experienced service providers.
- Partial monitoring is one of the biggest challenges INOC is best suited to address. During network expansions, companies often face the challenge of "partial monitoring," where only parts of the network are operational. Traditional approaches were binary, but modern solutions, like INOC's Ops 3.0 platform, address this by ensuring operational segments receive proper monitoring even during construction.
- Rapid network growth can strain technical resources, especially for smaller companies. By outsourcing daily operational duties to a NOC, companies can focus their technical teams on growth initiatives.
- Managing the vast amount of data from infrastructure assets and configurations is challenging. A robust Configuration Management Database (CMDB) is essential for efficiently capturing, storing, and managing this data.
- With the growth of optical networks, manual processes become inefficient. Automation, powered by AIOps, is the future of efficient optical network support. It not only speeds up processes but ensures precision and accuracy.
Let’s maximize uptime and performance across your fiber networks.
INOC takes your optical infrastructure support to the next level. Having worked on hundreds of optical fiber networks, we improve performance and solve problems before they affect end-users. ur engineers offer deep expertise in a wide range of optical network technologies: Ethernet (GigE), SONET, OCx ATM (100, 200,400, 800 Gig), any protocol—SONET, Ethernet, FibreChannel—over DWDM, GPON, and other technologies such as MPLS, PBB and OTN.
We support the optical platforms of a variety of equipment manufacturers, including Cisco, Juniper, Ciena, Alcatel-Lucent, Infinera, and Adtran. Contact us for optical network support that achieves meaningful ROI, and download our white paper below for a closer look at solving the top NOC challenges.
Free white paper Top 11 Challenges to Running a Successful NOC — and How to Solve Them
Download our free white paper and learn how to overcome the top challenges in running a successful NOC.