Vendor bloggers in Network Management; or

… how I  created a planet network management website to aggregate all of the network management related blogs I read. I was just going to do your bog standard list post, but then I thought it would be more fun to have a dedicated website instead.

I’d like to tell you how I’ve sweated for days  deep into the night, depriving myself of sleep and sustenance, whilst I slaved over a blinking terminal to get it working… well, I can’t because it only took a few hours of not very hard work plus an inexpensive domain name. 😉

The site uses the rather good planet software. The site is a bit stripped down at the moment, I will work on the look as things progress.

If you have any suggestions for blogs to be added, please get in touch. Alternatively, if you are the owner of a blog featured on the site and don’t wish to be, please get in touch as well.

Open source network management in Google 2001 vs Google 2008

Google have released a fully searchable version of their first available index from 2001 to celebrate their 10th birthday. I thought it would be interesting to compare and contrast a search for “open source network management” using the 2001 index and the current index.

The first thing that springs out are all of the adverts in the 2008 version. My guestimate is that you’re going to be bidding well north of $5 per click for the top spot on there.

The second thing that pops out is the number of results: 1,330,000 versus 11,900,000 results. That’s a heck of a big growth! Getting on for ten times more pages matching the search between 2001 and 2008.

The search results themselves seem better in 2008 than way back in 2001. In the sense that the search does actually provide results to things that are open source network management tools with the inevitable wikipedia article thrown in for good measure.

Things sure are more competitive now. 😉

How will cloud computing change network management

The big selling point with cloud computing is that computing capacity grows and shrinks depending upon the load being put upon it. You typically only pay for CPU (by the hour) and storage (by the month) you actually use.

If your website enjoys a sudden surge in traffic, by appearing on the front page of slashdot for instance, then extra servers will be provisioned automatically. Once the peak load passes and more normal traffic levels return, the extra servers are automatically de-provisioned.

How will cloud computing impact upon network management?

Service oriented monitoring

Cloud computing makes network and server infrastructure invisible. A large part of existing network management is involved with making sure that the network and server infrastructure is working properly.

The focus of network management in a cloud environment shifts away from managing infrastructure to managing service availability & performance.

One of the big sells of cloud computing is not only outsourcing the provision of a scalable, enterprise grade network, but also the necessity to manage it as well.

Root cause analysis will effectively come down to ringing your cloud vendor’s tech support team. Instead of your painstakingly planned network management system tracing the root cause of your outages, you’ll be relying upon your cloud vendor’s painstakingly planned network management system instead.

Standard cloud instrumentation

Your cloud vendor provides detailed usage statistics of your cloud through the cloud vendors management portal. In order to use this information in your own network management system you need it to be available in a format that can be read from your network management software.

Many network management systems have fine extensibility mechanisms so that you could wire up your network management system to use your vendor’s instrumentation.

A better solution would be for a standard to emerge that all cloud vendors implement. Not very likely given the diverse offerings in the cloud computing market. Amazon EC2 has little in common with Google Apps for instance.

The more likely scenario is that a winner will emerge eventually and that will become the de facto standard. Looks like Amazon at the moment, but I wouldn’t underestimate either Google or Microsoft. Microsoft in particular have a good deal to lose if they don’t allow Microsoft centric web developers take seemless advantage of cloud computing.

Configuration management

One of the major issues with compute clouds is the process of configuration management of the software image to be deployed. If the software running in the cloud has a bug, then I need to be able to revert to a previous image or upload a new one quickly.

In addition, controlling when new software is deployed is likely to be very important. I’m sure you don’t relish waiting around for an off peak time period to upload new software, it would be useful to have your network management system to do it for you.

Whilst configuration management is a function of network management, few tools outside of the big 4 are able to do it.

Managing the cloud in the cloud

The obvious place is to put your service oriented monitoring in the cloud right alongside your application. Whilst such an approach will work most of the time, what happens when your cloud vendor’s network dies? All of the major cloud vendors have had network outages so it isn’t a theoretical risk.

An alternative to deploying your own monitoring solution could be to use one of the many vendors promoting SaaS based online monitoring solutions. Whilst that is probably going to be a more robust solution, it may be difficult to know precisely how the vendor has deployed their solution. What if the vendor is using the same cloud computing vendor that you have chosen? Then it won’t be any better than deploying your own monitoring solution in the cloud.

One of the side effects of managing in a cloud world is that vendors will need to be more open about their infrastructure arrangements. If your management vendor is using a cloud then they need to be open about it.  Otherwise, when the 800 pound gorilla of the cloud computing space inevitably develops, there may be a danger that both management and managed services use exactly the same infrastructure.

Clouds are made of data centres, right?

Compute clouds are going to be housed in data centres, big ones too and lots of them. That’s good news for the enterprise network management vendors because those data centres are going to need managing.

People running small & medium sized data centres are likely to be the people most attracted to cloud computing. Therefore, there is likely to be a consolidation away from the small and medium data centres to large data centres. The only people who’ll be able to justify the cost of running a small or medium data centre are those with specialist requirements that cannot easily be accommodated using a cloud based solution.

It is hard to see the transition as anything other than party time for enterprise vendors. Open source enterprise vendors will be in a very good position to win new customers. The transition to cloud computing is a once in a lifetime disruption causing a lot of cloud vendors to look for new, more flexible tools to help them manage their new, ultra-flexible infrastructure. In addition, the new cloud vendors are utilising open source software extensively in building their offerings, so they will be more amenable to open source based network management solutions.

Conclusion

Cloud computing is very much in its infancy. Clouds in their current form are likely to appeal to companies who would otherwise have opted for shared or virtual machine hosting in the case of Google Apps or one or more dedicated servers in the case of Amazon EC2. In either case, neither would be in the market for network management systems.

The next wave, when the early adopters have ironed out the problems,  will be from companies replacing small or medium sized data centres with a compute cloud. These companies will still require a management system, but one tuned to managing in a cloud environment not a data centre environment. They’ll need to monitor the services they are providing and manage their interaction with the cloud.

On the other hand, the cloud computing vendors themselves will require full blown enterprise grade network management systems. A great opportunity is being presented to the more nimble vendors, able to quickly fine tune their products to the particular requirements of the cloud vendors. The open source, enterprise oriented network management vendors will find their offerings chime well with the cloud vendors. The cloud vendors are already heavy open source software users.

The network & systems management blogosphere

Thought I’d share the network & systems management blogs I read on a daily basis in the hope that you’d share yours with me.

Network & Systems Management

Cloud Computing

Open Source

Enterprise Management

Protocol Analysis

Data Centre

Aggregators

I’m always on the look out for good systems management blogs, so if you know of any I really should be reading, I’m all ears. 😉

Update 1: I’ve added the excellent blogs suggested by Matt Simmons & zennippon. Thanks! Anybody got any more that should be on the list above?

Update 2: Oops, forgot about Socialized Software… sorry Mark!

Distributed network monitoring interview with Robert Aronsson

Robert Aronsson is the CEO of Intellipool AB a company with over ten years experience of the network management market. Intellipool introduced a distributed network monitor over four years ago. I interviewed Robert with a view to getting some insight into Intellipool’s experience of implementing distributed network monitoring solutions with their customers.

The interview was conducted via email. My questions are in bold with Robert’s answers underneath.

Q: What key factors determine whether you need a distributed solution?

The most obvious factor is network location, doing direct network monitoring over, for example, a branch office VPN is not something I would recommend. Depending on what you are monitoring it can be a huge resource drain on a VPN that should be used for “normal” office work.

By placing a remote gateway at a branch office, INM can monitor all aspects of the remote servers to just a fraction of the “bandwith cost” compared to direct monitoring.

There is a second factor that might not be that obvious, but Intellipool’s take on distributed monitoring can be used as a “clustering” solution, splitting the monitoring workload over several machines if you have a very large network to monitor. Since all management is done from the central INM server it’s very easy to move monitored objects between different gateways when you for example needs to take down a machine for maintenance.

Q: What are the biggest barriers to successfully deploying a distributed network monitoring solution?

The two biggest practical problems that you will encounter is deciding if you need a dedicated machine to host each remote gateway and re-configuring firewall rules in both ends.

In Intellipool we have implemented the server/gateway system to make it as cost effective as possible for the customer, this means that:

  1. Remote gateways always connect to the central server, meaning that you have fewer firewall rules and also avoid the risk of deal breakers such as having to open a customer firewall for incoming traffic;
  2. Since the gateway connects to the server, the remote gateway can be placed on a dynamic IP;
  3. A typical Intellipool gateway consumes around 30 MB of memory, stores nothing but temporary work files on disk and can be remotely updated once it’s installed. Installation of a new gateway is done in a matter of minutes. The low resource footprint makes it possible to co-locate the remote gateway running on an already installed server.

Q: What is the best management structure for managing a distributed solution? Should each office/department be responsible for its part or should it be controlled centrally?

Control of the actual software (ie. the remote gateway) should be handled by the same group/entity that manages the Intellipool server. Management of monitored objects is another story, it’s quite possible to hand over the control over the object to a local sys admin. But in reality it’s best that everything is managed by the same group of people, the risk of responsibility grey zones is reduced if you have appointed one group of people to control the whole chain.

Q: What are your top 3 do’s and don’ts?

(I’ll formulate it in a bit different way here) There is a number of things to consider before going ahead with distributed monitoring.

  1. When considering distributed monitoring, make sure that the traffic to and from servers are encrypted and (something that is often forgotten) protected from man in the middle attacks;
  2. Make sure that you select a system where installation and upgrade of remote gateways can be done remotely;
  3. Distributed monitoring is not for everyone, even if your organization scattered around the globe connected with VPN’s. If you are just interested in basic monitoring (ping, simple HTTP monitoring etc) you are likely better off doing it from one central location, since in this case bandwidth is not an issue.

Thanks Robert. If you have any questions, please leave a comment.

In depth open source network management comparison

Jane Curry of the UK based Skills 1st network management training and consultancy company has written a rather good open source network management tool comparison. It is a large PDF file ~150 pages, so you have been warned!

Kudos to Jane for doing the comparison, it must have been a whole heap of work. Enjoy!


Distributed network monitoring introduction

A number of mid-level network monitoring products, like What’s Up Gold & Intellipool for instance, have recently implemented distributed monitoring features. Mid-level network monitoring products are now implementing  distributed monitoring so it is affordable by a lot more companies.

Single Poller Monitoring

With regular network monitoring you have a single poller measuring network and server performance from a single location on your network.

Architecture of a central polling in a distributed network
Architecture of a central poller in a distributed network

Single poller monitoring works well when the network is small or only has a single site. Every request is made from a single location to each of the resources being measured.

Whilst single poller network monitoring is well suited to single site performance monitoring, it does not scale well on larger, multi-campus networks.

What is Distributed Network Monitoring

Distributed network monitoring involves multiple pollers distributed around your network measuring performance from multiple locations on your network

Architecture of Distributed Polling in a Distributed Network
Architecture of Distributed Polling in a Distributed Network

Multi-campus networks typically have WAN links interlinking the various sites. WAN links are usually much slower and more expensive than LAN links. By placing your network monitoring probe in a single central location you are inevitably going to send more traffic over your WAN links.

Distributed network monitors permit you to locate your probes locally to the resources being monitored with only the statistics being synchronised en-masse back to a central Network Operations Centre (NOC).

Advantages of Distributed Network Monitoring

  • Real user view of network performance — with single point network monitoring you see the network from a single perspective. With distributed network monitoring you see the network from a number of different views across your network;
  • Helps with network troubleshooting — distributed network monitoring gives you multiple performance profiles giving you the ability to detect outages and bottlenecks more easily
  • Reduce bandwidth requirements over WANs — a central poller will send requests over your precious WAN links. A distributed network monitor will usually be configured to send requests to local resources and appropriate global resources;
  • Single consolidated NOC view — rather than have a number of separate network monitoring systems situated inside each campus, distributed network monitors allow you the best of both worlds. Monitor resources locally but consolidate all stats into a single NOC for analysis and storage.

Disadvantages of Distributed Network Monitoring

  • More expensive and complex — distributed monitors are more expensive than single poller monitors, sometimes quite a lot more. You also need to find the hardware upon which to deploy the remote pollers and the time for installation and configuration;
  • Unless carefully designed you may end up using more WAN bandwidth than a central network monitor — if you are not selective of which services you monitor and from where you will find no savings in bandwidth usage with a distributed network monitor. Unless polling a resource is going to buy you some insight into your systems performance then monitoring it from a remote site seems like a waste of bandwidth.

Recommendations

  • Multiple single poller monitors, one for each remote office, may be more appropriate if each office runs its IT systems autonomously with few shared systems. Distributed network monitoring comes into its own when a single NOC view of the entire network is required. If you are happy with multiple autonomous point tools then a distributed system may be overkill;
  • Only monitor resources remotely that are genuinely used remotely. This will not only save you the bandwidth required to periodically test the resource but mean that you do not need to deprecate your carefully designed security policy by making a resource more publicly available than is entirely necessary. In addition, your monitoring effort won’t tell you anything meaningful anyway because none of your users use the resource remotely;
  • When remotely monitoring a resource, do not set up a separate comms channel for the monitoring system to use. For a performance monitor to be of any use it needs to use the same infrastructure that your users utilise. If you’re not careful the network monitor just ends up effectively monitoring itself.

I’ll be investigating your open source distributed network monitoring options soon. In the meantime, if you’ve got any feedback, please leave a comment!