There are other SNMP variables monitored, but the majority of our monitoring is looking at these variables on the edge of the network.This amounts to about 500,000 snmp variables being monitored.We are using MRTG to poll the data and stuff it into rrd files.
We also only monitor ports that are currently active or have been active in the last 7 days, so it is closer to only 175,000 RRD files being active at any one time.It turns out that MRTG, properly configured, is an amazingly robust snmp poller.We run 6,000 "targets" per MRTG process, and each MRTG process has 4 forks for snmp polling.RRDtool is more or less the defacto way to store time-series data in the information technology world.Many sites are now monitoring more and more systems as a whole while increasing how much data they are collecting per system.
The ability to gather and store this much data is a testament to rrdtool's ease of use.
While RRDtool performs very well for small- and medium-sized installations, the RRD update mechanism requires a surprising number of I/O syscalls for each operation.
These syscalls, in turn result in cache-unfriendly (random-seeming) I/O access, which defeats most OS and hardware caching algorithms.
The main focus of this page is to examine how RRDtool performs in the large scale and propose some solutions.
At the University of Wisconsin at Madison, the central IT department is responsible for the majority of the campus network infrastructure which consists of approximatly 30 routers, 2,000 switches and 1,500 wireless access points.
For each interface we use SNMP to monitor byte counters, unicast, broadcast, and multicast packet counters, and interface errors.