Technical Details of the InterMapper Flows Implementation

InterMapper Flows displays a list with columns for Client and Server. How does it determine which host is which?

InterMapper Flows uses a heuristic rule to determine whether a host is operating as a client or server:

There is a built-in list of common Server ports. If a host is using an entry in the list, it is treated as the server.
If there is no match with a common Server port, the lower-numbered port is treated as a server.

What effect does InterMapper Flows have on server load? How much memory does InterMapper Flows require?

There are no hard and fast rules, but NetFlow processing takes considerably more horsepower than InterMapper's standard monitoring. In general, we don't recommend using a discarded piece of hardware found under a forgotten desk for InterMapper Flows.

Receiving and processing flows deals with summaries of large volumes of packets, so the collection and storage process will usually not be the bottleneck. At one customer, where traffic peaks up to 3Gbps, the server could process every packet on an off-the-shelf laptop, while still having plenty of CPU time to spare.

However, handling queries (e.g., reading the saved data to send information to the GUI) is real work for the server. Depending on the selection criteria, the server sometimes must process millions of records to create the summaries and the graphs. This goes faster if more RAM memory is configured for caching session records. Queries going farther back in time will have to read the session records from disk with resulting slower times.

For example, say you have a 200MBps border, with 15,000 hosts inside, all serviced by netflow-capable equipment. If you use InterMapper Flows only at the border router, a capable modern desktop, dual-core CPU, 4GBytes of RAM would do nicely. However, if you do decide to turn on NetFlow on each of 1000 Cisco devices throughout the network, you might need to get some serious hardware, with 4 CPUs, several fast disks, and 16GB RAM. Another alternative would be to run multiple InterMapper Flows copies, each monitoring a subsection of the flow exporters.

Another factor is what the administrator wants to see. If they only want to see the last hour of data, then the desktop might do fine to monitor all 15,000 hosts. If they need the ability to go back in time regularly (to last week, or even last month) they would be looking at expensive, top-of-the-line server hardware.

I have a chart that shows a peak of traffic around 160 KB/sec. If I click and drag across that peak, the resulting zoomed chart peaks at around 210 KB/sec. Why does this happen if InterMapper Flows is collecting all the data and not aggregating, averaging, or taking time samples? Or is it?

An InterMapper Flows chart can conceptually be seen as a bunch of buckets. These buckets are filled with sessions from the database. Since routers report sessions as a sequence of (start time, stop time, data volume) data points, it is not possible to know exactly how the transfers were distributed over the timeframe of the sessions. For instance, a webpage typically has the bulk of the traffic at the beginning of the session (when you load the page), and then very little the rest of the session (as you're reading it). This information is lost at the router level, before it reaches InterMapper Flows.

To better understand what happens in the charts, imagine each chart only consisted of 10 buckets. A chart that shows 10 minutes would divide all traffic into 1 minute buckets. This means that a session with a short, high spike will be "smoothed" out into a single-minute chunk. This has the effect of flattening the spike out.

When you drag across the chart to zoom into that minute, InterMapper Flows does something no other software does. It divides the new timeframe into another 10 (actually it's about 200) new buckets, which are then conceptually only 6 seconds wide. The spike will now be much more pronounced, as it falls in a much shorter timeframe, and its true height is much more accurate.

What happens here actually occurs with any network monitoring product. The reason you don't detect this elsewhere is that you generally cannot zoom in to get a more accurate view.

I have a 15 minute view and then re-center the view (still 15 minutes). Why would the size/shape of a couple of the peaks change?

This is the same effect as the previous question. When you recenter, some larger flows will fall a little bit more in one bucket, and a little less in another. That can affect the peaks somewhat, although it shouldn't be very drastic.

When the InterMapper Flows server is restarting and reloading the sessions, the Flows Window displays the number of records loaded so far vs. the total number of sessions. Sometimes the first number is larger than the second. What's going on?

The NetSAW server estimates the number of flows it will load into its cache, based on the flowrate that's learned from the actual records in the DB. Since this estimate is never perfect, you'll sometimes notice that the actual number of records exceeds the estimated records. Other times it'll fall short and finish early.

What information is available for troubleshooting or debugging a problem with InterMapper Flows?

There are several sets of information available:

The Copy to Clipboard button of the About window collects a great deal of relevant information about the configuration of InterMapper Flows. You can paste this into an e-mail or a bug report (Help -> Report a bug...)
InterMapper can provide some debug information via the Telnet server. To do this, turn on the Telnet server in the InterMapper Settings. Then telnet to the InterMapper server and use the "flows" command to list the exporters that InterMapper knows about. Use the "ext" command to check that InterMapper has its own connection to the IMFlows server. You can copy/paste the output of these two commands into a bug report (Help -> Report a Bug...).

Does InterMapper Flows work on LAN links? On WAN Links?

Yes, InterMapper Flows will work on any link where there's an "exporter" (the router/switch) to keep track of the traffic statistics. Many kinds of Cisco equipment can export flow records that summarize the data flowing through that device.

How much bandwidth will NetFlow consume? How frequent is the traffic flow?

A quick answer is "not much". The switch/router summarizes the flow information, and typically will send an update about the flows it has seen every 60 or 120 seconds (this is configurable).

It is easy to set up your Cisco gear to send flow records, so you can see the effect on the traffic. We have written a brief document at this URL that describes the commands:

Enabling NetFlow on Cisco Equipment

Can IM Flows act as a collector at each location so that the central server can pull the data from each collector and correlate the same?

No. In our first release, all the flow records are must be sent to one InterMapper Flows machine (the "collector"). You can have multiple exporters sending flow records to the single collector, though.