Network Troubleshooting Tools

by Joseph D. Sloan - O'Reilly Press

Packet Capture -- Part 1

Packet capture and analysis is the most powerful technique that will be discussed in this book -- it is the ultimate troubleshooting tool. If you really want to know what is happening on your network, you will need to capture traffic. No other tool provides more information.

On the other hand, no other tool requires the same degree of sophistication to use. If misused, it can compromise your system's security and invade the privacy of your users. Of the software described in this book, packet capture software is the most difficult to use to its full potential and requires a thorough understanding of the underlying protocols to be used effectively. As noted in , you must ensure that what you do conforms to your organization's policies and any applicable laws. You should also be aware of the ethical implications of your actions.

This chapter begins with a discussion of the type of tools available and various issues involved in traffic capture. Next I describe tcpdump, a ubiquitous and powerful packet capture tool. This is followed by a brief description of other closely related tools. Next is a discussion of ethereal, a powerful protocol analyzer that is rapidly gaining popularity. Next I describe some of the problems created by traffic capture. The chapter concludes with a discussion of packet capture tools available for use with Microsoft Windows platforms.

1.1. Traffic Capture Tools

Packet capture is the real-time collection of data as it travels over networks. Tools for the capture and analysis of traffic go by a number of names including packet sniffers, packet analyzers, protocol analyzers, and even traffic monitors. Although there is some inconsistency in how these terms are used, the primary difference is in how much analysis or interpretation is provided after a packet is captured. Packet sniffers generally do the least amount of analysis, while protocol analyzers provide the greatest level of interpretation. Packet analyzers typically lie somewhere in between. All have the capture of raw data as a core function. Traffic monitors typically are more concerned with collecting statistical information, but many support the capture of raw data. Any of these may be augmented with additional functions such as graphing utilities and traffic generators. This chapter describes tcpdump, a packet sniffer, several analysis tools, and ethereal, a protocol analyzer.

While packet capture might seem like a low-level tool, it can also be used to examine what is happening at higher levels, including the application level, because of the way data is encapsulated. Since application data is encapsulated in a generally transparent way by the lower levels of the protocol stack, the data is basically intact when examined at a lower level.[1] By examining network traffic, we can examine the data generated at the higher levels. (In general, however, it is usually much easier to debug an application using a tool designed for that application. Tools specific to several application-level protocols are described in .)

[1]There are two obvious exceptions. The data may be encrypted, or the data may be fragmented among multiple packets.

Packet capture programs also require the most technical expertise of any program we will examine. A thorough understanding of the underlying protocol is often required to interpret the results. For this reason alone, packet capture is a tool that you want to become familiar with well before you need it. When you are having problems, it will also be helpful to have comparison systems so you can observe normal behavior. The time to learn how your system works is before you have problems. This technique cannot be stressed enough -- do a baseline run for your network periodically and analyze it closely so you know what traffic you expect to see on your network before you have problems.

1.2. Access to Traffic

You can capture traffic only on a link that you have access to. If you can't get traffic to an interface, you can't capture it with that interface. While this might seem obvious, it may be surprisingly difficult to get access to some links on your network. On some networks, this won't be a problem. For example, 10Base2 and 10Base5 networks have shared media, at least between bridges and switches. Computers connected to a hub are effectively on a shared medium, and the traffic is exposed. But on other systems, watch out!

Clearly, if you are trying to capture traffic from a host on one network, it will never see the local traffic on a different network. But the problem doesn't stop there. Some networking devices, such as bridges and switches, are designed to contain traffic so that it is seen only by parts of the local network. On a switched network, only a limited amount of traffic will normally be seen at any interface.[2] Traffic will be limited to traffic to or from the host or to multicast and broadcast traffic. If this includes the traffic you are interested in, so much the better. But if you are looking at general network traffic, you will use other approaches.

[2]This assumes the switches have been running long enough to have a reasonably complete address table. Most switches forward traffic onto all ports if the destination address is unknown. So when they are first turned on, switches look remarkably like hubs.

Not being able to capture data on an interface has both positive and negative ramifications. The primary benefit is that it is possible to control access to traffic with an appropriate network design. By segmenting your network, you can limit access to data, improving security and enhancing privacy.

Lack of access to data can become a serious problem, however, when you must capture that traffic. There are several basic approaches to overcome this problem. First, you can try to physically go to the traffic by using a portable computer to collect the data. This has the obvious disadvantage of requiring that you travel to the site. This may not be desirable or possible. For example, if you are addressing a security problem, it may not be feasible to monitor at the source of the suspected attack without revealing what you are doing. If you need to collect data at multiple points simultaneously, being at different places at the same time is clearly not possible by yourself.

nother approach is to have multiple probe computers located throughout your network. For example, if you have computers on your network that you can reach using telnet, ssh, X Window software, or vnc, you can install the appropriate software on each. Some software has been designed with remote probing in mind. For example, Microsoft's netmon supports the use of a Windows platform as a probe for collecting traffic. Data from the agents on these machines can be collected by a central management station. Some RMON probes will also do this. (vnc and ssh are described in . netmon is briefly described later in this chapter, and RMON is described in .)

When dealing with switches, there are two common approaches you can take. (Several other techniques that I can't recommend are described later in this chapter.) One approach is to augment the switch with a spare hub. Attach the hub to the switch and move from the switch to the hub only the connections that need to be examined. You could try replacing the switch with a hub, but this can be disruptive and, since a hub inherently has a lower capacity, you may have more traffic than the hub can handle. Augmenting the switch with a hub is a better solution.

Buying a small portable hub to use in establishing a probe point into your network is certainly worth the expense. Because you will be connecting a hub to a switch, you will be using both crossover and patch cables. Be sure you work out the details of the cabling well before you have to try this approach on a problematic network. Alternately, there are several commercially available devices designed specifically for patching into networks. These devices include monitoring switches, fiber splitters, and devices designed to patch into 100- Mbps links or links with special protocols. If your hardware dictates such a need, these devices are worth looking into.

TIP
Here is a riddle for you -- when is a hub not a hub? In recent years, the distinction between hubs and switches has become blurred. For example, a 10/100 autoswitching hub may be implemented, internally, as a 10-Mbps hub and a 100- Mbps hub connected by a dual-port switch. With such a device, you may not be able to see all the traffic. In the next few years, true hubs may disappear from the market. You may want to keep this in mind when looking for a hub for traffic monitoring.

A second possibility with some switches is to duplicate the traffic from one port onto another port. If your switch supports this, it can be reconfigured dynamically to copy traffic to a monitoring port. Other ports continue functioning normally so the monitoring appears transparent to the rest of the switch's operation. This technique is known by a variety of names. With Bay Network products, this is known as conversation steering. Cisco refers to this as monitoring or using a spanning port. Other names include port aliasing and port mirroring.

Unfortunately, many switches either don't support this behavior or place limitations on what can be done. For instance, some switches will allow traffic to be redirected only to a high- speed port. Implementation details determining exactly what can be examined vary greatly. Another problem is that some types of errors will be filtered by the switch, concealing possible problems. For example, if there are any framing errors, these will typically be discarded rather than forwarded. Normally, discarding these packets is exactly what you want the switch to do, just not in this context. You'll have to consult the documentation with your switch to see what is possible.

1.3. Capturing Data

Packet capture may be done by software running on a networked host or by hardware/software combinations designed specifically for that purpose. Devices designed specifically for capturing traffic often have high-performance interfaces that can capture large amounts of data without loss. These devices will also capture frames with framing errors -- frames that are often silently discarded with more conventional interfaces. More conventional interfaces may not be able to keep up with high traffic levels so packets will be lost. Programs like tcpdump give summary statistics, reporting the number of packets lost. On moderately loaded networks, however, losing packets should not be a problem. If dropping packets becomes a problem, you will need to consider faster hardware or, better yet, segmenting your network.

Packet capture software works by placing the network interface in promiscuous mode.[3] In normal operations, the network interface captures and passes on to the protocol stack only those packets with the interface's unicast address, packets sent to a multicast address that matches a configured address for the interface, or broadcast packets. In promiscuous mode, all packets are captured regardless of their destination address.

[3]On a few systems you may need to manually place the interface in promiscuous mode with the ifconfig command before running the packet capture software.

While the vast majority of interfaces can be placed in promiscuous mode, a few are manufactured not to allow this. If in doubt, consult the documentation for your interface. Additionally, on Unix systems, the operating system software must be configured to allow promiscuous mode. Typically, placing an interface in promiscuous mode requires root privileges.

Next chapter