Theodore Baschak

BOFH. Open Source Guru. Founder/Operator of Hextet Systems, AS395089 and Network Architect for Daemon Defense Systems, AS55101.

Sleep Talking

Tue, 10 Feb 2015 11:10:17 -0600 » Security, IPv6, CLI, Networking, Network Monitoring, System Administration, Troubleshooting

This is a write up of an issue which I experienced on one of my networks. It is in response to When Good NICs Do Bad Things: A Blast of IPv6 Multicast Listener Discovery Queries posted on packetpushers.net.

The problem started early on a Friday morning – reports of dropped internal/external calls, as well as both LAN and internet outages. I wasn’t in the office when the problem was happening, so I didn’t get a packet capture of the event. Looking at my Observium graphs, it was soon obvious that some kind of Multicast packets were the culprit. This switch was seeing about 3300 packets/sec worth of Multicast packets.

Our network normally does not have a lot of Multicast traffic, we aren’t running IPv6 yet, and we don’t have a lot of Apple machines/devices on our network. I setup my Mac Pro desktop to capture all traffic using tcpdump, and waited for it to happen again. I ran:

tcpdump -ni en1 -s65535 -G 3600 -w 'trace_%Y-%m-%d_%H:%M:%S.pcap'

Near the end of the day I was rewarded with a giant capture file when the next report came in, about 2:20PM. I tore into the capture file with Wireshark and quickly discovered 3 machines were flooding the network with IPv6 Multicast Listener Reports.

Now that I finally had an idea what type of traffic we were seeing, I was able to search and while I didn’t find anyone else who had multicast traffic to ff02::1:ffbd:14b2 I discovered the above mentioned/linked blog which was exactly the same problem we were seeing. Machines with Intel I217-LM NICs would begin talking IPv6 when they were in S1 sleep, at fairly high PPS rates.

The fix of applying v19.0 (or v19.5, as this is the latest) on Monday seems to have resolved it – at least so far. Why the problem chose to show itself 8 months after the systems were installed, I don’t know, but this fix seems to have resolved it.

© Theodore Baschak - https://github.com/tbaschak - Powered by Jekyll.
Powered by Les.net.
CiscoDude.net is a personal website. Opinions expressed are not necessarily those of his employer.