Troubleshooting Clock Issues

Troubleshooting Clock Issues

About the Clock

Pops and clicks in Audio Over IP systems are almost always due to clock problems. In WheatNet-IP, our internal clock is called the "metronome." It's generated by whichever blade is currently set as the Clock Master blade. That's why this setting is ultra-important to your system. See Understanding Route and Clock Masters in WheatNet-IP for info on setting clock master and route master priorities for blades.

The WheatNet-IP metronome is sent by the clock master blade to all blades and drivers in your system. It keeps audio in sync when everything in your network is configured and working properly. The clock master sends out exactly two hundred metronome packets per second (that's one every 5 milliseconds). They are UDP packets and are 64 bytes in length. They are sent to every device that has joined the IGMP group for clock, which includes all WheatNet-IP devices. The multicast address is always 239.192.0.0 and the port used is always 51000.

Look at the E2 Logs

If you are experiencing pops and clicks in audio, there is a good chance that your blade is missing some of the clock packets that should be sent to it. You can get a good idea that this is happening by looking at the Wheatstone system's E2 logs:


Note that due to the nature of ethernet, an occasional lost metronome packet is to be expected and almost never results in a pop, click or other audio artifact. But when you see something like in the above screenshot, that's indicative of a problem.

Using Wireshark to Diagnose Clock Issues

The best way to confirm what's happening is to use Wireshark. See Using Wireshark to Diagnose AoIP Issues for some guidance on how to set up and use Wireshark.

You'll want to capture on the port that the blade which has the audio pop/click problem is connected to on your ethernet switch. Set up a SPAN (mirror) port and plug your Wireshark computer into that port. Make sure to select the proper NIC on your computer (if it has more than one) and be sure to turn off WiFi while doing the capture.

You only need to capture, say, 10,000 packets (or just a few seconds) to see if the clock is being received consistently by that blade. Don't use any capture filters. Set up the following display filter after you stop the capture:
  1. udp.srcport == 51000
By using that filter, it becomes easy to determine how many clock ticks per second your blade is receiving. You can either highlight all the packets for one second and look at the bottom of the Wireshark window to see how many you have highlighted:



Or, perhaps better, you can use Statistics | I/O Graphs:


You can see that as long as this capture was running, there is a straight line at 200 packets/second, which is what we want. The dropoff shown on the graph just shows that the capture was stopped after 1 second.

Here is a real-world example from a customer who was having some pretty serious audio artifacts with audio coming from one of their M4usb blades. Below is a chart of their metronome packets created by using the Statistics | I/O Graphs menu item in Wireshark:


We can see from this that most seconds during this capture had missing metronomes, some missing more than 70! Of course this is going to result in bad audio.

This customer moved the problematic blade to another switch and the problem was resolved; indicating that the switch was either bad, misconfigured, or just needed a reboot.

Basic Troubleshooting Technique

There are many ways to use Wireshark to help troubleshoot your audio system. If you have audio with a pop/click problem, start at the blade where the audio originates. Confirm that the blade itself isn't creating bad audio due to a software or hardware issue by routing the audio in question to that blade's headphone jack (or an analog or digital blade output on that same blade) and listening to it there. If it's good, then you are starting with good audio. The problem must be occurring when the audio gets onto the network.

You can strategically select other blades in the path of that audio signal and check each for missing metronomes. If you find that one or more blades connected to a particular ethernet switch have problems, but other blades in the system are able to reproduce the audio signal in question without pops/clicks, check that switch:
  1. Look for errors on the port the blade in question is connected to
    1. CRC errors nearly always indicate a bad cable
    2. If you find errors on the port, make an attempt to remedy the situation. and clear the error counters; then monitor until you're satisfied that the problem is resolved
  2. Misconfiguration of the port in question
    1. If you find a misconfiguration (search our knowledgebase for "Cisco" for information on how the switch you're using should be configured), make changes to the config as necessary. No switch reboot is required for changing settings on a port
  3. Maybe the switch needs a reboot
    1. Schedule a time when you can reboot the offending switch. Note that on Cisco switches such as 2960, 3850 and others of that generation, a switch reboot can take as long as ten minute
If your analysis tells you that the metronome is correct but you are still having audio problems, proceed thusly:
  1. Check your Wireshark methodology again to be sure you're monitoring the right port, using the right NIC on your Wireshark computer and that you're actually monitoring the blade that has problematic audio. Pops and clicks are nearly always a clock issue and you should be seeing that in your packet analysis
  2. If you're convinced that it's not a clock issue, and the originating audio is good, check with Wheatstone Tech Support for further troubleshooting assistance.

    • Related Articles

    • Using Wireshark to Diagnose AoIP Issues

      Back in the old days, you could find any signal in a radio/TV station with a Telco Butt-set. Now that everything is IP, the go-to tool is Wireshark. If you don't have it already, download here. Simple Troubleshooting Sometimes we can use the ...

    • WheatNet-IP Micro Blades Cannot Become Clock Master

      The WheatNet-IP Micro Blade, which is the WheatNet-IP card built into the Tieline Gateway, Eventide BD600W+, and Wheatstone Stagebox, is designed to only follow the WheatNet-IP metronome generated by the Clock Master Blade. It is not capable of ...

    • Wheatnet-IP Blade Troubleshooting: A field service guide

      The Wheatnet-IP blade system is the backbone of our IP infrastructure and the neural network of our operational systems. To date of this article, we have released 4 blade models, each one more advanced than it's predecessor. The purpose of this ...

    • Understanding Route and Clock Masters in WheatNet-IP

      Every WheatNet-IP system has a Route Master Blade and a Clock Master Blade. These Blades are automatically elected within your system. Depending upon how large your facility is, you may want to set certain Blades to be the preferred ones for your ...

    • How To Install The WNIP Audio Driver

      Installing The Audio Driver Before you begin, connect the PC/Server to the WNIP network and set a fixed IP address on the NIC you are using within the WNIP system’s subnet. The default WNIP subnet is 192.168.87.0; your system may be configured ...