If you have to debug at that level, and you're not designing hardware, things are really bad.<p>Some years back, Wes Irish at Xerox PARC tracked down one of the great mysteries of early coax Ethernet - why was throughput so much lower than theory predicted? For this, he got both ends of a building-length coax with many machines on it connected to one office, so he could plug both ends into a storage scope. If the waveforms disagreed, somebody was transmitting when they shouldn't. Storage scopes with large storage were rare then. It was an expensive LeCroy unit.<p>After the end of each Ethernet packet on coax, there is a brief "quiet time", and then the next packet can be sent, beginning with a sync pattern. The hardware detects if what it is sending does not match what it is receiving, which indicates a collision. Both senders stop, wait a random time so they don't collide again, and retry. This is how "carrier sense multiple access - collision detection", or CSMA-CD, works at the hardware level.<p>This setup revealed that something on the cable was transmitting a single spike after the end of each packet, during the "quiet time". That reset the "quiet time" timer in the network interface, which inhibited the transition to "look for sync" mode. So the next packet would be ignored.
The quiet time timer was at a very low level - software did not see this event.<p>What came out of looking at the waveforms was the surprising result that the spike during the quiet time was not coming from either the data source or the destination, but from something elsewhere on the cable. The spike was not synchronized to the packet just sent.
With the waveforms for both ends of the cable visible, speed of light lag revealed both that this was happening and where it was coming from, as distance along the cable.<p>It turned out that several brands of network interface used a part which contained the quiet time timer, the sync recognizer, and the transmitter power controller. When the timer ran out, the device did a state machine transition, and during that transition, for a nanosecond or so, the transmitter turned on. It wasn't supposed to do that. This generated a spike on the cable, resetting every quiet time timer and causing the next packet to be silently ignored by all stations.<p>The network interface didn't need to be active to do this. Being powered on was sufficient. One device with that part could halve the data rate on a coax Ethernet. Thousands of network interfaces had to be scrapped to fix this.