Any wireshark nerds in here?

[ScottJ]

So, I've got a situation where a customer is complaining of occasional packet loss on a network segment (which is transported on our layer-2 network), and I had thought this was due to possible over subscription of their service rate, which we have encountered before on this platform w/ other customers.

I was wondering if I connected a box w/ wire-shark, parked it on the same vlan as the customer, that perhaps it would capture pause frames (either tx or rx) to give me some indication that an end station is telling the other side to slow down.

There is another app called sflow which we haven't incorporated yet, which I gather does a pretty good job of reporting traffic intel, but thought I'd tap you guys on the shoulder for input!



--Scott

[roliath]

So, I've got a situation where a customer is complaining of occasional packet loss on a network segment (which is transported on our layer-2 network), and I had thought this was due to possible over subscription of their service rate, which we have encountered before on this platform w/ other customers.

I was wondering if I connected a box w/ wire-shark, parked it on the same vlan as the customer, that perhaps it would capture pause frames (either tx or rx) to give me some indication that an end station is telling the other side to slow down.

There is another app called sflow which we haven't incorporated yet, which I gather does a pretty good job of reporting traffic intel, but thought I'd tap you guys on the shoulder for input!



--Scott

I'll have to find out how we deploy headless laptops in the DC to monitor traffic. I know they're running on ubuntu, but I don't deal with taht side of the network so I don't really pay much attention. I'll ask tomorrow.

[ceyko]

So that'll likely be a fuck ton of data, unless you know specific src/dst addresses and protocols that you can filter by. With TCP you should see retransmits and such - but without more details it is hard to provide better details.

How did the customer determine packet loss?

[ScottJ]

that's the other hard part, their IP src/dst is their world and we don't know any of it (I guess we could ask them for that) to filter on certain flows.

They are using some flavor of performance assurance agent software (like Accedian or similar) to send test data end-to-end, so that they are always aware of available bandwidth, packet loss, latency, other performance data. They say this only happens occasionally, IMO I think it must be some bursty traffic that (briefly) over subscribes their service rate.

[ceyko]

That is likely. Are the Cisco devices or some other device that supports netflow? If so, do they have any NMS that can correlate the data to review later? Granted, netflow and its variants won't indicate a network failure type situation. However, if they have SNMP running and monitor all the interfaces and netflow - you'll get a better snapshot of what's going on during the reported time.

As you may know, they may say "this morning" or "it happened around 8" which the translation is anytime from midnight until 1500 or so.

[ScottJ]

We are asking them about things like netflow, and also more specifics on how their traffic happens.

Its actually a hybrid of sorts, which doesn't make troubleshooting it any easier :/

{CPE} U.S.A - 1000BaseLX - [Cisco 15454 ADM] -<STS-1 sized EoS circuit via OC-48>- [Cisco 15454 ADM] - 1000BaseLX - [client vlan access Force10 S4810 vlan trunk] - 1000BaseLX -<STM4 sized EoSDH via STM16>- 1000BaseLX - [vlan trunk Arista 7124SX client vlan access] - 1000BaseLX - {CPE} Brazil

So...w/o applying rate policing on the Force10 or Arista switches, the circuit is essentially constrained to the STS-1 sized EoS circuit (aka DS3 rate or 45Mbps).

We want the client to rate police their egress traffic before it touches our network, so any throttling from bursty traffic on their network, stays in their network.

[ceyko]

I've not really used that device or installed it a lot. More of a optical/SONET device and I'm GUESSING netflow will be a no go.

Police or shape? As I understand it, for critical/internal type traffic, shaping is more desirable. i.e. for WAN traffic for companies, we generally try to do traffic shaping. For customers that are more like an ISP, we'll setup policing.

The following diagram illustrates the key difference. Traffic policing propagates bursts. When the traffic rate reaches the configured maximum rate, excess traffic is dropped (or remarked). The result is an output rate that appears as a saw-tooth with crests and troughs. In contrast to policing, traffic shaping retains excess packets in a queue and then schedules the excess for later transmission over increments of time. The result of traffic shaping is a smoothed packet output rate.

http://www.cisco.com/en/US/tech/tk543/tk545/technologies_tech_note09186a00800a3a25.shtml#policingvsshaping

Either way for you the end result is the same. However, the customer may have better results on their end - most likely.

[ScottJ]

Thanks for the info ceyko I really appreciate it! We've got several other customers on this platform w/ no issues so we feel pretty confident the customer should be able to sort it out.