Diagnose slow connections with Wireshark

Diagnose slow connections with Wireshark

[lbmn_commentscount]

SYN retransmission - slow connectionsLast week, I was working with a customer for a large Internet site (over 20 million users) who was having some performance problems with some of their internal infrastructure. The issue: slow connections to a HTTPS service. After buying some new super-duper big-iron servers, this customer (using SteelApp Traffic Manager) started to move services off the “old and busted” and onto the “new hotness” and immediately started seeing slow connections. And by slow, I mean 12 seconds slow. On “old and busted”, these HTTPS transactions were humming through at around 230-250 ms from Go to Whoa. Adding insult to injury, the slow connections weren’t occurring for every transaction – it was more like every 1 in 4 connections was slow. Now the customer swears that the new version of the software is the root cause of the issue, but I doubted that to be so. I organised with the customer to get a tcpdump from the Linux host that was running SteelApp Traffic Manager, and got the customer to recreate the issue in order to have some hard data to work with.

From the packet trace I could see that the sessions were all coming from a single IP address – it was unlikely that a routing issue further upstream was causing the issue. I started collecting statistics to see how many sessions were being affected, and the first thing I looked for is how many total sessions are in this trace using the wireshark filter (tcp.flags ==2) && (ip.dst == a.b.c.d) – in other words, show me all the SYN (or connection start) packets that are destined to the IP Address a.b.c.d. Instantly I can see the proof of the slow connections. The image at the left is a grab of the actual trace – you can see the initial SYN (or the first time the client attempted to connect), and the client will wait patiently for the next part of the “Three Way Handshake” which is a SYN-ACK. The client doesn’t see a SYN-ACK and so it sends another SYN 1.05 seconds later.. and another 1.10 seconds later, and so on until the server finally sends the SYN-ACK 11.72 seconds later.

Here was the proof I needed. Incoming connections are managed by the operating system, the network stack. It is only once the “Three Way Handshake” is completed that the connection is passed up the stack to the application. The version of the software has no relationship, in this case. The root cause is much lower than the network stack, waaaayy down in the kernel.

In this scenario, the root cause was the network stack running out of available connection space in the receive queue. The underlying Linux operating system had not been tuned for a larger number of incoming connections. The root fix for this issue was to tune the network parameters, specifically setting  net.core.somaxconn=1024 from its default setting of 128. After making this change, the performance issue was resolved.

[lbmn_postpagination]

[lbmn_authorbio]

About us and this blog

We are a digital marketing company with a focus on helping our customers achieve great results across several key areas.

Request a free quote

We offer professional SEO services that help websites increase their organic search score drastically in order to compete for the highest rankings even when it comes to highly competitive keywords.

Subscribe to our newsletter!

Is IoT The Next Big Thing?

There's been a lot of discussion about Cloud, be it Hybrid or…
Continue reading
get your head out of your ass

Communication is key

As a customer-focussed IT professional, communication is key to ensuring a happy…
Continue reading

Looking for a photographer?

The website www.photographers.com.au/ came across my feed recently. If you're in Australia…
Continue reading
Vote Flux

The dreaded phone call

Yesterday it happened. Not just once. Twice! I got the dreaded phone…
Continue reading
Vote Flux

A State of Flux

Three days ago, the United Kingdom voted to leave the European Union.…
Continue reading

Martin Place one year on

One year ago today, I was holed up in my office near…
Continue reading

Pride comes before a camera replacement

 There's a lot to be said for not trying to over-reach your…
Continue reading

Up a tree

There are old pilots. And there are bold pilots. But there are…
Continue reading
[lbmn_commentscount]
    • Anderson Sales
    • September 24, 2014

    Great article!!!

  1. Pingback: Diagnose slow connections with Wireshark | Keeping all the balls in the air

Comments are closed.

 

2 thoughts on “Diagnose slow connections with Wireshark

Comments are closed.

%d bloggers like this: