Diagnose slow connections with Wireshark

SYN retransmission - slow connectionsLast week, I was working with a customer for a large Internet site (over 20 million users) who was having some performance problems with some of their internal infrastructure. The issue: slow connections to a HTTPS service. After buying some new super-duper big-iron servers, this customer (using SteelApp Traffic Manager) started to move services off the “old and busted” and onto the “new hotness” and immediately started seeing slow connections. And by slow, I mean 12 seconds slow. On “old and busted”, these HTTPS transactions were humming through at around 230-250 ms from Go to Whoa. Adding insult to injury, the slow connections weren’t occurring for every transaction – it was more like every 1 in 4 connections was slow. Now the customer swears that the new version of the software is the root cause of the issue, but I doubted that to be so. I organised with the customer to get a tcpdump from the Linux host that was running SteelApp Traffic Manager, and got the customer to recreate the issue in order to have some hard data to work with.

From the packet trace I could see that the sessions were all coming from a single IP address – it was unlikely that a routing issue further upstream was causing the issue. I started collecting statistics to see how many sessions were being affected, and the first thing I looked for is how many total sessions are in this trace using the wireshark filter (tcp.flags ==2) && (ip.dst == a.b.c.d) - in other words, show me all the SYN (or connection start) packets that are destined to the IP Address a.b.c.d. Instantly I can see the proof of the slow connections. The image at the left is a grab of the actual trace – you can see the initial SYN (or the first time the client attempted to connect), and the client will wait patiently for the next part of the “Three Way Handshake” which is a SYN-ACK. The client doesn’t see a SYN-ACK and so it sends another SYN 1.05 seconds later.. and another 1.10 seconds later, and so on until the server finally sends the SYN-ACK 11.72 seconds later.

Here was the proof I needed. Incoming connections are managed by the operating system, the network stack. It is only once the “Three Way Handshake” is completed that the connection is passed up the stack to the application. The version of the software has no relationship, in this case. The root cause is much lower than the network stack, waaaayy down in the kernel.

In this scenario, the root cause was the network stack running out of available connection space in the receive queue. The underlying Linux operating system had not been tuned for a larger number of incoming connections. The root fix for this issue was to tune the network parameters, specifically setting  net.core.somaxconn=1024 from its default setting of 128. After making this change, the performance issue was resolved.

 

Beauty and the Beast – Looking back

The Rose - Beauty and the BeastIt’s almost one month since the closing night of Packemin Productions “Beauty and the Beast“. Just over 3 months went into the actual rehearsal and production phases of “Beauty and the Beast”, and it’s been one of most memorable experiences of my life.

I’m no stranger to large scale productions, having been part of Hillsong Church Christmas productions for many years – either as part of the vocals team or choir, or one of the acting talent.  But “Beauty” was my first `theatre’ production, and I wanted to learn as much as I could from as many people as I could.

The first thing that I noticed was the difference between having a completed script and music, as compared to being in the midst of a work in progress. Arrangements were already dotted, parts already allocated, lines already written and all of it to be adhered to. This was backed up with a musical director who knew exactly what he wanted before hitting the rehearsal space, alongside a choreographer who also knew exactly what she wanted, and coordinated by a director and production team who also knew what they wanted. And that’s the benefit of a pre-written show – the framework is already there – it’s` just‘ a matter of putting the muscle, tendons, sinews and skin over the top of it. Added into this mix is the cast, all of whom have auditioned with a solo vocal piece and a group dance audition, and have beaten out another 250-ish hopefuls for their place, and you have the formula for a core of people who want to be there.

Packemin Productions have done a remarkable job at creating, and maintaining, the culture of the `Packemin family’. As a newbie, I was a little nervous about how I was going to integrate, and find my feet. From the very first rehearsal, I felt included and welcomed and, dare I say it, needed by the rest of the cast. Rehearsals quickly became my ‘other’ family – a time of working with talented people, sharing stories from our respective lives. Laughing together as we developed our characters. And shedding tears together, when one of the cast/crew passed away not long into the rehearsal process, and then seeing all the production team and cast wearing and walking through their grief and loss. The sense of family and community was definitely assisted by the private Facebook groups allowing sharing of comments, jokes, requests and chit-chat even outside of the rehearsal rooms.

I had the privilege of being in the “Main male cast” dressing room, and the pleasure of assisting Scott Irwin (Beast) and Adam Scicluna (Cogsworth) in their costume changes. While “what happens in the dressing room stays in the dressing room“, I will say that Scott and Adam are two of the most genuine, humble and down-to-earth people I’ve had the honour of working alongside, freely willing to encourage and pass-on their hard-won experience from the “School of Hard Knocks”. In addition to Scott and Adam, I have learnt so much from all these guys – Luke Lamond, Patrick Lee, Jim Mitchell, Levi Gardner, Andrew Tucker, Michael Johnson, Boshko Maksimovic, Mark Power, Danny Folpp, and David Collins. Oh, and Kamahl – I learnt lots from you too.

On the Friday night before we closed on Saturday, I was warned by Luke Lamond to expect the onset of “PSD” or Post-show depression. I knew what PSD felt like – I just didn’t know it had a name, let alone a definition. From the Urban Dictionary: Post Show Depression “The feeling after a musical is over and you realize you have no life. After putting months into making a show perfect it is all over. It is a feeling of emptiness and sadness. Usually during the finale is when this begins. It can continue from then till weeks or months after the shows finished. You get little pangs when you see something that reminds you of the musical or when your sitting at home on a night when you would usually be performing or rehearsing. This depression is generally shared by most of the cast and the last show and cast party generally involves a lot of hugging, crying and beautiful parting words. When you run into one of your show family after this depression it usually involves a lot of hugging and crying and comes back for another couple of days. The only way to fully recover is to go head on into another project and remember all the good memories. It is a bittersweet feeling. It is only really understood by fellow theatre dorks.”

I came down pretty hard with PSD, event though I knew it was on it’s way. Maybe because I’d worked so long on this production. Maybe because it was my first theatre production. Maybe because of all the wonderful people I’d met. I know that I’ve loved every moment being part of Beauty and the Beast, and that I’ve been well and truly bitten by the theatre bug. So while this `look back’ has been a little emotional, it’s my closure, my farewell to a friend, the filing of the chart music for another day.

So where to from here? Get ready for the audition for “Phantom of the Opera” *smile*

This slideshow requires JavaScript.

 

 

Transformers star in handcuffs

Transformers star escorted out in handcuffs

What is the go with this guy? A few years ago, Shia LaBeouf has Hollywood at his feet, having just starred in a couple of the Transformers movies, and looked to be really becoming the go-to actor for the young yet mature, innocent yet strong male roles. And then Shia kicks in “I’m a Diva!” mode, pulling out of stage roles and going on a social media rampage saying he was ‘retiring from public life because of attacks against his artistic integrity‘. And then he follows it up with what appears to be another diva-esque performance at a performance of CABARET in New York, and has to be escorted out of the theatre by handcuffs by the New York Police Department.

Seriously Shia, take a page out of the Transformers script, and transform yourself out of the Diva, and back into a real person. Lose the victim mentality about your “artistic integrity“, and maybe take the view that your fame and notoriety could be used for something other than making you look and sound like a little brat. Get your eyes off yourself, and have a look around at the world around you. Pick a cause, and add your weight to it, and maybe.. just maybe.. you’ll regain some credibility in the eyes of others.

Reference: Broadway World – Breaking News Shia LaBeouf Escorted Out of CABARET in Handcuffs 

 

Beauty and the Beast – three weeks to go

Packemin Productions Beauty and the Beast - Rouse Hill TimesA few months ago I posted that I had been accepted into the Ensemble cast for the Packemin Productions “Beauty and the Beast”. Since the beginning of March, each week has consisted of rehearsals on Monday and Wednesday (with the occasional cancellation due to scheduling conflicts and the like). After meeting so many new people, learning lines and choreography, it’s a little startling to find that the production is just over three weeks away to opening night.

Being involved in a professional production has opened my eyes to a lot of things. Ways to improve rehearsals and retention/practice of songs and choreography, promotions and the mechanics of getting ticket sales, the differences between “building and creating a show” as opposed to crafting and tweaking an “existing show”. As you can see from the thumbnail image, I was invited to be part of the promotional shots for the Rouse Hill area, with the article appearing in the Rouse Hill Times on the 4th June, and even that process was an interesting insight into how all of that “front of stage” process works as well.

Continue reading…

 

Be a better SysAdmin – know your applications

how to be a better sysadmin

It is a source of constant amazement to me that most SysAdmins (shorthand for System Administrators) have so little understanding of the applications running on their iron, apart from a passing “that’s the mail server”. Knowing exactly what your server is doing in normal operation makes it easier to troubleshoot when things aren’t “normal”.

Baselining

Everyone hates documenting system builds. It’s as much a truism as “the sky is blue”, “politicians always lie” and “whatever can go wrong will go wrong in the most spectacular way at the most inopportune moment”. Something as simple as a capture of what a server is doing just prior to deployment can make fire fighting much easier later on. There’s a bare minimum of information that I like to have on DropBox/Google Drive/Evernote for each server that I manage.

  • Hostnamenetstat-windows
  • DNS and LDAP/Active Directory domain names
  • DNS server
  • Authentication server (LDAP / Kerberos / Local)
  • Local administration username/password
  • Edited output of “netstat -ao” (Windows) or “netstat -ap” (Linux)
  • Edited output of “tasklist” (Windows) or “ps -ef” (Linux)

Continue reading…

 

A Day in Support – The King had a dream

IT Support Frustration

Frustration

There are certain times in my work life when I feel like I am beating my head against a wall in frustration. This is the story of one such time.

Many times, in the IT Support arena, you get to see products used in ways that defy logic. In ways that make you want to find the sales droid involved in making the sale and shake his hand for the barefaced audacity they have shown in getting that sale over the line. In ways that make you want to find the architect and slap them sideways for being ridiculous.

To be fair, I understand that sometimes these deployments are “thin edge of wedge” deployments – just get the product in the door, get it used in production, and then Sales can go back and sell more product, or expand the use of the product’s existing feature set. I get that.

But there are certain types of customers that make me want to (in no particular order of preference: Continue reading…

 

Regularly do something that scares you

Beauty and The Beast - something that scares youA few weeks ago, I did something that scares me. It was something that pushed me to make a choice to step through the barrier of fear. I auditioned for a role in a Pro-Amateur production of “Beauty and The Beast“.

I learned that courage was not the absence of fear, but the triumph over it. The brave man is not he who does not feel afraid, but he who conquers that fear. – Nelson Mandela

In the two weeks between being allocated an audition time and doing the actual audition, I wish I could say that I was unwavering, wholly committed to going through with the audition. I would love to say that I didn’t suffer from self-doubt, and second-guess my every decision.

That wasn’t the case.

Continue reading…

 

MH370 – how stupid is the media

mh370 media stupidity and identity safetyIf you’ve seen any news over the weekend, you will have seen that Malaysian Airlines flight MH370 disappeared off radar screens less than one hour after takeoff. The Internet has been awash with conspiracy theories about the causes; terrorist bombing, the new Bermuda triangle, spontaneous decompression, the long-term results of a ground-based collision two years ago.

The investigation and search process is still under-way, and the focus now is that at least two people boarded MH370 using stolen passports. Click through to this news piece, and fast forward in to 26 seconds of the video clip; better still click on the image to the right. How much more information do you need to forge a passport? How many times does this guy’s name need to show up in Interpol’s database of “Lost and Stolen Travel Documents” before he ends up on the “Do Not Fly With This Guy” ™ list?

Identity theft is a very real issue is this day and age. And our news outlets spruik about the dangers of identity theft on one hand, and then openly publicize private data on the other. I love consistency.. and that ain’t it.

 

Use OpenDNS to reduce malware and virus infection

Use OpenDNS to reduce malware and virus infections

I have been a long-time user of OpenDNS for my home network, and have recommend it to my friends and family. The “Home Parental Controls” allows me to easily add another layer of protection to my home network, not only to setup content filtering but also to reduce the likelihood of malware and virus infections. I must say, right up front that I have no affiliation, financial or otherwise, with OpenDNS other than a satisfied user. Your mileage may vary.

OpenDNS is just one of the tools that I use to enforce a `perimeter’ around my home network. It is just one of the many layers that I have setup to defend my network. OpenDNS is so easy to setup that I personally think it is bordering on negligent if you don’t use it.

Continue reading…