Be a better SysAdmin – know your applications

Be a better SysAdmin – know your applications

[lbmn_commentscount]

how to be a better sysadmin

It is a source of constant amazement to me that most SysAdmins (shorthand for System Administrators) have so little understanding of the applications running on their iron, apart from a passing “that’s the mail server”. Knowing exactly what your server is doing in normal operation makes it easier to troubleshoot when things aren’t “normal”.

Baselining

Everyone hates documenting system builds. It’s as much a truism as “the sky is blue”, “politicians always lie” and “whatever can go wrong will go wrong in the most spectacular way at the most inopportune moment”. Something as simple as a capture of what a server is doing just prior to deployment can make fire fighting much easier later on. There’s a bare minimum of information that I like to have on DropBox/Google Drive/Evernote for each server that I manage.

  • Hostnamenetstat-windows
  • DNS and LDAP/Active Directory domain names
  • DNS server
  • Authentication server (LDAP / Kerberos / Local)
  • Local administration username/password
  • Edited output of “netstat -ao” (Windows) or “netstat -ap” (Linux)
  • Edited output of “tasklist” (Windows) or “ps -ef” (Linux)

You’re probably wondering “Why is the output of netstat so helpful?”.  In one very concise text file, I can see:

  • what network ports are open, which gives me an idea of what applications are runningtasklist
  • what network addresses an application listens on
  • what hosts a server is talking to, which gives me an idea of the rest of the infrastructure, and what other dependencies might be at play
  • combined with the “tasklist” / “ps -ef” output, gives me the executable and process name

Adding the “-o” (Windows) or “-p” (Linux) gives one really useful piece of information – the Process ID (PID) of the application that is holding that network port open. Using “tasklist” (Windows), you can then match the network port+PID to the application, whereas the “netstat -ap” output already has the application name listed next to the PID.

Another really helpful tool for Windows servers is Process Explorer – it’s a more interactive way of drilling down into processes, ports open files, and a bunch of other useful bits that I may talk about in a future post.

Traffic Analysis

No, that is not a typo – if you run a server, you should know what the traffic going in and out looks like. A good SysAdmin should be able to:wireshark Example

  • understand (and be able to explain clearly) the TCP/IP addressing and subnetting in use on the network
  • capture packet traces using tools like “Wireshark” or tcpdump
  • filter a packet capture to narrow down a particular “stream” of traffic for analysis
  • read through and understand, at a basic level, the session setup, protocol and data communications, and session tear down

Before you come after me with torches and pitchforks, let me explain why this is a useful skill. At some point in time, any SysAdmin is going to hear “Why is the <insert application> server running so slow?”. Now an entry-level SysAdmin would take a look at the monitoring systems (you do have something monitoring your servers right?) or jump onto the server in question and checks all the basics such as CPU load, Memory usage, disk I/O, network I/O and then blame it on the NetAdmins and wash your hands of it. Until the NetAdmins dump that little burning coal right back into your lap.

But you’re not an entry-level SysAdmin so you take it to a “Whole Nutha Level” and start a packet capture on the <insert application> server AND on a client system that is trying to access the <insert application> server. After a little more digging, your Wireshark analysis skills shows that the NIC (Network Interface Card) is showing a high number of retransmissions occurring, even though the NIC is nowhere near capacity. Send that burning coal to the NetAdmins, with the packet capture attached, and you’ve now shown that a) you’ve actually got some useful hard data to back up your hypothesis, and b) put the NetAdmins on the back foot before they can come back with “It’s a server issue.”

In all seriousness, the judicious use of Wireshark can help nail down those tricky application performance issues by detecting network congestion, faulty NICs, faulty or misconfigured network switch ports, dodgy cabling, or latency issues.

The following YouTube video is a great starting point on how to setup your Wireshark environment. For full disclosure, I work for Riverbed Technology, and this video was posted by Riverbed Technology.

I’d also like to point you at another blog called PacketBomb – it’s written by a colleague of mine who REALLY knows how to fly Wireshark. Drop by and say “Hi!” to Kary.

If you’re feeling a little out of your depth with the addressing, subnetting and other arcana of TCP/IP networking, please take a look at the links below:

As always, I’d love to hear from you about:

  • how do you document your systems?
  • other online resources for network analysis


[lbmn_postpagination]

[lbmn_authorbio]

About us and this blog

We are a digital marketing company with a focus on helping our customers achieve great results across several key areas.

Request a free quote

We offer professional SEO services that help websites increase their organic search score drastically in order to compete for the highest rankings even when it comes to highly competitive keywords.

Subscribe to our newsletter!

Is IoT The Next Big Thing?

There's been a lot of discussion about Cloud, be it Hybrid or…
Continue reading
get your head out of your ass

Communication is key

As a customer-focussed IT professional, communication is key to ensuring a happy…
Continue reading

Looking for a photographer?

The website www.photographers.com.au/ came across my feed recently. If you're in Australia…
Continue reading
Vote Flux

The dreaded phone call

Yesterday it happened. Not just once. Twice! I got the dreaded phone…
Continue reading
Vote Flux

A State of Flux

Three days ago, the United Kingdom voted to leave the European Union.…
Continue reading

Martin Place one year on

One year ago today, I was holed up in my office near…
Continue reading

Pride comes before a camera replacement

 There's a lot to be said for not trying to over-reach your…
Continue reading

Up a tree

There are old pilots. And there are bold pilots. But there are…
Continue reading
[lbmn_commentscount]
    • Dave
    • June 20, 2014

    speaking on behalf of NetAdmins everywhere, Amen..

    In my experience having a known working baseline is vital to any investigation.

    Additional documentation you should also collect is regarding other applications/services you’re server is reliant on.

    I’ve witnessed multiple catastrophic issues, caused by a slow/non responsive DNS server, all blamed on “Network Issues” That detailed documentation and thorough investigation would have stopped.

    Whilst I’m up here on the soap box, providing detailed analysis to your IT colleagues is a non negotiable. Don’t make them guess what you’ve investigated. Make it clear.

  1. Pingback: Be a better SysAdmin – know your applications | Keeping all the balls in the air

Comments are closed.

 

2 thoughts on “Be a better SysAdmin – know your applications

  1. speaking on behalf of NetAdmins everywhere, Amen..

    In my experience having a known working baseline is vital to any investigation.

    Additional documentation you should also collect is regarding other applications/services you’re server is reliant on.

    I’ve witnessed multiple catastrophic issues, caused by a slow/non responsive DNS server, all blamed on “Network Issues” That detailed documentation and thorough investigation would have stopped.

    Whilst I’m up here on the soap box, providing detailed analysis to your IT colleagues is a non negotiable. Don’t make them guess what you’ve investigated. Make it clear.

Comments are closed.

%d bloggers like this: