The Linux Setup – Aaron Toponce, Director of System Administration

I first ran into Aaron via his essay about leaving Ubuntu. Aaron’s obviously a thoughtful technologist and that thoughtfulness comes through in his description of his setup, which is fantastic. His IRC workflow alone could almost be the subject of a book. And I love that he works in an office environment that prohibits Windows. That’s the dream for a lot of us (especially those of us who have spent some time with Windows 8).

You can find more of The Linux Setup here.

You can follow Linux Rig on Google+ here and follow me on Twitter here.

  1. Who are you, and what do you do?

    My name is Aaron Toponce. I am the Director of System Administration for a Utah based ISP named XMission. I am in charge of a couple of hundred servers, in varying degrees: mail servers, web servers, shared hosting, hypervisors, storage, etc. I even act as a failover network engineer for smallish problems. Just about everything we run is Ubuntu LTS on the server, and workstations must either be a GNU/Linux operating system or Mac OS X. Windows is too vulnerable to trust on our office network.

    Pete Ashdown is my boss. You’ve probably heard of him. He has been getting a lot of press lately standing up to the NSA, not allowing them access to the infrastructure at all. It’s awesome working for him, and working for such a great company.

  2. Why do you use Linux?

    I started using GNU/Linux back in 1999, shortly after I got married. We were in need of a new computer, and being poor newly weds, I didn’t have the money to fork over for a brand new computer. So, I purchased a used one, but it had Windows 95 on it. My wife was a big Mac OS user (pre- OS X even), and I didn’t care for the Windows 95 interface. So, I was interested in something new.

    While at an electronics company, and browsing the bundled software aisle, I saw a box for Red Hat Linux 6, for $40. I had heard of Linux before, and knew it was an operating system, or something, so I figured I would give it a try. I installed it on our used computer, only to accidentally remove the Windows 95 installation. Eventually, I learned about dual booting, partitions, and formatting filesystems. 🙂

    After getting a handle on things, I learned about Free Software and the GNU movement. I agreed with the ethics of Free Software, and in 2001, dropped all proprietary software from my personal computers, running only Free Software. My wife still had a Windows XP install until 2008, when it got a nasty virus, and she switched back to OS X.

    I use GNU/Linux now, rather than Mac OS X or Windows, because it offers everything I need as a system administrator and light developer. I’m familiar with the interfaces, the tools, and getting around the operating system. I know how to troubleshoot anything that is thrown my way, without much trouble, and usually, I can fix the problem. I still strongly believe in the ethics of Free Software, and that also drives my decision to choose the software that I do.

  3. What distribution do you run on your main desktop/laptop?

    I run Debian GNU/Linux on all of my machines. This includes my desktop, my Lenovo T61 laptop, and my HP netbook. If I could run Debian on my phone, I would.

  4. What desktop environment do you use and why do you use it?

    I prefer Awesome. I’ve always been a big tiling window manager user. On my netbook, I ran DWM for a couple years because it just got out of the way, and allowed me to take as much advantage of screen real estate as I could. Then I discovered Awesome, and have been running that since.

    Configuring power management on a laptop with something that doesn’t do it automatically for you, like GNOME or KDE however is a challenge. But once you get everything in shape, Awesome WM is very, very pleasant to use.

  5. What one piece of software do you depend upon with this distribution? Why is it so important?

    I rely on ZNC, WeeChat, tmux and Bitlbee more than anything. But, they aren’t running on the computer from which the screenshot came from.

    I’ve been an IRC junky since roughly 2002, and have been permanently connected since 2005. All of my main communication go through it one way or another. Further, XMission uses IRC for all internal communication, and
    has Nagios, SMS, fax, and other bots configured to use it also (XMission was started in 1993, before XMPP or other more “modern” chat solutions existed).

    As such, I’ve created a Python script that sends an SMS alert to my phone while I am marked as “/away” in the IRC client. This way, I will always get notified of highlights or private messages. This has been critical for keeping on top of issues at work, and has also allowed me to see who is sharing my blog, and other things, in the many IRC channels that I hang out in.

    My setup has the ZNC bouncer running on a virtual machine. Bitlbee is running locally also, and is always connected to ZNC. Furthermore, I’ve launched WeeChat behind tmux on the same virtual machine, so I can take advantage of some WeeChat scripts. When I disconnect from tmux, then WeeChat will mark me as away automatically, which means I will then get push notifications of highlights and private messages, to my phone. In Bitlbee, I have some XMPP bots for various push notifications also.

    Long story short, this virtual machine is a messaging hub that I rely on for live push notifications that are important to me. And they all use SMS, so I don’t need to keep a running data connection on my phone, to save battery.

  6. What kind of hardware do you run this setup on?

    I have two hypervisors that are using a shared ZFS storage using GlusterFS connected via 20 Gb Inifiniband. Each hypervisor has 8 AMD cores, with 32GB of DDR2 ECC registered ram. Each server has 4×1 TB drives in a ZFS RAID 1+0. Both hypervisors are running Debian GNU/Linux stable, and using KVM for the virtualization layer. The virtual machines are using image files rather than block devices to take advantage of the clustering for HA. My messaging VM has 1GB of RAM with 2 cores and 100GB of disk.

    You can read more about the setup here: https://plus.google.com/+AaronToponce/posts/CTDeruUFMse

  7. Will you share a screenshot of your desktop?

    I have three monitors connected using Xinerama (yeah, I don’t care about the wobbly windows or GPU acceleration). They are each Dell U2410 connected via DVI.

    On the left monitor is my main “monitoring” view. I have a custom Perl script I wrote for showing Nagios/Icinga alerts in a terminal. Working for an ISP means I’m also monitoring border traffic. So, I have a custom Perl script monitoring that. On the other workspaces are Nagstamon, virt-manager, and some custom tools for work.

    In the middle monitor, is my main messaging monitor. You can see IRC occupying most of the screen, with three separate Mutt instances on the right: personal, work and shool. On the other workspaces are Bitmessage, Thunderbird, and Hotot.

    On the right monitor is mainly my browser. In the screenshot, it’s viewing a Munin installation, in this case looking into a disk warning with one of our Exim servers. On the other workspaces are KeepassX and Zim.

Aaron Toponce's desktop

Interview conducted November 19, 2013


The Linux Setup is a feature where I interview people about their Linux setups. The concept is borrowed, if not outright stolen, from this site. If you’d like to participate, drop me a line.

You can follow Linux Rig on Google+ here, follow me on Twitter here, and subscribe to the feed here.