Linux on Evan Hoffman

Home NAS build - 2026 Edition

Thu, 05 Feb 2026 06:18:40 +0000

My journey building a new home NAS in 2026 I outgrew the one I had previously built in 2017. I discuss CPU and disk selection, benchmarking raid levels, optimizing configurations with docker and how to leverage AI assistants to simplify configurations.

Home NAS build, 2017 Edition

Thu, 09 Feb 2017 20:04:14 +0000

Many years ago, before WiFi or even routers were common in homes, I was faced with the problem of how to connect multiple computers in my home to the internet when our cable provider only provided a single IP address. Being the late 1990s-early 2000s, this was a problem largely faced only by super-nerds like me, and the generally accepted solution was to build your own router, generally using a Linux box with two Ethernet ports, ipchains (or iptables, if you were bleeding-edge) to do NAT, DHCP and DNS. I had an old computer laying around so that’s what I did. This box served me well for many years, ultimately being replaced by a Linksys WRT54g.

My brief love affair with Synology

Tue, 24 Jan 2017 20:11:59 +0000

For several years I’ve been missing the loss of my Linux file server and contemplating either building a new one or buying a home NAS. About 3 weeks ago I made the plunge and ordered a Synology DS416 from Newegg.

Setup was very easy, though I was surprised to learn the device doesn’t work at all without a drive in it. I expected there would be a built on OS volume for running native services but apparently everything gets copied onto the data volume once the drives are inserted.

How (the hell) do you set up Splunk Cloud on Linux?

Wed, 29 Oct 2014 14:11:40 +0000

This took me way longer than I would’ve thought, mostly due to horrible documentation. Here’s my TL;DR version:

Sign up for Splunk Cloud
Download and install the forwarder binary from here.- Log in here and note the URL of your Splunk instance:

In the above picture, assume the URL is https://prd-p-jxxxxxxxx.splunk6.splunktrial.com.

Make sure your instances can connect to port tcp/9997 on your input host. Your input host is the hostname from above with “input-” prepended to it. So in our example, the input host is input-prd-p-jxxxxxxxx.splunk6.splunktrial.com. To ensure you can connect, try telnet input-prd-p-jxxxxxxxx.splunk6.splunktrial.com 9997. If it can’t connect you may need to adjust your firewall rules / Security groups to allow outbound tcp/9997

Below are the actual commands I used to get data into our Splunk Cloud trial instance:

Can I create an EC2 MySQL slave to an RDS master?

Wed, 08 Oct 2014 15:25:10 +0000

No.

Here’s what happens if you try:

mysql> grant replication slave on *.* to 'ec2-slave'@'%';
ERROR 1045 (28000): Access denied for user 'rds_root'@'%' (using password: YES)
mysql> update mysql.user set Repl_slave_priv='Y' WHERE user='rds_root' AND host='%';
ERROR 1054 (42S22): Unknown column 'ERROR (RDS): REPLICA SLAVE PRIVILEGE CANNOT BE GRANTED OR MAINTAINED' in 'field list'
mysql>

Note: this is for MySQL 5.5, which is unfortunately what I’m currently stuck with.

The m3.medium is terrible

Tue, 12 Aug 2014 13:51:08 +0000

I’ve been doing some testing of various instance types in our staging environment, originally just to see if Amazon’s t2.* line of instances is usable in a real-world scenario. In the end, I found that not only are the t2.mediums viable for what I want them to do, but they’re far better suited than the m3.medium, which I wouldn’t use for anything that you ever expect to reach any load.

Using OpenSWAN to connect two VPCs in different AWS regions

Thu, 07 Aug 2014 13:16:19 +0000

Amazon has a pretty decent writeup on how to do this (here), but in trying to establish Postgres replication across regions, I found some weird behavior where I could connect to the port directly (telnet to 5432) but psql (or pg_basebackup) didn’t work. tcpdump showed this:

16:11:28.419642 IP 10.121.11.47.35039 > 10.1.11.254.postgresql: Flags [P.], seq 9:234, ack 2, win 211, options [nop,nop,TS val 11065893 ecr 1811434], length 225
16:11:28.419701 IP 10.121.11.47.35039 > 10.1.11.254.postgresql: Flags [P.], seq 9:234, ack 2, win 211, options [nop,nop,TS val 11065893 ecr 1811434], length 225
16:11:28.421186 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [.], ack 234, win 219, options [nop,nop,TS val 1811520 ecr 11065893,nop,nop,sack 1 {9:234}], length 0
16:11:28.425273 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811522 ecr 11065893], length 1375
16:11:28.425291 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:28.697397 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811590 ecr 11065893], length 1375
16:11:28.697438 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:29.241311 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811726 ecr 11065893], length 1375
16:11:29.241356 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:30.333438 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811999 ecr 11065893], length 1375
16:11:30.333488 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:32.513418 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1812544 ecr 11065893], length 1375
16:11:32.513467 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:36.881409 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1813636 ecr 11065893], length 1375
16:11:36.881460 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556

After quite a bit of Google and mucking in network ACLs and security groups, the fix ended up being this:

OpenVPN CLI Cheat Sheet

Tue, 22 Jul 2014 10:58:21 +0000

Adding a regular user called testing

/usr/local/openvpn_as/scripts/sacli -u testing -k type -v user_connect UserPropPut

Add an autologin user called knock

/usr/local/openvpn_as/scripts/sacli -u knock -k prop_autologin -v true UserPropPut

Create CloudWatch alerts for all Elastic Load Balancers

Fri, 18 Jul 2014 10:29:57 +0000

I manage a bunch of ELBs but we were missing an alert on a pretty basic metric: how many errors the load balancer was returning. Rather than wade through the UI to add these alerts I figured it would be easier to do it via the CLI.

Assuming aws-cli is installed and the ARN for your SNS topic (in my case, just an email alert) is $arn:

for i in `aws elb describe-load-balancers | grep LoadBalancerName | 
perl -ne 'chomp; my @a=split(/s+/); $a[2] =~ s/[",]//g ; print "$a[2] ";' ` ; 
do aws cloudwatch put-metric-alarm --alarm-name "$i ELB 5XX Errors" --alarm-description 
"High $i ELB 5XX error count" --metric-name HTTPCode_ELB_5XX --namespace AWS/ELB 
--statistic Sum --period 300 --evaluation-periods 1 --threshold 50 
--comparison-operator GreaterThanThreshold --dimensions Name=LoadBalancerName,Value=$i 
--alarm-actions $arn --ok-actions $arn ; done

That huge one-liner creates a CloudWatch notification that sends an alarm when the number of 5XX errors returned by the ELB is greater than 50 over 5 minutes, and sends an “ok” message via the same SNS topic. The for loop creates/modifies the alarm for every ELB.

Super quick wordpress exploit stopper

Fri, 25 Oct 2013 09:27:38 +0000

I got an email yesterday from my host (DigitalOcean) that I was running a phishing website. So, I’m not, but I quickly guessed what happened: my Wordpress got hacked. This is just one of the risks of running silly little PHP apps. I logged in, deleted the themes directories, reinstalled clean ones, and ensured this doesn’t happen again by doing the following:

useradd apache_ro
chown -R apache_ro:apache_ro $WP/wp-content/themes

Now apache can’t write to those directories. This means you can’t update Wordpress via the web UI, but I’m ok with that.

The not-so-secret secret to Postgres performance

Mon, 07 Oct 2013 10:19:05 +0000

I manage a bunch of Postgres DBs and one of the things I almost always forget to do when setting up a new one is set the readahead up from the default of 256. I created this script and run it out of /etc/rc.local and sometimes cron it too. The 3 commands at the top are only really relevant on systems with “huge” memory - probably over 64 GB. We ran into some memory problems with CentOS 6 on a box with 128 GB ram which we ended up working around by reinstalling CentOS 5, but the /sys/kernel/mm/redhat_transparent_hugepage/ options below should fix them in 6.x (though we haven’t actually tried it on that DB, we haven’t seen any problems in other large DBs).

Goodbye, pg_dump

Thu, 26 Sep 2013 09:12:05 +0000

I’ve been a Postgres user and administrator for a while. Over the years, my views on backups have evolved.

Originally, like most people, I started out with good old pg_dump. With a reasonably small database (under 50 GB) dumping to a flat text file is a fine option. I’d generally do something like pg_dump -Upostgres dbname | gzip > dbname.sql.gz to compress it on the fly and save space. For years this seemed perfect: dumping the entire database in a single transaction into a single file that can be restored anywhere.

Linux 'date' stuff I didn't know until recently.

Fri, 26 Jul 2013 12:47:32 +0000

I’ve been using Linux for a long time but I had no idea you could do this.

evan@evbox:~$ date -d yesterday
Thu Jul 25 12:44:39 EDT 2013
evan@evbox:~$ date -d '42 days ago'
Fri Jun 14 12:44:51 EDT 2013
evan@evbox:~$ date -d '65 minutes ago'
Fri Jul 26 11:41:18 EDT 2013

As epoch:

evan@evbox:~$ date -d '65 minutes ago' +%s
1374853307

Amazing.

Digital Ocean - First Impressions

Wed, 24 Jul 2013 08:59:06 +0000

For the past few years I’ve been hosting this site on an old desktop in my basement on my FiOS connection. This was one of the things I really liked when I switched from Cablevision to Verizon - they don’t block port 80 inbound, so I didn’t have to pay for separate hosting. My “server” was an old AMD desktop with 1 gig ram and a sata drive. It was ok; my site was slow but I was ok with that. I configured Nginx to cache the static assets which sped most things up to “ok” levels but it was never fast.

XFS write speeds: software RAID 0/5/6 across 45 spindles

Wed, 02 Jan 2013 16:05:07 +0000

We’re currently building a new storage server to store low-priority data (tertiary backups, etc). One of the requirements for the project is that it needs to be on cheap storage (as opposed to expensive enterprise SAN/NAS). After some research we decided to build a Backblaze pod. Backblaze used 3TB Hitachi drives in their system, but the ones they listed in their blog post are discontinued and the reviews for all other 3TB+ drives were terrible, so we went with Samsung ST2000DL004 2TB 7200 RPM drives. Like Backblaze, we’re going with software raid, but I figured a good first step would be to figure out what RAID level we want to use, and if we want to use the mdadm/LVM mish-mosh Backblaze uses, or find something simpler. For my testing I created a RAID6 of all 45 drives and created a single XFS volume (XFS’s size limit is ~8 exabytes vs ext4’s 16TB). Ext4 may present some performance advantages, but the management overhead is probably not worth it in our case.

Using Nginx as a caching proxy in front of Wordpress

Wed, 07 Mar 2012 16:43:28 +0000

Why you may want to use Nginx as a caching reverse-proxy in front of Apache for Wordpress, and config examples for doing so.

Reorganizing photos in 1 line with exiftool

Fri, 02 Mar 2012 10:01:11 +0000

A few years ago I wrote a utility in Java to find all JPG files in a directory and move them into a date-based directory structure like /YYYY/MM/DD/ based on the date the photo was taken, extracted from the exif metadata in the file. Well, apparently that was a huge waste of time, as I just discovered that exiftool, an awesome perl utility I’ve used for years to edit/extract the metadata on the command line, can also do this natively. So my entire program can be replaced with this simple command:

Slow HTTP downloads through Cisco ASA 5500

Tue, 28 Feb 2012 16:15:20 +0000

Recently we noticed weird behavior downloading files from certain sites. The transfer would start out fast (around 10 MB/s), then after a couple of seconds it would plummet to around 9 KB/s. It didn’t happen for every file or every site: downloads from S3 buckets were still particularly fast. But some files that I remember being particularly fast were now showing this weird fast/slow/fast/slow behavior, for example the Sun JDK and ISOs from rit.edu that used to saturate our pipe were now getting all cRAzY.

Using rrdtool to generate server load & bandwidth graphs

Fri, 17 Feb 2012 15:32:25 +0000

I’ve been using MRTG and routers2.cgi for years to graph the various aspects of a server that warrant monitoring. I’ve long known that they used something called rrdtool to do… well, something, but never had a need or desire to figure out exactly what that was.

But, having just moved my site to a new server, I was curious how the server would handle the load. Rather than setting up some behemoth like Nagios or Zabbix, which are full monitoring/alerting suites, I just wanted graphing. As I said, in the past I’ve used MRTG or routers2.cgi for this but both of them were overkill for me in this case. Since both of them used rrdtool, I figured that was a good place to look.

Load balancing in EC2 with Nginx and HAProxy

Mon, 09 Jan 2012 17:00:22 +0000

We wanted to setup a loadbalanced web cluster in AWS for expansion. My first inclination was to use ELB for this, but I soon learned that ELB doesn’t let you allocate a static IP, requiring you to refer to it only by DNS name. This would be OK except for the fact that our current DNS provider, Dyn, requires IP addresses when using their GSLB (geo-based load balancer) service.

Rather than let this derail the whole project, I decided to look into the software options available for loadbalancing in EC2. I’ve been a fan of hardware load balancers for a while, sort of looking down at software-based solutions without any real rationale, but in this case I really had no choice so I figured I’d give it a try.

Installing Sun (Oracle) JDK 1.5 on an EC2 instance

Wed, 16 Nov 2011 11:52:55 +0000

I’m currently working on moving a Tomcat-based application into EC2. The code was written for Java 5.0. While Java 6 would probably work, I’d like to keep everything as “same” as possible, since EC2 presents its own challenges. I spun up a couple of t1.micro instances and copied everything over, including the Java 5 JDK, jdk-1_5_0_22-linux-amd64.rpm. Installing from RPM was easy, but the EC2 instance defaults to using OpenJDK 1.6:

[root@ec2 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.10) (amazon-52.1.9.10.40.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

There were a couple of things I had to do to get the system to accept the Sun JDK as its “real” java.

Rescan SATA bus (aka hot-adding a SATA disk on a Linux guest in VMware without rebooting)

Mon, 24 Oct 2011 15:20:51 +0000

Linux supports hot-adding disks but whenever I add a new vdisk in VMware the new disk doesn’t show up unless I reboot, which defeats the purpose of hot-add. This command forces a rescan of the bus:

echo "- - -" > /sys/class/scsi_host/host0/scan

dmesg shows the new disk has been found:

Vendor: VMware Model: Virtual disk Rev: 1.0 
 Type: Direct-Access ANSI SCSI revision: 02
 target0:0:2: Beginning Domain Validation
 target0:0:2: Domain Validation skipping write tests
 target0:0:2: Ending Domain Validation
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
 sdd: unknown partition table
sd 0:0:2:0: Attached scsi disk sdd
sd 0:0:2:0: Attached scsi generic sg3 type 0

Now, why there’s no “rescan_sata” command is something I can’t fathom, but that’s Linux for you.

Fedora Core 15 & Gnome 3

Fri, 23 Sep 2011 10:27:22 +0000

Shrinking partitions, installing/upgrading to FC15, and systemd/chkconfig.

Making sure SSLv2 is disabled in Apache (and Nginx)

Tue, 20 Sep 2011 14:06:38 +0000

Edit Jan 24, 2012: Deleted all the crap from this story and just left the recommended Apache and Nginx SSL cipher suites for maximum security without SSLv2 and without BEAST vulnerability (at least according to Qualys). Apache httpd

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
SSLHonorCipherOrder on

nginx

ssl_protocols SSLv3 TLSv1;
 ssl_ciphers ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
 ssl_prefer_server_ciphers on;

Source:

64-bit Firefox 6 on CentOS 6 x86_64 (or Fedora) with Flash 11

Wed, 31 Aug 2011 16:15:14 +0000

So I’ve gone back to Linux from Mac, due to the SSD issues I had with my Macbook Pro basically making VMware unusable. A Win7 VM would grind the guest and host to a halt on the 7200 RPM SATA OEM drive, and the SSD wouldn’t work, so I put the SSD back in my HP and installed CentOS 6 x86_64. Not really ideal, but at least it works.

CentOS 6 ships with Firefox 3.6.9, which is really old by now. Fine for a server, but I wanted FF6. I grabbed the Firefox 6.0.1 bz2 from VoxCast, unzipped it and copied it to /usr/lib64/firefox-6. Then I ran yum remove firefox to remove 3.6.9 and avoid any issues. I tried the instructions from my older post on this subject but for whatever reason it didn’t work - I’m guessing because I’m not using the distro’s Firefox RPM.

33 hours to restore a Postgres DB

Tue, 26 Jul 2011 18:10:01 +0000

Restoring a huge DB takes a long time.

Benchmarking disk IO on ext3 vs ext4 vs xfs with fio

Fri, 22 Jul 2011 07:15:05 +0000

With the old database phased out, I figured this was a good time to benchmark IO on it before either repurposing it or mothballing it. In the past I’ve used dd for elementary sequential read/write testing, but I recently found fio which is much more versatile. Apparently developed by FusionIO, fio lets you control concurrency, blocksize and many other parameters and reports lots more data, most notably IOPS.

I created 4 fio config files, basically the same as the one below, for direct / buffered disk access and sequential / random writes. All have 8k blocksize since that’s what Postgres uses by default.

MongoDB logrotate script

Thu, 21 Jul 2011 10:10:26 +0000

Daily mongodb log rotation via logrotate.

VMWare 5's new licensing model.

Sun, 17 Jul 2011 10:20:56 +0000

After reading up on the new VMware licensing&pricing model I understand the uproar. Limiting vRAM is a reasonable constraint, but 32GB per socket for Enterprise? 48 GB for “Enterprise Plus”? If you have a dual CPU server with 144 GB (easily configurable last year), with 4.1 you’d only need 2 enterprise licenses to use all 144 GB, since in 4.1 an “Enterprise” license covered 1 CPU (up to 6 cores) and up to 256 GB memory on the host.

Using WAL archiving & Compellent snapshots for PostgreSQL backups

Wed, 13 Jul 2011 13:46:40 +0000

Changing my Postgres DB backup strategy from nightly pg_dumps to WAL archiving with SAN snapshots.