Aws on Evan Hoffman

Installing a new SSL certificate in your ELB via CLI

Tue, 14 Oct 2014 12:01:49 +0000

For future me:

Create the key and CSR:

$ openssl req -out wildcard.site.com.csr -new -newkey rsa:2048 -nodes -keyout wildcard.site.com.key

Upload the CSR to your SSL vendor (in this case, DigiCert) and obtain the signed SSL certificate.
Create a PEM-encoded version of the signing key. This is required for AWS/IAM certs. To check if your key is already PEM-encoded, just “head -1 site.key”. If the first line says “—–BEGIN PRIVATE KEY—–” then it’s NOT PEM-encoded. The first line should be “—–BEGIN RSA PRIVATE KEY—–”.

The m3.medium is terrible

Tue, 12 Aug 2014 13:51:08 +0000

I’ve been doing some testing of various instance types in our staging environment, originally just to see if Amazon’s t2.* line of instances is usable in a real-world scenario. In the end, I found that not only are the t2.mediums viable for what I want them to do, but they’re far better suited than the m3.medium, which I wouldn’t use for anything that you ever expect to reach any load.

Using OpenSWAN to connect two VPCs in different AWS regions

Thu, 07 Aug 2014 13:16:19 +0000

Amazon has a pretty decent writeup on how to do this (here), but in trying to establish Postgres replication across regions, I found some weird behavior where I could connect to the port directly (telnet to 5432) but psql (or pg_basebackup) didn’t work. tcpdump showed this:

16:11:28.419642 IP 10.121.11.47.35039 > 10.1.11.254.postgresql: Flags [P.], seq 9:234, ack 2, win 211, options [nop,nop,TS val 11065893 ecr 1811434], length 225
16:11:28.419701 IP 10.121.11.47.35039 > 10.1.11.254.postgresql: Flags [P.], seq 9:234, ack 2, win 211, options [nop,nop,TS val 11065893 ecr 1811434], length 225
16:11:28.421186 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [.], ack 234, win 219, options [nop,nop,TS val 1811520 ecr 11065893,nop,nop,sack 1 {9:234}], length 0
16:11:28.425273 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811522 ecr 11065893], length 1375
16:11:28.425291 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:28.697397 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811590 ecr 11065893], length 1375
16:11:28.697438 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:29.241311 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811726 ecr 11065893], length 1375
16:11:29.241356 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:30.333438 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1811999 ecr 11065893], length 1375
16:11:30.333488 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:32.513418 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1812544 ecr 11065893], length 1375
16:11:32.513467 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556
16:11:36.881409 IP 10.1.11.254.postgresql > 10.121.11.47.35039: Flags [P.], seq 2:1377, ack 234, win 219, options [nop,nop,TS val 1813636 ecr 11065893], length 1375
16:11:36.881460 IP 10.1.96.20 > 10.1.11.254: ICMP 10.121.11.47 unreachable - need to frag (mtu 1422), length 556

After quite a bit of Google and mucking in network ACLs and security groups, the fix ended up being this:

OpenVPN CLI Cheat Sheet

Tue, 22 Jul 2014 10:58:21 +0000

Adding a regular user called testing

/usr/local/openvpn_as/scripts/sacli -u testing -k type -v user_connect UserPropPut

Add an autologin user called knock

/usr/local/openvpn_as/scripts/sacli -u knock -k prop_autologin -v true UserPropPut

Create CloudWatch alerts for all Elastic Load Balancers

Fri, 18 Jul 2014 10:29:57 +0000

I manage a bunch of ELBs but we were missing an alert on a pretty basic metric: how many errors the load balancer was returning. Rather than wade through the UI to add these alerts I figured it would be easier to do it via the CLI.

Assuming aws-cli is installed and the ARN for your SNS topic (in my case, just an email alert) is $arn:

for i in `aws elb describe-load-balancers | grep LoadBalancerName | 
perl -ne 'chomp; my @a=split(/s+/); $a[2] =~ s/[",]//g ; print "$a[2] ";' ` ; 
do aws cloudwatch put-metric-alarm --alarm-name "$i ELB 5XX Errors" --alarm-description 
"High $i ELB 5XX error count" --metric-name HTTPCode_ELB_5XX --namespace AWS/ELB 
--statistic Sum --period 300 --evaluation-periods 1 --threshold 50 
--comparison-operator GreaterThanThreshold --dimensions Name=LoadBalancerName,Value=$i 
--alarm-actions $arn --ok-actions $arn ; done

That huge one-liner creates a CloudWatch notification that sends an alarm when the number of 5XX errors returned by the ELB is greater than 50 over 5 minutes, and sends an “ok” message via the same SNS topic. The for loop creates/modifies the alarm for every ELB.

Setting hostname in an EC2 instance from the name tag

Mon, 19 May 2014 15:24:09 +0000

# pip install awscli
# HOSTNAME=`aws ec2 describe-tags --region us-east-1 --filters Name=resource-id,Values=`curl http://169.254.169.254/latest/meta-data/instance-id 2> /dev/null` Name=key,Values=Name --output text --query 'Tags[*].Value'`
# hostname $HOSTNAME
# hostname > /etc/hostname

Goodbye, pg_dump

Thu, 26 Sep 2013 09:12:05 +0000

I’ve been a Postgres user and administrator for a while. Over the years, my views on backups have evolved.

Originally, like most people, I started out with good old pg_dump. With a reasonably small database (under 50 GB) dumping to a flat text file is a fine option. I’d generally do something like pg_dump -Upostgres dbname | gzip > dbname.sql.gz to compress it on the fly and save space. For years this seemed perfect: dumping the entire database in a single transaction into a single file that can be restored anywhere.

Load balancing in EC2 with Nginx and HAProxy

Mon, 09 Jan 2012 17:00:22 +0000

We wanted to setup a loadbalanced web cluster in AWS for expansion. My first inclination was to use ELB for this, but I soon learned that ELB doesn’t let you allocate a static IP, requiring you to refer to it only by DNS name. This would be OK except for the fact that our current DNS provider, Dyn, requires IP addresses when using their GSLB (geo-based load balancer) service.

Rather than let this derail the whole project, I decided to look into the software options available for loadbalancing in EC2. I’ve been a fan of hardware load balancers for a while, sort of looking down at software-based solutions without any real rationale, but in this case I really had no choice so I figured I’d give it a try.

Integrating Amazon Simple Email Service with postfix for SMTP smarthost relaying.

Thu, 28 Apr 2011 12:23:47 +0000

How I configured an EC2 instance to relay mail through Amazon’s SES, and some of the bumps I encountered along the way.

Amazon EC2 - ext3 mkfs takes 30+ minutes?

Mon, 02 Aug 2010 11:33:42 +0000

I’ve been playing around with Amazon EC2 for a new project I’m working on and so far I’m really impressed. One thing I’ve noticed, however, is that it takes forever to create an ext3 filesystem on a new volume. For example, the below command took over 30 minutes to create the filesystem on a 300 GB volume:

mke2fs -j -m0 /dev/sdf1

mke2fs 1.40.4 (31-Dec-2007) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 39321600 inodes, 78642183 blocks 0 blocks (0.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 2400 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616