Database on Evan Hoffman

Goodbye, pg_dump

Thu, 26 Sep 2013 09:12:05 +0000

I’ve been a Postgres user and administrator for a while. Over the years, my views on backups have evolved.

Originally, like most people, I started out with good old pg_dump. With a reasonably small database (under 50 GB) dumping to a flat text file is a fine option. I’d generally do something like pg_dump -Upostgres dbname | gzip > dbname.sql.gz to compress it on the fly and save space. For years this seemed perfect: dumping the entire database in a single transaction into a single file that can be restored anywhere.

Problems with PostgreSQL 9.0 on CentOS 6.3 with a 2.6.32-series kernel

Tue, 25 Sep 2012 13:20:36 +0000

Note: this is cross-posted from the Yodle Tech Blog. We recently migrated our Postgres database to a significantly more powerful machine. While our existing server was doing okay load-wise, we were approaching 90% full on its Postgres volume. We’d also recently acquired an awesome new PureStorage SSD-based SAN and were excited to move our database onto it. For comparison, here are the the specs for the two systems:


	Old server	New server
Nickname	DB1	DB2
CPU	2x 6-core Xeon X5690 CPUs at 3.47GHz	2x 8-core E5-2690 CPUs at 2.90 GHz
Memory	96 GB 1333 MHz	128 GB 1600 MHz
Storage	- Dell PERC H800 SAS HBA - Dell MD1220 drive enclosure - 24x 15k RPM SAS drives configured in RAID 10	- QLogic QLE2562 dual-port 8 Gbit FC HBA - PureStorage SSD-based SAN
OS	CentOS 5.6 x86_64	CentOS 6.3 x86_64
Kernel	2.6.18-238.9.1.el5	2.6.32-279.2.1.el6.x86_64

As you can probably guess, moving from spinning disk onto SSD was the centerpiece of the upgrade. We’d tested the PureStorage system pretty extensively and used it in some secondary databases and seen some amazing improvements. We were excited to see how it would handle our production workload.

33 hours to restore a Postgres DB

Tue, 26 Jul 2011 18:10:01 +0000

Restoring a huge DB takes a long time.

Benchmarking disk IO on ext3 vs ext4 vs xfs with fio

Fri, 22 Jul 2011 07:15:05 +0000

With the old database phased out, I figured this was a good time to benchmark IO on it before either repurposing it or mothballing it. In the past I’ve used dd for elementary sequential read/write testing, but I recently found fio which is much more versatile. Apparently developed by FusionIO, fio lets you control concurrency, blocksize and many other parameters and reports lots more data, most notably IOPS.

I created 4 fio config files, basically the same as the one below, for direct / buffered disk access and sequential / random writes. All have 8k blocksize since that’s what Postgres uses by default.

Using WAL archiving & Compellent snapshots for PostgreSQL backups

Wed, 13 Jul 2011 13:46:40 +0000

Changing my Postgres DB backup strategy from nightly pg_dumps to WAL archiving with SAN snapshots.

Migrating a PostgreSQL DB to a new machine without doing a dump & restore

Mon, 11 Jul 2011 14:39:16 +0000

rsync over NFS can be so fast.

Do I still need swap space?

Thu, 11 Nov 2010 14:36:49 +0000

About three years ago I replaced our primary database. For years we’d been plagued by awful performance in the database and we were never able to diagnose the problem. The original server was a real beast at the time: 8 Opterons (single core), 32 gigs ram, and a fibre channel RAID connected via a QLogic HBA. This was back in 2005, so those specs don’t probably sound that impressive today, but this was a crazy configuration (with a crazy price tag to match). On paper it looked like this server should be basically invincible but the performance was awful, slowing down every process within the company. We contacted a few different companies (including CommandPrompt, which employs several of the core Pg devs) to see if they could assist us in diagnosing the problems but tuning only helped to a point. There was just something wrong with the box, maybe having to do with the FC HBA itself (which nobody knew much about).