Perl on Evan Hoffman

Reorganizing photos in 1 line with exiftool

Fri, 02 Mar 2012 10:01:11 +0000

A few years ago I wrote a utility in Java to find all JPG files in a directory and move them into a date-based directory structure like /YYYY/MM/DD/ based on the date the photo was taken, extracted from the exif metadata in the file. Well, apparently that was a huge waste of time, as I just discovered that exiftool, an awesome perl utility I’ve used for years to edit/extract the metadata on the command line, can also do this natively. So my entire program can be replaced with this simple command:

Graphing SSH dictionary attacks with HighCharts

Fri, 18 Nov 2011 14:06:03 +0000

After my 10-year-old basement Linux server died this week from a power outage, I took the sad step of giving up on it. It’s died before and I’ve patched it back together with a new power supply here or an addon PCI SATA card there, but I finally decided to throw in the towel since I had a newer old computer that had been idle for several years. The one that died was an Athlon K7 750 MHz with 512 MB ram. The new one is an Athlon 2 GHz (3200+) with 1 gig. For my uses, specs don’t really matter that much, but it’s nice to have more power for free.

Logging RT username in Apache access_log

Mon, 08 Aug 2011 17:34:18 +0000

RT has its own internal accounting & tracking system for logging activity, but I was interested in even more granular stuff, like seeing who looked at which tickets. I figured it wouldn’t be that hard to log this in Apache. Well, I was kind of right, in that it wasn’t “hard,” but it took me a long time to find the right place to do it. I did finally get it though.

Amazon SES: "illegal headers" with ses-send-email.pl (followup)

Tue, 02 Aug 2011 14:39:45 +0000

A few people have emailed me asking me to integrate the perl code snippet into I wrote to strip illegal headers when sending email via Amazon SES into something actually usable. I’ve done so! I haven’t really tested this beyond sending some test emails, but here it is. Use this at your own risk, I make no warranty, blah blah blah. This fix requires editing ses-send-email.pl, so I’d advise making a backup copy, though I imagine you can always get a fresh version from Amazon if necessary.

Amazon SES - "Illegal Header" errors

Mon, 16 May 2011 12:37:49 +0000

My thoughts on the “illegal header” error some people are encountering when relaying mail through postfix/SES. Plus some perl code that should help fix it.

Integrating Amazon Simple Email Service with postfix for SMTP smarthost relaying.

Thu, 28 Apr 2011 12:23:47 +0000

How I configured an EC2 instance to relay mail through Amazon’s SES, and some of the bumps I encountered along the way.

More fun parsing BIND query logs

Mon, 13 Dec 2010 18:21:59 +0000

“But wouldn’t it be cool if it also…” That phrase usually triggers a lot of wasted cycles in my brain, though it sometimes comes up with something neat. I added a super lame graph to the DNS QPS parser. Makes it really easy to see peaks & troughs in usage:

2010-12-12 00:00 to 00:59 => 3237, rate 0.899 queries/sec	***********
2010-12-12 01:00 to 01:59 => 2709, rate 0.752 queries/sec	**********
2010-12-12 02:00 to 02:59 => 2473, rate 0.687 queries/sec	*********
2010-12-12 03:00 to 03:59 => 2171, rate 0.603 queries/sec	********
2010-12-12 04:00 to 04:59 => 2048, rate 0.569 queries/sec	*******
2010-12-12 05:00 to 05:59 => 1918, rate 0.533 queries/sec	*******
2010-12-12 06:00 to 06:59 => 2120, rate 0.589 queries/sec	********
2010-12-12 07:00 to 07:59 => 2477, rate 0.688 queries/sec	*********
2010-12-12 08:00 to 08:59 => 2994, rate 0.832 queries/sec	**********
2010-12-12 09:00 to 09:59 => 3704, rate 1.029 queries/sec	*************
2010-12-12 10:00 to 10:59 => 4297, rate 1.194 queries/sec	***************
2010-12-12 11:00 to 11:59 => 4744, rate 1.318 queries/sec	****************
2010-12-12 12:00 to 12:59 => 5106, rate 1.418 queries/sec	******************
2010-12-12 13:00 to 13:59 => 5311, rate 1.475 queries/sec	******************
2010-12-12 14:00 to 14:59 => 5083, rate 1.412 queries/sec	*****************
2010-12-12 15:00 to 15:59 => 4855, rate 1.349 queries/sec	*****************
2010-12-12 16:00 to 16:59 => 5179, rate 1.439 queries/sec	******************
2010-12-12 17:00 to 17:59 => 4959, rate 1.377 queries/sec	*****************
2010-12-12 18:00 to 18:59 => 4693, rate 1.304 queries/sec	****************
2010-12-12 19:00 to 19:59 => 4792, rate 1.331 queries/sec	****************
2010-12-12 20:00 to 20:59 => 4799, rate 1.333 queries/sec	****************
2010-12-12 21:00 to 21:59 => 5068, rate 1.408 queries/sec	*****************
2010-12-12 22:00 to 22:59 => 4672, rate 1.298 queries/sec	****************
2010-12-12 23:00 to 23:59 => 4514, rate 1.254 queries/sec	****************
2010-12-13 00:00 to 00:59 => 3917, rate 1.088 queries/sec	**************
2010-12-13 01:00 to 01:59 => 3458, rate 0.961 queries/sec	************
2010-12-13 02:00 to 02:59 => 2874, rate 0.798 queries/sec	**********
2010-12-13 03:00 to 03:59 => 2715, rate 0.754 queries/sec	**********
2010-12-13 04:00 to 04:59 => 2690, rate 0.747 queries/sec	*********
2010-12-13 05:00 to 05:59 => 2719, rate 0.755 queries/sec	**********
2010-12-13 06:00 to 06:59 => 2831, rate 0.786 queries/sec	**********
2010-12-13 07:00 to 07:59 => 3416, rate 0.949 queries/sec	************
2010-12-13 08:00 to 08:59 => 4962, rate 1.378 queries/sec	*****************
2010-12-13 09:00 to 09:59 => 6943, rate 1.929 queries/sec	************************
2010-12-13 10:00 to 10:59 => 8296, rate 2.304 queries/sec	****************************
2010-12-13 11:00 to 11:59 => 8938, rate 2.483 queries/sec	******************************
2010-12-13 12:00 to 12:59 => 8926, rate 2.479 queries/sec	******************************
2010-12-13 13:00 to 13:59 => 8950, rate 2.486 queries/sec	******************************
2010-12-13 14:00 to 14:59 => 2173, rate 0.604 queries/sec	********

You can see pretty quickly that 4-5 AM (Eastern) is the period of lowest activity, as you’d probably expect for a US-based US-centric site. Modified perl script below.

Perl script to calculate DNS queries per second in BIND (named)

Tue, 07 Dec 2010 18:53:43 +0000

I’m pricing out DNS providers and was asked what our current queries-per-second currently are. Sadly I had no idea. After lots of Googling I decided there was really no good way to get this information so I decided to parse the logfile myself. First, I turned on logging with timestamp in named.conf:

logging
{
...
 # Query logging 2010-12-07
 channel query-log {
 file "data/queries.log" versions 3 size 10m;
 print-time yes;
 };
 category queries { query-log; };
};
options
{
...
 # Query logging 2010-12-07
 querylog yes;

};

Blocking comment spammers by IP

Fri, 13 Aug 2010 11:59:27 +0000

I use Akismet to block comment spam, but it still annoys me that it even exists. Last night I put a simple IP ban into my httpd config. But who to block?

I used a grep & Perl to get a rough guess of which IPs were submitting the most comments (working on the assumption that one IP address submits many spam comments) It took me about 20 minutes to write this mess but it does what I wanted to do: [root@lunix ~]# zgrep POST /var/log/httpd/evanhoffman-access_log-201008??.gz | grep comment | perl -ne 'chomp; $_ =~ m/(?:\d{1,3}\.){3}\d{1,3}/; print "$&\n";' | perl -e '%a = (); while (<>) { chomp; $a{$_} += 1; } while (my ($key, $value) = each (%a)) { if ($value > 1) { print "$value\t=>\t$key\n";}}' 2 => 218.6.9.140 180 => 91.201.66.34 2 => 213.5.67.41 2 => 188.187.102.74 [root@lunix ~]#

Running MRTG cfgmaker across your entire subnet?

Wed, 27 Jan 2010 13:00:30 +0000

I realized recently that I had a bunch of newly-provisioned VMs that weren’t being monitored by MRTG (one of the tools we use to monitor network usage and other fun stats). Rather than manually run cfgmaker against all the new machines, I decided to script my way out of this.

Step 1: Build a list of hosts to monitor

For this I used an nmap ping test to generate a list of “up” hosts:

Victory! Change Active Directory Password via LDAP through browser

Wed, 13 Jan 2010 13:51:43 +0000

I had to give up on PHP and go to Perl, but it turned out not to be so bad. Users can now change their Active Directory passwords via a self-service web page that doesn’t require admin credentials. The Perl code is below. Authentication to the script is done via .htaccess LDAP authentication, so the REMOTE_USER env variable is assumed to contain the user’s username (sAMAccountName) by the time this script is called. There is a simple check for $ENV{HTTPS} to ensure the script is called via SSL, and AD requires password changes to be done via ldaps, so the whole thing should be encrypted end to end.

Flac to MP3 (flac2mp3.pl)

Mon, 07 Nov 2005 00:00:44 +0000

I wrote a crappy perl script to read all the .flac files in a directory and convert them to MP3 using LAME, extracting the meta-data and inserting it as ID3v2 tags. I figured this might be somewhat useful to others, so here you go: <flac2mp3.pl>

https://gist.github.com/1642637.js?file=flac2mp3.pl