Check Apache Httpd MPM Config Limits

A little while ago I had to reboot a client’s VM because the web server forked too many processes. They were making use of PHP, but the web server had not been configured for the resulting larger process size. I searched for a tool that would analyze the size of running httpd processes, and project the impact of starting the maximum number of processes allowed by MaxClients or ServerLimit, but didn’t find anything, so ended-up writing my own.

The following check_httpd_limits.pl script compares the size of running Apache httpd processes, the configured prefork/worker/event MPM limits, and the server’s available memory. The script exits with a warning or error message if the configured limits exceed the server’s available memory.

check_httpd_limits.pl does not use any 3rd-party perl modules, unless the --save/days/max command-line options are used, in which case you will need to have the DBD::SQLite module installed. It should work on any UNIX server that provides /proc/meminfo, /proc//exe, /proc//stat, and /proc//statm files. You will probably have to run the script as root for it to read the /proc//exe symbolic links.

Process Description

When executed, the script will follow this general process.

Read the /proc/meminfo file for server memory values (total memory, free memory, etc.).
Read the /proc/*/exe symbolic links to find the matching httpd binaries. By default, the script will use the first httpd binary found in the @httpd_paths array. You can specify an alternate binary path using the --exe=/path command-line option.
Read the /proc/*/stat files for pid, process name, ppid, and rss values.
Read the /proc/*/statm files for the shared memory size.
Execute the httpd binary with “-V” to get the config file path and MPM info.
Read the httpd configuration file to get MPM (prefork or worker) settings.
Calculate the average and total HTTP process sizes, taking into account the shared memory used.
Calculates possible changes to MPM settings based on available memory and process sizes.
Display all the values found and settings calculated if the --verbose command-line option is used.
Exit with OK (0), WARNING (1), or ERROR (2) based on the projected memory used by all httpd processes running.
- OK: The maximum number of httpd processes fit within the available RAM.
- WARNING: The maximum number of httpd processes exceed the available RAM, but still fits within the free swap.
- ERROR: The maximum number of httpd processes exceed the available RAM and swap space.

Command-Line Options

A few command-line options can modify the behavior of the script.

--help Display a summary of command-line options.
--debug Show debugging messages as the script is executing.
--verbose Display a detailed report of all values found and calculated.
--exe=/path The complete path to an httpd binary file. By default the script will look in the following locations, and use the first executable found: /usr/sbin/httpd, /usr/local/sbin/httpd, /usr/sbin/apache2, /usr/local/sbin/apache2.
--swappct=# The percent of free swap that is allowed to be used before exiting with a WARNING condition. The default is 0% — if the projected size of all httpd processes allowed exceeds the available RAM, the script will exit with a WARNING message.

The DBD::SQLite perl module must be installed if any of the following command-line options are used. By default, the script will analyze and report on the current httpd process sizes. Since httpd processes may grow over time, these options allow you to save historical information to a database file, and use it to predict memory use based on the process sizes over a number of days.

--save Save the current process average sizes to an SQLite database file (/var/tmp/check_httpd_limits.sqlite).
--days=# Remove all database entries that are older than # days. The default is 30 days and using --days=0 will remove all entries from the database.
--max=realavg Use the largest HttpdRealAvg size value from the database, or calculated from the current httpd processes.
--max=running Use the HttpdRealAvg size value associated with the maximum number of running httpd processes saved in the database.

All three command-line options must be used to report on historical data. By itself, the --save option only saves current process information, the --days=# option will only remove old database entries, and --max will only use HttpdRealAvg from the database to predict memory usage.

Use --max=running if the size and number of httpd processes on the web server increase and decrease rapidly or unpredictably. The --max=realavg setting should be more accurate for web servers that have stable httpd sizes, and progressive increase / decrease in the number of httpd processes.

The check_httpd_limits.pl Script

#!/usr/bin/perl

# Copyright 2012 - Jean-Sebastien Morisset - http://surniaulula.com/
#
# This script is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
# Foundation; either version 3 of the License, or (at your option) any later
# version.
#
# This script is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
# details at http://www.gnu.org/licenses/.

# Perl script to compare the size of running Apache httpd processes, the
# configured prefork/worker limits, and the available server memory. Exits with
# a warning or error message if the configured limits exceed the server's
# memory.
#
# Syntax: check_httpd_limits.pl --help

# The script performs the following tasks:
#
# - Reads the /proc/meminfo file for server memory values.
# - Reads the /proc/*/exe symbolic links to find the matching httpd binaries.
# - Reads the /proc/*/stat files for pid, process name, ppid, and rss.
# - Reads the /proc/*/statm files for the shared memory size.
# - Executes HTTP binary with "-V" to get the config file path and MPM info.
# - Reads the HTTP config file to get MPM (prefork or worker) settings.
# - Calculates the average and total HTTP process sizes, taking into account
#   the shared memory used.
# - Calculates possible changes to MPM settings based on available memory and
#   process sizes.
# - Displays all the values found and settings calculated if the --verbose
#   parameter is used.
# - Exits with OK (0), WARNING (1), or ERROR (2) based on projected memory use
#   with all (allowed) HTTP processes running.
#        OK: Maximum number of HTTP processes fit within available RAM.
#   WARNING: Maximum number of HTTP processes exceeds available RAM, but still
#            fits within the free swap.
#     ERROR: Maximum number of HTTP processes exceeds available RAM and swap.

# Changes:
#
# v2.4:
# - Added config for Apache Httpd v2.5 and 2.6 (identical to 2.4).
# - Added config for 'eventopt' MPM (identical to 'event' MPM).

use strict;
use warnings;
use POSIX;
use Getopt::Long;

no warnings 'once';	# no warning for $DBI::err

my $VERSION = '2.4';
my $pagesize = POSIX::sysconf(POSIX::_SC_PAGESIZE);
my @stathrefs;
my $err = 0;
my %mem = (
	'MemTotal' => '',
	'MemFree' => '',
	'Cached' => '',
	'SwapTotal' => '',
	'SwapFree' => '',
);
my %httpd = (
	'EXE' => '',
	'ROOT' => '',
	'CONFIG' => '',
	'MPM' => '',
	'VERSION' => '',
);
my $cf_IfModule = '';
my $cf_MaxName = '';	# defined based on httpd version (MaxClients or MaxRequestWorkers)
my $cf_LimitName = '';	# defined once MPM is determined (MaxClients/MaxRequestWorkers or ServerLimit)
my $cf_ver = '';
my $cf_min = '2.2';
my $cf_mpm = '';
my %cf_read = ();
my %cf_changed = ();
my %cf_defaults = (
	'2.2' => {
		'prefork' => {
			'StartServers' => 5,
			'MinSpareServers' => 5,
			'MaxSpareServers' => 10,
			'ServerLimit' => 256,
			'MaxClients' => 256,
			'MaxRequestsPerChild' => 10000,
		},
		'worker' => {
			'StartServers' => 3,
			'MinSpareThreads' => 75,
			'MaxSpareThreads' => 250,
			'ThreadsPerChild' => 25,
			'ServerLimit' => 16,
			'MaxClients' => 400,
			'MaxRequestsPerChild' => 10000,
		},
	},
	'2.4' => {
		'prefork' => {
			'StartServers' => 5,
			'MinSpareServers' => 5,
			'MaxSpareServers' => 10,
			'ServerLimit' => 256,
			'MaxRequestWorkers' => 256,	# aka MaxClients
			'MaxConnectionsPerChild' => 0,	# aka MaxRequestsPerChild
		},
		'worker' => {
			'StartServers' => 3,
			'MinSpareThreads' => 75,
			'MaxSpareThreads' => 250,
			'ThreadsPerChild' => 25,
			'ServerLimit' => 16,
			'MaxRequestWorkers' => 400,	# aka MaxClients
			'MaxConnectionsPerChild' => 0,	# aka MaxRequestsPerChild
		},
	},
);
$cf_defaults{'2.5'} = $cf_defaults{'2.4'};
$cf_defaults{'2.6'} = $cf_defaults{'2.5'};

# The event MPM config is identical to the worker MPM config
# Uses a hashref instead of copying the hash elements
for my $ver ( keys %cf_defaults ) {
	$cf_defaults{$ver}{'event'} = $cf_defaults{$ver}{'worker'};
	$cf_defaults{$ver}{'eventopt'} = $cf_defaults{$ver}{'event'};
}
# easiest way to copy the three-dimensional hash without using a module
for my $ver ( keys %cf_defaults ) {
	for my $mpm ( keys %{$cf_defaults{$ver}} ) {
		for my $el ( keys %{$cf_defaults{$ver}{$mpm}} ) {
			$cf_read{$ver}{$mpm}{$el} = $cf_defaults{$ver}{$mpm}{$el};
			$cf_changed{$ver}{$mpm}{$el} = $cf_defaults{$ver}{$mpm}{$el};
		}
	}
}
my %cf_comments = (
	'2.2' => {
		'prefork' => {
			'ServerLimit' => 'MaxClients',
			'MaxClients' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',
		},
		'worker' => {
			'ServerLimit' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',
			'MaxClients' => 'ServerLimit * ThreadsPerChild',
		},
	},
	'2.4' => {
		'prefork' => {
			'ServerLimit' => 'MaxRequestWorkers',
			'MaxRequestWorkers' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',
		},
		'worker' => {
			'ServerLimit' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',
			'MaxRequestWorkers' => 'ServerLimit * ThreadsPerChild',
		},
	},
);
$cf_comments{'2.5'} = $cf_comments{'2.4'};
$cf_comments{'2.6'} = $cf_comments{'2.5'};

# the event MPM config is identical to the worker MPM config
# uses a hashref instead of copying the hash elements
for my $ver ( keys %cf_comments ) {
	$cf_comments{$ver}{'event'} = $cf_comments{$ver}{'worker'};
	$cf_comments{$ver}{'eventopt'} = $cf_comments{$ver}{'event'};
}

my %calcs = (
	'HttpdRealAvg' => 0,
	'HttpdSharedAvg' => 0,
	'HttpdRealTot' => 0,
	'HttpdRunning' => 0,
	'OtherProcsMem' => '',
	'FreeMemNoHttpd' => '',
	'MaxLimitHttpdMem' => '',
	'AllProcsTotalMem' => '',
);

# comment string when MaxLimitHttpdMem is calculated from DB values
my $mcs_from_db = '';

# common location for httpd binaries if not sepcified on command-line
my @httpd_paths = (
	'/usr/sbin/httpd',
	'/usr/local/sbin/httpd',
	'/opt/apache/bin/httpd',
	'/opt/apache/sbin/httpd',
	'/usr/lib/apache2/mpm-prefork/apache2',
	'/usr/sbin/apache2',
	'/usr/local/sbin/apache2',
);
my $dbname = '/var/tmp/check_httpd_limits.sqlite';
my $dbuser = '';
my $dbpass = '';
my $dbtable = 'HttpdProcInfo';
my $dsn = "DBI:SQLite:dbname=$dbname";
my $dbh;
my %dbrow = (
	'DateTimeAdded' => 0,
	'HttpdRealAvg' => 0,
	'HttpdSharedAvg' => 0,
	'HttpdRealTot' => 0,
	'HttpdRunning' => 0,
);
my %opt = ();
GetOptions(\%opt, 
	'help',
	'debug',
	'verbose',
	'exe=s',
	'swappct=i',
	'save',
	'days=i',
	'max=s',
);
$opt{'swappct'} = 0 unless ( $opt{'swappct'} );
$opt{'max'} = $opt{'max'} ? lc($opt{'max'}) : "";
&ShowUsage() if ( $opt{'help'} );

if ( $opt{'verbose'} ) {
	print "\nCheck Apache Httpd MPM Config Limits (Version $VERSION)\n";
	print "by Jean-Sebastien Morisset - http://surniaulula.com/\n\n";
}

#
# READ MAXIMUM FROM DATABASE
#
if ( $opt{'save'} || $opt{'days'} || $opt{'max'} ) {
	$opt{'days'} = 30 unless ( defined $opt{'days'} );
	print "Saving Httpd Averages to $dsn\n\n" 
		if ( $opt{'save'} && $opt{'verbose'} );

	require DBD::SQLite;
	print "DEBUG: Connecting to database $dsn.\n" if ( $opt{'debug'} );
	$dbh = DBI->connect($dsn, $dbuser, $dbpass);
	die "ERROR: $DBI::errstr\n" if ($DBI::err);

	$dbh->do("PRAGMA foreign_keys = ON;");

	$dbh->do("CREATE TABLE IF NOT EXISTS $dbtable ( 
		DateTimeAdded DATE PRIMARY KEY, 
		HttpdRealAvg INTEGER NOT NULL, 
		HttpdSharedAvg INTEGER NOT NULL,
		HttpdRealTot INTEGER NOT NULL,
		HttpdRunning INTEGER NOT NULL);");

	# Use an array instead of a hash to keep the column order. If you're
	# using MySQL, you may want to add an 'AFTER ColumnName' to the
	# definiton string. 'AFTER' is not supported by SQLite, so always add
	# new columns to the end of the array.
	my @dbcol = (
		{ 'name' => 'DateTimeAdded',  'definition' => 'DATE', },
		{ 'name' => 'HttpdRealAvg',   'definition' => 'INTEGER', },
		{ 'name' => 'HttpdSharedAvg', 'definition' => 'INTEGER', },
		{ 'name' => 'HttpdRealTot',   'definition' => 'INTEGER', },
		{ 'name' => 'HttpdRunning',   'definition' => 'INTEGER', },
	);
	my @dbidx = (
		{ 'name' => 'HttpdRealAvgIdx', 'table' => 'HttpdRealAvg', },
		{ 'name' => 'HttpdRunningIdx', 'table' => 'HttpdRunning', },
	); 
	# Use hashes to quickly define (and lookup) which tables/indexes already exist.
	my %dbcol_exists = ();
	my %dbidx_exists = ();
	for ( @{ $dbh->selectall_arrayref( "PRAGMA TABLE_INFO($dbtable)") } ) { $dbcol_exists{$_->[1]} = 1; };
	for ( @{ $dbh->selectall_arrayref( "PRAGMA INDEX_LIST($dbtable)") } ) { $dbidx_exists{$_->[1]} = 1; };

	# Create any missing columns.
	for my $col ( @dbcol ) {
		unless ( $dbcol_exists{$col->{'name'}} ) {
			print "DEBUG: Adding missing column $col->{'name'} as $col->{'definition'}.\n" if ( $opt{'debug'} );
			$dbh->do("ALTER TABLE $dbtable ADD COLUMN $col->{'name'} $col->{'definition'};");
			$dbh->do("UPDATE $dbtable SET $col->{'name'} = 0 WHERE $col->{'name'} = NULL;");
		}
	}

	# Create any missing indexes.
	for my $idx ( @dbidx ) {
		unless ( $dbidx_exists{$idx->{'name'}} ) {
			print "DEBUG: Adding missing index $idx->{'name'} for $idx->{'table'}.\n" if ( $opt{'debug'} );
			$dbh->do("CREATE INDEX $idx->{'name'} ON $dbtable ($idx->{'table'});");
		}
	}

	print "DEBUG: Removing DB rows older than $opt{'days'} days.\n" if ( $opt{'debug'} );
	$dbh->do("DELETE FROM $dbtable WHERE DateTimeAdded < DATETIME('NOW', '-$opt{'days'} DAYS');");

	if ( $opt{'max'} eq 'realavg' ) {

		print "DEBUG: Selecting largest HttpdRealAvg value in past $opt{'days'} days.\n" if ( $opt{'debug'} );
		( $dbrow{'DateTimeAdded'}, $dbrow{'HttpdRealAvg'}, $dbrow{'HttpdSharedAvg'}, $dbrow{'HttpdRealTot'}, $dbrow{'HttpdRunning'} ) = 
			$dbh->selectrow_array("SELECT DateTimeAdded, HttpdRealAvg, HttpdSharedAvg, HttpdRealTot, HttpdRunning
				FROM $dbtable ORDER BY HttpdRealAvg DESC, DateTimeAdded DESC LIMIT 1;");

	} elsif ( $opt{'max'} eq 'running' ) {

		print "DEBUG: Selecting largest HttpdRunning value in past $opt{'days'} days.\n" if ( $opt{'debug'} );
		( $dbrow{'DateTimeAdded'}, $dbrow{'HttpdRealAvg'}, $dbrow{'HttpdSharedAvg'}, $dbrow{'HttpdRealTot'}, $dbrow{'HttpdRunning'} ) = 
			$dbh->selectrow_array("SELECT DateTimeAdded, HttpdRealAvg, HttpdSharedAvg, HttpdRealTot, HttpdRunning
				FROM $dbtable ORDER BY HttpdRunning DESC, HttpdRealAvg DESC, DateTimeAdded DESC LIMIT 1;");
	}

	if ( $opt{'max'} && %dbrow ) {
		# make sure HttpdRunning (a column added later) has a value
		$dbrow{'HttpdRunning'} = 0 unless( $dbrow{'HttpdRunning'} );
		if ( $opt{'debug'} ) {
			print "DEBUG: DateTimeAdded=$dbrow{'DateTimeAdded'}\n";
			print "DEBUG: HttpdRealAvg=$dbrow{'HttpdRealAvg'}\n";
			print "DEBUG: HttpdSharedAvg=$dbrow{'HttpdSharedAvg'}\n";
			print "DEBUG: HttpdRealTot=$dbrow{'HttpdRealTot'}\n";
			print "DEBUG: HttpdRunning=$dbrow{'HttpdRunning'}\n";
		}
	}

}

# ---------------------------
# READ THE SERVER MEMORY INFO
# ---------------------------
#
print "DEBUG: Open /proc/meminfo\n" if ( $opt{'debug'} );
open ( my $mem_fh, "<", "/proc/meminfo" ) or die "ERROR: /proc/meminfo - $!\n";
while (<$mem_fh>) {
	if ( /^[[:space:]]*([a-zA-Z]+):[[:space:]]+([0-9]+)/) {
		if ( defined $mem{$1} ) {
			$mem{$1} = sprintf ( "%0.2f", $2 / 1024 );
			print "DEBUG: Found $1 = $mem{$1}.\n" if ( $opt{'debug'} );
		}
	}
}
close ( $mem_fh );

# -----------------------
# LOCATE THE HTTPD BINARY
# -----------------------
#
if ( defined $opt{'exe'} ) {
	$httpd{'EXE'} = $opt{'exe'};
	print "DEBUG: Using command-line exe \"$httpd{'EXE'}\".\n" if ( $opt{'debug'} );
} else {
	for ( @httpd_paths ) { 
		if ( $_ && -x $_ ) { 
			$httpd{'EXE'} = $_;
			print "DEBUG: Using httpd array exe \"$httpd{'EXE'}\".\n" if ( $opt{'debug'} );
			last;
		} 
	}
}
die "ERROR: No executable Apache HTTP binary found!\n"
	unless ( defined $httpd{'EXE'} && -x $httpd{'EXE'} );

# -----------------------------------------
# READ PROCESS INFORMATION FOR HTTPD BINARY
# -----------------------------------------
#
print "DEBUG: Opendir /proc\n" if ( $opt{'debug'} );
opendir ( my $proc_fh, "/proc" ) or die "ERROR: /proc - $!\n";
while ( my $pid = readdir( $proc_fh ) ) {
	my $exe = readlink( "/proc/$pid/exe" );
	next unless ( defined $exe );
	print "DEBUG: Readlink /proc/$pid/exe ($exe)" if ( $opt{'debug'} );
	if ( $exe eq $httpd{'EXE'} ) {
		print " - matched ($httpd{'EXE'})\n" if ( $opt{'debug'} );
		print "DEBUG: Open /proc/$pid/stat\n" if ( $opt{'debug'} );
		open ( my $stat_fh, "<", "/proc/$pid/stat" ) or die "ERROR: /proc/$pid/stat - $!\n";
		my @pid_stat = split (/ /, readline( $stat_fh )); close ( $stat_fh );

		print "DEBUG: Open /proc/$pid/statm\n" if ( $opt{'debug'} );
		open ( my $statm_fh, "<", "/proc/$pid/statm" ) or die "ERROR: /proc/$pid/statm - $!\n";
		my @pid_statm = split (/ /, readline( $statm_fh )); close ( $statm_fh );

		my %all_stats = ( 
			'pid' => $pid_stat[0],
			'name' => $pid_stat[1],
			'ppid' => $pid_stat[3],
			'rss' => $pid_stat[23] * $pagesize / 1024 / 1024,
			'share' => $pid_statm[2] * $pagesize / 1024 / 1024,
		);
		if ( $opt{'debug'} ) {
			print "DEBUG:";
			for (sort keys %all_stats) { print " $_:$all_stats{$_}"; }
			print "\n";
		}
		push ( @stathrefs, \%all_stats );
	} else { print "\n" if ( $opt{'debug'} ); }
}
close ( $proc_fh );
die "ERROR: No $httpd{'EXE'} processes found in /proc/*/exe! Are you root?\n" 
	unless ( @stathrefs );

# -------------------------------------
# READ THE HTTPD BINARY COMPILED VALUES 
# -------------------------------------
#
print "DEBUG: Open $httpd{'EXE'} -V\n" if ( $opt{'debug'} );
open ( my $set_fh, "-|", "$httpd{'EXE'} -V" ) or die "ERROR: $httpd{'EXE'} - $!\n";
while ( <$set_fh> ) {
	$httpd{'ROOT'} = $1 if (http://cdn6.static.surniaulula.com/^.*HTTPD_ROOT="(.*)"$/);
	$httpd{'CONFIG'} = $1 if (http://cdn5.static.surniaulula.com/^.*SERVER_CONFIG_FILE="(.*)"$/);
	$httpd{'VERSION'} = $1 if (/^Server version:[[:space:]]+Apache\/([0-9]\.[0-9]).*$/);
	$httpd{'MPM'} = lc($1) if (http://cdn5.static.surniaulula.com/^Server MPM:[[:space:]]+(.*)$/);
	$httpd{'MPM'} = lc($1) if (/APACHE_MPM_DIR="server\/mpm\/([^"]*)"$/);
}
close ( $set_fh );

if ( $opt{'debug'} ) {
	print "DEBUG: HTTPD ROOT = $httpd{'ROOT'}\n";
	print "DEBUG: HTTPD CONFIG = $httpd{'CONFIG'}\n";
	print "DEBUG: HTTPD VERSION = $httpd{'VERSION'}\n";
	print "DEBUG: HTTPD MPM = $httpd{'MPM'}\n";
}

$httpd{'CONFIG'} = "$httpd{'ROOT'}/$httpd{'CONFIG'}" 
	unless ( $httpd{'CONFIG'} =~ /^\// );

die "ERROR: Cannot determine httpd version number.\n" 
	unless ( $httpd{'VERSION'} && $httpd{'VERSION'} > 0 );

die "ERROR: Cannot determine httpd server MPM type.\n" 
	unless ( $httpd{'MPM'} );

# determine the config version number to use
if ( $cf_defaults{$httpd{'VERSION'}} ) {
	$cf_ver = $httpd{'VERSION'};
} elsif ( $httpd{'VERSION'} < $cf_min ) {
	$cf_ver = $cf_min;
	print "INFO: Httpd version $httpd{'VERSION'} not configured - using $cf_ver values instead.\n";
} else { 
	die "ERROR: Httpd version $httpd{'VERSION'} configuration values not defined.\n";
}

if ( $cf_defaults{$cf_ver}{$httpd{'MPM'}} ) { $cf_mpm = $httpd{'MPM'}; }
else { die "ERROR: Httpd server MPM \"$httpd{'MPM'}\" is unknown.\n"; }

# --------------------------
# READ THE HTTPD CONFIG FILE
# --------------------------
#
print "DEBUG: Open $httpd{'CONFIG'}\n" if ( $opt{'debug'} );
open ( my $conf_fh, "<", $httpd{'CONFIG'} ) or die "ERROR: $httpd{'CONFIG'} - $!\n";
my $conf = do { local $/; <$conf_fh> };
close ( $conf_fh );

# Read the MPM config values
if ( $conf =~ /^[[:space:]]*<IfModule ($cf_mpm\.c|mpm_$cf_mpm\_module)>([^<]*)/im ) {
	$cf_IfModule = $1; my $cf_Content = $2;
	print "DEBUG: IfModule $cf_IfModule\n$cf_Content\n" if ( $opt{'debug'} );
	for ( split (/\n/, $cf_Content) ) {
		if ( /^[[:space:]]*([a-zA-Z]+)[[:space:]]+([0-9]+)/) {
			print "DEBUG: $1 = $2\n" if ( $opt{'debug'} );
			$cf_read{$cf_ver}{$cf_mpm}{$1} = $2;
			$cf_changed{$cf_ver}{$cf_mpm}{$1} = $2;
		}
	}
}

if ( $cf_ver <= $cf_min ) {
	$cf_MaxName = 'MaxClients';
} else {
	$cf_MaxName = 'MaxRequestWorkers';
	my %dep = (
		'MaxClients' => 'MaxRequestWorkers',
		'MaxRequestsPerChild' => 'MaxConnectionsPerChild',
	);
	for ( sort keys %dep ) {
		if ( defined $cf_read{$cf_ver}{$cf_mpm}{$_} ) {
			print "INFO: $_($cf_read{$cf_ver}{$cf_mpm}{$_}) is deprecated - renaming to $dep{$_}.\n";
			$cf_read{$cf_ver}{$cf_mpm}{$dep{$_}} = $cf_read{$cf_ver}{$cf_mpm}{$_};
			$cf_changed{$cf_ver}{$cf_mpm}{$dep{$_}} = $cf_changed{$cf_ver}{$cf_mpm}{$_};
			delete $cf_read{$cf_ver}{$cf_mpm}{$_};
			delete $cf_changed{$cf_ver}{$cf_mpm}{$_};
		}
	}
}

# If using prefork MPM, base the caculation on MaxClients/MaxRequestWorkers instead of ServerLimit
# When using prefork, MaxClients/MaxRequestWorkers determines how many processes can be started
$cf_LimitName = $cf_mpm eq 'prefork' ? $cf_MaxName : 'ServerLimit';

# Exit with an error if any value is not > 0
for my $set ( sort keys %{$cf_changed{$cf_ver}{$cf_mpm}} ) {
	die "ERROR: $set value is 0 in $httpd{'CONFIG'}!\n" 
		unless ( $cf_changed{$cf_ver}{$cf_mpm}{$set} > 0 || 
			$set =~ /^(MaxRequestsPerChild|MaxConnectionsPerChild)$/ );
}

# -----------------------
# CALCULATE SIZE AVERAGES
# -----------------------
#
my @procs;
for my $stref ( @stathrefs ) {

	my $real = ${$stref}{'rss'} - ${$stref}{'share'};
	my $share = ${$stref}{'share'};
	my $proc_msg = sprintf ( " - %-22s: %7.2f MB / %6.2f MB shared", 
		"PID ${$stref}{'pid'} ${$stref}{'name'}", ${$stref}{'rss'}, $share );

	if ( ${$stref}{'ppid'} > 1 ) {
		$calcs{'HttpdRealAvg'} = $real if ( $calcs{'HttpdRealAvg'} == 0 );
		$calcs{'HttpdSharedAvg'} = $share if ( $calcs{'HttpdSharedAvg'} == 0 );
		$calcs{'HttpdRealAvg'} = ( $calcs{'HttpdRealAvg'} + $real ) / 2;
		$calcs{'HttpdSharedAvg'} = ( $calcs{'HttpdSharedAvg'} + $share ) / 2;
	} else {
		$proc_msg .= " [excluded from averages]";
	}
	$calcs{'HttpdRealTot'} += $real;
	print "DEBUG: $proc_msg\n" if ( $opt{'debug'} );
	print "DEBUG: Avg $calcs{'HttpdRealAvg'}, Shr $calcs{'HttpdSharedAvg'}, Tot $calcs{'HttpdRealTot'}\n" if ( $opt{'debug'} );
	push ( @procs, $proc_msg);
}

# round off the calcs
$calcs{'HttpdRealAvg'} = sprintf ( "%0.2f", $calcs{'HttpdRealAvg'} );
$calcs{'HttpdSharedAvg'} = sprintf ( "%0.2f", $calcs{'HttpdSharedAvg'} );
$calcs{'HttpdRealTot'} = sprintf ( "%0.2f", $calcs{'HttpdRealTot'} );
$calcs{'HttpdRunning'} = $#procs + 1;

# save the new averages to the database
if ( $opt{'save'} ) {
	if ( $opt{'debug'} ) {
		print "DEBUG: Adding to database: HttpdRealAvg($calcs{'HttpdRealAvg'}), ";
		print "HttpdSharedAvg($calcs{'HttpdSharedAvg'}), HttpdRealTot($calcs{'HttpdRealTot'}), ";
		print "HttpdRunning($calcs{'HttpdRunning'}).\n" 
	}
	my $sth = $dbh->prepare( "INSERT INTO $dbtable VALUES ( DATETIME('NOW'), ?, ?, ?, ? )" );
	$sth->execute( $calcs{'HttpdRealAvg'}, $calcs{'HttpdSharedAvg'}, $calcs{'HttpdRealTot'}, $calcs{'HttpdRunning'} );
	$sth->finish;
}
if ( $opt{'save'} || $opt{'days'} || $opt{'max'} ) {
	print "DEBUG: Disconnecting from database." if ( $opt{'debug'} );
	$dbh->disconnect;
}

# use max averages from database if --max used (and the database average is larger than current)
if ( $opt{'max'} eq 'realavg' && $dbrow{'HttpdRealAvg'} && $dbrow{'HttpdSharedAvg'} && $dbrow{'HttpdRealAvg'} > $calcs{'HttpdRealAvg'} ) {
	$mcs_from_db = " [Avg from $dbrow{'DateTimeAdded'}]";
	$calcs{'MaxLimitHttpdMem'} = $dbrow{'HttpdRealAvg'} * $cf_changed{$cf_ver}{$cf_mpm}{$cf_LimitName} + $dbrow{'HttpdSharedAvg'};
	print "DEBUG: DB HttpdRealAvg: $dbrow{'HttpdRealAvg'} > Current HttpdRealAvg: $calcs{'HttpdRealAvg'}.\n" if ( $opt{'debug'} );
} else {
	$calcs{'MaxLimitHttpdMem'} = $calcs{'HttpdRealAvg'} * $cf_changed{$cf_ver}{$cf_mpm}{$cf_LimitName} + $calcs{'HttpdSharedAvg'};
}

$calcs{'OtherProcsMem'} = $mem{'MemTotal'} - $mem{'Cached'} - $mem{'MemFree'} - $calcs{'HttpdRealTot'} - $calcs{'HttpdSharedAvg'};
$calcs{'FreeMemNoHttpd'} = $mem{'MemFree'} + $mem{'Cached'} + $calcs{'HttpdRealTot'} +  $calcs{'HttpdSharedAvg'};
$calcs{'AllProcsTotalMem'} = $calcs{'OtherProcsMem'} + $calcs{'MaxLimitHttpdMem'};

# ---------------------------------
# CALCULATE NEW HTTPD CONFIG VALUES
# ---------------------------------
#
$cf_changed{$cf_ver}{$cf_mpm}{'ServerLimit'} = sprintf ( "%0.2f", 
	( $mem{'MemFree'} + $mem{'Cached'} + $calcs{'HttpdRealTot'} + $calcs{'HttpdSharedAvg'} ) / $calcs{'HttpdRealAvg'} );

if ( $cf_mpm eq 'prefork' ) {
	$cf_changed{$cf_ver}{$cf_mpm}{$cf_MaxName} = $cf_changed{$cf_ver}{$cf_mpm}{'ServerLimit'};
} else {
	$cf_changed{$cf_ver}{$cf_mpm}{$cf_MaxName} =  sprintf ( "%0.2f",
		$cf_changed{$cf_ver}{$cf_mpm}{'ServerLimit'} * $cf_changed{$cf_ver}{$cf_mpm}{'ThreadsPerChild'} );
}

# ----------------------
# DISPLAY VERBOSE REPORT
# ----------------------
#
if ( $opt{'verbose'} ) {
	print "Httpd Binary\n\n";
	for ( sort keys %httpd ) { printf ( " - %-22s: %s\n", $_, $httpd{$_} ); }

	print "\nHttpd Processes\n\n";
	for ( @procs ) { print $_, "\n"; }
	print "\n";
	printf ( " - %-22s: %7.2f MB [excludes shared]\n", "HttpdRealAvg", $calcs{'HttpdRealAvg'} );
	printf ( " - %-22s: %7.2f MB\n", "HttpdSharedAvg", $calcs{'HttpdSharedAvg'} );
	printf ( " - %-22s: %7.2f MB [excludes shared]\n", "HttpdRealTot", $calcs{'HttpdRealTot'} );
	printf ( " - %-22s: %7.0f\n", "HttpdRunning", $calcs{'HttpdRunning'} );

	if ( $opt{'max'} && %dbrow ) {
		print "\nDatabase Values\n\n";
		printf ( " - DB %-19s: %s\n", "DateTimeAdded", $dbrow{'DateTimeAdded'} );
		printf ( " - DB %-19s: %7.2f MB [excludes shared]\n", "HttpdRealAvg", $dbrow{'HttpdRealAvg'} );
		printf ( " - DB %-19s: %7.2f MB\n", "HttpdSharedAvg", $dbrow{'HttpdSharedAvg'} );
		printf ( " - DB %-19s: %7.2f MB [excludes shared]\n", "HttpdRealTot", $dbrow{'HttpdRealTot'} );
		printf ( " - DB %-19s: %7.0f\n", "HttpdRunning", $dbrow{'HttpdRunning'} );
	}

	print "\nHttpd Config\n\n";
	# sort in reverse to make sure ServerLimit is before MaxClients
	for my $set ( reverse sort keys %{$cf_read{$cf_ver}{$cf_mpm}} ) {
		printf ( " - %-22s: %d\n", $set, $cf_read{$cf_ver}{$cf_mpm}{$set} );
	}
	print "\nServer Memory\n\n";
	for ( sort keys %mem ) { printf ( " - %-22s: %8.2f MB\n", $_, $mem{$_} ); }

	print "\nCalculations Summary\n\n";
	printf ( " - %-22s: %8.2f MB (MemTotal - Cached - MemFree - HttpdRealTot - HttpdSharedAvg)\n", "OtherProcsMem", $calcs{'OtherProcsMem'} );
	printf ( " - %-22s: %8.2f MB (MemFree + Cached + HttpdRealTot + HttpdSharedAvg)\n", "FreeMemNoHttpd", $calcs{'FreeMemNoHttpd'} );
	printf ( " - %-22s: %8.2f MB (HttpdRealAvg * $cf_LimitName + HttpdSharedAvg)%s\n", "MaxLimitHttpdMem", $calcs{'MaxLimitHttpdMem'}, $mcs_from_db );
	printf ( " - %-22s: %8.2f MB (OtherProcsMem + MaxLimitHttpdMem)\n", "AllProcsTotalMem", $calcs{'AllProcsTotalMem'} );

	print "\nMaximum Values for MemTotal ($mem{'MemTotal'} MB)\n\n";
	print "   <IfModule $cf_IfModule>\n";
	# sort in reverse to make sure ServerLimit is before MaxClients
	for my $set ( reverse sort keys %{$cf_changed{$cf_ver}{$cf_mpm}} ) {
		printf ( "\t%-22s %5.0f\t# ", $set, $cf_changed{$cf_ver}{$cf_mpm}{$set} );
		if ( $cf_read{$cf_ver}{$cf_mpm}{$set} != $cf_changed{$cf_ver}{$cf_mpm}{$set} ) {
			printf ( "(%0.0f -> %0.0f)", $cf_read{$cf_ver}{$cf_mpm}{$set}, $cf_changed{$cf_ver}{$cf_mpm}{$set} );
		} else { print "(no change)"; }

		if ( $cf_comments{$cf_ver}{$cf_mpm}{$set} ) {
			print " $cf_comments{$cf_ver}{$cf_mpm}{$set}" 
		} elsif ( $cf_defaults{$cf_ver}{$cf_mpm}{$set} ne '' ) {
			print " Default is $cf_defaults{$cf_ver}{$cf_mpm}{$set}" 
		}
		print "\n";
	}
	print "   </IfModule>\n";
	print "\nResult\n\n";
}

# ------------------------
# EXIT WITH RESULT MESSAGE
# ------------------------
#
my $result_prefix = sprintf ( "AllProcsTotalMem (%0.2f MB)$mcs_from_db", $calcs{'AllProcsTotalMem'} );
my $result_availram = "MemTotal ($mem{'MemTotal'} MB)";

if ( $calcs{'AllProcsTotalMem'} <= $mem{'MemTotal'} ) {

	print "OK: $result_prefix fits within $result_availram.\n";
	$err = 0;

} elsif ( $calcs{'AllProcsTotalMem'} <= ( $mem{'MemTotal'} + ( $mem{'SwapFree'} * $opt{'swappct'} / 100 ) ) ) {

	print "OK: $result_prefix exceeds $result_availram, but fits within $opt{'swappct'}% of free swap ";
	printf ( "(uses %0.2f MB of %0.0f MB).\n", $calcs{'AllProcsTotalMem'} - $mem{'MemTotal'}, $mem{'SwapFree'} );
	$err = 1;

} elsif ( $calcs{'AllProcsTotalMem'} <= ( $mem{'MemTotal'} + $mem{'SwapFree'} ) ) {

	print "WARNING: $result_prefix exceeds $result_availram, but still fits within free swap ";
	printf ( "(uses %0.2f MB of %0.0f MB).\n", $calcs{'AllProcsTotalMem'} - $mem{'MemTotal'}, $mem{'SwapFree'} );
	$err = 1;
} else {
	print "ERROR: $result_prefix exceeds $result_availram and free swap ($mem{'SwapFree'} MB) ";
	printf ( "by %0.2f MB.\n", $calcs{'AllProcsTotalMem'} - ( $mem{'MemTotal'} + $mem{'SwapFree'} ) );
	$err = 2;
}
print "\n" if ( $opt{'verbose'} );

if ( $opt{'debug'} ) {
	print "DEBUG: OtherProcsMem($calcs{'OtherProcsMem'}) + MaxLimitHttpdMem($calcs{'MaxLimitHttpdMem'})";
	print " = AllProcsTotalMem($calcs{'AllProcsTotalMem'}) vs MemTotal($mem{'MemTotal'}) + SwapFree($mem{'SwapFree'})\n";
}

exit $err;

# ---------------
# BEGIN FUNCTIONS
# ---------------
#
sub ShowUsage {
              #------------------------------------------------------------------------------
	print "\nPurpose:\n\n";
	print "This script will attempt to predict the memory used by Apache Httpd processes\n";
	print "when the maximum configured limits are reached. The prediction is based on the\n";
	print "(calculated) HttpdRealAvg value -- an average of the memory used by each\n";
	print "running Httpd process. To see the HttpdRealAvg value, and all other calculated\n";
	print "variables, use the \"verbose\" command-line argument. There are no additional\n";
	print "modules required, unless you use the save/days/max command-line argument(s).\n";
	print "\nSyntax:\n\n";
	print "$0 [--help] [--debug] [--verbose] \\\n";
	print "    [--exe=/path/to/httpd] [--swappct=#] --save] [--days=#] \\\n";
	print "    [--max=realavg|running]\n\n";
	printf ("%-15s: %s\n", "--help", "This syntax summary.");
	printf ("%-15s: %s\n", "--debug", "Show debugging messages as the script is executing.");
	printf ("%-15s: %s\n", "--verbose", "Display a detailed report of all values found and calculated.");
	printf ("%-15s: %s\n", "--exe=/path", "Path to httpd binary file (if non-standard).");
	printf ("%-15s: %s\n", "--swappct=#", "% of FREE swap use allowed before WARNING condition (default 0).");
	printf ("%-15s: %s\n", "--save", "Save average sizes to database ($dbname).");
	printf ("%-15s: %s\n", "--days=#", "Remove database entries older than # days (default 30).");
	printf ("%-15s: %s\n", "--max=realavg", "Use largest HttpdRealAvg size from current procs or database.");
	printf ("%-15s: %s\n", "--max=running", "Use HttpdRealAvg size from the largest MaxRunning recorded.");
              #------------------------------------------------------------------------------
	print "\nThe save/days/max command-line arguments require the DBD::SQLite perl module.\n";
	print "Use --max=running if the size and number of httpd processes increases and\n";
	print "decreases rapidly or unpredictably. The --max=realavg setting should be more\n";
	print "accurate for servers that have stable httpd sizes, and progressive increase /\n";
	print "decrease in the number of httpd processes.\n";
	print "\nExample:\n\n";
	print "/usr/local/bin/check_httpd_limits.pl --save --days=14 --max=realavg --swappct=25\n\n";
	exit $err;
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

#!/usr/bin/perl

# This script is free software; you can redistribute it and/or modify it under

# the terms of the GNU General Public License as published by the Free Software

# Foundation; either version 3 of the License, or (at your option) any later

# version.

# This script is distributed in the hope that it will be useful, but WITHOUT

# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS

# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more

# details at http://www.gnu.org/licenses/.

# Perl script to compare the size of running Apache httpd processes, the

# configured prefork/worker limits, and the available server memory. Exits with

# a warning or error message if the configured limits exceed the server's

# memory.

# Syntax: check_httpd_limits.pl --help

# The script performs the following tasks:

# - Reads the /proc/meminfo file for server memory values.

# - Reads the /proc/*/exe symbolic links to find the matching httpd binaries.

# - Reads the /proc/*/stat files for pid, process name, ppid, and rss.

# - Reads the /proc/*/statm files for the shared memory size.

# - Executes HTTP binary with "-V" to get the config file path and MPM info.

# - Reads the HTTP config file to get MPM (prefork or worker) settings.

# - Calculates the average and total HTTP process sizes, taking into account

# the shared memory used.

# - Calculates possible changes to MPM settings based on available memory and

# process sizes.

# - Displays all the values found and settings calculated if the --verbose

# parameter is used.

# - Exits with OK (0), WARNING (1), or ERROR (2) based on projected memory use

# with all (allowed) HTTP processes running.

# OK: Maximum number of HTTP processes fit within available RAM.

# WARNING: Maximum number of HTTP processes exceeds available RAM, but still

# fits within the free swap.

# ERROR: Maximum number of HTTP processes exceeds available RAM and swap.

# Changes:

# v2.4:

# - Added config for Apache Httpd v2.5 and 2.6 (identical to 2.4).

# - Added config for 'eventopt' MPM (identical to 'event' MPM).

use strict;

use warnings;

use POSIX;

use Getopt::Long;

no warnings 'once'; # no warning for $DBI::err

my $VERSION = '2.4';

my $pagesize = POSIX::sysconf(POSIX::_SC_PAGESIZE);

my @stathrefs;

my $err = 0;

my %mem = (

'MemTotal' => '',

'MemFree' => '',

'Cached' => '',

'SwapTotal' => '',

'SwapFree' => '',

);

my %httpd = (

'EXE' => '',

'ROOT' => '',

'CONFIG' => '',

'MPM' => '',

'VERSION' => '',

);

my $cf_IfModule = '';

my $cf_MaxName = ''; # defined based on httpd version (MaxClients or MaxRequestWorkers)

my $cf_LimitName = ''; # defined once MPM is determined (MaxClients/MaxRequestWorkers or ServerLimit)

my $cf_ver = '';

my $cf_min = '2.2';

my $cf_mpm = '';

my %cf_read = ();

my %cf_changed = ();

my %cf_defaults = (

'2.2' => {

'prefork' => {

'StartServers' => 5,

'MinSpareServers' => 5,

'MaxSpareServers' => 10,

'ServerLimit' => 256,

'MaxClients' => 256,

'MaxRequestsPerChild' => 10000,

'worker' => {

'StartServers' => 3,

'MinSpareThreads' => 75,

'MaxSpareThreads' => 250,

'ThreadsPerChild' => 25,

'ServerLimit' => 16,

'MaxClients' => 400,

'MaxRequestsPerChild' => 10000,

'2.4' => {

'prefork' => {

'StartServers' => 5,

'MinSpareServers' => 5,

'MaxSpareServers' => 10,

'ServerLimit' => 256,

'MaxRequestWorkers' => 256, # aka MaxClients

'MaxConnectionsPerChild' => 0, # aka MaxRequestsPerChild

'worker' => {

'StartServers' => 3,

'MinSpareThreads' => 75,

'MaxSpareThreads' => 250,

'ThreadsPerChild' => 25,

'ServerLimit' => 16,

'MaxRequestWorkers' => 400, # aka MaxClients

'MaxConnectionsPerChild' => 0, # aka MaxRequestsPerChild

);

$cf_defaults{'2.5'} = $cf_defaults{'2.4'};

$cf_defaults{'2.6'} = $cf_defaults{'2.5'};

# The event MPM config is identical to the worker MPM config

# Uses a hashref instead of copying the hash elements

for my $ver ( keys %cf_defaults ) {

$cf_defaults{$ver}{'event'} = $cf_defaults{$ver}{'worker'};

$cf_defaults{$ver}{'eventopt'} = $cf_defaults{$ver}{'event'};

}

# easiest way to copy the three-dimensional hash without using a module

for my $ver ( keys %cf_defaults ) {

for my $mpm ( keys %{$cf_defaults{$ver}} ) {

for my $el ( keys %{$cf_defaults{$ver}{$mpm}} ) {

$cf_read{$ver}{$mpm}{$el} = $cf_defaults{$ver}{$mpm}{$el};

$cf_changed{$ver}{$mpm}{$el} = $cf_defaults{$ver}{$mpm}{$el};

}

my %cf_comments = (

'2.2' => {

'prefork' => {

'ServerLimit' => 'MaxClients',

'MaxClients' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',

'worker' => {

'ServerLimit' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',

'MaxClients' => 'ServerLimit * ThreadsPerChild',

'2.4' => {

'prefork' => {

'ServerLimit' => 'MaxRequestWorkers',

'MaxRequestWorkers' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',

'worker' => {

'ServerLimit' => '(MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg',

'MaxRequestWorkers' => 'ServerLimit * ThreadsPerChild',

);

$cf_comments{'2.5'} = $cf_comments{'2.4'};

$cf_comments{'2.6'} = $cf_comments{'2.5'};

# the event MPM config is identical to the worker MPM config

# uses a hashref instead of copying the hash elements

for my $ver ( keys %cf_comments ) {

$cf_comments{$ver}{'event'} = $cf_comments{$ver}{'worker'};

$cf_comments{$ver}{'eventopt'} = $cf_comments{$ver}{'event'};

}

my %calcs = (

'HttpdRealAvg' => 0,

'HttpdSharedAvg' => 0,

'HttpdRealTot' => 0,

'HttpdRunning' => 0,

'OtherProcsMem' => '',

'FreeMemNoHttpd' => '',

'MaxLimitHttpdMem' => '',

'AllProcsTotalMem' => '',

);

# comment string when MaxLimitHttpdMem is calculated from DB values

my $mcs_from_db = '';

# common location for httpd binaries if not sepcified on command-line

my @httpd_paths = (

'/usr/sbin/httpd',

'/usr/local/sbin/httpd',

'/opt/apache/bin/httpd',

'/opt/apache/sbin/httpd',

'/usr/lib/apache2/mpm-prefork/apache2',

'/usr/sbin/apache2',

'/usr/local/sbin/apache2',

);

my $dbname = '/var/tmp/check_httpd_limits.sqlite';

my $dbuser = '';

my $dbpass = '';

my $dbtable = 'HttpdProcInfo';

my $dsn = "DBI:SQLite:dbname=$dbname";

my $dbh;

my %dbrow = (

'DateTimeAdded' => 0,

'HttpdRealAvg' => 0,

'HttpdSharedAvg' => 0,

'HttpdRealTot' => 0,

'HttpdRunning' => 0,

);

my %opt = ();

GetOptions(\%opt,

'help',

'debug',

'verbose',

'exe=s',

'swappct=i',

'save',

'days=i',

'max=s',

);

$opt{'swappct'} = 0 unless ( $opt{'swappct'} );

$opt{'max'} = $opt{'max'} ? lc($opt{'max'}) : "";

&ShowUsage() if ( $opt{'help'} );

if ( $opt{'verbose'} ) {

print "\nCheck Apache Httpd MPM Config Limits (Version $VERSION)\n";

print "by Jean-Sebastien Morisset - http://surniaulula.com/\n\n";

}

# READ MAXIMUM FROM DATABASE

if ( $opt{'save'} || $opt{'days'} || $opt{'max'} ) {

$opt{'days'} = 30 unless ( defined $opt{'days'} );

print "Saving Httpd Averages to $dsn\n\n"

if ( $opt{'save'} && $opt{'verbose'} );

require DBD::SQLite;

print "DEBUG: Connecting to database $dsn.\n" if ( $opt{'debug'} );

$dbh = DBI->connect($dsn, $dbuser, $dbpass);

die "ERROR: $DBI::errstr\n" if ($DBI::err);

$dbh->do("PRAGMA foreign_keys = ON;");

$dbh->do("CREATE TABLE IF NOT EXISTS $dbtable (

DateTimeAdded DATE PRIMARY KEY,

HttpdRealAvg INTEGER NOT NULL,

HttpdSharedAvg INTEGER NOT NULL,

HttpdRealTot INTEGER NOT NULL,

HttpdRunning INTEGER NOT NULL);");

# Use an array instead of a hash to keep the column order. If you're

# using MySQL, you may want to add an 'AFTER ColumnName' to the

# definiton string. 'AFTER' is not supported by SQLite, so always add

# new columns to the end of the array.

my @dbcol = (

{ 'name' => 'DateTimeAdded', 'definition' => 'DATE', },

{ 'name' => 'HttpdRealAvg', 'definition' => 'INTEGER', },

{ 'name' => 'HttpdSharedAvg', 'definition' => 'INTEGER', },

{ 'name' => 'HttpdRealTot', 'definition' => 'INTEGER', },

{ 'name' => 'HttpdRunning', 'definition' => 'INTEGER', },

);

my @dbidx = (

{ 'name' => 'HttpdRealAvgIdx', 'table' => 'HttpdRealAvg', },

{ 'name' => 'HttpdRunningIdx', 'table' => 'HttpdRunning', },

);

# Use hashes to quickly define (and lookup) which tables/indexes already exist.

my %dbcol_exists = ();

my %dbidx_exists = ();

for ( @{ $dbh->selectall_arrayref( "PRAGMA TABLE_INFO($dbtable)") } ) { $dbcol_exists{$_->[1]} = 1; };

for ( @{ $dbh->selectall_arrayref( "PRAGMA INDEX_LIST($dbtable)") } ) { $dbidx_exists{$_->[1]} = 1; };

# Create any missing columns.

for my $col ( @dbcol ) {

unless ( $dbcol_exists{$col->{'name'}} ) {

print "DEBUG: Adding missing column $col->{'name'} as $col->{'definition'}.\n" if ( $opt{'debug'} );

$dbh->do("ALTER TABLE $dbtable ADD COLUMN $col->{'name'} $col->{'definition'};");

$dbh->do("UPDATE $dbtable SET $col->{'name'} = 0 WHERE $col->{'name'} = NULL;");

}

# Create any missing indexes.

for my $idx ( @dbidx ) {

unless ( $dbidx_exists{$idx->{'name'}} ) {

print "DEBUG: Adding missing index $idx->{'name'} for $idx->{'table'}.\n" if ( $opt{'debug'} );

$dbh->do("CREATE INDEX $idx->{'name'} ON $dbtable ($idx->{'table'});");

}

print "DEBUG: Removing DB rows older than $opt{'days'} days.\n" if ( $opt{'debug'} );

$dbh->do("DELETE FROM $dbtable WHERE DateTimeAdded < DATETIME('NOW', '-$opt{'days'} DAYS');");

if ( $opt{'max'} eq 'realavg' ) {

print "DEBUG: Selecting largest HttpdRealAvg value in past $opt{'days'} days.\n" if ( $opt{'debug'} );

( $dbrow{'DateTimeAdded'}, $dbrow{'HttpdRealAvg'}, $dbrow{'HttpdSharedAvg'}, $dbrow{'HttpdRealTot'}, $dbrow{'HttpdRunning'} ) =

$dbh->selectrow_array("SELECT DateTimeAdded, HttpdRealAvg, HttpdSharedAvg, HttpdRealTot, HttpdRunning

FROM $dbtable ORDER BY HttpdRealAvg DESC, DateTimeAdded DESC LIMIT 1;");

} elsif ( $opt{'max'} eq 'running' ) {

print "DEBUG: Selecting largest HttpdRunning value in past $opt{'days'} days.\n" if ( $opt{'debug'} );

( $dbrow{'DateTimeAdded'}, $dbrow{'HttpdRealAvg'}, $dbrow{'HttpdSharedAvg'}, $dbrow{'HttpdRealTot'}, $dbrow{'HttpdRunning'} ) =

$dbh->selectrow_array("SELECT DateTimeAdded, HttpdRealAvg, HttpdSharedAvg, HttpdRealTot, HttpdRunning

FROM $dbtable ORDER BY HttpdRunning DESC, HttpdRealAvg DESC, DateTimeAdded DESC LIMIT 1;");

}

if ( $opt{'max'} && %dbrow ) {

# make sure HttpdRunning (a column added later) has a value

$dbrow{'HttpdRunning'} = 0 unless( $dbrow{'HttpdRunning'} );

if ( $opt{'debug'} ) {

print "DEBUG: DateTimeAdded=$dbrow{'DateTimeAdded'}\n";

print "DEBUG: HttpdRealAvg=$dbrow{'HttpdRealAvg'}\n";

print "DEBUG: HttpdSharedAvg=$dbrow{'HttpdSharedAvg'}\n";

print "DEBUG: HttpdRealTot=$dbrow{'HttpdRealTot'}\n";

print "DEBUG: HttpdRunning=$dbrow{'HttpdRunning'}\n";

}

# ---------------------------

# READ THE SERVER MEMORY INFO

# ---------------------------

print "DEBUG: Open /proc/meminfo\n" if ( $opt{'debug'} );

open ( my $mem_fh, "<", "/proc/meminfo" ) or die "ERROR: /proc/meminfo - $!\n";

while (<$mem_fh>) {

if ( /^[[:space:]]*([a-zA-Z]+):[[:space:]]+([0-9]+)/) {

if ( defined $mem{$1} ) {

$mem{$1} = sprintf ( "%0.2f", $2 / 1024 );

print "DEBUG: Found $1 = $mem{$1}.\n" if ( $opt{'debug'} );

}

close ( $mem_fh );

# -----------------------

# LOCATE THE HTTPD BINARY

# -----------------------

if ( defined $opt{'exe'} ) {

$httpd{'EXE'} = $opt{'exe'};

print "DEBUG: Using command-line exe \"$httpd{'EXE'}\".\n" if ( $opt{'debug'} );

} else {

for ( @httpd_paths ) {

if ( $_ && -x $_ ) {

$httpd{'EXE'} = $_;

print "DEBUG: Using httpd array exe \"$httpd{'EXE'}\".\n" if ( $opt{'debug'} );

last;

}

die "ERROR: No executable Apache HTTP binary found!\n"

unless ( defined $httpd{'EXE'} && -x $httpd{'EXE'} );

# -----------------------------------------

# READ PROCESS INFORMATION FOR HTTPD BINARY

# -----------------------------------------

print "DEBUG: Opendir /proc\n" if ( $opt{'debug'} );

opendir ( my $proc_fh, "/proc" ) or die "ERROR: /proc - $!\n";

while ( my $pid = readdir( $proc_fh ) ) {

my $exe = readlink( "/proc/$pid/exe" );

next unless ( defined $exe );

print "DEBUG: Readlink /proc/$pid/exe ($exe)" if ( $opt{'debug'} );

if ( $exe eq $httpd{'EXE'} ) {

print " - matched ($httpd{'EXE'})\n" if ( $opt{'debug'} );

print "DEBUG: Open /proc/$pid/stat\n" if ( $opt{'debug'} );

open ( my $stat_fh, "<", "/proc/$pid/stat" ) or die "ERROR: /proc/$pid/stat - $!\n";

my @pid_stat = split (/ /, readline( $stat_fh )); close ( $stat_fh );

print "DEBUG: Open /proc/$pid/statm\n" if ( $opt{'debug'} );

open ( my $statm_fh, "<", "/proc/$pid/statm" ) or die "ERROR: /proc/$pid/statm - $!\n";

my @pid_statm = split (/ /, readline( $statm_fh )); close ( $statm_fh );

my %all_stats = (

'pid' => $pid_stat[0],

'name' => $pid_stat[1],

'ppid' => $pid_stat[3],

'rss' => $pid_stat[23] * $pagesize / 1024 / 1024,

'share' => $pid_statm[2] * $pagesize / 1024 / 1024,

);

if ( $opt{'debug'} ) {

print "DEBUG:";

for (sort keys %all_stats) { print " $_:$all_stats{$_}"; }

print "\n";

}

push ( @stathrefs, \%all_stats );

} else { print "\n" if ( $opt{'debug'} ); }

}

close ( $proc_fh );

die "ERROR: No $httpd{'EXE'} processes found in /proc/*/exe! Are you root?\n"

unless ( @stathrefs );

# -------------------------------------

# READ THE HTTPD BINARY COMPILED VALUES

# -------------------------------------

print "DEBUG: Open $httpd{'EXE'} -V\n" if ( $opt{'debug'} );

open ( my $set_fh, "-|", "$httpd{'EXE'} -V" ) or die "ERROR: $httpd{'EXE'} - $!\n";

while ( <$set_fh> ) {

$httpd{'ROOT'} = $1 if (/^.*HTTPD_ROOT="(.*)"$/);

$httpd{'CONFIG'} = $1 if (/^.*SERVER_CONFIG_FILE="(.*)"$/);

$httpd{'VERSION'} = $1 if (/^Server version:[[:space:]]+Apache\/([0-9]\.[0-9]).*$/);

$httpd{'MPM'} = lc($1) if (/^Server MPM:[[:space:]]+(.*)$/);

$httpd{'MPM'} = lc($1) if (/APACHE_MPM_DIR="server\/mpm\/([^"]*)"$/);

}

close ( $set_fh );

if ( $opt{'debug'} ) {

print "DEBUG: HTTPD ROOT = $httpd{'ROOT'}\n";

print "DEBUG: HTTPD CONFIG = $httpd{'CONFIG'}\n";

print "DEBUG: HTTPD VERSION = $httpd{'VERSION'}\n";

print "DEBUG: HTTPD MPM = $httpd{'MPM'}\n";

}

$httpd{'CONFIG'} = "$httpd{'ROOT'}/$httpd{'CONFIG'}"

unless ( $httpd{'CONFIG'} =~ /^\// );

die "ERROR: Cannot determine httpd version number.\n"

unless ( $httpd{'VERSION'} && $httpd{'VERSION'} > 0 );

die "ERROR: Cannot determine httpd server MPM type.\n"

unless ( $httpd{'MPM'} );

# determine the config version number to use

if ( $cf_defaults{$httpd{'VERSION'}} ) {

$cf_ver = $httpd{'VERSION'};

} elsif ( $httpd{'VERSION'} < $cf_min ) {

$cf_ver = $cf_min;

print "INFO: Httpd version $httpd{'VERSION'} not configured - using $cf_ver values instead.\n";

} else {

die "ERROR: Httpd version $httpd{'VERSION'} configuration values not defined.\n";

}

if ( $cf_defaults{$cf_ver}{$httpd{'MPM'}} ) { $cf_mpm = $httpd{'MPM'}; }

else { die "ERROR: Httpd server MPM \"$httpd{'MPM'}\" is unknown.\n"; }

# --------------------------

# READ THE HTTPD CONFIG FILE

# --------------------------

print "DEBUG: Open $httpd{'CONFIG'}\n" if ( $opt{'debug'} );

open ( my $conf_fh, "<", $httpd{'CONFIG'} ) or die "ERROR: $httpd{'CONFIG'} - $!\n";

my $conf = do { local $/; <$conf_fh> };

close ( $conf_fh );

# Read the MPM config values

if ( $conf =~ /^[[:space:]]*<IfModule ($cf_mpm\.c|mpm_$cf_mpm\_module)>([^<]*)/im ) {

$cf_IfModule = $1; my $cf_Content = $2;

print "DEBUG: IfModule $cf_IfModule\n$cf_Content\n" if ( $opt{'debug'} );

for ( split (/\n/, $cf_Content) ) {

if ( /^[[:space:]]*([a-zA-Z]+)[[:space:]]+([0-9]+)/) {

print "DEBUG: $1 = $2\n" if ( $opt{'debug'} );

$cf_read{$cf_ver}{$cf_mpm}{$1} = $2;

$cf_changed{$cf_ver}{$cf_mpm}{$1} = $2;

}

if ( $cf_ver <= $cf_min ) {

$cf_MaxName = 'MaxClients';

} else {

$cf_MaxName = 'MaxRequestWorkers';

my %dep = (

'MaxClients' => 'MaxRequestWorkers',

'MaxRequestsPerChild' => 'MaxConnectionsPerChild',

);

for ( sort keys %dep ) {

if ( defined $cf_read{$cf_ver}{$cf_mpm}{$_} ) {

print "INFO: $_($cf_read{$cf_ver}{$cf_mpm}{$_}) is deprecated - renaming to $dep{$_}.\n";

$cf_read{$cf_ver}{$cf_mpm}{$dep{$_}} = $cf_read{$cf_ver}{$cf_mpm}{$_};

$cf_changed{$cf_ver}{$cf_mpm}{$dep{$_}} = $cf_changed{$cf_ver}{$cf_mpm}{$_};

delete $cf_read{$cf_ver}{$cf_mpm}{$_};

delete $cf_changed{$cf_ver}{$cf_mpm}{$_};

}

# If using prefork MPM, base the caculation on MaxClients/MaxRequestWorkers instead of ServerLimit

# When using prefork, MaxClients/MaxRequestWorkers determines how many processes can be started

$cf_LimitName = $cf_mpm eq 'prefork' ? $cf_MaxName : 'ServerLimit';

# Exit with an error if any value is not > 0

for my $set ( sort keys %{$cf_changed{$cf_ver}{$cf_mpm}} ) {

die "ERROR: $set value is 0 in $httpd{'CONFIG'}!\n"

unless ( $cf_changed{$cf_ver}{$cf_mpm}{$set} > 0 ||

$set =~ /^(MaxRequestsPerChild|MaxConnectionsPerChild)$/ );

}

# -----------------------

# CALCULATE SIZE AVERAGES

# -----------------------

my @procs;

for my $stref ( @stathrefs ) {

my $real = ${$stref}{'rss'} - ${$stref}{'share'};

my $share = ${$stref}{'share'};

my $proc_msg = sprintf ( " - %-22s: %7.2f MB / %6.2f MB shared",

"PID ${$stref}{'pid'} ${$stref}{'name'}", ${$stref}{'rss'}, $share );

if ( ${$stref}{'ppid'} > 1 ) {

$calcs{'HttpdRealAvg'} = $real if ( $calcs{'HttpdRealAvg'} == 0 );

$calcs{'HttpdSharedAvg'} = $share if ( $calcs{'HttpdSharedAvg'} == 0 );

$calcs{'HttpdRealAvg'} = ( $calcs{'HttpdRealAvg'} + $real ) / 2;

$calcs{'HttpdSharedAvg'} = ( $calcs{'HttpdSharedAvg'} + $share ) / 2;

} else {

$proc_msg .= " [excluded from averages]";

}

$calcs{'HttpdRealTot'} += $real;

print "DEBUG: $proc_msg\n" if ( $opt{'debug'} );

print "DEBUG: Avg $calcs{'HttpdRealAvg'}, Shr $calcs{'HttpdSharedAvg'}, Tot $calcs{'HttpdRealTot'}\n" if ( $opt{'debug'} );

push ( @procs, $proc_msg);

}

# round off the calcs

$calcs{'HttpdRealAvg'} = sprintf ( "%0.2f", $calcs{'HttpdRealAvg'} );

$calcs{'HttpdSharedAvg'} = sprintf ( "%0.2f", $calcs{'HttpdSharedAvg'} );

$calcs{'HttpdRealTot'} = sprintf ( "%0.2f", $calcs{'HttpdRealTot'} );

$calcs{'HttpdRunning'} = $#procs + 1;

# save the new averages to the database

if ( $opt{'save'} ) {

if ( $opt{'debug'} ) {

print "DEBUG: Adding to database: HttpdRealAvg($calcs{'HttpdRealAvg'}), ";

print "HttpdSharedAvg($calcs{'HttpdSharedAvg'}), HttpdRealTot($calcs{'HttpdRealTot'}), ";

print "HttpdRunning($calcs{'HttpdRunning'}).\n"

}

my $sth = $dbh->prepare( "INSERT INTO $dbtable VALUES ( DATETIME('NOW'), ?, ?, ?, ? )" );

$sth->execute( $calcs{'HttpdRealAvg'}, $calcs{'HttpdSharedAvg'}, $calcs{'HttpdRealTot'}, $calcs{'HttpdRunning'} );

$sth->finish;

}

if ( $opt{'save'} || $opt{'days'} || $opt{'max'} ) {

print "DEBUG: Disconnecting from database." if ( $opt{'debug'} );

$dbh->disconnect;

}

# use max averages from database if --max used (and the database average is larger than current)

if ( $opt{'max'} eq 'realavg' && $dbrow{'HttpdRealAvg'} && $dbrow{'HttpdSharedAvg'} && $dbrow{'HttpdRealAvg'} > $calcs{'HttpdRealAvg'} ) {

$mcs_from_db = " [Avg from $dbrow{'DateTimeAdded'}]";

$calcs{'MaxLimitHttpdMem'} = $dbrow{'HttpdRealAvg'} * $cf_changed{$cf_ver}{$cf_mpm}{$cf_LimitName} + $dbrow{'HttpdSharedAvg'};

print "DEBUG: DB HttpdRealAvg: $dbrow{'HttpdRealAvg'} > Current HttpdRealAvg: $calcs{'HttpdRealAvg'}.\n" if ( $opt{'debug'} );

} else {

$calcs{'MaxLimitHttpdMem'} = $calcs{'HttpdRealAvg'} * $cf_changed{$cf_ver}{$cf_mpm}{$cf_LimitName} + $calcs{'HttpdSharedAvg'};

}

$calcs{'OtherProcsMem'} = $mem{'MemTotal'} - $mem{'Cached'} - $mem{'MemFree'} - $calcs{'HttpdRealTot'} - $calcs{'HttpdSharedAvg'};

$calcs{'FreeMemNoHttpd'} = $mem{'MemFree'} + $mem{'Cached'} + $calcs{'HttpdRealTot'} + $calcs{'HttpdSharedAvg'};

$calcs{'AllProcsTotalMem'} = $calcs{'OtherProcsMem'} + $calcs{'MaxLimitHttpdMem'};

# ---------------------------------

# CALCULATE NEW HTTPD CONFIG VALUES

# ---------------------------------

$cf_changed{$cf_ver}{$cf_mpm}{'ServerLimit'} = sprintf ( "%0.2f",

( $mem{'MemFree'} + $mem{'Cached'} + $calcs{'HttpdRealTot'} + $calcs{'HttpdSharedAvg'} ) / $calcs{'HttpdRealAvg'} );

if ( $cf_mpm eq 'prefork' ) {

$cf_changed{$cf_ver}{$cf_mpm}{$cf_MaxName} = $cf_changed{$cf_ver}{$cf_mpm}{'ServerLimit'};

} else {

$cf_changed{$cf_ver}{$cf_mpm}{$cf_MaxName} = sprintf ( "%0.2f",

$cf_changed{$cf_ver}{$cf_mpm}{'ServerLimit'} * $cf_changed{$cf_ver}{$cf_mpm}{'ThreadsPerChild'} );

}

# ----------------------

# DISPLAY VERBOSE REPORT

# ----------------------

if ( $opt{'verbose'} ) {

print "Httpd Binary\n\n";

for ( sort keys %httpd ) { printf ( " - %-22s: %s\n", $_, $httpd{$_} ); }

print "\nHttpd Processes\n\n";

for ( @procs ) { print $_, "\n"; }

print "\n";

printf ( " - %-22s: %7.2f MB [excludes shared]\n", "HttpdRealAvg", $calcs{'HttpdRealAvg'} );

printf ( " - %-22s: %7.2f MB\n", "HttpdSharedAvg", $calcs{'HttpdSharedAvg'} );

printf ( " - %-22s: %7.2f MB [excludes shared]\n", "HttpdRealTot", $calcs{'HttpdRealTot'} );

printf ( " - %-22s: %7.0f\n", "HttpdRunning", $calcs{'HttpdRunning'} );

if ( $opt{'max'} && %dbrow ) {

print "\nDatabase Values\n\n";

printf ( " - DB %-19s: %s\n", "DateTimeAdded", $dbrow{'DateTimeAdded'} );

printf ( " - DB %-19s: %7.2f MB [excludes shared]\n", "HttpdRealAvg", $dbrow{'HttpdRealAvg'} );

printf ( " - DB %-19s: %7.2f MB\n", "HttpdSharedAvg", $dbrow{'HttpdSharedAvg'} );

printf ( " - DB %-19s: %7.2f MB [excludes shared]\n", "HttpdRealTot", $dbrow{'HttpdRealTot'} );

printf ( " - DB %-19s: %7.0f\n", "HttpdRunning", $dbrow{'HttpdRunning'} );

}

print "\nHttpd Config\n\n";

# sort in reverse to make sure ServerLimit is before MaxClients

for my $set ( reverse sort keys %{$cf_read{$cf_ver}{$cf_mpm}} ) {

printf ( " - %-22s: %d\n", $set, $cf_read{$cf_ver}{$cf_mpm}{$set} );

}

print "\nServer Memory\n\n";

for ( sort keys %mem ) { printf ( " - %-22s: %8.2f MB\n", $_, $mem{$_} ); }

print "\nCalculations Summary\n\n";

printf ( " - %-22s: %8.2f MB (MemTotal - Cached - MemFree - HttpdRealTot - HttpdSharedAvg)\n", "OtherProcsMem", $calcs{'OtherProcsMem'} );

printf ( " - %-22s: %8.2f MB (MemFree + Cached + HttpdRealTot + HttpdSharedAvg)\n", "FreeMemNoHttpd", $calcs{'FreeMemNoHttpd'} );

printf ( " - %-22s: %8.2f MB (HttpdRealAvg * $cf_LimitName + HttpdSharedAvg)%s\n", "MaxLimitHttpdMem", $calcs{'MaxLimitHttpdMem'}, $mcs_from_db );

printf ( " - %-22s: %8.2f MB (OtherProcsMem + MaxLimitHttpdMem)\n", "AllProcsTotalMem", $calcs{'AllProcsTotalMem'} );

print "\nMaximum Values for MemTotal ($mem{'MemTotal'} MB)\n\n";

print " <IfModule $cf_IfModule>\n";

# sort in reverse to make sure ServerLimit is before MaxClients

for my $set ( reverse sort keys %{$cf_changed{$cf_ver}{$cf_mpm}} ) {

printf ( "\t%-22s %5.0f\t# ", $set, $cf_changed{$cf_ver}{$cf_mpm}{$set} );

if ( $cf_read{$cf_ver}{$cf_mpm}{$set} != $cf_changed{$cf_ver}{$cf_mpm}{$set} ) {

printf ( "(%0.0f -> %0.0f)", $cf_read{$cf_ver}{$cf_mpm}{$set}, $cf_changed{$cf_ver}{$cf_mpm}{$set} );

} else { print "(no change)"; }

if ( $cf_comments{$cf_ver}{$cf_mpm}{$set} ) {

print " $cf_comments{$cf_ver}{$cf_mpm}{$set}"

} elsif ( $cf_defaults{$cf_ver}{$cf_mpm}{$set} ne '' ) {

print " Default is $cf_defaults{$cf_ver}{$cf_mpm}{$set}"

}

print "\n";

}

print " </IfModule>\n";

print "\nResult\n\n";

}

# ------------------------

# EXIT WITH RESULT MESSAGE

# ------------------------

my $result_prefix = sprintf ( "AllProcsTotalMem (%0.2f MB)$mcs_from_db", $calcs{'AllProcsTotalMem'} );

my $result_availram = "MemTotal ($mem{'MemTotal'} MB)";

if ( $calcs{'AllProcsTotalMem'} <= $mem{'MemTotal'} ) {

print "OK: $result_prefix fits within $result_availram.\n";

$err = 0;

} elsif ( $calcs{'AllProcsTotalMem'} <= ( $mem{'MemTotal'} + ( $mem{'SwapFree'} * $opt{'swappct'} / 100 ) ) ) {

print "OK: $result_prefix exceeds $result_availram, but fits within $opt{'swappct'}% of free swap ";

printf ( "(uses %0.2f MB of %0.0f MB).\n", $calcs{'AllProcsTotalMem'} - $mem{'MemTotal'}, $mem{'SwapFree'} );

$err = 1;

} elsif ( $calcs{'AllProcsTotalMem'} <= ( $mem{'MemTotal'} + $mem{'SwapFree'} ) ) {

print "WARNING: $result_prefix exceeds $result_availram, but still fits within free swap ";

printf ( "(uses %0.2f MB of %0.0f MB).\n", $calcs{'AllProcsTotalMem'} - $mem{'MemTotal'}, $mem{'SwapFree'} );

$err = 1;

} else {

print "ERROR: $result_prefix exceeds $result_availram and free swap ($mem{'SwapFree'} MB) ";

printf ( "by %0.2f MB.\n", $calcs{'AllProcsTotalMem'} - ( $mem{'MemTotal'} + $mem{'SwapFree'} ) );

$err = 2;

}

print "\n" if ( $opt{'verbose'} );

if ( $opt{'debug'} ) {

print "DEBUG: OtherProcsMem($calcs{'OtherProcsMem'}) + MaxLimitHttpdMem($calcs{'MaxLimitHttpdMem'})";

print " = AllProcsTotalMem($calcs{'AllProcsTotalMem'}) vs MemTotal($mem{'MemTotal'}) + SwapFree($mem{'SwapFree'})\n";

}

exit $err;

# ---------------

# BEGIN FUNCTIONS

# ---------------

sub ShowUsage {

#------------------------------------------------------------------------------

print "\nPurpose:\n\n";

print "This script will attempt to predict the memory used by Apache Httpd processes\n";

print "when the maximum configured limits are reached. The prediction is based on the\n";

print "(calculated) HttpdRealAvg value -- an average of the memory used by each\n";

print "running Httpd process. To see the HttpdRealAvg value, and all other calculated\n";

print "variables, use the \"verbose\" command-line argument. There are no additional\n";

print "modules required, unless you use the save/days/max command-line argument(s).\n";

print "\nSyntax:\n\n";

print "$0 [--help] [--debug] [--verbose] \\\n";

print " [--exe=/path/to/httpd] [--swappct=#] --save] [--days=#] \\\n";

print " [--max=realavg|running]\n\n";

printf ("%-15s: %s\n", "--help", "This syntax summary.");

printf ("%-15s: %s\n", "--debug", "Show debugging messages as the script is executing.");

printf ("%-15s: %s\n", "--verbose", "Display a detailed report of all values found and calculated.");

printf ("%-15s: %s\n", "--exe=/path", "Path to httpd binary file (if non-standard).");

printf ("%-15s: %s\n", "--swappct=#", "% of FREE swap use allowed before WARNING condition (default 0).");

printf ("%-15s: %s\n", "--save", "Save average sizes to database ($dbname).");

printf ("%-15s: %s\n", "--days=#", "Remove database entries older than # days (default 30).");

printf ("%-15s: %s\n", "--max=realavg", "Use largest HttpdRealAvg size from current procs or database.");

printf ("%-15s: %s\n", "--max=running", "Use HttpdRealAvg size from the largest MaxRunning recorded.");

#------------------------------------------------------------------------------

print "\nThe save/days/max command-line arguments require the DBD::SQLite perl module.\n";

print "Use --max=running if the size and number of httpd processes increases and\n";

print "decreases rapidly or unpredictably. The --max=realavg setting should be more\n";

print "accurate for servers that have stable httpd sizes, and progressive increase /\n";

print "decrease in the number of httpd processes.\n";

print "\nExample:\n\n";

print "/usr/local/bin/check_httpd_limits.pl --save --days=14 --max=realavg --swappct=25\n\n";

exit $err;

}

Download the check_httpd_limits.pl script here or
visit the check-httpd-limits project on Google Code.

Examples

Here are a few examples of check_httpd_limits.pl’s usage and screen output.


$ sudo ./check_httpd_limits.pl --help

Purpose:

This script will attempt to predict the memory used by Apache Httpd processes
when the maximum configured limits are reached. The prediction is based on the
(calculated) HttpdRealAvg value -- an average of the memory used by each
running Httpd process. To see the HttpdRealAvg value, and all other calculated
variables, use the "verbose" command-line argument. There are no additional
modules required, unless you use the save/days/max command-line argument(s).

Syntax:

./check_httpd_limits.pl [--help] [--debug] [--verbose] \
    [--exe=/path/to/httpd] [--swappct=#] --save] [--days=#] \
    [--max=realavg|running]

--help         : This syntax summary.
--debug        : Show debugging messages as the script is executing.
--verbose      : Display a detailed report of all values found and calculated.
--exe=/path    : Path to httpd binary file (if non-standard).
--swappct=#    : % of FREE swap use allowed before WARNING condition (default 0).
--save         : Save average sizes to database (/var/tmp/check_httpd_limits.sqlite).
--days=#       : Remove database entries older than # days (default 30).
--max=realavg  : Use largest HttpdRealAvg size from current procs or database.
--max=running  : Use HttpdRealAvg size from the largest MaxRunning recorded.

The save/days/max command-line arguments require the DBD::SQLite perl module.
Use --max=running if the size and number of httpd processes increases and
decreases rapidly or unpredictably. The --max=realavg setting should be more
accurate for servers that have stable httpd sizes, and progressive increase /
decrease in the number of httpd processes.

Example:

/usr/local/bin/check_httpd_limits.pl --save --days=14 --max=realavg --swappct=25

$ sudo ./check_httpd_limits.pl --help

Purpose:

This script will attempt to predict the memory used by Apache Httpd processes

when the maximum configured limits are reached. The prediction is based on the

(calculated) HttpdRealAvg value -- an average of the memory used by each

running Httpd process. To see the HttpdRealAvg value, and all other calculated

variables, use the "verbose" command-line argument. There are no additional

modules required, unless you use the save/days/max command-line argument(s).

Syntax:

./check_httpd_limits.pl [--help] [--debug] [--verbose] \

[--exe=/path/to/httpd] [--swappct=#] --save] [--days=#] \

[--max=realavg|running]

--help : This syntax summary.

--debug : Show debugging messages as the script is executing.

--verbose : Display a detailed report of all values found and calculated.

--exe=/path : Path to httpd binary file (if non-standard).

--swappct=# : % of FREE swap use allowed before WARNING condition (default 0).

--save : Save average sizes to database (/var/tmp/check_httpd_limits.sqlite).

--days=# : Remove database entries older than # days (default 30).

--max=realavg : Use largest HttpdRealAvg size from current procs or database.

--max=running : Use HttpdRealAvg size from the largest MaxRunning recorded.

The save/days/max command-line arguments require the DBD::SQLite perl module.

Use --max=running if the size and number of httpd processes increases and

decreases rapidly or unpredictably. The --max=realavg setting should be more

accurate for servers that have stable httpd sizes, and progressive increase /

decrease in the number of httpd processes.

Example:

/usr/local/bin/check_httpd_limits.pl --save --days=14 --max=realavg --swappct=25

An example of a prefork MPM with a MaxClients / ServerLimit that exceeds the available RAM, but still fits within the free swap space. The first execution generates a WARNING message, and the second uses the --swappct command-line option to allow up to 20% use of the free swap space before exiting with a WARNING.


$ sudo ./check_httpd_limits.pl
WARNING: AllProcsTotalMem (16736.13 MB) exceeds MemTotal (16040.57 MB), but still fits within free swap (uses 695.56 MB of 5984 MB).

$ sudo ./check_httpd_limits.pl --swappct=20
OK: AllProcsTotalMem (16735.64 MB) exceeds MemTotal (16040.57 MB), but fits within 20% of free swap (uses 695.07 MB of 5984 MB).

$ sudo ./check_httpd_limits.pl

WARNING: AllProcsTotalMem (16736.13 MB) exceeds MemTotal (16040.57 MB), but still fits within free swap (uses 695.56 MB of 5984 MB).

$ sudo ./check_httpd_limits.pl --swappct=20

OK: AllProcsTotalMem (16735.64 MB) exceeds MemTotal (16040.57 MB), but fits within 20% of free swap (uses 695.07 MB of 5984 MB).

An example of a prefork MPM with a MaxClients / ServerLimit that exceeds the available RAM and swap space. The --save/days/max command-line options are used to predict memory use based on the maximum process size averages, instead of just the current process list.


$ sudo ./check_httpd_limits.pl --save --days=30 --max=realavg
ERROR: AllProcsTotalMem (24679.18 MB) [Avg from 2012-11-28 19:05:01] exceeds MemTotal (16040.57 MB) and free swap (5983.86 MB) by 2654.75 MB.

$ sudo ./check_httpd_limits.pl --save --days=30 --max=realavg

ERROR: AllProcsTotalMem (24679.18 MB) [Avg from 2012-11-28 19:05:01] exceeds MemTotal (16040.57 MB) and free swap (5983.86 MB) by 2654.75 MB.

check_httpd_limits.pl can also be executed with --verbose to get detailed information on the httpd processes, configuration values, and the server’s memory. The MaxClients / ServerLimit in this example is low enough to fit within available RAM.


$ sudo ./check_httpd_limits.pl --verbose

Check Apache Httpd MPM Config Limits (Version 2.3)
by Jean-Sebastien Morisset - http://surniaulula.com/

Httpd Binary

 - CONFIG                : /etc/httpd/conf/httpd.conf
 - EXE                   : /usr/sbin/httpd
 - MPM                   : prefork
 - ROOT                  : /etc/httpd
 - VERSION               : 2.2

Httpd Processes

 - PID 3159 (httpd)      :  158.66 MB / 101.06 MB shared
 - PID 3951 (httpd)      :  141.54 MB /  90.79 MB shared
 - PID 25940 (httpd)     :   18.60 MB /   7.90 MB shared [excluded from averages]
 - PID 32674 (httpd)     :  184.47 MB /  99.45 MB shared
 - PID 32676 (httpd)     :  159.85 MB / 100.11 MB shared
 - PID 32678 (httpd)     :  178.31 MB / 119.04 MB shared
 - PID 32679 (httpd)     :  154.46 MB /  98.32 MB shared
 - PID 32680 (httpd)     :  170.37 MB / 106.61 MB shared
 - PID 32681 (httpd)     :  159.76 MB /  96.75 MB shared
 - PID 32687 (httpd)     :  159.78 MB / 102.75 MB shared
 - PID 32690 (httpd)     :  160.80 MB /  98.82 MB shared
 - PID 32691 (httpd)     :  166.86 MB / 103.38 MB shared
 - PID 32692 (httpd)     :  166.59 MB / 105.75 MB shared
 - PID 32693 (httpd)     :  155.40 MB /  98.08 MB shared
 - PID 32694 (httpd)     :  160.34 MB / 102.43 MB shared
 - PID 32695 (httpd)     :  161.81 MB / 106.31 MB shared
 - PID 32696 (httpd)     :  168.48 MB / 107.70 MB shared

 - HttpdRealAvg          :   58.93 MB [excludes shared]
 - HttpdSharedAvg        :  105.85 MB
 - HttpdRealTot          :  980.84 MB [excludes shared]
 - HttpdRunning          :      17

Httpd Config

 - StartServers          : 8
 - ServerLimit           : 220
 - MinSpareServers       : 8
 - MaxSpareServers       : 16
 - MaxRequestsPerChild   : 10000
 - MaxClients            : 220

Server Memory

 - Cached                : 13551.30 MB
 - MemFree               :    89.12 MB
 - MemTotal              : 16040.57 MB
 - SwapFree              :  5983.86 MB
 - SwapTotal             :  5983.99 MB

Calculations Summary

 - OtherProcsMem         :  1313.46 MB (MemTotal - Cached - MemFree - HttpdRealTot - HttpdSharedAvg)
 - FreeMemNoHttpd        : 14727.11 MB (MemFree + Cached + HttpdRealTot + HttpdSharedAvg)
 - MaxLimitHttpdMem      : 13070.45 MB (HttpdRealAvg * MaxClients + HttpdSharedAvg)
 - AllProcsTotalMem      : 14383.91 MB (OtherProcsMem + MaxLimitHttpdMem)

Maximum Values for MemTotal (16040.57 MB)

   <IfModule prefork.c>
        StartServers               8    # (no change) Default is 5
        ServerLimit              250    # (220 -> 250) MaxClients
        MinSpareServers            8    # (no change) Default is 5
        MaxSpareServers           16    # (no change) Default is 10
        MaxRequestsPerChild    10000    # (no change) Default is 10000
        MaxClients               250    # (220 -> 250) (MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg
   </IfModule>

Result

OK: AllProcsTotalMem (14383.91 MB) fits within MemTotal (16040.57 MB).

$ sudo ./check_httpd_limits.pl --verbose

Check Apache Httpd MPM Config Limits (Version 2.3)

by Jean-Sebastien Morisset - http://surniaulula.com/

Httpd Binary

- CONFIG : /etc/httpd/conf/httpd.conf

- EXE : /usr/sbin/httpd

- MPM : prefork

- ROOT : /etc/httpd

- VERSION : 2.2

Httpd Processes

- PID 3159 (httpd) : 158.66 MB / 101.06 MB shared

- PID 3951 (httpd) : 141.54 MB / 90.79 MB shared

- PID 25940 (httpd) : 18.60 MB / 7.90 MB shared [excluded from averages]

- PID 32674 (httpd) : 184.47 MB / 99.45 MB shared

- PID 32676 (httpd) : 159.85 MB / 100.11 MB shared

- PID 32678 (httpd) : 178.31 MB / 119.04 MB shared

- PID 32679 (httpd) : 154.46 MB / 98.32 MB shared

- PID 32680 (httpd) : 170.37 MB / 106.61 MB shared

- PID 32681 (httpd) : 159.76 MB / 96.75 MB shared

- PID 32687 (httpd) : 159.78 MB / 102.75 MB shared

- PID 32690 (httpd) : 160.80 MB / 98.82 MB shared

- PID 32691 (httpd) : 166.86 MB / 103.38 MB shared

- PID 32692 (httpd) : 166.59 MB / 105.75 MB shared

- PID 32693 (httpd) : 155.40 MB / 98.08 MB shared

- PID 32694 (httpd) : 160.34 MB / 102.43 MB shared

- PID 32695 (httpd) : 161.81 MB / 106.31 MB shared

- PID 32696 (httpd) : 168.48 MB / 107.70 MB shared

- HttpdRealAvg : 58.93 MB [excludes shared]

- HttpdSharedAvg : 105.85 MB

- HttpdRealTot : 980.84 MB [excludes shared]

- HttpdRunning : 17

Httpd Config

- StartServers : 8

- ServerLimit : 220

- MinSpareServers : 8

- MaxSpareServers : 16

- MaxRequestsPerChild : 10000

- MaxClients : 220

Server Memory

- Cached : 13551.30 MB

- MemFree : 89.12 MB

- MemTotal : 16040.57 MB

- SwapFree : 5983.86 MB

- SwapTotal : 5983.99 MB

Calculations Summary

- OtherProcsMem : 1313.46 MB (MemTotal - Cached - MemFree - HttpdRealTot - HttpdSharedAvg)

- FreeMemNoHttpd : 14727.11 MB (MemFree + Cached + HttpdRealTot + HttpdSharedAvg)

- MaxLimitHttpdMem : 13070.45 MB (HttpdRealAvg * MaxClients + HttpdSharedAvg)

- AllProcsTotalMem : 14383.91 MB (OtherProcsMem + MaxLimitHttpdMem)

Maximum Values for MemTotal (16040.57 MB)

StartServers 8 # (no change) Default is 5

ServerLimit 250 # (220 -> 250) MaxClients

MinSpareServers 8 # (no change) Default is 5

MaxSpareServers 16 # (no change) Default is 10

MaxRequestsPerChild 10000 # (no change) Default is 10000

MaxClients 250 # (220 -> 250) (MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg

</IfModule>

Result

OK: AllProcsTotalMem (14383.91 MB) fits within MemTotal (16040.57 MB).

Did you find this post useful? Share it with your circles / friends, or leave a quick note bellow.

Thank you,

js.

Sean Robertson says:

Thursday, April 11th, 2013 at 5:37 pm EDT

This is very cool. Hoping it’ll help me get my server running a bit more smoothly. One question, though: after making the changes recommended and running it again, I got different, significantly lower recommended MaxClients number – any idea why?

Jean-Sebastien Morisset says:

Friday, April 12th, 2013 at 8:19 am EDT

The script bases its calculations on the running processes (or the information it saves over time). Depending on when you started Apache, how many requests each process is allowed to service before it gets re-spawned (if ever), if your using PHP / APC, etc., and what your traffic is like, the calculated maximum values for the configuration file will change slightly.

BTW, if you’re using PHP, you might want to have a look at APC. I’ve found that it actually saved me memory and allowed me to run more Apache processes. ;-) See http://surniaulula.com/2012/11/22/save-memory-with-alternative-php-cache-apc/ for more info.

js.

Iguana50 says:

Tuesday, March 12th, 2013 at 3:28 pm EDT

I found this script to pretty neat. A couple of issues though.

The “plain text” view of the code inserts your CDN address into three lines, beginning on line 402. That threw me for a loop.
We do not use the default httpd.conf file, and unfortunately this script relies on this. I added the ability to specify the config file at the command line. I am sure I am not the only person that may find this useful. Here are my additions:

    210 GetOptions(\%opt,
    211         'help',
    212         'debug',
    213         'verbose',
    214         'exe=s',
    215         'config=s',
    216         'swappct=i',
    217         'save',
    218         'days=i',
    219         'max=s',
    220 );
...
    402 while (  ) {
    403         $httpd{'ROOT'} = $1 if (http://cdn6.static.surniaulula.com/^.*HTTPD_ROOT="(.*)"$/);
    404
    405         # -----------------------
    406         # LOCATE THE HTTPD CONFIG
    407         # -----------------------
    408         #
    409         if ( defined $opt{'config'} ) {
    410                 $httpd{'CONFIG'} = "conf/$opt{'config'}";
    411                 print "DEBUG: Using command-line config \"$httpd{'CONFIG'}\".\n" if ( $opt{'debug'} );
    412         } else {
    413                 $httpd{'CONFIG'} = $1 if (http://cdn5.static.surniaulula.com/^.*SERVER_CONFIG_FILE="(.*)"$/);
    414         }
    415
    416         #$httpd{'CONFIG'} = $1 if (http://cdn5.static.surniaulula.com/^.*SERVER_CONFIG_FILE="(.*)"$/);

210 GetOptions(\%opt,

211 'help',

212 'debug',

213 'verbose',

214 'exe=s',

215 'config=s',

216 'swappct=i',

217 'save',

218 'days=i',

219 'max=s',

220 );

...

402 while ( ) {

403 $httpd{'ROOT'} = $1 if (/^.*HTTPD_ROOT="(.*)"$/);

405 # -----------------------

406 # LOCATE THE HTTPD CONFIG

407 # -----------------------

408 #

409 if ( defined $opt{'config'} ) {

410 $httpd{'CONFIG'} = "conf/$opt{'config'}";

411 print "DEBUG: Using command-line config \"$httpd{'CONFIG'}\".\n" if ( $opt{'debug'} );

412 } else {

413 $httpd{'CONFIG'} = $1 if (/^.*SERVER_CONFIG_FILE="(.*)"$/);

414 }

416 #$httpd{'CONFIG'} = $1 if (http://cdn5.static.surniaulula.com/^.*SERVER_CONFIG_FILE="(.*)"$/);

Iguana50 says:

Tuesday, March 12th, 2013 at 3:32 pm EDT
Heh, that did not post well. Anyhow, I think you get the idea. Email me if you have a question.
- Jean-Sebastien Morisset says:
  
  Friday, April 12th, 2013 at 8:11 am EDT
  
  I fixed it for you (changed the “<code>>” to “<pre>” instead) and will look at applying these changes to the code shortly.
  
  Thanks for your suggestion.
  
  js.

Lantrix says:

Wednesday, December 19th, 2012 at 2:58 am EDT

A pretty useful script. Saved me already where I was running into SWAP.

Sam says:

Sunday, November 25th, 2012 at 6:54 pm EDT

# ./check_httpd_limits.pl ERROR: No /usr/sbin/apache2 processes found in /proc/*/exe! Are you root? # ps -ef | grep apache www-data 1034 10974 0 10:50 ? 00:00:00 /usr/sbin/apache … … .. /proc/10974# ls | grep exe exe

Jean-Sebastien Morisset says:

Sunday, November 25th, 2012 at 8:31 pm EDT
Sam, The script must have found a /usr/sbin/apache2 executable on disk, and is looking for it in /proc/*/exe. Since you’re using /usr/sbin/apache and not /usr/sbin/apache2, you’ll have to use the --exe=/usr/sbin/apache parameter to tell the script specifically which binary you’re using. Let me know if that helps. Thanks, js.

Surnia Ulula