LPE

The third post in my Nginx flow, or whatever.
A very good configuration example if you’re proxying requests to different backends and wants to “failover” if the backend server returns a 404 error code.

server {
        root /var/www/site1.com;
        server_name localhost;
 
        location / {
                error_page 404 = @try_backup_path;
                proxy_intercept_errors on;
                proxy_pass http://localhost:8000;                                                                                                                             
        }
 
        location @try_backup_path {
                proxy_pass http://localhost:8001;                                                                                                                             
        }
}

A small snippet I’m using to put sites in to maintenance mode, even though I admit it’s not something you should do in 2012, it’s still needed from time to time and this minimize the downtime atleast a bit.

Theoretically, if you put a file called ‘maintenance.html’ in to your wwwroot it automatically puts the site in maintenance mode and serves all remote addresses a 503 except the ones you specify in the geo section, and when you’re done with your update or whatever, just remove the file and everything is back to normal, no downtime or restarting daemons.

geo $maintenance_ip {
        default 0;
        your.ip.address.here 1;
        }
 
if ($maintenance_ip = 0) {
        set $maintenance y;
        }
 
if (-f /var/www/site1.com/maintenance.html) {
        set $maintenance y$maintenance;
        }
 
if ($maintenance = yy) {
         return 503;
        }
 
error_page 503 @maintenance;
         location @maintenance {
         rewrite ^(.*)$ /maintenance.html break;
        }

Nearly more standard then not, running Nginx infront of Apache to cache static content is today a very common way of lowering the load on your server, and it’s truly an extremely easy setup, which I’ll show.

Theoretically Nginx (listening on port 80) responds to every HTTP request coming in to the machine, it later decides if it should respond to it itself or make Apache do the response. If it decides to pass the request to Apache it sends it down to localhost (in this particular example, atleast) to Apache (listening on localhost and port 8080), gets a response back and then proxies it back out to the client.

Technically, it looks like this in nginx.conf.
We cache whatever is matched by the regex, which should be self explanatory.

server {
        listen 80;
        server_name site1.com;
        access_log /var/log/nginx/site1.com.log;
 
        # static content folders
        location ^~ /(images|css|js) {
                root /var/www/site1.com/current;
                access_log /var/log/nginx/site1.com.static.log;
        }
 
        # static content files
        location ~* \.(js|css|rdf|xml|ico|txt|jpg|gif|png|jpeg)$ {
                root /var/www/site1.com/current;
                access_log /var/log/nginx/site1.com.static.log;
        }
 
        # proxy the rest to apache
        location / {
 
            # proxy settings
            proxy_pass         http://127.0.0.1:8080/;
            proxy_redirect     off;
 
            proxy_set_header   Host             $host;
            proxy_set_header   X-Real-IP        $remote_addr;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
 
            client_max_body_size       10m;
            client_body_buffer_size    128k;
 
            proxy_connect_timeout      90;
            proxy_send_timeout         90;
            proxy_read_timeout         90;
 
            proxy_buffer_size          4k;
            proxy_buffers              4 32k;
            proxy_busy_buffers_size    64k;
            proxy_temp_file_write_size 64k;
        }

… And in httpd.conf.

 
NameVirtualHost *:8080
Listen 8080
 
<VirtualHost *:8080>
        ....
        ....
</VirtualHost>

A quite brain dead error message, but it basically means that your performance counters are fubar and this is the solution.

lodctr /R

On your host and then restart the agent, this command basically restarts the performance counters.

A few days ago we had a very peculiar situation at work regarding the file size of /var/log/lastlog and I decided to find out why.

This was the initial output that made me very confused:


[root@dev ~]# du -sh /var/log/lastlog
52K /var/log/lastlog
[root@dev ~]# ls -alh /var/log/lastlog
-rw-r--r-- 1 root root 85G Jan 11 14:52 /var/log/lastlog


As you can see, the file size clearly differs depending on what command I use, I also ran ‘df’ and since it reported that my partition was not bigger then 30G’s in size, and not even half of it was used, I understood that it wasn’t really a problem, but just something I hadn’t came across earlier.

After the regular minutes on Google and IRC, I quickly understood that it was a sparse file, and for those of you that aren’t familiar with sparse files, this is Wikipedias explanation which I found very fitting.

“In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently when blocks allocated to the file are mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual “empty” space which makes up the block, using less disk space. The full block size is written to disk as the actual size only when the block contains “real” (non-empty) data.”

After understanding this, my heart rate dropped back to normal, but I still wanted to find out why it was showing me such a size as 85GB, which is millions times more then just 52K.

After consulting with Peter van Dijk, who tends to have the answer to everything, I managed to understand why, and here it is.

This is a snippet from the lastlog source code (lastlog.c) which I hope is pretty self explanatory.


/*
* Read the right structure.
*/
fseek(fp, pwd->pw_uid * sizeof(struct lastlog), 0);
fread(&ll, sizeof(struct lastlog), 1, fp);


This means that the program takes the uid (type ‘id’ to find out) of your user, which in my case (connected to Active Directory through LikeWise Open) was 311428236, and multiply that with 292 bytes which is the size of the lastlog structure, and from there adds another 292 bytes, and there’s your final file size.

In short:

311428236*292+292 = 90937045204
And output from ls without -h (human readable out) is … you guessed it.
-rw-r–r– 1 root root 90937045204 Jan 11 17:00 lastlog

I hope this shed some light on why you suddenly find a huge file on your system and you don’t know why.
After reading up on this I’ve managed to realize that specifically lastlog is always a sparse file, they even mention it in the man page.

“NOTE
The lastlog file is a database which contains info on the last login of each user. You should not rotate it. It is a sparse file, so its size on the disk
is usually much smaller than the one shown by “ls -l” (which can indicate a really big file if you have in passwd users with a high UID). You can display
its real size with “ls -s”.”

Good luck.

Our current setup is an internal PDNS server with the MySQL back-end and three BIND slaves scattered across the country, and this is how we transfer zones from the master to the slaves.

On the master

 
<?php
ob_start();
if(!isset($_SERVER['PHP_AUTH_USER'])
        || $_SERVER['PHP_AUTH_USER'] !== '%CLIENTUSER%'
        || $_SERVER['PHP_AUTH_PW'] !== '%CLIENTPASSWD%') die();
 
if(!isset($_REQUEST['hostname'])
        || empty($_REQUEST['hostname'])) die();
 
$configTarget = $_REQUEST['hostname'];
 
$masters_default = array('master.company.com' => 'XXX.XXX.XXX.XXX');
 
$internal_ns = array(
        'ns1.company.com' => 'XXX.XXX.XXX.XXX',
        'ns2.company.com' => 'XXX.XXX.XXX.XXX',
        'ns3.company.com' => 'XXX.XXX.XXX.XXX',
);
$external_ns = array(
        'external.othercompany.com' => 'XXX.XXX.XXX.XXX',
);
 
$mysql = new mysqli('%DBHOST%','%DBUSER%','%DBPASSWD%','%DBNAME%',%DBPORT%);
if($mysql->connect_error) die($mysql->connect_error);
 
$sql = 'SELECT DISTINCT d.name,d.account,GROUP_CONCAT(r.content SEPARATOR \';\') AS ns'
        . ' FROM domains AS d'
        . ' RIGHT JOIN records AS r ON d.id=r.domain_id'
        . ' WHERE r.name=d.name AND r.type=\'NS\''
        . 'GROUP BY d.id';
$result = $mysql->query($sql);
 
/*
 * Internal named.conf
 */
$result->data_seek(0);
while($domain = $result->fetch_object()) {
        $ns = split(';', $domain->ns);
        printf("# Domain: %s\n", $domain->name);
        //printf("# Account: %s\n", empty($domain->account) ? 'n/a' : $domain->account);
        if(!in_array($configTarget, $ns)) {
                printf("# WARNING: %s not in %s\n", $configTarget, $domain->name);
        }
        $masters = $masters_default;
        $allow_transfer = $internal_ns;
        unset($allow_transfer[$configTarget]);
        foreach($external_ns as $exthost => $extip) {
                if(in_array($exthost, $ns)) {
                        $allow_transfer[$exthost] = $extip;
                }
        }
 
        printf("zone \"%s\" {\n", $domain->name);
        printf("\ttype slave;\n");
        printf("\tfile \"slaves/%s\";\n", str_replace('/', '_', $domain->name));
        printf("\tnotify no;\n");
        printf("\tmasters { %s; };\n", implode('; ', $masters));
        printf("\tallow-transfer { %s; };\n", implode('; ', $allow_transfer));
        printf("\tallow-notify { %s; };\n", implode('; ', $allow_transfer));
        printf("};\n");
}
 
$output = ob_get_contents();
ob_end_clean();
printf("# %s %s\n%s",
        sha1($output),
        date('Y-m-d H:i:s'),
        $output);
?>

On the slaves

#!/bin/bash
ts=`date +%s`
hostname=`hostname`
url="https://%CLIENTUSER%:%CLIENTPASSWD%@master.company.com/getconfig.php?hostname=${hostname}"
 
tmpfile=`mktemp /tmp/download.XXXX`
newconf=`mktemp /tmp/${hostname}.named.conf.XXXX`
sysconf="/etc/named/master-zones.conf"
 
curl --cacert /etc/pki/tls/certs/master.crt  -s "${url}" > "${tmpfile}"
if [[ "$?" != "0" ]]; then
  echo "download failed"
  rm -f "${tmpfile}" "${newconf}"
  exit 1
fi
tail -n+2 "${tmpfile}" > "${newconf}"
 
hash1=`head -n1 ${tmpfile} | cut -d' ' -f2`
hash2=`sha1sum ${newconf} | cut -d' ' -f1`
 
if [[ "${hash1}" == "${hash2}" ]]; then
  cmp -s ${sysconf} ${newconf}
  if [[ "$?" != "0" ]]; then
    mv "${sysconf}" "${sysconf}-${ts}"
    install -m640 -o root -g named "${newconf}" "${sysconf}"
 
    named-checkconf "${sysconf}"
    if [[ "$?" == "0" ]]; then
      rndc reload
      diff -u "${sysconf}-${ts}" "${sysconf}"
    else
      echo "named-checkconf failed, aborting update"
      mv "${sysconf}-${ts}" "${sysconf}"
    fi
  fi
else
  echo "HASH FAIL. aborted."
  echo "hash1 $hash1"
  echo "hash2 $hash2"
fi
rm -f "${tmpfile}" "${newconf}"

Include the config file on the slaves, put this in /etc/named.conf

include "/etc/named/master-zones.conf";

MAILTO=logwatch@company.com
*/5 * * * * root /usr/local/system_scripts/update-zones.sh

Hopefully it’s pretty self-explanatory, if not, leave a comment or drop a mail.

A horribly ugly hack to find snapshots on mounted volumes on a VMWare ESXi 4.1 host.

Create a script on random host with this

#!/bin/sh
for i in $(find /vmfs -name "*.vmsn");do eval $(echo $i | sed -n "s|^\(.*volumes\)/\([^/]*\)/\(.*\)$|ls -l \1 \| grep \2 \| grep '\\\->' \| awk '{ printf \$9 }';echo \/\3|p"); done

Then just run it with

ssh root@esxhost ./snapshotfinder.sh

And why this script just doesn’t do a find / -name “*.vmsn” is ‘couse the output won’t include the symlinks name but merely the UID of the volume, and that doesn’t help very much. And unfortunately ESX comes with a busybox binary of find, which doesn’t have the -L option.

OpenVPN have a whole armada of neat features to discover, but in this post we’ll be shedding a light on the most basic one, connect a client to a server.

Install OpenVPN on the server (you need EPEL for this, search for EPEL on this site)

yum install openvpn

Move the scripts to a more proper location.
This step is optional, but as said before, it “feels” better”

cp -r /usr/share/openvpn/easy-rsa/2.0/ /etc/openvpn/ca/

Edit the file named ‘vars’ and edit the variables at the bottom to fit your company and situation

export KEY_COUNTRY="SE"
export KEY_PROVINCE="X"
export KEY_CITY="Gavle"
export KEY_ORG="yourcompany"
export KEY_EMAIL="you@yourcompany.com"

Source the script and if you want to make sure it worked type ‘export’ and check for those variables

. vars

Clean up

./clean-all

Now generate the necessary certificates

./build-ca
./build-key-server servername
./build-key clientname

Edit your configuration (/etc/openvpn/server.conf), an example configuration could look like this,
if you have questions about these options you may browse the official OpenVPN site.

port 10001
proto udp
dev tun
ca ca/keys/ca.crt
cert ca/keys/servername.crt
key ca/keys/servername.key  # This file should be kept secret
dh dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3

Generate Diffie-Hellman parameters

./build-dh

Now start the service

service openvpn start

Now on to the Windows side of things, ie. the dark side.

Install OpenVPN from OpenVPN
Copy ca.crt, client.crt and client.key to your config folder.

Create a file called client.ovpn and edit it after this template

ca ca.crt
cert client.crt
key client.key
comp-lzo
client
dev tun
proto udp
remote 10.0.0.1 10001 # this beeing the IP to the centosserver as well as port
resolv-retry infinite
nobind
persist-key
persist-tun
verb 3
route-method exe

Now rightclick your file and choose “Start with OpenVPN” and you should be done.

In short, EPEL is an extra repository outside of CentOS and Redhats regular repositories.
It’s mostly based on Fedora packages and works for both CentOS, Redhat and Scientific Linux.

It’s extremely simple to use, merely one command.
Pick your correct architecture and version from the EPEL website (for example, http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm) and then type this and you’re done:

yum install --nogpg http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

A very short tutorial on how to set up an IPSec VPN between two hosts to make the traffic encrypted.
As usual, my weapon of choice would be CentOS with some help from EPEL.
(Having trouble understanding what EPEL is or how to configure it? See http://mute.nu/2011/what-is-epel-and-how-to-use-it-on-centos/)

Some prerequisites are that both machines are either on the same subnet, have public routeable addresses or is behind firewalls with IPSec Passthrough.

Basically, Openswan works in a “left” and “right” manner, meaning that one machine is called “left” and the other “right”.
The machine we’ll start configuring is “left” in this guide.

At first, install the necessary packages on both machines.

yum install openswan nss-tools

We’ll be configuring this without passwords for simplicity, so we remove the old database files and regenerate new ones.

cd /etc/ipsec.d/
mkdir db_old
mv *.db db_old/
certutil -N -d .

Now create the necessary keys on both machines but make sure to change the hostname to the proper hostname of the server.

ipsec newhostkey --output /etc/ipsec.d/HOSTNAME.secrets --configdir /etc/ipsec.d/ --verbose

Now remove the comment on the last line of /etc/ipsec.conf so it includes /etc/ipsec.d/*.conf on both servers.

Now create a file on your left server called HOSTNAME.conf after this template and edit the fields for your setup.
Type in ipsec showhostkey –left and copy that key to leftrsasigkey=
as well as ipsec showhostkey –right on your other server and copy that key to rightsasigkey=.

conn host-to-host
    left=10.0.0.11
    leftid=@server1
    leftrsasigkey=0dsaA2...
    leftnexthop=%defaultroute
    right=10.0.0.12
    rightid=@server22
    rightrsasigkey=0sAQOuP...
    rightnexthop=%defaultroute
    auto=add

Copy that file to the other server and rename it.

scp /etc/ipsec.d/server1.conf root@10.0.0.12:/etc/ipsec.d/server2.conf

Now start it.

service ipsec start

If you encounter any errors they will most probably be printed directly to you or you can have a see in /var/log/messages

If you want to test your connection out, try a ping from server1 to server2 and capture it using tcpdump on server2.

tcpdump -ni eth0 -X -v host 10.0.0.12

You should now see every other packet using the ESP protocol and the rest TCP.
Remember, you’ll only see the encrypted outgoing packets.

If you want it to start on boot you either change “auto=add” to “auto=start” in HOSTNAME.conf or type in “chkconfig ipsec on”