LPE

A few days ago we had a very peculiar situation at work regarding the file size of /var/log/lastlog and I decided to find out why.

This was the initial output that made me very confused:


[root@dev ~]# du -sh /var/log/lastlog
52K /var/log/lastlog
[root@dev ~]# ls -alh /var/log/lastlog
-rw-r--r-- 1 root root 85G Jan 11 14:52 /var/log/lastlog


As you can see, the file size clearly differs depending on what command I use, I also ran ‘df’ and since it reported that my partition was not bigger then 30G’s in size, and not even half of it was used, I understood that it wasn’t really a problem, but just something I hadn’t came across earlier.

After the regular minutes on Google and IRC, I quickly understood that it was a sparse file, and for those of you that aren’t familiar with sparse files, this is Wikipedias explanation which I found very fitting.

“In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently when blocks allocated to the file are mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual “empty” space which makes up the block, using less disk space. The full block size is written to disk as the actual size only when the block contains “real” (non-empty) data.”

After understanding this, my heart rate dropped back to normal, but I still wanted to find out why it was showing me such a size as 85GB, which is millions times more then just 52K.

After consulting with Peter van Dijk, who tends to have the answer to everything, I managed to understand why, and here it is.

This is a snippet from the lastlog source code (lastlog.c) which I hope is pretty self explanatory.


/*
* Read the right structure.
*/
fseek(fp, pwd->pw_uid * sizeof(struct lastlog), 0);
fread(&ll, sizeof(struct lastlog), 1, fp);


This means that the program takes the uid (type ‘id’ to find out) of your user, which in my case (connected to Active Directory through LikeWise Open) was 311428236, and multiply that with 292 bytes which is the size of the lastlog structure, and from there adds another 292 bytes, and there’s your final file size.

In short:

311428236*292+292 = 90937045204
And output from ls without -h (human readable out) is … you guessed it.
-rw-r–r– 1 root root 90937045204 Jan 11 17:00 lastlog

I hope this shed some light on why you suddenly find a huge file on your system and you don’t know why.
After reading up on this I’ve managed to realize that specifically lastlog is always a sparse file, they even mention it in the man page.

“NOTE
The lastlog file is a database which contains info on the last login of each user. You should not rotate it. It is a sparse file, so its size on the disk
is usually much smaller than the one shown by “ls -l” (which can indicate a really big file if you have in passwd users with a high UID). You can display
its real size with “ls -s”.”

Good luck.

Our current setup is an internal PDNS server with the MySQL back-end and three BIND slaves scattered across the country, and this is how we transfer zones from the master to the slaves.

On the master

 
<?php
ob_start();
if(!isset($_SERVER['PHP_AUTH_USER'])
        || $_SERVER['PHP_AUTH_USER'] !== '%CLIENTUSER%'
        || $_SERVER['PHP_AUTH_PW'] !== '%CLIENTPASSWD%') die();
 
if(!isset($_REQUEST['hostname'])
        || empty($_REQUEST['hostname'])) die();
 
$configTarget = $_REQUEST['hostname'];
 
$masters_default = array('master.company.com' => 'XXX.XXX.XXX.XXX');
 
$internal_ns = array(
        'ns1.company.com' => 'XXX.XXX.XXX.XXX',
        'ns2.company.com' => 'XXX.XXX.XXX.XXX',
        'ns3.company.com' => 'XXX.XXX.XXX.XXX',
);
$external_ns = array(
        'external.othercompany.com' => 'XXX.XXX.XXX.XXX',
);
 
$mysql = new mysqli('%DBHOST%','%DBUSER%','%DBPASSWD%','%DBNAME%',%DBPORT%);
if($mysql->connect_error) die($mysql->connect_error);
 
$sql = 'SELECT DISTINCT d.name,d.account,GROUP_CONCAT(r.content SEPARATOR \';\') AS ns'
        . ' FROM domains AS d'
        . ' RIGHT JOIN records AS r ON d.id=r.domain_id'
        . ' WHERE r.name=d.name AND r.type=\'NS\''
        . 'GROUP BY d.id';
$result = $mysql->query($sql);
 
/*
 * Internal named.conf
 */
$result->data_seek(0);
while($domain = $result->fetch_object()) {
        $ns = split(';', $domain->ns);
        printf("# Domain: %s\n", $domain->name);
        //printf("# Account: %s\n", empty($domain->account) ? 'n/a' : $domain->account);
        if(!in_array($configTarget, $ns)) {
                printf("# WARNING: %s not in %s\n", $configTarget, $domain->name);
        }
        $masters = $masters_default;
        $allow_transfer = $internal_ns;
        unset($allow_transfer[$configTarget]);
        foreach($external_ns as $exthost => $extip) {
                if(in_array($exthost, $ns)) {
                        $allow_transfer[$exthost] = $extip;
                }
        }
 
        printf("zone \"%s\" {\n", $domain->name);
        printf("\ttype slave;\n");
        printf("\tfile \"slaves/%s\";\n", str_replace('/', '_', $domain->name));
        printf("\tnotify no;\n");
        printf("\tmasters { %s; };\n", implode('; ', $masters));
        printf("\tallow-transfer { %s; };\n", implode('; ', $allow_transfer));
        printf("\tallow-notify { %s; };\n", implode('; ', $allow_transfer));
        printf("};\n");
}
 
$output = ob_get_contents();
ob_end_clean();
printf("# %s %s\n%s",
        sha1($output),
        date('Y-m-d H:i:s'),
        $output);
?>

On the slaves

#!/bin/bash
ts=`date +%s`
hostname=`hostname`
url="https://%CLIENTUSER%:%CLIENTPASSWD%@master.company.com/getconfig.php?hostname=${hostname}"
 
tmpfile=`mktemp /tmp/download.XXXX`
newconf=`mktemp /tmp/${hostname}.named.conf.XXXX`
sysconf="/etc/named/master-zones.conf"
 
curl --cacert /etc/pki/tls/certs/master.crt  -s "${url}" > "${tmpfile}"
if [[ "$?" != "0" ]]; then
  echo "download failed"
  rm -f "${tmpfile}" "${newconf}"
  exit 1
fi
tail -n+2 "${tmpfile}" > "${newconf}"
 
hash1=`head -n1 ${tmpfile} | cut -d' ' -f2`
hash2=`sha1sum ${newconf} | cut -d' ' -f1`
 
if [[ "${hash1}" == "${hash2}" ]]; then
  cmp -s ${sysconf} ${newconf}
  if [[ "$?" != "0" ]]; then
    mv "${sysconf}" "${sysconf}-${ts}"
    install -m640 -o root -g named "${newconf}" "${sysconf}"
 
    named-checkconf "${sysconf}"
    if [[ "$?" == "0" ]]; then
      rndc reload
      diff -u "${sysconf}-${ts}" "${sysconf}"
    else
      echo "named-checkconf failed, aborting update"
      mv "${sysconf}-${ts}" "${sysconf}"
    fi
  fi
else
  echo "HASH FAIL. aborted."
  echo "hash1 $hash1"
  echo "hash2 $hash2"
fi
rm -f "${tmpfile}" "${newconf}"

Include the config file on the slaves, put this in /etc/named.conf

include "/etc/named/master-zones.conf";

MAILTO=logwatch@company.com
*/5 * * * * root /usr/local/system_scripts/update-zones.sh

Hopefully it’s pretty self-explanatory, if not, leave a comment or drop a mail.

A horribly ugly hack to find snapshots on mounted volumes on a VMWare ESXi 4.1 host.

Create a script on random host with this

#!/bin/sh
for i in $(find /vmfs -name "*.vmsn");do eval $(echo $i | sed -n "s|^\(.*volumes\)/\([^/]*\)/\(.*\)$|ls -l \1 \| grep \2 \| grep '\\\->' \| awk '{ printf \$9 }';echo \/\3|p"); done

Then just run it with

ssh root@esxhost ./snapshotfinder.sh

And why this script just doesn’t do a find / -name “*.vmsn” is ‘couse the output won’t include the symlinks name but merely the UID of the volume, and that doesn’t help very much. And unfortunately ESX comes with a busybox binary of find, which doesn’t have the -L option.

OpenVPN have a whole armada of neat features to discover, but in this post we’ll be shedding a light on the most basic one, connect a client to a server.

Install OpenVPN on the server (you need EPEL for this, search for EPEL on this site)

yum install openvpn

Move the scripts to a more proper location.
This step is optional, but as said before, it “feels” better”

cp -r /usr/share/openvpn/easy-rsa/2.0/ /etc/openvpn/ca/

Edit the file named ‘vars’ and edit the variables at the bottom to fit your company and situation

export KEY_COUNTRY="SE"
export KEY_PROVINCE="X"
export KEY_CITY="Gavle"
export KEY_ORG="yourcompany"
export KEY_EMAIL="you@yourcompany.com"

Source the script and if you want to make sure it worked type ‘export’ and check for those variables

. vars

Clean up

./clean-all

Now generate the necessary certificates

./build-ca
./build-key-server servername
./build-key clientname

Edit your configuration (/etc/openvpn/server.conf), an example configuration could look like this,
if you have questions about these options you may browse the official OpenVPN site.

port 10001
proto udp
dev tun
ca ca/keys/ca.crt
cert ca/keys/servername.crt
key ca/keys/servername.key  # This file should be kept secret
dh dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3

Generate Diffie-Hellman parameters

./build-dh

Now start the service

service openvpn start

Now on to the Windows side of things, ie. the dark side.

Install OpenVPN from OpenVPN
Copy ca.crt, client.crt and client.key to your config folder.

Create a file called client.ovpn and edit it after this template

ca ca.crt
cert client.crt
key client.key
comp-lzo
client
dev tun
proto udp
remote 10.0.0.1 10001 # this beeing the IP to the centosserver as well as port
resolv-retry infinite
nobind
persist-key
persist-tun
verb 3
route-method exe

Now rightclick your file and choose “Start with OpenVPN” and you should be done.

In short, EPEL is an extra repository outside of CentOS and Redhats regular repositories.
It’s mostly based on Fedora packages and works for both CentOS, Redhat and Scientific Linux.

It’s extremely simple to use, merely one command.
Pick your correct architecture and version from the EPEL website (for example, http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm) and then type this and you’re done:

yum install --nogpg http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

A very short tutorial on how to set up an IPSec VPN between two hosts to make the traffic encrypted.
As usual, my weapon of choice would be CentOS with some help from EPEL.
(Having trouble understanding what EPEL is or how to configure it? See http://mute.nu/2011/what-is-epel-and-how-to-use-it-on-centos/)

Some prerequisites are that both machines are either on the same subnet, have public routeable addresses or is behind firewalls with IPSec Passthrough.

Basically, Openswan works in a “left” and “right” manner, meaning that one machine is called “left” and the other “right”.
The machine we’ll start configuring is “left” in this guide.

At first, install the necessary packages on both machines.

yum install openswan nss-tools

We’ll be configuring this without passwords for simplicity, so we remove the old database files and regenerate new ones.

cd /etc/ipsec.d/
mkdir db_old
mv *.db db_old/
certutil -N -d .

Now create the necessary keys on both machines but make sure to change the hostname to the proper hostname of the server.

ipsec newhostkey --output /etc/ipsec.d/HOSTNAME.secrets --configdir /etc/ipsec.d/ --verbose

Now remove the comment on the last line of /etc/ipsec.conf so it includes /etc/ipsec.d/*.conf on both servers.

Now create a file on your left server called HOSTNAME.conf after this template and edit the fields for your setup.
Type in ipsec showhostkey –left and copy that key to leftrsasigkey=
as well as ipsec showhostkey –right on your other server and copy that key to rightsasigkey=.

conn host-to-host
    left=10.0.0.11
    leftid=@server1
    leftrsasigkey=0dsaA2...
    leftnexthop=%defaultroute
    right=10.0.0.12
    rightid=@server22
    rightrsasigkey=0sAQOuP...
    rightnexthop=%defaultroute
    auto=add

Copy that file to the other server and rename it.

scp /etc/ipsec.d/server1.conf root@10.0.0.12:/etc/ipsec.d/server2.conf

Now start it.

service ipsec start

If you encounter any errors they will most probably be printed directly to you or you can have a see in /var/log/messages

If you want to test your connection out, try a ping from server1 to server2 and capture it using tcpdump on server2.

tcpdump -ni eth0 -X -v host 10.0.0.12

You should now see every other packet using the ESP protocol and the rest TCP.
Remember, you’ll only see the encrypted outgoing packets.

If you want it to start on boot you either change “auto=add” to “auto=start” in HOSTNAME.conf or type in “chkconfig ipsec on”

Download the correct version of Java to your computer.

Java Download URL

Run the binary

./jre-6u27-linux-x64.bin

Move the files to a better location.
This step is optional, but in my opinion it sort of “feels” better.

mv jre1.6.0_27/ /usr/lib/jvm/

And now link it to Chromium.

ln -sf /usr/lib/jvm/jre1.6.0_27/lib/amd64/libnpjp2.so /usr/lib/chromium-browser/plugins/

And now restart your browser, voila!

I came across this today when a customer called in and asked why he could browse to a website that we’ve yet to make available to the public.

Our scenario today is that we allowed 4 specific IP-address and then chose “Deny” under “Edit Feature Settings”, but then I also added localhost, and I added it as a range using a prefix.
Adding 127.0.0.0 with 8 as prefix makes IIS discard the rule you set in “Edit Feature Settings”.

If you write /8 instead of just 8, it gives you this text in the error message:

'/8' is an invalid subnet mask. It must be a valid IP address or an integer value between 0-32 for IPv4 and 0-128 for IPv6"

So obviously you’re supposed to be able to write 8 and get a proper mask (255.0.0.0).
On the other hand, if you add the mask 255.0.0.0 instead of the prefix, it works as intended.

Hopefully this will be fixed sooner or later.

A smaller batchscript to show current logged on remote users using ‘query session’ in Windows, I suspect it’s quite self explanatory.
This is set for three servers named server-01, server-02 and server-03.

@ECHO OFF
setlocal EnableDelayedExpansion
 
:: PRINT SESSION OUTPUTS FROM TERMINAL SERVERS TO TEMPORARY SESSION FILES
FOR /L %%G IN (1,1,3) DO query session /server:server-ts0%%G > sess.tmp%%G
 
:: COUNTING MATCHING ROWS FOR "tcp-rdp" IN TEMPORARY SESSION FILE
FOR %%G IN (1,2,3) DO (
	set "cmd=findstr /R /N "^.*rdp-tcp.*$" sess.tmp%%G | FIND /C ":""
	FOR /F %%B in ('!cmd!') DO (echo SERVER-TS0%%G: %%B)
)
 
:: DELETE TEMPORARY SESSION FILES
DEL sess.tmp*
 
PAUSE

Output will be

SERVER-01: 35
SERVER-02: 48
SERVER-03: 12

A few regular expressions proven quite handy in day-to-day work, creds to kll at http://labs.spritelink.net/

Matching IPv4:
for example: 192.0.2.1

(((2(5[0-5]|[0-4][0-9])|[01]?[0-9][0-9]?)\.){3}(2(5[0-5]|[0-4][0-9])|[01]?[0-9][0-9]?))

Matching IPv4 CIDR slash-notation:
for example: 192.0.2.1/32

(((2(5[0-5]|[0-4][0-9])|[01]?[0-9][0-9]?)\.){3}(2(5[0-5]|[0-4][0-9])|[01]?[0-9][0-9]?)(/(3[12]|[12]?[0-9])))

Matching IPv6:

/((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))
|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:
[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:
[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:
[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:
[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})
|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?/

The regex matches the following IPv6 address forms. Note that these are all the same address:

fe80:0000:0000:0000:0204:61ff:fe9d:f156 // full form of IPv6
fe80:0:0:0:204:61ff:fe9d:f156 // drop leading zeroes
fe80::204:61ff:fe9d:f156 // collapse multiple zeroes to :: in the IPv6 address
fe80:0000:0000:0000:0204:61ff:254.157.241.086 // IPv4 dotted quad at the end
fe80:0:0:0:0204:61ff:254.157.241.86 // drop leading zeroes, IPv4 dotted quad at the end
fe80::204:61ff:254.157.241.86 // dotted quad at the end, multiple zeroes collapsed

In addition, the regular expression matches these IPv6 forms:

::1 // localhost
fe80:: // link-local prefix
2000:: // global unicast prefix

One could optionally append the following to match on a slash-notation style prefix.

(/[0-9][0-9]?|1([01][0-9]|2[0-8]))?