wjd.nu

2009-12-28 - mysql ubuntu readline delete-char / karmic

A couple of months ago, I wrote about how to get the delete key (and reverse history search) to work again on mysql-client-5.1 that comes with Ubuntu/Jaunty.

That trick doesn't work anymore with Ubuntu/Karmic and here's why:

As you can see from these highlighted differences between mysql 5.1.31 and 5.1.37 (the versions used in Jaunty and Karmic respectively), you can see that it previously tested for the issetugid(2) syscall but ignored it if it doesn't exist. At some point after 5.1.31 did the mysql guys change that to returning early if the function doesn't exist.

This means that in Ubuntu/Karmic which doesn't have an issetugid function (availability seems limited to BSD's and Solarises) the .editrc file from the homedir will not be read.

So.. how to fix this? There are three places where a committed fix will turn up in Karmic: (1) in the mysql source, (2) in the debian package, (3) in the ubuntu package. The fix itself is simple enough: emulate issetugid with syscalls that do exist. Google turns up plenty of examples where getuid is compared against geteuid and getgid is compared against getegid. If one of those doesn't match, issetugid returns nonzero (the value 1).

Something like this patch should to the trick. I have it compiled into a nice ubuntu package (for amd64 only!):
mysql-client-5.1_5.1.37-1ubuntu5wjd1_amd64.deb

References:
Debian bug 552003: debian mysql-client uses libedit instead of libreadline
MySQL bug 49967: built-in libedit doesn't read .editrc on linux
Ubuntu bug 451801: some keys like the delete-char do not work in mysql-client

Things to remember:
$ apt-cache source / showsrc ...
$ apt-get build-dep ...
$ dch -l wjd
$ DEB_BUILD_OPTIONS=nocheck debuild -us -uc
$ dpkg -x / -e ... path
$ dpkg -b path ...

2009-11-30 - unoverridable language default / ubuntu

Just now I logged on to a fresh Ubuntu server install for work. He who installed it had chosen a Dutch locale as default, which is fine, as long as my ssh environment can override it. It couldn't.

The ssh (on my client) and sshd (on the server) were okay:

• SendEnv LANG LC_* in ssh_config and,
• AcceptEnv LANG LC_* in sshd_config.

The /etc/environment file on the server was clean, so that couldn't break anything either.

A quick search on the web yielded a bug from 2005 which is not fixed, it seems. Javier Serrano Polo was kind enough to provide a working fix:

Edit /etc/default/locale and replace "LANG=nl_NL.UTF-8" with "LANG DEFAULT=nl_NL.UTF-8 OVERRIDE=${LANG}".

2009-11-04 - ssh and sshd trojaned

So one of my systems has had customized password-logging ssh and sshd applications running for quite a while. Yes, I've been r00ted :-(

How did I find out?

When using svn+ssh protocol subversion repositories, I observed the "Killed by signal 15" message as reported in debian bug #366391, like this:

$ svn up
At revision 44.
Killed by signal 15.

The message is harmless and is caused by a specific combination of the versions of subversion and ssh. However, according to the bugreport this should've been fixed in openssh version 1:4.2p1-6 and I was running latest debian/lenny (openssh-client version 1:5.1p1-5).

That's odd. So I decided to strace the app to find out more. There I noticed how one of the processes opened /usr/lib/libv.o and wrote to it.

That's even more odd. Why should I as a normal user have write powers in /usr/lib? When you think about it, /usr should during normal operations even be read only for root. Why is this a .o (object file for static linking) file? And, worst of all, why hasn't google heard of /usr/lib/libv.o?

Looking at the data in /usr/lib/libv.o, it was not human readable, yet it used only a slight subset of the available character set. Perhaps a very trivial obfuscation method? I took the first 80 bytes and checked if I could add an integer and get something readable. No? Okay, deducted by an integer then? Yes, success.

$ od -t x1 /usr/lib/libv.o | head -n8
0000000 96 91 c5 df ca c7 d1 cd cd cd d1 ce ce d1 cd df
0000020 f6 c5 df 8d 90 90 8b df f6 c5 df 9b 9a 8b 9e 96
0000040 91 9a 9a 8c f5 96 91 c5 df ca c7 d1 cd cd cd d1
0000060 ce ce d1 cd df f6 c5 df 8d 90 90 8b df f6 c5 df
0000100 8f 9e 8d 9e 8c 9e 96 93 96 91 98 f5 96 91 c5 df
0000120 ca c7 d1 cd cd cd d1 ce ce d1 cd df f6 c5 df 8d
0000140 90 90 8b df f6 c5 df 8c 97 96 96 8b 9e 94 9a 8c
0000160 f5 96 91 c5 df ca c7 d1 cd cd cd d1 ce ce d1 cd
$ dd if=/usr/lib/libv.o of=garbled.dat count=1 bs=80
$ for x in `seq 1 255` ; do echo -n "$x: "; perl -pe 's/(.)/chr((ord($1)-'$x')%256)/ge' garbled.dat | cat -A ; echo ; done
<only garbage>
$ for x in `seq 1 255` ; do echo -n "$x: "; perl -pe 's/(.)/chr(('$x'-ord($1))%256)/ge' garbled.dat | cat -A ; echo ; done
<skipping 1 through 252>
253: gl8^^36,000,//,0^^^G8^^pmmr^^^G8^^bcr_glccq^Hgl8^^36,000,//,0^^^G8^^pmmr^^^G8^^n_p_q_gjgle^Hgl8^^
254: hm9^_47-111-00-1^_^H9^_qnns^_^H9^_cds`hmddr^Ihm9^_47-111-00-1^_^H9^_qnns^_^H9^_o`q`r`hkhmf^Ihm9^_
255: in: 58.222.11.2 ^I: root ^I: detainees$
in: 58.222.11.2 ^I: root ^I: parasailing$
in: 

Drat.. that looks like passwords. And mine were in there too. And yes, both ssh and sshd seemed to be the culprits:

$ strings /usr/bin/ssh | grep '^out:'
out: %s   : %s  : %s
out: %s   : %s  : %s  (%s)
$ strings /usr/sbin/sshd | grep '^in:'
in: %s  : %s  : %s
in: %s  : %s  : %s  (%s)

Had the "Killed by signal 15" issue not alerted me, I might not have found out that I was running a trojaned ssh. The behaviour of ssh being different from its supposed version, gave it away. This can also be observed in these following help and version messages:

$ /usr/bin/ssh -V
OpenSSH_5.1p1, OpenSSL 0.9.8g 19 Oct 2007
$ /usr/bin/ssh.safe -V
OpenSSH_5.1p1 Debian-5, OpenSSL 0.9.8g 19 Oct 2007
$ /usr/sbin/sshd -V
sshd: illegal option -- V
OpenSSH_5.1p1, OpenSSL 0.9.8g 19 Oct 2007
usage: sshd [-46Ddeiqt] [-b bits] [-f config_file] [-g login_grace_time]
            [-h host_key_file] [-k key_gen_time] [-o option] [-p port] [-u len]
$ /usr/sbin/sshd.safe -V
sshd: illegal option -- V
OpenSSH_5.1p1 Debian-5, OpenSSL 0.9.8g 19 Oct 2007
usage: sshd [-46DdeiqTt] [-b bits] [-C connection_spec] [-f config_file]
            [-g login_grace_time] [-h host_key_file] [-k key_gen_time]
            [-o option] [-p port] [-u len]

Other tell-tale signs — besides the world-writable libv.o and the messages being different — were extra ext2 attributes on the ssh binaries, statically linked files (the programs being larger but having less dependencies):

$ find /usr -type f -perm -o+w
/usr/lib/libv.o
$ lsattr /usr/bin/scp /usr/bin/sftp /usr/bin/ssh /usr/bin/ssh-keygen /usr/sbin/sshd
s---ia------------- /usr/bin/scp
s---ia------------- /usr/bin/sftp
s---ia------------- /usr/bin/ssh
s---ia------------- /usr/bin/ssh-keygen
s---ia------------- /usr/sbin/sshd
$ ls -l /usr/bin/sshd /usr/bin/sshd.safe
-rwxrwxr-x 1 root root 1424243 2009-01-14 02:07 /usr/sbin/sshd
-rwxr-xr-x 1 root root  474640 2009-01-14 02:07 /usr/sbin/sshd.safe
$ ldd /usr/sbin/sshd | wc -l
11
$ ldd /usr/sbin/sshd.safe | wc -l
20

Further inspection of the ssh binary shows both "/usr/lib/libv.o" in the .rodata section, and embedded in the code (.text section):

$ objdump -D /usr/bin/ssh | grep movb
...
  410fe2: c6 84 24 b0 01 00 00  movb   $0x2f,0x1b0(%rsp)
  410fea: c6 84 24 b1 01 00 00  movb   $0x75,0x1b1(%rsp)
  410ff7: c6 84 24 b2 01 00 00  movb   $0x73,0x1b2(%rsp)
  410fff: c6 84 24 b3 01 00 00  movb   $0x72,0x1b3(%rsp)
  411007: c6 84 24 b4 01 00 00  movb   $0x2f,0x1b4(%rsp)
  41100f: c6 84 24 b5 01 00 00  movb   $0x6c,0x1b5(%rsp)
  411017: c6 84 24 b6 01 00 00  movb   $0x69,0x1b6(%rsp)
  41101f: c6 84 24 b7 01 00 00  movb   $0x62,0x1b7(%rsp)
  411027: c6 84 24 b8 01 00 00  movb   $0x2f,0x1b8(%rsp)
  41102f: c6 84 24 b9 01 00 00  movb   $0x6c,0x1b9(%rsp)
  411037: c6 84 24 ba 01 00 00  movb   $0x69,0x1ba(%rsp)
  41103f: c6 84 24 bb 01 00 00  movb   $0x62,0x1bb(%rsp)
  411047: c6 84 24 bc 01 00 00  movb   $0x76,0x1bc(%rsp)
  41104f: c6 84 24 bd 01 00 00  movb   $0x2e,0x1bd(%rsp)
  411057: c6 84 24 be 01 00 00  movb   $0x6f,0x1be(%rsp)
...
$ objdump -D /usr/bin/ssh \
  | sed -e '/[[:blank:]]movb/!d;s/.*movb[[:blank:]]*\$\(0x[0-9a-f]\{1,2\}\).*/\1/' \
  | uniq | perl -pe 's/(.*)/chr(eval $1)/e' | tr -d '\n' | cat -A ; echo
^@^A^@^A^@^@man^@mkdir^@/usr/lib/libv.o^@man^@mkdir^@/usr/lib/libv.o^@man^@mkdir^@/usr/lib/libv.o^@Z
^@^E^@^A^@aeiouybcdfghklmnprstvzx^@-^@ A^@^A^B^@^A^@^A^@=^@=a^@/A^@\^@\^\s\r\f\v\n\t\b\a\0\-M?^@\^@9
^_M-^C^@@^@M-d^H^@

Very unpleasant, all of this. But a good incentive to dive into examining binaries ;-)

2009-10-18 - rtpproxy with opensips / nat traversal

OpenSIPS combined with RtpProxy can be used to relay traffic between a SIP provider on the WAN and your SIP PBX on the LAN. The steps:

Download, compile and install rtpproxy. Then do the following to create an application level gateway user:

# useradd --system --home-dir /var/run/alg --create-home --skel /dev/null alg

Apply these patches to the sysinit files (debian) using patch -l. The DAEMON_OPTS in /etc/default/rtpproxy should have the external IP first, and the internal IP second.

--- ~/src/rtpproxy-1.2.0/debian/rtpproxy-default.ex 2007-11-29 01:46:26.000000000 +0100
+++ /etc/default/rtpproxy 2009-10-18 20:56:39.000000000 +0200
@@ -7,4 +7,4 @@
 #
 
 # Additional options that are passed to the Daemon.
-DAEMON_OPTS=""
+DAEMON_OPTS="-l 1.2.3.4/192.168.1.1 -s unix:/var/run/alg/rtpproxy.sock -d WARN"

--- ~/src/rtpproxy-1.2.0/debian/rtpproxy.init 2007-11-29 01:46:26.000000000 +0100
+++ /etc/init.d/rtpproxy  2009-10-18 14:34:19.000000000 +0200
@@ -11,9 +11,11 @@
 #
 
 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
-DAEMON=/usr/bin/rtpproxy
+DAEMON=/usr/local/bin/rtpproxy
 NAME=rtpproxy
 DESC=rtpproxy
+USER=alg
+PIDFILE=/var/run/alg/$NAME.pid
 
 test -x $DAEMON || exit 0
 
@@ -29,13 +31,13 @@
 case "$1" in
   start)
  echo -n "Starting $DESC: "
- start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
-   --exec $DAEMON -- $DAEMON_OPTS
+ start-stop-daemon --start --quiet --pidfile $PIDFILE \
+   --chuid $USER --exec $DAEMON -- $DAEMON_OPTS -p $PIDFILE
  echo "$NAME."
  ;;
   stop)
  echo -n "Stopping $DESC: "
- start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
+ start-stop-daemon --stop --quiet --pidfile $PIDFILE \
    --exec $DAEMON
  echo "$NAME."
  ;;
@@ -58,11 +60,11 @@
  # just the same as "restart".
  #
  echo -n "Restarting $DESC: "
- start-stop-daemon --stop --quiet --pidfile \
-   /var/run/$NAME.pid --exec $DAEMON
+ start-stop-daemon --stop --quiet --pidfile $PIDFILE \
+   --exec $DAEMON
  sleep 1
- start-stop-daemon --start --quiet --pidfile \
-   /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS
+ start-stop-daemon --start --quiet --pidfile $PIDFILE \
+   --chuid $USER --exec $DAEMON -- $DAEMON_OPTS -p $PIDFILE
  echo "$NAME."
  ;;
   *)

Next, download, compile and install opensips. Then apply these patches:

--- ~/src/opensips-1.6.0-notls/packaging/debian/opensips.default  2009-10-16 02:37:58.000000000 +0200
+++ /etc/default/opensips 2009-10-18 21:26:05.000000000 +0200
@@ -3,13 +3,13 @@
 #
 
 # Set to yes to enable opensips, once configured properly.
-RUN_OPENSIPS=no
+RUN_OPENSIPS=yes
 
 # User to run as
-USER=opensips
+USER=alg
 
 # Group to run as
-GROUP=opensips
+GROUP=alg
 
 # Amount of memory to allocate for the running OpenSIPS server (in Mb)
 MEMORY=64

--- ~/src/opensips-1.6.0-notls/packaging/debian/opensips.init 2009-10-16 02:37:58.000000000 +0200
+++ /etc/init.d/opensips  2009-10-18 21:27:27.000000000 +0200
@@ -16,19 +16,20 @@
 # Should-Stop:       postgresql mysql radius
 
 PATH=/sbin:/bin:/usr/sbin:/usr/bin
-DAEMON=/usr/sbin/opensips
+DAEMON=/usr/local/sbin/opensips
 NAME=opensips
 DESC=opensips
-HOMEDIR=/var/run/opensips
+HOMEDIR=/var/run/alg
 PIDFILE=$HOMEDIR/$NAME.pid
 DEFAULTS=/etc/default/opensips
 RUN_OPENSIPS=no
+CONFIG=/usr/local/etc/opensips/opensips.cfg
 
 # Do not start opensips if fork=no is set in the config file
 # otherwise the boot process will just stop
 check_fork ()
 {
-    if grep -q "^[[:space:]]*fork[[:space:]]*=[[:space:]]*no.*" /etc/opensips/opensips.cfg; then
+    if grep -q "^[[:space:]]*fork[[:space:]]*=[[:space:]]*no.*" $CONFIG; then
         echo "Not starting $DESC: fork=no specified in config file; run /etc/init.d/opensips debug instead"
         exit 1
     fi

Last but not least, the opensips.cfg config file. This is far from perfect, I'm probably violating the SIP RFC, but it seems to work. Note that this only handles INVITE's and related messages. Note that you'll have to replace the example IP addresses sprinkled all over the config with real IP addresses.

####### Global Parameters #########

debug=0
log_stderror=no
log_facility=LOG_LOCAL0

fork=yes
children=4

disable_tcp=yes
dns_try_ipv6=no
auto_aliases=no

####### Modules Section ########

mpath="/usr/local/lib64/opensips/modules/"

loadmodule "rr.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "tm.so"
loadmodule "uri.so"
loadmodule "nathelper.so"
#loadmodule "uac.so"
#loadmodule "xlog.so"

modparam("nathelper", "rtpproxy_sock", "unix:/var/run/alg/rtpproxy.sock")

####### Routing Logic ########

route {
    #xlog("route in [$fu/$tu/$ru/$ci]\n");

    # Sanity checks
    if (msg:len >= 2048) {
        sl_send_reply("513", "Message too big");
        exit;
    }

    # Only support INVITE, ACK, BYE and CANCEL
    if (method != "INVITE" && method != "ACK" && method != "BYE" && method != "CANCEL") {
        sl_send_reply("503", "Service Unavailable");
        exit;
    }

    # Add record routing headers if this is the first message in a dialog,
    # we need to stay in the middle. And, set the new $du by $si.
    if (!has_totag()) {
        record_route();
        route("relay_by_source");
    }

    # If this an in-dialog message, the message should have Route headers.
    if (has_totag()) {
        if (!loose_route()) {
            if (t_check_trans()) {
                t_relay();
                exit;
            }
            exit;
        }
    }

    # Have rtpproxy handle SDP relaying
    if (has_body("application/sdp")) {
        if (method == "INVITE") {
            route("rtp_offer");
            t_on_reply("rtp_answer");
        } else if (method == "ACK") {
            rtpproxy_answer();
        }
    } else if (method == "INVITE") {
        t_on_reply("rtp_offer");
    }

    # Cancel RTP stream
    if (method == "BYE" || method == "CANCEL") {
        unforce_rtp_proxy();
    }

    t_relay();
}

route[relay_by_source] {
    if ($si == "192.168.1.30") {
        #uac_replace_from("", "sip:$fU@sip.voip-provider.tld");
        #uac_replace_to("", "sip:$tU@sip.voip-provider.tld");
        force_send_socket("1.2.3.4");
        rewritehost("sip.voip-provider.tld");
    } else {
        force_send_socket("192.168.1.1");
        rewritehost("192.168.1.30");
    }
}

route[rtp_offer] {
    if ($si == "192.168.1.30")
        rtpproxy_offer("ei");
    else
        rtpproxy_offer("ie");
}

onreply_route[rtp_offer] {
    if (has_body("application/sdp"))
        route("rtp_offer");
}

onreply_route[rtp_answer] {
    if (has_body("application/sdp"))
        rtpproxy_answer();
}

Great. And first after posting this entry, I find the alg.cfg example file which is pretty similar to this one :-)

####### Global Parameters #########

debug=3
log_stderror=no
log_facility=LOG_LOCAL0

fork=yes
children=8

disable_tcp=yes
dns_try_ipv6=no
auto_aliases=no

####### Modules Section ########

mpath="/usr/lib/opensips/modules/"

loadmodule "rr.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "tm.so"
loadmodule "uri.so"
loadmodule "nathelper.so"
loadmodule "uac.so"
loadmodule "xlog.so"

modparam("nathelper", "rtpproxy_sock", "unix:/var/run/voipalg/rtpproxy.sock")

####### Routing Logic ########

route {
    #xlog("route in [$fu/$tu/$ru/$ci]\n");

    # Sanity checks
    if (msg:len >= 2048) {
        sl_send_reply("513", "Message too big");
        exit;
    }

    # Only support INVITE, ACK, BYE and CANCEL
    if (method != "INVITE" && method != "ACK" && method != "BYE" && method != "CANCEL") {
        sl_send_reply("503", "Service Unavailable");
        exit;
    }

    # We can redirect inbound calls here if we want...
    if (method == "INVITE" && !($Ri =~ "^192\.168\.")) {
        if ($rU == "31123456789") {
            rewriteuri("sip:0612345678@sip.voip-provider.tld");
            sl_send_reply("302", "Moved");
            exit;
        }
    }

    # We do two things:
    # (1) rewrite SDP to use the RTP proxy
    # (2) forward all incoming messages to either 192.168.101.14 or sip.voip-provider.tld

    # Rewrite the SDP
    if (method == "INVITE") {
        if (has_body("application/sdp")) {
            # (unsure whether we need the FA options)
            if ($Ri =~ "^192\.168\.") {
                force_rtp_proxy("FAEI");
            } else {
                force_rtp_proxy("FAIE");
            }
        }
    } else if (method == "BYE" || method == "CANCEL") {
        unforce_rtp_proxy();
    }

    # Always stay in the middle
    if (method == "INVITE") {
        record_route();
    }

    # Find the correct destination
    if (!has_totag()) {
        route("relay_by_source");
    } else if (!loose_route()) {
        sl_send_reply("503", "Expected routing headers");
    }

    # Relay the message
    if (!t_relay()) {
        sl_reply_error();
    }
}

route[relay_by_source] {
    if ($Ri =~ "^192\.168\.") {
        uac_replace_from("", "sip:$fU@sip.voip-provider.tld");
        uac_replace_to("", "sip:$tU@sip.voip-provider.tld");
        force_send_socket("1.2.3.4");
        rewritehost("sip.voip-provider.tld");
    } else {
        force_send_socket("192.168.101.30");
        rewritehost("192.168.101.14");
    }
}

onreply_route {
    if ($rm == "INVITE" && (status =~ "^18" || status =~ "^200")) {
        # (unsure about the FA options)
        force_rtp_proxy("FA");
    }
}

2009-09-08 - mysql ubuntu readline delete-char

No, editing your ~/.inputrc does not work. And yes "\e[3~" is the right sequence. The way to get the DEL (forward delete) character to work in the mysql (mysql-client-5.1) on Ubuntu/Jaunty is to edit ~/.editrc and place the following lines in it:

bind "\e[3~" ed-delete-next-char
bind "^R" em-inc-search-prev

chuckh1958 explains on the ubuntu forum that the mysql client is not using readline but editline.

And as for using strace(1) to find out what config files you might need: mysql uses isatty to check whether it should do any readline-ish stuff at all. So you'll need to cheat around that, for example like this:

$ strace mysql -uuser -ppass somedb </dev/null 2>&1 | grep open.*/home
$ cat >isatty.c
int isatty(int i) { return 1; }
^D
$ gcc -fPIC -shared -o isatty.so isatty.c
$ LD_PRELOAD=./isatty.so strace mysql -uuser -ppass somedb </dev/null 2>&1 | grep open.*/home
open("/home/walter/.editrc", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/walter/.mysql_history", O_RDONLY) = 4
open("/home/walter/.mysql_history.TMP", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

2009-08-17 - recoding avi to dvd using mencoder

The quick and easy way to burn your Midsomer Murders avi's to DVD. (Mind the "rm -rf DVD/" in there!)

$ cat >dvdauthor.xml <<EOF
<dvdauthor dest="DVD">
  <vmgm />
  <titleset>
    <titles>
      <pgc>
        <vob file="dvd_movie.mpg" chapters="0,12:00,24:00,36:00,48:00,1:00:00,1:12:00,1:24:00,1:36:00,1:48:00"/>
      </pgc>
    </titles>
  </titleset>
</dvdauthor>
EOF
$ mencoder -oac lavc -ovc lavc -of mpeg -mpegopts format=dvd:tsaf \
    -srate 48000 -af lavcresample=48000 \
    -vf scale=720:450,expand=720:576,harddup \
    -lavcopts vcodec=mpeg2video:vrc_buf_size=1835:vrc_maxrate=9800:vbitrate=5000:dc=10:trell:mbd=2:keyint=15:vstrict=0:aspect=4/3:acodec=ac3:abitrate=192 \
    -ofps 25 -o dvd_movie.mpg \
    Midsomer\ Murders.s01e02.Written\ In\ Blood.avi ; \
  rm -rf DVD/ ; \
  dvdauthor -x dvdauthor.xml ; \
  growisofs -dvd-compat -Z /dev/dvdrw1 -dvd-video -input-charset utf8 DVD/

2009-08-10 - prepending dates to program output

This convenient dateify snippet is useful when a program (in my case lightcount) doesn't prepend dates/times to its logging output. Of course you could edit the program and add an strftime yourself, but this is almost as cheap when the program does not produce much output and way cheaper to implement.

#!/usr/bin/env perl
# dateify(1) uses getc to avoid all perl input buffering
# and prepends the current local time to every line read
use POSIX 'strftime';
$| = 1; # disable output buffering (useful for tee(1))
while (1) {
  my ($s, $c) = ("", -1);
  $s .= $c while (($c = getc) ne "\n" && !eof);
  print strftime("%Y-%m-%d %H:%M:%S: ", localtime) . "$s\n";
  last if eof;
}

#!/usr/bin/env perl
# dateify(1) uses getc to avoid all perl input buffering
# and prepends the current local time to every line read
# (fixed code on 2010-03-12)
use POSIX 'strftime';
$| = 1; # disable output buffering (useful for tee(1))
while (1) {
  my ($s, $c) = ("", -1);
  $s .= $c while (($c = getc(STDIN)) ne "\n" and !eof(STDIN));
  print strftime("%Y-%m-%d %H:%M:%S: ", localtime) . "$s\n"
    unless $s eq "" and $c eq "";
  last if eof(STDIN);
}

Example:

$ ( echo abc ; sleep 1 ; echo def ; sleep 1 ; echo ghi ) | dateify 
2009-08-10 13:35:14: abc
2009-08-10 13:35:15: def
2009-08-10 13:35:16: ghi

2009-06-25 - wrong resolution / hp probook 4510s

My work got me a HP ProBook 4510s laptop recently. Ubuntu Jaunty amd64 installed fine and all drivers (all Intel and one Marvell NIC) work fine out of the box.

There's just one minor issue: often when booting, GNOME is loaded in a 800x600 display mode. Which is way too small, obviously. Restarting gdm fixes it, but that becomes a pain after a while.

The problem can be seen in the following diff of Xorg.0.log.

--- /var/log/Xorg.0.log  2009-06-25 09:43:10.643346190 +0200
+++ /var/log/Xorg.0.log  2009-06-25 09:44:52.279321015 +0200
@@ -181,10 +181,9 @@
 (II) intel(0): Output VGA disconnected
 (II) intel(0): Output LVDS connected
 (II) intel(0): Output HDMI-1 disconnected
-(II) intel(0): Output TV connected
-(II) intel(0): Using fuzzy aspect match for initial modes
-(II) intel(0): Output LVDS using initial mode 800x600
-(II) intel(0): Output TV using initial mode 800x600
+(II) intel(0): Output TV disconnected
+(II) intel(0): Using exact sizes for initial modes
+(II) intel(0): Output LVDS using initial mode 1366x768
 (II) intel(0): detected 512 kB GTT.
 (II) intel(0): detected 65532 kB stolen memory.
 (==) intel(0): video overlay key set to 0x101fe

The fix is quite simple. Define the LVDS monitor (the LCD display) as used. My xorg.conf now looks like this:

Section "Device"
        Identifier      "Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07)"
        Driver          "intel"
        Option          "monitor-LVDS" "LCD-1366x768"
EndSection

Section "Monitor"
        Identifier      "LCD-1366x768"
        Option          "Position" "0 0"
        Modeline        "1366x768"  69.30  1366 1382 1416 1466  768 770 776 788 -hsync -vsync
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Device          "Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (rev 07)"
        #Monitor         "LCD-1366x768"
EndSection

# echo 'options snd-hda-intel model=laptop' >> /etc/modprobe.d/custom.conf

2009-06-21 - recipe / chicken paksoi noodles

Source: http://www.keep-in-health.com/item/recipe_prawn_paksoi_stir-fry.html

Prawn and paksoi stir-fry by Georgette Henson, adapted.

Ingredients:

• egg noodles, 250 (g)
• arachis oil (peanut oil), 3+ (T) (other oils should work fine too)
• sesame oil, 1+ (T)
• garlic cloves, 2, sliced
• optionally a sliced chilli pepper
• chicken breast, cut up in cubes, 200 (g)
• ginger, freshly grated, 1 (T)
• onions, 1+ (large), coarsely sliced (spring onions are even better)
• paksoi, 200+ (g), rinsed
• pine nuts, a handful, roasted
• optional masala/curry sauce, 1+ (T)
• optionally coriander, a couple of leaves

Put the noodles in a bowl, add enough boiling water (more than one liter) and leave to soak for 4 minutes. Drain and set aside.

Heat the oil in a wok — make it hot — add the garlic, the sesame oil and optionally the chilli pepper. Add the chicken and stir-fry (brown) for a minute or so.

Add ginger and the onions. Add the chopped white of the paksoi.

Roast the pine nuts in a separate clean dry frying pan at medium heat. Don't forget to toss them often.

Add the green of the paksoi, the optional masala/curry sauce, the roasted pine nuts and the noodles. Stir well.

When these last ingredients are all warm, you're done. Serve in a bowl with the optional coriander leaves on top.

2009-06-20 - slowloris / quick fix

If you're worried about slowloris tying up all your webserver's — most notably apache's — resources, the quick fix is to limit connections by IP address. This will of course limit functionality for all your clients using a proxy, but it's the easiest workaround workaround and works fine in the simplest of cases.

All you need is the the connlimit (xt_connlimit) iptables(8) module:

# iptables -I INPUT -p tcp --syn --dport 80 -m connlimit \
    --connlimit-above 10 -j DROP # or REJECT

If you don't have this particular module, you can try to use the hashlimit module — if you have that — but it's not by far as effective, and if you're unlucky it might only slow the attack down.

# iptables -I INPUT -p tcp --syn --dport 80 -j DROP # or REJECT
# iptables -I INPUT -p tcp --syn --dport 80 -m hashlimit \
    --hashlimit-name allow-syn-80 --hashlimit 1/sec --hashlimit-burst 10 \
    --hashlimit-mode srcip -j ACCEPT

I don't want to give more attention to slowloris than necessary. I'm not convinced that full disclosure after only one — short-sighted, I admit — response from the apache team was the way to go. Perhaps limited disclosure on an apache mailing list would've been possible. But the loris is out of the bag now (very poor joke, I know, he he he), so it doesn't matter. What does matter is that there is a way more readable version of the tool out in python: pyloris. Use it to your advantage.

2009-06-19 - GNU as Hello World / jumps

The Hello World assembly snippet as found on the 2008 notes, extended to check the return value of sys_write(2). This shows the usage of jmp and that %eax gets overwritten by the sys_write(2) call with its return value.

--- hello.s 2009-06-19 09:27:10.667095772 +0200
+++ hello2.s  2009-06-19 10:21:38.062595546 +0200
@@ -8,8 +8,15 @@
     movl    len, %edx   /* message length */
     int     $0x80       /* call kernel */
 
+    xorl    len, %eax   /* check return value */
+    jnz     fail        /* jump to fail if non-zero */
+    xorl    %ebx, %ebx  /* set ebx to zero (EXIT_SUCCESS) */
+    jmp     done        /* jump to exit part */
+fail:
+    movl    $1, %ebx    /* set ebx to non-zero (EXIT_FAILURE) */
+done:
+
     movl    $1, %eax    /* system call number (sys_exit) */
-    xorl    %ebx, %ebx  /* return value success */
     int     $0x80       /* call kernel */
 
 .data

A more readable version using cmpl instead of xorl:

--- hello2.s  2009-06-19 10:21:38.062595546 +0200
+++ hello2a.s 2009-06-19 10:33:20.375095609 +0200
@@ -8,8 +8,8 @@
     movl    len, %edx   /* message length */
     int     $0x80       /* call kernel */
 
-    xorl    len, %eax   /* check return value */
-    jnz     fail        /* jump to fail if non-zero */
+    cmpl    len, %eax   /* compare return value */
+    jne     fail        /* jump to fail if not equal */
     xorl    %ebx, %ebx  /* set ebx to zero (EXIT_SUCCESS) */
     jmp     done        /* jump to exit part */
 fail:

Observe that jnz and jne are actually the same jump, as can be seen from the following gdb(1) output.

$ gdb ./hello2
[snip]
(gdb) disass _start
Dump of assembler code for function _start:
0x00000000004000b0 <_start+0>:  mov    $0x4,%eax
0x00000000004000b5 <_start+5>:  mov    $0x1,%ebx
0x00000000004000ba <_start+10>: mov    $0x6000e4,%ecx
0x00000000004000bf <_start+15>: mov    0x6000f2,%edx
0x00000000004000c6 <_start+22>: int    $0x80
0x00000000004000c8 <_start+24>: xor    0x6000f2,%eax
0x00000000004000cf <_start+31>: jne    0x4000d5 <fail>
0x00000000004000d1 <_start+33>: xor    %ebx,%ebx
0x00000000004000d3 <_start+35>: jmp    0x4000da <done>
End of assembler dump.

2009-05-27 - postfix / aliases / myhostname

If you specify myhostname in your main.cf postfix configuration, make sure you add that hostname to mydestination as well. If you don't, your /etc/aliases will be ignored, and you might end up sending mail to root@myhostname when you want it all for yourself.

This can happen in the following common scenario:

• You have your workstation at your mycompany.tld intranet called mydesktop. Because it's an intranet, you can't receive mail from the outside.
• You have postfix as your mailer daemon that uses mail.mycompany.tld as relay machine.
• The relay machine validates sender addresses and doesn't like myname@mydesktop as FROM-address ("Sender address rejected: Domain not found (in reply to RCPT TO command)").
• To address the FROM-address problem, you set myhostname to mycompany.tld. Mail to myname@mycompany.tld does get to your mailbox properly. At the same time this makes sure that recipients of your mail will be able to reply to your mail. Note that /etc/mailname or myorigin sets the HELO value while myhostname sets the MAIL FROM domain.
• Now, this all works fine, unless you have set mydestination to something like: mydesktop, localhost.localdomain, localhost. If you have, you're missing mycompany.tld and mail to your own users will bypass local mail handling: /etc/aliases won't be used like you expect it to.
• The solution: add mycompany.tld to the mydestination.

To summarize:

# Updated 2009-06-05
myorigin = mycompany.tld
canonical_maps = hash:/etc/aliases
#myhostname = mycompany.tld

# Original
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = mycompany.tld, mydesktop, localhost.localdomain, localhost
relayhost = [mail.mycompany.tld]

Two other unrelated postfix caveats:

• Postfix runs in a chroot jail by default (or at least on Debian/Linux it does). If you change /etc/resolv.conf you must restart/reload it or your new DNS settings will not be used.
• If you set relayhost, decide whether you want the MX or the A record of that host to be used. In the configuration mentioned above, that would be either mycompany.tld or [mail.mycompany.tld]. Usually you will use the latter — the A record — so you shouldn't forget the brackets. When your DNS changes to not resolve MX records of subdomains to the MX of your toplevel domain, you'll be glad you didn't.

2009-05-24 - recipe / dads brown beans

Source: Dads recipes

Ingredients:

• brown beans, dried, 500 (g)
• molasses, 6+ table spoons
• salt, 3+ tea spoons
• vinegar, distilled, 6+ table spoons

Sort the beans (removing small rocks and bad beans) and soak the beans in room-temperature water for 8+ hours. Discard the soak water (this reduces flatulence).

Cook the beans in about 1.5 liters of water. You can start out with less, but you'll have to check periodically to make sure the beans do not dry out. This can take up to two hours as you want the beans to be overcooked, releasing starch, thickening the cooking liquid. Stir every now and then.

When the beans are almost completely dissolved, add the salt, the molasses and the vinegar. Depending on your taste and the types of ingredients, you might need more. You should now have a thick substance closer to mashed potatoes than to soup, with a sweet-sour taste.

Serve with crisp thick sliced bacon and potatoes.

2009-04-08 - thunderbird advanced config settings

A couple of useful Thunderbird about:config settings:

;; Check all folders for new mail (default: false)
mail.check_all_imap_folders_for_new = true
;; Sort all folders descending by default (default: 1)
mailnews.default_sort_order = 2
;; Some UTF-8 settings to pester non-UTF-8-users (defaults: false, ISO-8859-1)
mailnews.reply_in_default_charset = true
mailnews.send_default_charset = UTF-8

2009-04-05 - reading rfc documents on sony prs505

I recently bought the Sony PRS-505 reader — a digital book — and I am pretty content with it. The resolution and weight are fine. It handles darn slow, but it really is no problem having to wait one second for the next page... and it probably saves a lot of battery life.

The only two drawbacks I have found, is that it renders text (.txt) documents in a variable width font and it reboots on certain — not many — (broken?) pages in a couple of pdf files.

The first of these drawbacks is easily fixable by converting text documents to rich text files (.rtf) with a monospaced font. The embedded font/resolution then yields 71 (almost 72) characters per line in the small font mode. Fortunately, most RFC files are already limited to 72 characters and have enough double spaces that can be trimmed down to a single space.

Download this python script to convert text files to rich text files, adjusted to fit the Sony PRS-505:
txt2prs505rtf.py (view)
See this example for an already converted RFC:
rfc4930.rtf

2009-01-26 - unicodeencodeerror / python redirect pipe

Python has excellent Unicode support. Most of the time it just works. However, when you want to redirect non-ASCII characters to a different program through shell redirection, you will run into a UnicodeEncodeError.

See the following example:

$ locale | grep LC_CTYPE
LC_CTYPE="en_US.UTF-8"
$ python -c 'print "\u20ac 3.50"'
\u20ac 3.50
$ python -c 'print u"\u20ac 3.50"'
€ 3.50
$ python -c 'print u"\u20ac 3.50"' | cat
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)

Python calls something like isatty(3) ("is a TTY?") on the standard out file descriptor, and when that returns 0 (false) it forces stdout to use the ascii codec. This can be quite annoying if you want to use other shell tools (awk(1), grep(1), sed(1), tr(1) etc..) to quickly search through or modify the output stream.

The solution is to replace the stdout object with one that doesn't assume that a non-terminal only likes ASCII. Add the following to your python code:

import codecs, locale, sys
sys.stdout = codecs.getwriter(locale.getdefaultlocale()[1])(sys.stdout, 'replace')

It's not real pretty, but it works :-)

$ python -c 'import codecs, locale, sys
> sys.stdout = codecs.getwriter(locale.getdefaultlocale()[1])(sys.stdout, "replace")
> print u"\u20ac 3.50"' | cat
€ 3.50

2009-01-26 - gless / open binary and unknown files / gnome

Are you tired of gedit asking you what type of encoding a file is, when you just want to open the file to take a peek at the contents?

Use gless(1):

#!/bin/sh
gnome-terminal -x less -f "$@"

You need the -f or it will bypass the question of whether you want to open binary files and assume no. (Resulting in a gnome-terminal(1) that immediately closes.)

#!/bin/bash
file="$1" ; shift 
if [ "${file:0:7}" == 'file://' ]; then
  file="${file:7}"
fi
gnome-terminal -x less -f -- "$file"

2009-01-05 - more or less useless tips and tricks

It's a new year, so it's time for some more more or less useless/useful tips and tricks, bundled together. Like last time, they aren't worthy of a box div on their own. They only get a li each.

• Fixing so you're printing A4 and not Letter in Ubuntu/GNOME
  If you have your language settings set to en_US — like me — you're bound to have noticed that the default printing size is Letter. Letter is not A4, like you know, and A4 comes out of most printers here in Europe.
  
  Solutions:

  # echo a4 > /etc/papersize
  # echo 'LC_PAPER=en_GB.UTF-8' >> /etc/environment  # you do have en_US.UTF-8 set as LANG in there, right?
  

  And, if those two do not provide the correct results, Firefox has a print.postscript.paper_size setting that likes an "a4" string as well.
• Using Unix (GNU) utilities on Windows (download)
  You might've spotted the UnxUtils.zip package on sourceforge.net, or you might not have because it is not always as easy to find the correct download location (it hasn't been updated since 2000-10-28). Also, the package contains some programs/files that most people do not need or that conflict with already existing windows binary names.
  
  Therefore, for your pleasure and mine, I have repackaged the files. They are a bit old, but they work just fine. You get grep, tar, less, patch, diff, find (renamed), wc, wget, netcat and much more.
  
  Get unxutilseasy.zip now and unzip it in your WINDOWS or SYSTEM32 directory so the files are in your path.
• Did somebody send you a message consisting of only ones and zeroes?
  Don't worry! Perl is your friend!

  echo 0100100001100101011011000110110001101111 | perl -pe 's/([01]{8})/chr unpack C,pack B8,$1/ge'
  

  I leave encoding to binary as an exercise to the reader.
• Need to access something (a website) on a remote lan when you only have ssh access?
  Sometimes you are forced — for example by stupid web interfaces that insist on redirecting you to some LAN address — to connect to a LAN address that is not on your LAN.
  
  Assume the following scenario:
  1. I have shell (ssh) access to some machine (machine roger.example.com) that's connected to a LAN, with IPs 10.11.22.0/24.
  2. I want to reach some internal machine there, let's say 10.11.22.33 on port 80.
  3. You can use ssh forwarding for this: ssh -L 9999:10.11.22.33:80 roger.example.com — after logging in like always, you also get a tunnel from 127.0.0.1:9999 to 10.11.22.33:80.
  4. Sounds good. But wait, when logging on to the web interface on localhost, the site might decide to redirect you to 10.11.22.33:80. That address is not in your routing table, so you're screwed.
  5. iptables comes to the rescue: iptables -t nat -I OUTPUT -d 10.11.22.33 -p tcp --dport 80 -j DNAT --to 127.0.0.1:9999
  6. Now you can connect to http://10.11.22.33 and iptables will reroute it to localhost before it gets to the routing table. Pretty neat.
• Some ssh peculiarities worth remembering
  
  1. /etc/environment should not be treated as a shell script. Use only lines like KEY=value.
  2. If you somehow find that you have trouble using scp or sftp, make sure that your shell initialization scripts (.bashrc and the likes, ran for non-login shells) do not print any information to stdout or you get an error similar to this: Received message too long 1752131850 or no files written in case of scp.
     
     Obviously you are allowed to print stuff after the mandatory [ -z "$PS1" ] && return line that makes sure that non-shells exit before you do them any harm ;-)