Notes to self, 2010

2010-11-30 - faxable images / asterisk pbx

Because creating images that the open source PBX Asterisk(tm) will properly fax using the SendFAX() application, was a pain in the ass, I'd like to share my findings.

The HOWTO for creating TIFF images that are laid out so the spandsp fax back-end in asterisk, is embedded in the shell script, below.

In short, you need to have images of 1728x2292, in 2 colors, with the correct DPI (204x196) and the right compression.

This script automates the process of scaling, resizing, adding canvas and merging more images into a single multi-page image. It takes two or more arguments: the destination TIFF and all source images. It uses the multi-purpose ImageMagick convert utility for all the necessary operations.

# -- Walter Doekes 2010
# vim: set ts=8 sw=4 sts=4 et:

# Get arguments / show help

IMCONVERT="`which convert`"
if [ "$#" -lt 2 -o -z "$IMCONVERT" ]; then
    cat >&2 << __EOF__
Converts a collection of input images to a single multipage tiff to be
sent using the asterisk SendFAX application.

Requires ImageMagick convert(1), found at: ${IMCONVERT:-NOT FOUND}
    exit 1
OUTPUT="$1" ; shift

# Utility functions
# (mktemp and mktemp_tiff return filenames safe to use without double quotes)

mktemp_tiff() {
    # Bah, this is ugly.
    mv -n $file $file.tiff
    if [ $? = 0 ]; then
        rm $file
        file=`mktemp_tiff` # filename in use? try again
    echo $file

resize_image_to_1728x2292() {
    # (-alpha: remove transparency)
    # (-resample: scale to correct proportions)
    # (-scale: resize up/down so it fits)
    # (-extent: add white canvas)
    "$IMCONVERT" "$input" -alpha off -resample 204x196 -scale 1728x2292 \
        -extent 1728x2292 $output
    if [ $? != 0 ]; then
        rm $output
        exit 1
    echo $output

convert_image_to_tiff() {
    "$IMCONVERT" "$input" -compress Fax -units PixelsPerInch -density 204x196 \
        -antialias -resize '1728x2292!' -dither FloydSteinberg -monochrome \
        -dither FloydSteinberg -monochrome $output
    if [ $? != 0 ]; then
        rm $output
        exit 1
    echo $output

# Main: convert individual images and combine them

for image in $@; do
    # Resize image
    resized=`resize_image_to_1728x2292 "$image"`
    if [ $? != 0 ]; then
        echo "Failure resizing $image. Aborting..." >&2
        [ -n "$converted_list" ] && rm $converted_list
        exit 1

    # Convert image
    converted=`convert_image_to_tiff $resized`
    if [ $? != 0 ]; then
        echo "Failure converting $image. Aborting..." >&2
        rm $converted_list $resized
        exit 1

    # Append to list
    rm $resized
    converted_list="$converted_list $converted"

"$IMCONVERT" '(' -coalesce $converted_list ')' "$OUTPUT"
rm $converted_list
exit $status

Update 2010-12-06

In some cases — this has been observed when converting PDF to TIFF — the polarity can be mixed up, creating a white-on-black fax which can be quite expensive ;-) Fix this using the -define quantum:polarity=min-is-white option.

2010-11-23 - nat / switch external source port

When reproducing an issue with IP phones speaking SIP, I ran into the question of how to switch source port on my linux NAT router.

The problem the clients were having, were a result of a failing (or reset) NAT-gateway. The NAT-gateway would change the external source port mid-dialog (some SIP dialogs can persist for quite a long time).

So, how do you go about switching source port on an UDP connection on your Linux NAT router without resetting it (or disturbing anyone else using it)? The answer: conntrack(8)

Step (1): add a static SNAT mapping for a custom port, before the standard SNAT rules, for your specific application. In my case the phone is at, the destination at and my external IP is

# iptables -t nat -nvL POSTROUTING
Chain POSTROUTING (policy ACCEPT 25M packets, 1534M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 348K   31M SNAT       0    --  *      eth2             to: 
# iptables -t nat -I POSTROUTING -o eth2 -s -d -p udp --dport 5060 -j SNAT --to

Step (2): start the dialog. It is now connection tracked:

# conntrack -L -s -p udp --reply-port-dst 7060
udp      17 165 src= dst= sport=5060 dport=5060 packets=15 bytes=7670 src= dst= sport=5060 dport=7060 packets=37 bytes=10179 [ASSURED] mark=0 use=1

Step (3): remove static NAT mapping, optionally replacing it with a new one and kill the tracked connection.

# iptables -t nat -D POSTROUTING -o eth2 -s -d -p udp --dport 5060 -j SNAT --to
# iptables -t nat -I POSTROUTING -o eth2 -s -d -p udp --dport 5060 -j SNAT --to
# conntrack -D -s -d -p udp --orig-port-src 5060 --orig-port-dst 5060

Step (4): watch how the dialog continues on the new external source port. You're done.

# conntrack -L -s -p udp --reply-port-dst 7061
udp      17 64 src= dst= sport=5060 dport=5060 packets=5 bytes=3208 src= dst= sport=5060 dport=7061 packets=6 bytes=2618 [ASSURED] mark=0 use=1

2010-08-30 - sip totag / grandstream / register

SIP Question: The Grandstream GXP2000 sends SIP REGISTER requests with a To tag. Is my proxy wrong in refusing the request?

REGISTER sip:server SIP/2.0
Via: SIP/2.0/UDP;branch=z9hG4bK0b90873d634698eb
From: "phone 123" <sip:123@server>;tag=c29eb9104c6a5a86
To: <sip:123@server>;tag=as77984b6
Contact: <sip:123@;transport=udp>
Supported: path
Authorization: Digest username="123", realm="server", algorithm=MD5, uri="sip:server", nonce="0997652c", response="3b91afb768c11ae0a0405e1bed41bc23"
Call-ID: 3033205eb0b5d203@
CSeq: 56349 REGISTER
Expires: 3600
User-Agent: Grandstream GXP2000
Max-Forwards: 70
Content-Length: 0

Answer: no, the proxy is right. RFC 3261 says this about it. To

   A request outside of a dialog MUST NOT contain a To tag; the tag in
   the To field of a request identifies the peer of the dialog.  Since
   no dialog is established, no tag is present.

10.2 Constructing the REGISTER Request

   A REGISTER request does not establish a dialog.

Grandstream has already fixed this bug long ago (in version, beginning of 2007) as you can see in the release notes on You should upgrade the firmware.

2010-08-04 - nxclient / locale passing

So, I achieved victory on getting the compose key to work in the NX session. On to get a proper English language setting on our Terminal Server.

The configuration suffers from two problems:
(1) /etc/environment had values set (LANG=nl_NL.UTF-8 and LANGUAGE=nl_NL:nl). (2) nxssh does not pass the LANG/LC_* environment variables.

If I were to remove the /etc/environment variables and configure everything like in a previous post of mine, everyone gets the POSIX locale (nxssh doesn't pass anything). That's not good. I can't find a way to get nxssh(1) to send the needed environment, so we'll have to look elsewhere.

The fix: configure sshd(1) on the server to PermitUserEnvironment yes and create a custom ~/.ssh/environment in my homedir on the server with the following settings:


My environment gets loaded after the global environment, so I get my custom locale. At the same time I haven't broken it for everyone else. (Observe that the PermitUserEnvironment can have security implications in some cases.)

2010-08-03 - altgr / nxclient / compose key

Like various reports on the internet suggest, the AltGr compose key doesn't work properly or not at all from an NXClient connected to an NXServer (FreeNX in my case).

Note that this is a different issue from the one where Alt_R (and Super_L, Super_R, Ctrl_R en Menu) remains pressed after which no normal typing is possible. That issue is described in Alt Gr keeps stuck and involves a new int sendKey = 0; in nxagent that should be reverted.

I run Ubuntu Lucid Lynx on my desktop with the NXClient, and we run the same on the terminal server. Normally getting the Compose Key to work is simple: go to the System menu, then Preferences, then Keyboard. Choose Layouts, Options and set the Compose Key to Right Alt:

Keyboard Preferences: compose key

However, on the server this has no effect. When comparing output of the xev(1) program, we see the following when pressing AltGr, single-quote, e (to get é):

$ diff -uw client_xev server_xev
--- client_xev 2010-08-02 12:38:23.923838872 +0200
+++ server_xev 2010-08-02 12:44:50.843839010 +0200
@@ -1,11 +1,11 @@
 KeyPress event, serial NN, synthetic NO, window 0xXXX,
-    state 0x10, keycode 108 (keysym 0xff20, Multi_key), same_screen YES,
+    state 0x10, keycode 113 (keysym 0xffea, Alt_R), same_screen YES,
     XLookupString gives 0 bytes: 
     XmbLookupString gives 0 bytes: 
-    XFilterEvent returns: True
+    XFilterEvent returns: False
 KeyRelease event, serial NN, synthetic NO, window 0xXXX,
-    state 0x10, keycode 108 (keysym 0xff20, Multi_key), same_screen YES,
+    state 0x18, keycode 113 (keysym 0xffea, Alt_R), same_screen YES,
     XLookupString gives 0 bytes: 
     XFilterEvent returns: False
@@ -13,7 +13,7 @@
     state 0x10, keycode 48 (keysym 0x27, apostrophe), same_screen YES,
     XLookupString gives 1 bytes: (27) "'"
     XmbLookupString gives 1 bytes: (27) "'"
-    XFilterEvent returns: True
+    XFilterEvent returns: False
 KeyRelease event, serial NN, synthetic NO, window 0xXXX,
     state 0x10, keycode 48 (keysym 0x27, apostrophe), same_screen YES,
@@ -24,12 +24,6 @@
     state 0x10, keycode 26 (keysym 0x65, e), same_screen YES,
     XLookupString gives 1 bytes: (65) "e"
     XmbLookupString gives 1 bytes: (65) "e"
-    XFilterEvent returns: True
-KeyPress event, serial NN, synthetic NO, window 0xXXX,
-    state 0x10, keycode 0 (keysym 0xe9, eacute), same_screen YES,
-    XLookupString gives 0 bytes: 
-    XmbLookupString gives 2 bytes: (c3 a9) "é"
     XFilterEvent returns: False
 KeyRelease event, serial NN, synthetic NO, window 0xXXX,

Two things stand out here:
(1) The right alt press is seen as Alt_R and not Multi_key and XFilterEvent returns False (meaning that the receiving application should take action on the keypress).
(2) The keycode observed is 113 instead of 108.

This information is enough to fix the issue. First get the Multi_key setting for xmodmap(1):

$ xmodmap -pke | grep Multi_key
keycode 108 = Multi_key Multi_key Multi_key Multi_key

Put that line in ~/.xmodmaprc on the server, changing the 108 to 113. You test if it works by running xmodmap(1) manually with the .xmodmaprc as argument. If everything works like it should, you should now have a working compose key again: ¡Vìctørý!

2010-06-20 - python2.6 features / python2.5

Today I'll show you some quick and dirty python2.5 compatibility fixes. Of course you're developing on python2.6 or even python3.x, but your customer still lives in the dark ages. Here are two fixes that might come in handy.

ImportError: No module named ssl

Falling back to python2.5 socket.SSL if there is no python2.6 ssl through a small wrap_socket replacement:

import socket
    from ssl import wrap_socket
except ImportError:
    class wrap_socket:
        def __init__(self, socket):
            self.socket = socket
        def __getattr__(self, name):
            return getattr(self.socket, name)
        def connect(self, *args, **kwargs):
            self.socket.connect(*args, **kwargs)
            self.ssl = socket.ssl(self.socket)
        def read(self, *args, **kwargs):
            return*args, **kwargs)
        recv = read
        def write(self, *args, **kwargs):
            return self.ssl.write(*args, **kwargs)
        send = write

AttributeError: 'unicode' object has no attribute 'format'

Falling back to python2.5 Template if there is no python2.6 string.format. It does not do any fancy formatting, but in many cases you're not using that either. (Yes, for template syntax, {variable} wins in readability over $variable or, even worse, %(variable)s.)

if not hasattr('', 'format'):
    from string import Template
    import re
    def str_format(string, *args, **kwargs):
        tpl = Template(string)
        tpl.pattern =
        return tpl.substitute(*args, **kwargs) = re.compile(r'\{(?P<named>[A-Za-z0-9_]+)\}')


    value = template.format(**values)
except AttributeError:
    value = str_format(template, **values)

2010-06-17 - uninitialized globals / C language

As per the C language spec., uninitialized globals are initialized to zero (0). Nandu310 tells us why on his blog about the memory areas in the C language.

Data segment: the data segment is to hold the value of those variables that need to be available throughout the life time of the program. [...]

There are two parts in this segment. The initialized data segment and uninitialized data segment. When variables are initialized to some value (other than 0 or which is different value), they are allocated in the initialized segment (.data). When the variables are uninitialized they get allocated in the uninitialized data segment (.bss). [...] [The] job of initialization of variables to zero is left to the operating system to take care of. This bulk initialization can greatly reduce the time required to load.

When we want to run an executable program, the OS starts a program known as loader. When this loads the file into memory, it takes the BSS segment and initializes the whole thing to zeros. That is why the uninitialized global data and static data always get the default value of zero.

Nandu310 explains the segments in more detail and uses the nm tool. Using objdump you can get a more verbose view.

--- uninited.c  2010-06-17 09:14:59.364090777 +0200
+++ initone.c 2010-06-17 09:15:57.144090752 +0200
@@ -1,5 +1,5 @@
 int my_int;
-int my_int2;
+int my_int2 = 3;
 int main() {
     return my_int + my_int2;

Calling objdump on the compiled C programs shows you how my_int2 moves from the .bss section to the .data section between the two binaries.

$ objdump -zD uninited | sed -ne '/<main>/,/^$/p;/of section \.data/,/of section \.comment/p' 
Disassembly of section .bss:
000000000060102c <my_int2>:
  60102c: 00 00                 add    %al,(%rax)
  60102e: 00 00                 add    %al,(%rax)
$ objdump -zD initone | sed -ne '/<main>/,/^$/p;/of section \.data/,/of section \.comment/p'
Disassembly of section .data:
0000000000601018 <my_int2>:
  601018: 03 00                 add    (%rax),%eax
  60101a: 00 00                 add    %al,(%rax)

The moral of the story: globals in C are always initialized.

And, for the record, when using gcc — even without optimizations — there is no difference between initializing to 0 and leaving the variable uninitialized: they both go in the .bss. This means that you get no benefit from leaving a variable uninitialized. For readability, it is often better to explicitly initialize it anyway.

2010-05-07 - unexpanded tabs / mercurial web / diff

The hgweb mercurial web interface on current Debian/Squeeze (mercurial-common 1.5.1-2) lists tab characters as-is in the diff view.

Every line is prefixed not only by a plus or a minus (unified diff), but also by file and line numbers. This can cause a tab (0x9) character to appear as a single space. This does not look nice.

The following patch can be applied to expand the tab character so the intentation looks right again.

--- /usr/share/pyshared/mercurial/hgweb/ 2010-05-07 09:41:17.000000000 +0200
+++ /usr/share/pyshared/mercurial/hgweb/      2010-05-07 09:47:53.000000000 +0200
@@ -177,6 +177,7 @@ def diffs(repo, tmpl, ctx, files, parity
                 ltype = "difflineat"
                 ltype = "diffline"
+            l = l.expandtabs()
             yield tmpl(ltype,
                        lineid="l%s" % lineno,

2010-04-27 - asian site visits / trend micro

Today I noticed some odd traffic from a couple of chinese and japanese IP addresses on a website that I maintain.

A known visitor visited the site with (anonymized) IP address and a minute later the chinese and japanese IP addresses started following suit:
netblock - "Japan Network Information Center" and
netblock - "TREND MICRO INCORPORATED". - -        [25/Apr/2010:09:14:32 +0200] GET /oper/waitingroom/ - -        [25/Apr/2010:09:14:24 +0200] GET /media/252/js/jquery-1.3.2.min.js - -        [25/Apr/2010:09:14:24 +0200] GET /media/252/js/userchat.js - -        [25/Apr/2010:09:14:24 +0200] GET /media/252/js/jquery.cookie.js - -        [25/Apr/2010:09:14:33 +0200] GET /poll?_=1272179685670 
... - - [25/Apr/2010:09:15:42 +0200] GET /oper/waitingroom/ - - [25/Apr/2010:09:16:28 +0200] GET /media/252/js/userchat.js - - [25/Apr/2010:09:16:30 +0200] GET /media/252/js/jquery-1.3.2.min.js - - [25/Apr/2010:09:16:31 +0200] GET /media/252/js/jquery.cookie.js - -   [25/Apr/2010:09:16:33 +0200] GET /poll?_=1272179685670

I contacted the known visitor and indeed she was running some kind of Trend Micro security-all-in-one-tool. So, this probably is "legal" traffic designed to detect malicious websites.

The asian web requests did not supply the session cookies, so these foreign requests do not present an immediate security concern. What does concern me however is that (1) the traffic from is now doubled — or at least the request counts are — and (2) googling for these netblocks or IP-addresses does not turn up any info that these requests are legitimate.

FYI, the User-Agent string reported by these bots is Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1).

2010-04-09 - mysql utf8 collation / conversion

On a clean MySQL install — on a Debian or Ubuntu system at least — the MySQL server gets the latin1_swedish_ci with latin1 character set by default. Every time you set up a new machine, you must remember to either fix the defaults in my.cnf config file or to supply character set and collation options when creating databases.

Of course you'll opt to set this by default in my.cnf first:

default-character-set = utf8

init_connect = 'SET collation_connection = utf8_unicode_ci'
default-character-set = utf8
character-set-server = utf8
collation-server = utf8_unicode_ci

Unfortunately, sometimes you forget to do this, and now you have a database in some legacy (non-utf-8) character set. Well. Here is the fix. Create this stored procedure and run it supplying the the schema_name. It converts the database(*) and the tables to utf-8 with the utf8_unicode_ci collation.

delimiter //


CREATE PROCEDURE convert_to_utf8 (schema_name VARCHAR(64))
  DECLARE table_name VARCHAR(64);
    SELECT t.table_name FROM information_schema.tables t
    WHERE CAST(table_schema AS CHAR CHARACTER SET utf8) COLLATE utf8_unicode_ci
        = CAST(schema_name AS CHAR CHARACTER SET utf8) COLLATE utf8_unicode_ci
      AND table_collation NOT LIKE 'utf8%';

  SET @statement = CONCAT('ALTER DATABASE ', schema_name,
    ' CHARACTER SET utf8 COLLATE utf8_unicode_ci');
  -- "This command is not supported in the prepared statement protocol yet"
  -- PREPARE executable FROM @statement;
  -- EXECUTE executable;
  -- DEALLOCATE PREPARE executable;
  SELECT @statement AS `Run this by hand!`;

  OPEN schema_cur;
    FETCH schema_cur INTO table_name;
    IF NOT done THEN
      SET @statement = CONCAT('ALTER TABLE ', schema_name, '.', table_name,
        ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci');
      PREPARE executable FROM @statement;
      EXECUTE executable;
      DEALLOCATE PREPARE executable;
    END IF;

After switching back the delimiter to ';', you can call this stored procedure like this:

CALL convert_to_utf8('meetbees_db');

This converts the meetbees_db database to the utf-8 characters set with the utf8_unicode_ci collation.

(*) The ALTER DATABASE command must be run by hand as the MySQL version that I've tried it on reports that “[this] command is not supported in the prepared statement protocol yet”. That's one command to copy-paste, so you should be able to manage that.

2010-03-06 - anwb rijopleiding cdrom / linux

Regelmatig wordt er niet aan de Linux-gebruikers gedacht als ergens een custom applicatie wordt gemaakt. Zo is het natuurlijk ook bij de Examentraining Rijbewijs B. De 2005 versie bevat een binary voor Windows en MAC OS9. De CD-ROM uit 2008 bevat eentje voor Windows en voor MAC-OSX.

Waar dit keer echter wel aan gedacht is, is het niet nodeloos moeilijk verpakken van de data op de CD.

$ ls /media/cdrom
anwb.ico                       ANWB_Rijopleiding_PC.exe  Desktop DB  Fdata
ANWB_Rijopleiding_MAC_OSX.osx  AUTORUN.INF               Desktop DF  xtras
$ ls /media/cdrom/Fdata/Data/
$ head -n4 /media/cdrom/Fdata/Data/DATA1.CFG
001 11 03 --------------------------
De maximumsnelheid voor een motorvoertuig bedraagt op deze rijbaan:
30 30,0 30,00
$ ls /media/cdrom/Fdata/Gdata/ | head -n3

Mooi! Als het zo makkelijk is, kunnen we daar net zo goed een scriptje omheen vouwen. Dan hoeven we niet een Windows VM te regelen.

Zo gezegd, zo gedaan: (view)

Het gebruik ziet er als volgt uit:

Linux versie van ANWB Examentraining

Het script neemt 3 of 4 argumenten:
- het pad naar de ANWB CD-ROM
- de eerste vraag (bijvoorbeeld 1 of 451)
- de laatste vraag (bijvoorbeeld 50 of 500)
- optioneel of de vragen in een willekeurige volgorde gesteld moeten

$ ./ /media/cdrom 1 2
Attempting to parse ANWB files in /media/cdrom...
Parsing DATA1.CFG (offset 0)...
Parsing DATA10.CFG (offset 450)...
Parsing DATA8.CFG (offset 350)...
Parsing DATA9.CFG (offset 400)...
Found and parsed 500 questions...
Found 500 corresponding images...
<... beantwoord de eerste twee vragen ...>
Right answers: 2
Wrong answers: 0


Het script ook de geluidsfragmenten met de vraag laten afspelen — ja, er is ook een Fdata/Sdata met 500 sounds — laat ik over als exercitie voor de lezer.

2010-01-30 - dutch public broadcasting teletext

Laying around for almost a year, gathering dust, for your enjoyment here is:
a parsable plain text version of the Dutch Public Broadcasting teletext pages.

As the format hardly ever changes, it can be used as an data aggregation source. For example, for fetching the current Eredivisie ranking or the scores for latest Eredivisie games.


 819     Teletekst za 30 jan           
    OETBAL -----------------------------
 +-+      eredivisie,stand per 29/01    
  1. PSV          19 16  3  0  51  47-13
  2. FC Twente    19 15  4  0  49  38-15
  3. Ajax         19 13  3  3  42  55-16
  4. Feyenoord    19 11  5  3  38  30-16
  5. Heracles Al. 19 10  2  7  32  27-26
  6. FC Utrecht   19  8  7  4  31  23-18
  7. AZ           19  9  1  9  28  32-22
  8. NAC Breda    19  7  5  7  26  26-29
  9. VVV-Venlo    19  4  9  6  21  26-27
 10. FC Groningen 19  5  6  8  21  22-28
 11. Vitesse      19  6  3 10  21  23-33
 12. Roda JC      19  5  5  9  20  25-34
 13. Heerenveen   19  6  2 11  20  21-34
 14. Sparta R'dam 19  5  4 10  19  19-35
 15. Willem II    20  5  2 13  17  24-39
 16. NEC          19  3  7  9  16  21-35
 17. ADO Den Haag 19  3  6 10  15  18-35
 18. RKC Waalwijk 20  4  0 16  12  17-39
   programma op 818 / topscorers op 832 
 volgende   nieuws   sport   voetbal    

The teletekst viewer also features colored (html) versions. The source — the website uses the images from the NOS pages as data source — is freely available.