wjd.nu

2007-11-21 - the hiatus explained

As you can see, there is a minor temporal gap in the Notes to self section. There are still enough practical tips and solutions to come. I just haven't had time to jot them down; the main reason being that I've become a parent.

2007-09-23 - more or less useless tips and tricks

More or less useless/useful tips and tricks, bundled together. They weren't worthy of a box div on their own. I gave them only a li each.

• DOS / cmd.exe scripting does have an escape character.
  It exists since windows NT. It's the caret (^), it's the NT equivalent of the unix backslash.
  You didn't know that, did you? ;-) Now you can escape that darned ampersand in URLs.
  For more info, see Escape Characters.
• Serving an unknown filetype with Microsoft IIS 6.0 yields a 404 in the default configuration.
  Your IIS does not know about the .PSB (Photoshop Large Document Format) file extension. “But the file is there, how can it give a 404?,” you might think.
  It's ... drum ruffle ... Microsoft style security:
  “Help! It's an unknown file. IT MUST BE DANGEROUS!1oneone”. *sigh*
  Solution: add the wildcard mime type in the IIS configuration.
• Unmounting a hung NFS mount can be a pain.
  If you have an NFS mount that's stuck because the peer fails to respond anymore, you can unfreeze the mounts quite easily. Simply making the peers IP reappear on any machine in the vicinity that either RSTs (closed port) the NFS port (2049) or speaks NFS should do the trick. The kernel realises that it has a peer again, but the peers configuration has changed, so it unmounts. After/during the unmount all hung processes get unfrozen. (The processes probably get an EINTR on their read(2) call or whatever they were doing, and die because of the error condition.)
  To give yourself an extra IP the old way: ifconfig eth0:1 inet <ip>
  The new way: ip addr add <ip>/32 dev eth0
  (Remove them with ifconfig eth0:1 del <ip> or ip addr del <ip>/32 dev eth0 respectively.)
• You can start putty with a session name instead of a user+host on the command line.
  You have putty.exe stored in a file in your %PATH% and you do WINDOWS+R and type putty user@host normally.
  If the hosts ssh daemon runs on a different port, you wouldn't be able to do that. You can't specify the port on the command line. But if you have a session stored, you can call that one by name instead.
  Use putty @<sessionName> or putty -load <sessionName>.
• The GNOME ability to use CTRL+SHIFT+<HEX> to type Unicode disappeared.
  After one Ubuntu/Linux upgrade, the CTRL+SHIFT+unicode_number stopped working. I had used that before to type the Swedish “å” (a-ring, pronounced like the “o” in “hot”), with either CTRL+SHIFT+E5 (lowercase) or CTRL+SHIFT+C5 (uppercase). I didn't know that Compose (usually right-alt), * (star), a, got the same character so I was not amused. A bit of googling around turned up ISO 14755 and from that I found Special character keyboard shortcuts in Edgy? describing that it now is CTRL+SHIFT+U, CTRL+SHIFT+<HEX> because the GNOME team didn't want to steal 16 possible CTRL+SHIFT keyboard shortcut combinations. Okay, I can live with that. (Yes, I know this is old. I don't dist-upgrade my Ubuntu that often.)
• It's possible to rename computers in your domain from the domain controller.
  You don't have to physically walk up to every computer in your domain you want to rename. With netdom.exe — it is installed by default on Windows 2003 server — you can do it from any computer in the domain even:
  C:\Documents and Settings\Administrator>netdom renamecomputer <oldname>
  /newname:<newname> /force /reboot:1
  /userd:administrator /passwordd:<password>

  Update 2008-08-25

  If you don't have netdom.exe, you must install the Windows 2003 SP1 Support Tools. Short syntax:
  "C:\Program Files\Support Tools\netdom.exe" renamecomputer <oldname>
  /newname:<newname> /ud:administrator /pd:<password>
  /force /reboot:1

2007-09-23 - parsing vi.nl soccer scores xml with perl

Parsing an XML file with perl and XML::DOM takes just a couple of minutes once you know the API.

Voetbal International has a score board of current soccer matches in the Netherlands. They use an XML file to refresh the data periodically on the front page. Using this data is no problem. The XML looks like:

<games>
  <configuration>
  ...
  </configuration>
  ...
  <game>
    <id>167934</id>
    ...
    <info type="live">
      <left>23-9-07</left>
      <center>Eerste helft</center>
      <right>14:30</right>
    <info>
    <home>
      <name>PSV</name>
      <score></score>
      <score></score>
      <score></score>
      <score></score>
    </home>
    <away>
      <name>Feyenoord</name>
    </away>
  </game>
  <game>
    ...
  </game>
</games>

In twenty or so lines of perl, you can make that look like this. For example for quick lookup in your irc channel.

07-9-23 12:40:               AZ 2-3 Ajax            
07-9-23 14:30:              PSV 4-0 Feyenoord       
07-9-23 14:30:        Excelsior 1-3 FC Groningen    
07-9-23 14:30:        FC Twente 3-0 NEC             
07-9-23 14:30:       Heerenveen 2-2 FC Utrecht

How? Like this. Enjoy!

use XML::DOM;
sub get_vi_dot_nl_score {
        my $file = 'http://www.vi.nl/matchcenterv2.xml';
        my $parser = XML::DOM::Parser->new();
        my $doc = $parser->parsefile($file);
        my @ret;
        foreach my $game ($doc->getElementsByTagName('game')) {
                my $info = $game->getElementsByTagName('info')->item();
                my $date = $info->getElementsByTagName('left')->item()->getFirstChild()->toString();
                my $time = $info->getElementsByTagName('right')->item()->getFirstChild()->toString();
                my $homename = $game->getElementsByTagName('home')->item()
                                ->getElementsByTagName('name')->item()->getFirstChild()->toString();
                my $awayname = $game->getElementsByTagName('away')->item()
                                ->getElementsByTagName('name')->item()->getFirstChild()->toString();
                my $homescore = $game->getElementsByTagName('home')->item()->getElementsByTagName('score')->getLength();
                my $awayscore = $game->getElementsByTagName('away')->item()->getElementsByTagName('score')->getLength();
                my @date = split /-/, $date;
                $date = "$date[2]-$date[1]-$date[0]";
                push @ret, sprintf "%s %s: %16s %i-%i %-16s", $date, $time, $homename, $homescore, $awayscore, $awayname;
        }
        @ret = sort @ret;
        return join "\n", @ret;
}

2007-08-09 - unexpected bashslashes / template php

If you're suddenly seeing backslashes in old php generated pages, check if your php got upgraded to a version after 5.1.1. In my case I was seeing odd backslashes in pages generated with a modified Yapter template engine.

The cause could be found in the PHP manual:

Again, if you try to escape any other character, the backslash will be printed too! Before PHP 5.1.1, backslash in \{$var} hasn't been printed.

That was exactly what was going on. The modifications to Yapter.php, replacing "\{$var}" with "{"."$var}":

$ diff classes/Yapter.php.old classes/Yapter.php -uw
--- classes/Yapter.php.old      2007-08-09 11:33:41.000000000 +0200
+++ classes/Yapter.php  2007-08-09 11:33:17.000000000 +0200
@@ -163,7 +163,7 @@
                                        }
                                        for ($j = $i; $j >= $currblockstart; $j--) {
                                                if ($j == $currblockstart && $currblocktype == 'BLOCK') {
-                                                       $block['content'][$j] = "\{$currblockname}\n";
+                                                       $block['content'][$j] = "{"."$currblockname}\n";
                                                        } else {
                                                        unset($block['content'][$j]);
                                                }
@@ -209,9 +209,9 @@
                                        }
                                        // Make this line a variable...
                                        if ($this->debug) {
-                                               $this->log .= "$this->prelog...turning line $i into variable (<i>\{$name}</i>)<br>\n";
+                                               $this->log .= "$this->prelog...turning line $i into variable (<i>{"."$name}</i>)<br>\n";
                                        }
-                                       $block['content'][$i] = "\{$name}\n";
+                                       $block['content'][$i] = "{"."$name}\n";
                                        // ...and include the given file...
                                        if ($this->debug) {
                                                $this->callSubFunc("$this->prelog<font color=\"green\">...calling addBlockFromFile() for this INCLUDE-tag</font><br>\n");
@@ -227,9 +227,9 @@
                                        // Make this line a variable...
                                        if ($this->debug) {
                                                $this->log .= "$this->prelog...REUSE-tag found on line $i<br>\n";
-                                               $this->log .= "$this->prelog...turning line $i into variable (<i>\{$name}</i>)<br>\n";
+                                               $this->log .= "$this->prelog...turning line $i into variable (<i>{"."$name}</i>)<br>\n";
                                        }
-                                       $block['content'][$i] = "\{$matches[4]}\n";
+                                       $block['content'][$i] = "{"."$matches[4]}\n";
                                        // ...and get this REUSE value from the block definition list...
                                        if ($this->debug) {
                                                $this->callSubFunc("$this->prelog<font color=\"green\">...calling addBlockFromDef() for this tag</font><br>\n");

2007-08-02 - php sessions / integer keys produce alzheimer

Somehow, integer keys — or strings that look like integers — are forgotten by the PHP $_SESSION variable. This might be logical if you take into account that once upon a time, every one used register_globals. If you're not expecting it, you'll be amazed that not only some of your dictionary keys disappear, but also that everything after disappears as well.

In the Examples section — the examples? Why isn't this marked with one of those big red warnings? — you can find:

The keys in the $_SESSION associative array are subject to the same limitations as regular variable names in PHP, i.e. they cannot start with a number and must start with a letter or underscore. For more details see the section on variables in this manual.

Not only does it not state that that it does a complete Alzheimer on you if you supply an integer-like key, the sentence about that it must have a legal variable name is not enforced. See this example:

<?php
session_start();
if (sizeof($_SESSION)) {
        echo 'GETTING: ';
        print_r($_SESSION);
        session_unset();
} else {
        echo 'SETTING: ';
        $_SESSION['normal_string_key'] = 1;
        $_SESSION['1234.5'] = 2; // Not a legal variable name, but a legal key
        $_SESSION['1234'] = 2;   // Same as previous
        $_SESSION['another_normal_key'] = 3; // Legal, but gets forgotten
        print_r($_SESSION);
}
?>

This produces, alternating:

SETTING: Array
(
    [normal_string_key] => 1
    [1234.5] => 2
    [1234] => 2
    [another_normal_key] => 3
)
GETTING: Array
(
    [normal_string_key] => 1
    [1234.5] => 2
)

If you read the quote, one would expect to only get 'normal_string_key' and 'another_normal_key', but as you can see, it's not quite that. You have been warned ;-)

2007-07-23 - apache / rewrite rule / ip based

Let's say you've installed a new version an http service that needs just that last bit of online testing. Once again, mod_rewrite comes to the rescue. You can let those in your network at work test it without them having to change anything — and without having to wait for dns cache flushes. Rewrite rules can in a simple and effective way let only a limited set of users see the new system by using the REMOTE_ADDR as RewriteCond.

Imagine you have http://someservice.mynet.net. Add a vhost for newservice.mynet.net and add the following in a .htaccess. Replace the IP address with your IP or network using a regular expression. It will 302 (temporary redirect) all requests for your original service to the testing location.

RewriteEngine on
RewriteCond %{REMOTE_ADDR} ^12.34.56.78$
RewriteRule ^(.*)$ http://newservice.mynet.net/$1

2007-07-02 - terminal services / restart / change port

After changing the listening port for your RDP (Remote Desktop, Terminal Services) server in the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp\PortNumber, you'll need to poke the server to get it to listen on the new port.

According to http://support.microsoft.com/kb/187623 you need to:

NOTE: You must restart the Terminal Server before the new listening port becomes active, or recreate the RDP listener via Terminal Services configuration.

Solution 1: Go to Services, select Terminal Services, right-click and select Restart.
... WRONG. You're not allowed to restart it.

Solution 2: Delete a listener.. somewhere.. and remember what listener settings I have. Troublesome.

Solution 3: Run tsadmin.exe (yes.. there are a bunch of ts*.exe files in the system32 folder), click on the “RDP-Tcp (listener)” and select Reset from the Action menu. Better.. but why isn't this documented? (And why is there no killall -HUP "terminal services"? But that's a different discussion.)

2007-06-14 - photoshop / crappy png support / wrong gamma

As you might know, Photoshop has had lousy PNG support for a while now. That it does not compress images as well as it should, I can live with, but the fact that it stores the gamma value when saving for the web (I wasn't able to find any way around this) I cannot.

Somewhere along the line, Internet Explorers ability to read PNG files got the ability to read and use that gamma value. Normally one would think that this is a good thing. But when combining PNGs with CSS background colors and other image types to form a nice looking page style, having overly bright PNGs is bad.

My designer colleague refuses to use anything but Photoshop for raster images — understandable from his point of view — but I want PNGs instead of GIFs that have a limited palette and do not compress as well (unless you're using Photoshop of course, haha).

But, since I'm roaming around in the shell 70% of the time, this shell script does the last bits that Photoshop and my designer can't manage. Supply all the PNG files on the command line and it will use pngcrush(1) to (losslessly) compress them and remove the gamma value that we didn't want.

walter@wza:0:~$ cat /usr/local/bin/wjd_pngcrush
#!/bin/sh

for orig in "$@" ; do
    temp="`mktemp`"
    cp -f "$orig" "$temp" 2>/dev/null \
        && pngcrush -quiet -rem gAMA "$temp" "$orig" >/dev/null \
        && echo "OK:     $orig" \
        || echo "FAILED: $orig" >&2
    rm -f "$temp"
done

2007-06-12 - logcheck / mail too large

Instead of the normal logcheck messages, I got this in a mail from the Cron Daemon:

postdrop: warning: uid=109: File too large
send-mail: fatal: logcheck(109): Message file too big Can't send mail: sendmail process failed with error code 69

Solution: run the following command to get the log that you would normally get by mail in a file (/tmp/logcheck.txt). Note that this will take some time.

root@server:0:~# su -m -c '/usr/sbin/logcheck -o >/tmp/logcheck.txt' logcheck

2007-06-03 - chown / missing suid/sgid bits

When your non-*nix guru colleague accidentally hits chown -R www-data / on your entire system, you're bound to be unhappy. Even after fixing the ownership of the files, you may still see the odd failures of some tools. It's very possible that your OS decided to clear the SUID and SGID bits.

From man 2 chown:

When the owner or group of an executable file are changed by a non-superuser, the S_ISUID and S_ISGID mode bits are cleared. POSIX does not specify whether this also should happen when root does the chown(); the Linux behaviour depends on the kernel version.

I wasn't about to install a new system over this. Luckily I had similar systems. The following snippet gave me enough info to restore most of the needed functionality.

$ find -H /bin /usr /lib -type f -and '(' -perm -u=sx -or -perm -g=sx ')' -print0 \
  | xargs -0 ls -l \
  | perl -e 'for(<>){s/^...(.)..(.)\S+\s+\d+\s+(\S+)\s+(\S+)\s+\d+\s.{17}(.*?)\r?\n?$/chown $3:$4 "$5";chmod u+"$1",g+"$2" "$5"/;print "$_\n";}'
chown root:root "/bin/mount";chmod u+"s",g+"x" "/bin/mount"
chown root:root "/bin/ping";chmod u+"s",g+"x" "/bin/ping"
chown root:root "/bin/ping6";chmod u+"s",g+"x" "/bin/ping6"
chown root:root "/bin/su";chmod u+"s",g+"x" "/bin/su"
chown root:root "/bin/umount";chmod u+"s",g+"x" "/bin/umount"
chown daemon:daemon "/usr/bin/at";chmod u+"s",g+"s" "/usr/bin/at"
chown root:tty "/usr/bin/bsd-write";chmod u+"x",g+"s" "/usr/bin/bsd-write"
chown dcc:dcc "/usr/bin/cdcc";chmod u+"s",g+"x" "/usr/bin/cdcc"
... etc ...

2007-04-26 - opening smb/cifs links / firefox / opera

For those using my smburl.sh script to open smb:// URLs in Windows: I've got an updated VBScript version. This one doesn't need external binaries (sh.exe) and this one doesn't flash the cmd.exe window like the old one did. And, this one allows you to register and unregister itself from the registry.

Install steps:

• Download and save it in a directory of your liking.
  E.g. in: C:\WINDOWS
• Run it with "register" as argument.
  E.g.: Start -> Run -> C:\WINDOWS\smburl3.vbs register

2007-04-22 - ipac-ng / fetchipac / mrtg / configuration

See the previous Note to get ipac-ng-1.31 to work on debian/etch.

The configuration examples in ipac-ng-1.31 are a little outdated. You can still use them for clues, but the following config will help you with the most basic mrtg setup.

walter@wza:0:~$ cat /etc/ipac-ng/ipac.conf
# This is the main ipac-ng configuration file.  It contains the
# configuration directives that give the ipac-ng its instructions.

## specify access agent
# supported are: 'files'.
access agent = files

## accouting agent
# supported are: 'iptables', 'ipchains'
account agent = iptables

## storage
# supported are: 'gdbm', 'postgre', 'plain-file' (plain-file is not recommended)
# postgre is the best & fastest method now
storage = plain-file

# set the hostname, used to store\fetch\another work with database
# get from hostname() if not specified here
hostname =

## rules file
rules file = /etc/ipac-ng/rules.conf

## login all users at startup (only those who have enough cash)
# this is not used at all - needs removal from source
#login at start = yes

## support for traffic passing to\from auth host 
# this is not used at all - needs removal from source
#auth host = sibinet.com

## don't store lines contains only zeroes to speedup processing and to save space
drop zero lines = yes

## This parameters controls database location
# left blank 'db host', 'db port' for local database
# as now, both databases (access and storage) configured by these parameters
db host = localhost
db name = ipac
db user = root
db port = 5432
db pass = ""

walter@wza:0:~$ cat /etc/ipac-ng/rules.conf
## Example config file with accounting rules
## Install as /etc/ipac-ng/rules.conf
##
## Format:
## Name of rule|direction|interface|protocol|source|destination|extension|
## WARNING!!!! spaces are not allowed before and after '|'.
##
## where
## Name of rule         Any string to identify this rule, use only A-Za-z0-9 and space
## direction            ipac~fi  - forward in
##                      ipac~fo  - forward out
##                      ipac~i   - outgoing from machine with ipac-ng to other host(/net)
##                                      (or incoming to otherhost)
##                      ipac~o   - incoming to machine with ipac-ng
##                                      (or outgoing from otherhost)
##
## interface            interface name, empty means all interfaces (dont try to use ip numbers here!)
## protocol             tcp | udp | icmp | all
## source               \
## destination          both as described in ipfwadm(8) (a.b.c.d[/n]), or empty
## extension            optional match (e.g. mac --mac-source 00:10:20:30:40:50)
##                                      (see /lib/iptables/ for clues)
##
#
IN  wza.all|ipac~o||all||||
OUT wza.all|ipac~i||all||||
IN  natmachine1.all|ipac~fi||all|0/0|10.0.0.2||
OUT natmachine1.all|ipac~fo||all|10.0.0.2|0/0||
IN  natmachine2.all|ipac~fi||all|0/0|10.0.0.3||
OUT natmachine2.all|ipac~fo||all|10.0.0.3|0/0||

If you hadn't installed mrtg already, you might want it, along with the mrtgutils.

root@wza:0:~# apt-get install mrtg mrtgutils

/usr/bin/mrtg-ip-acct from mrtgutils can already fix graphs for traffic per device. You'll need a small script to get the same output for ipacsum and your rules defined in /etc/ipac-ng/rules.conf.

walter@wza:0:~$ cat /usr/local/bin/ipaccount
#!/usr/bin/perl -w
#
# /usr/local/bin/ipaccount
#
# ipaccount reads data from ipac-ng logs and updates mrtg log file(s).
# ipaccount is used with ipac-ng and mrtg to create ip traffic graphs.
#
# The "fetchipac" (part of ipac-ng) application creates log files that
# contain historical IP traffic data depending on rules set in ipac.conf.
#
# The mrtg reads data from a unique log file for each "target" ($TARGET.log)
# The "target" log files depend on rules set in the mrtg.cfg file.
# mrtg can use this data for creation of graphs.
#
# Read ipac-ng and mrtg manuals for more information.
#########################################################################

$ENV{PATH}="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin";

die "Usage: ipaccount <relative time> <chain>\n" if $#ARGV < 1;

@output = `/usr/local/sbin/ipacsum --exact -s $ARGV[0]`;
die "Can't execute ipacsum: $!\n" if !defined @output;

$bytesin=0;
$bytesout=0;

foreach (@output)
{
    # Incoming...
    $bytesin = $1 if (/^[\* ]\s+IN\s+\Q$ARGV[1]\E\s+\:\s+(\d+)/);

    # Outgoing...
    $bytesout = $1 if (/^[\* ]\s+OUT\s+\Q$ARGV[1]\E\s+\:\s+(\d+)/);
}

print "$bytesin\n$bytesout\n";
print `/usr/bin/uptime`;
print `/bin/hostname -f`;

With that, you get similar output that can be fed to mrtg. See this:

walter@wza:0:~$ /usr/bin/mrtg-ip-acct eth1
4048186296
3736514381
  4:19pm  up 1 day, 21:10,  6 users,  load average: 0.01, 0.02, 0.00
wza.selwerd.lan
walter@wza:0:~$ /usr/local/bin/ipaccount 5m natmachine2.all
352309
61750
  16:19:53 up 1 day, 21:10,  6 users,  load average: 0.01, 0.02, 0.00
wza.selwerd.lan

All that is left is to define these targets in /etc/mrtg.cfg.

walter@wza:0:~$ cat /etc/mrtg.cfg
# Created by 
# /usr/bin/cfgmaker --global 'WorkDir: /var/www/mrtg' --output /etc/mrtg.cfg public@::1

RunAsDaemon: no

### Global Config Options

#  for Debian
WorkDir: /var/www/mrtg
  
### Global Defaults

Options[_]: growright
EnableIPv6: no
WorkDir: /var/www/mrtg

### ...

Title[^]: Traffic Analysis for WZA
YSize[^]: 180
MaxBytes[^]: 13500000
PageTop[^]: Traffic Analysis

### Ifaces

Target[wza.eth0]: `/usr/bin/mrtg-ip-acct eth0`
Target[wza.eth1]: `/usr/bin/mrtg-ip-acct eth1`
Target[wza.all]: `/usr/local/bin/ipaccount 5m wza.all`
Target[natmachine1.all]: `/usr/local/bin/ipaccount 5m natmachine1.all`
Target[natmachine2.all]: `/usr/local/bin/ipaccount 5m natmachine2.all`

2007-04-21 - fetchipac / dlopen / mysql / iptables / name clashes

Skip to the end of this piece to just get ipac-ng to work under debian/etch.

Getting fetchipac to work on debian/etch after the upgrade from debian/sarge was far from trivial. First of all it tried to allocate some ~3GB of memory, exiting like this (use ltrace and strace to find out exactly what it does before it dies):

root@wza:0:~# /usr/sbin/fetchipac -S
 calloc failed
: Cannot allocate memory

After a bit of digging with the ipac-ng source and gdb, I found out that the struct iptables_target had gotten bigger and that a size struct member was now most likely pointing to a char * because the iptables guys had added a u_int8_t revision field.

iptables had changed quite a bit, so ipac-ng-1.27-5.1 wasn't going to cut it. ipac-ng-1.31, unfortunately, wasn't up to date either, but Peter Warasin had made a patch already. First mentioned on the ipac-ng developer list and later with the patches on the IPCop tracker. These patches fix the the problem of the enlarged struct. So, on to the next problem.

Somehow the register_target function didn't get called. This function is supposed to get called by the iptables module when it gets loaded. (For the standard targets — ACCEPT, DROP, QUEUE and RETURN — by /lib/iptables/libipt_standard.so.) Having an “onload”-function is supported by special gcc attributes; see GNU/Linux library constructor and destructor functions.

The problem, it seems, was that libmysqlclients shared library has a function with the same name as iptables’ initialization function; namely my_init.

See the following code examples demonstrating how my my_init function silently gets overridden by libmysqlclients my_init even though my my_init has the special __attribute__ ((constructor)) directive.

walter@wza:0:~/src/test$ cat shared.c
extern void callback(int);
void __attribute__ ((constructor))my_init(void) {
  callback(0x1234);
}

walter@wza:0:~/src/test$ cat dlopen.c
#include <stdio.h>
#include <dlfcn.h>
extern void callback(int n) {
  fprintf(stderr, "callback: 0x%x\n", n);
}
int main() {
  dlopen("./shared.so", RTLD_NOW);
  return 0;
}

walter@wza:0:~/src/test$ gcc shared.c -o shared.so -Wall -shared
walter@wza:0:~/src/test$ gcc dlopen.c -o dlopen -Wall -rdynamic -ldl
walter@wza:0:~/src/test$ ./dlopen
callback: 0x1234
walter@wza:0:~/src/test$ gcc dlopen.c -o dlopen -Wall -rdynamic -ldl -lmysqlclient
walter@wza:0:~/src/test$ ./dlopen
walter@wza:0:~/src/test$ 

Notice the lack of output from the second dlopen that was dynamically linked with libmysqlclient.

Now, we can't blame mysql for using my_init as a function name. But it is a real problem which forces me to not use mysql in any application that loads shared libraries that have a initialization function called my_init.

Summarizing, to get ipac-ng (with fetchipac) to work under debian/etch, do the following:

1. Download ipac-ng-1.31.tar.bz2 (mirror) and unpack.
2. Apply ipac-ng-1.31-iptables-1.3.1.patch (mirror) which updates it for the iptables change (patch -p0).
3. Apply ipcop-ipac-ng-1.31-fetchcounter.patch (mirror) which fixes some counter issue.
4. Apply my ipac-ng-1.31-debian-etch.patch which fixes one compile error, a couple of warnings and disables MySQL support.
5. Build by entering the directory, running ./configure --with-postgresql-inc=/usr/include/postgresql and make and make install. You can skip the previous 4 steps by downloading this already patched ipac-ng.

See my next Note for help with configuration files.

2007-03-22 - lvalue / rvalue / define / google

Recently an intern asked me about a PHP error he got in a piece of seemingly good code. Trimmed down, it looked like this:

if (empty(trim($some_var))) {
    // do something
}

With PHP4 one gets “Parse error: parse error, unexpected T_STRING, expecting T_VARIABLE or '$' in FILE on line LINE”. With PHP5 one gets “Fatal error: Can't use function return value in write context in FILE on line LINE”.

If you read the function description — or language construct description actually — it becomes obvious why the above doesn't work. empty needs an l-value as argument. trim($some_var) is an r-value.

While explaining this, the alt-d, tab, define:lvalue sequence (go to the google search bar and enter “define:lvalue”) revealed only one result, the one from wikipedia. The shown definition summary does not cover l-value, it covers the superset value. That's not what I was looking for.

Now, in my curiosity to see if google picks up on Definition Lists from any old site, in this case this one, I'm providing the definition here:

lvalue

L-values are values (or expressions) that have addresses (a place in memory). They're called l-values because they can be used on the left side of an assignment, i.e. they can be assigned to.
GOOD: an_l_value = some_value + 4.
BAD: 4 = some_value.
In the C language *(&some_value + 10) (the storage space at 10 values further in memory than the location of some_value) is a valid l-value, while &some_value (the address number where some_value is stored) is not.

rvalue

R-values are all values or value-returning expressions. (Or the aforementioned minus all l-values, depending on who you ask). They can be full blown expressions like some_func(some_value) + 4. They're called r-values because they can be used on the right side of an assignment.
GOOD: some_value = an_r_value.
ALSO GOOD: some_value = some_other_value + 4.

2007-03-18 - linuxprinting / firefox / brother hl1230

After succesfully setting up this linuxprinting, my html documents printed with Mozilla Firefox had funky margins and were missing the title/URL and pagenumbering/date.

Firefox knows of two settings for margins: under File -> Page Setup -> Margins & Header/Footer the Top, Left, Right and Bottom settings and under File -> Print -> Printer Properties -> Gap from edge of paper to Margin again four values. The former are specified in millimetres and the latter in inches.

The Gap from edge settings specify the space between the edge of the paper and the closest writable position. This is the offset for the headers and footers. If you set these values too low, parts of your headers will not get printed.

The Margins settings specify the space between the edge of the paper and the html page. Once again, if you set these too low, you'll end up with missing letters, or - equally bad - you'll end up writing on top of your headers and footers.. Fortunately you'll usually want a larger margin here, as otherwise you wouldn't have any white space to hold the paper by or make tiny notes on.

The Gap from edge settings I had to find using trial and error. Not only because I didn't know how close to the edge of the paper my printer would print, but also because the values do not make sense (to me). The Margins settings are almost correct, so you could easily modify the margin temporarily for a custom document.

For the Brother HL-1230 I settled for the following:

2007-03-17 - linuxprinting / cups / brother hl1230

Hm.. that was trickier than I thought, getting CUPS to work properly with my clients.

The steps:

1. root@server# apt-get install cupsys

2. Edit /etc/cups/cupsd.conf and add a couple of Allow From lines. (Don't add the addresses to the existing line, create a new line per IP-range.)

   Allow From 127.0.0.1
   Allow From 10.102.221.0/24

3. Add myself to the lpadmin group. Edit /etc/group.
4. Restart/reload cupsd (/etc/init.d/cupsys restart).
5. Surf to http://server:361/, go to admin panel and add my Brother HL-1230 printer. (Although it seemed to detect the hl-1230, I was asked to specify the driver myself. But that's okay, I guess.)
6. Install a network printer on client-winxp. Use http://server:631/printers/brother-hl1230 (the printer identifier) as the URI. Set up the correct driver and succesfully print a document. W00t! That was easy. (In my case the 1230 wasn't in the driver list, but the 1250 was. I selected that, had a misprint and went to the printer properties where I could select the 1230 driver. I suspect it wasn't in the new-drivers list because this printer had been installed locally already. But I'm not sure.)
7. Okay, Windows can talk to my cupsd fine. Getting client-ubuntu to play along should be 2 clicks... Wrong. Attempt one: open gedit, type a few words, hit print, add printer, network printer, enter the same URI, and then ... nothing.
8. I went to the System menu. In Administration there is a Printing settings applet. A popup with "The CUPS server could not be contac" and a quit.
9. Time to check the cups client settings. Yes, in /etc/cups/client.conf I got to specify my print server.

   #
   # ServerName: the hostname of your server.  By default CUPS will use the
   # hostname of the system or the value of the CUPS_SERVER environment
   # variable.  ONLY ONE SERVER NAME MAY BE SPECIFIED AT A TIME.  To use
   # more than one server you must use a local scheduler with browsing
   # and possibly polling.
   #
   
   ServerName server
   

10. This time around the Printing applet opened up fine and I saw my printer. Success! Alas, every print job I initiated got the aborted state in the “Completed Jobs” list. “Restart Job” returned a client-error-not-possible.
11. Next. Checking /var/log/cups/error_log revealed a hint, literally.

    E [17/Mar/2007:13:54:21 +0100] PID 5385 stopped with status 1!
    I [17/Mar/2007:13:54:21 +0100] Hint: Try setting the LogLevel to "debug" to find out more.
    

12. Yes, edit /etc/cups/cupsd.conf, do as the hint says and restart cupsys.

    LogLevel debug

13. Checking the error log while creating a new job:

    D [17/Mar/2007:14:07:35 +0100] [Job 8] renderer PID kid4=6038
    D [17/Mar/2007:14:07:35 +0100] [Job 8] renderer command: gs -q -dBATCH -dPARANOIDSAFER -dNOPAUSE -sDEVICE=hl1250 -dEconoMode=0 -dSourceTray=0 -sOutputFile=- 
    -
    D [17/Mar/2007:14:07:35 +0100] [Job 8] 
    D [17/Mar/2007:14:07:35 +0100] [Job 8] Closing renderer
    D [17/Mar/2007:14:07:35 +0100] [Job 8] foomatic-gswrapper: gs '-dBATCH' '-dPARANOIDSAFER' '-dNOPAUSE' '-sDEVICE=hl1250' '-dEconoMode=0' '-dSourceTray=0' '-sO
    utputFile=/dev/fd/3' '/dev/fd/0' 3<&1 1>&2
    D [17/Mar/2007:14:07:35 +0100] [Job 8] sh: line 1: gs: command not found
    D [17/Mar/2007:14:07:35 +0100] [Job 8] renderer return value: 127
    D [17/Mar/2007:14:07:35 +0100] [Job 8] renderer received signal: 127
    

14. Aha, no ghostscript. Install and try again.

    root@server# apt-get install gs

    D [17/Mar/2007:14:09:03 +0100] [Job 9] renderer PID kid4=6236
    D [17/Mar/2007:14:09:03 +0100] [Job 9] renderer command: gs -q -dBATCH -dPARANOIDSAFER -dNOPAUSE -sDEVICE=hl1250 -dEconoMode=0 -dSourceTray=0 -sOutputFile=-
    -
    D [17/Mar/2007:14:09:03 +0100] [Job 9] foomatic-gswrapper: gs '-dBATCH' '-dPARANOIDSAFER' '-dNOPAUSE' '-sDEVICE=hl1250' '-dEconoMode=0' '-dSourceTray=0' '-sO
    utputFile=/dev/fd/3' '/dev/fd/0' 3>&1 1>&2
    D [17/Mar/2007:14:09:03 +0100] [Job 9]
    D [17/Mar/2007:14:09:03 +0100] [Job 9] Closing renderer
    D [17/Mar/2007:14:09:03 +0100] [Job 9] GPL Ghostscript 8.01 (2004-01-30)
    D [17/Mar/2007:14:09:03 +0100] [Job 9] Copyright (C) 2004 artofcode LLC, Benicia, CA.  All rights reserved.
    D [17/Mar/2007:14:09:03 +0100] [Job 9] This software comes with NO WARRANTY: see the file PUBLIC for details.
    D [17/Mar/2007:14:09:03 +0100] [Job 9] Unknown device: hl1250
    D [17/Mar/2007:14:09:03 +0100] [Job 9] renderer return value: 1
    D [17/Mar/2007:14:09:03 +0100] [Job 9] renderer received signal: 1
    

    Better, but still no ink on my paper.

15. root@server# gs -h | grep hl12

    No results. That's unfortunate. CUPS documentation tells me that “you can see the available Ghostscript drivers on your system by running 'gs -h'. If the driver you need is not listed, you need to obtain a new Ghostscript package which includes this driver, or compile Ghostscript yourself. ESP Ghostscript includes the largest set of drivers, so is probably the best choice. For some third-party drivers, the driver authors distribute special Ghostscript packages including the needed driver; give that a shot first.” Hm..

    root@server# apt-get install gs-eps
    root@server# gs-esp -h | grep hl12
       hl1240 hl1250 hl7x0 hpdj1120c hpdj310 hpdj320 hpdj340 hpdj400 hpdj500
    

    Much better.
16. A test page. W00t! A second try on client-ubuntu with gedit. And more success.
17. Don't forget to change the LogLevel back to info.

I guess I won't be recommending linux to my mother this time around either.

2007-03-09 - recipe / lamb vindaloo

Source: http://www.sheepscreek.com/recipes/lamb_vindaloo.html

An Everyday Vindaloo from Curries Without Worries by Sudha Koul,
adapted.

Very hot Lamb Vindaloo for 3+ people. Excellently suited for “hete vrijdag”. Dutch ingredient names are between square brackets.

• 1 (kg) boneless lamb [lamsbout, zonder bot]
  Get it at your Halal butcher, as the nearest Dutch butcher didn't have lamb, nor knew why it didn't... odd.
• Rice [rijst]
• Regular vinegar [azijn]
• Vegetable oil [plantaardige olie]
• Water [water]

• 3 large onions [ui], chopped
• 6 garlic cloves [knoflookteentje], chopped
• 32 red/green “bird's eye chillies” [Rawit] (Capsicum frutescens), chopped
  Any pepper that is hot but not too taste-rich will do fine. Note that 32 is a multiple of 8 (ACHT!).
• 30 (g) fresh coriander/cilantro [koriander], chopped
  It's okay to use a little extra.
• 2 (T) (=tablespoon) fresh ginger [gember], chopped
• 1 (T) cumin seeds [komijnzaad], coarsely ground
• 5 (cm) cinnamon stick [kaneelstokje], coarsely ground
  Don't overdo it, as the cinnamon smell will dominate while preparing.

• 1 (T) tumeric [kurkuma/koenjit/geelwortel], ground
• 1 (t) (=teaspoon) black pepper [zwarte peper], ground
• Salt [zout] to taste
• 1 (T) sugar [suiker]
• 1/2 (t) cloves [kruidnagel], ground

Chop the meat into pieces of 2.5 x 5 x 5 (cm). Marinate it in 3 (T) of vinegar for a couple of hours in the refrigerator. Be sure to remove the larger pieces of skin after marinating.

Blend the onions, garlic, chilli, coriander, ginger, cumin and cinnamon with another 4 (T) of vinegar and a drop of water. Insert into hot saucepan (wokpan met deksel, dut.) with 6+ (T) of oil. Fry briskly for a couple of minutes.

Add the meat, tumeric and the black pepper. Fry well for 10 minutes, taking care not to burn the meat. Stir briskly, constantly.

Add a bit of salt, the sugar, cloves and two cups of water. Stir, cover, reduce heat to medium, and cook for about 45 minutes, until the gravy thickens and the meat is completely tender. You may need to add water and/or oil from time to time.

Remember to cook the rice in time. And don't be stingy with the amount of rice because someone might puss out on the incendiary hot food. If you like, you can add a couple of large chunks of precooked potatoes to the vindaloo, about ten minutes before its done. Enjoy!

2007-02-21 - microsoft exchange / changing from address / rant

Microsoft Exchange is doing From address checking in your internal e-mail. But why? In the default configuration, the From: header may be spoofed all you like when it comes from the outside. But when I want to change the percieved sender to myname@my-other-company.tld in my mail client, the Exchange server decides that it should suddenly do filtering.

Where I work, we have several e-mail addresses for the different sites that we maintain. They're all myname@<sitename> and our Microsoft Exchange server knows exactly which e-mail address gets sent to which user. But when you try to change your From-header in your mail client to something other than your primary mail address, you get the following returned mail crap:

Your message did not reach some or all of the intended recipients.

Subject: mysubject
Sent: 2007-02-20 11:02

The following recipient(s) could not be reached:

MyRecipient on 2007-02-20 11:02
You do not have permission to send to this recipient. For assistance, contact your system administrator.
MSEXCH:MSExchangeIS:/DC=local/DC=MyDomain:MYSERVER

“It stops people from sending mail pretending to be the CEO.” Bah! For small (and medium) sized businesses this is not a scenario that is likely to cause any trouble. Moreover, in most cases one can still send mail through an SMTP server after which - in the default configuration - no checking is done to ensure that you aren't spoofing your CEO.

Where to turn this nonsense off? Nowhere? I could find some ChooseFromOWA for Outlook Web Access that's supposedly fixing things. That one costs money and consists of third party binaries and registry hocus-pocus that I don't trust offhand on our server. I could also find Send As functionality that allows me to change my From address to that of a different user in the company. Still I'm only allowed to use their primary e-mail address. Where is the damn don't-be-stupid-now switch when you need it?

The only other option that I could find (and think of) was to add an e-mail account for EACH AND EVERY CLIENT that wants to switch From addresses every now and then. (See also: http://www.webservertalk.com/archive128-2006-9-1672180.html)

So, the crappy solution. Add a new account for POP3 or IMAP. Enter bogus data as the POP3/IMAP server (of course you aren't allowed to click Next before filling out data you're not going to use *sigh*) and add a working SMTP server. Now you're allowed to enter whichever (From) e-mail address you like. Now tell your Outlook to not check this non-existent incoming mail server by unchecking this account in Define Send/Recieve-groups (under Tools).

This would be one of those features I'd like to call a bug :-(

2007-02-08 - mssql 7.0 / backup database job / success and failure

Somehow some database backup jobs were succeeding with MSSQL 7 and some were failing with the following extremely helpful message in the Event log:

SQL Server Scheduled Job 'MYDATABASE backup' (0x9963013EAE380C4546ECF7C2B2D12F4A) - Status: Failed - Invoked on: 8-2-2007 0:30:00 - Message: The job failed. The Job was invoked by Schedule 10 (Schedule 1). The last step to run was step 1 (Step 1).

The jobs that were succeeding were the backups of the smaller databases (around 20 MiB or less), the failing jobs were of databases of 64 MiB or more. Oddly, backing these up by hand - using the SQL Server Enterprise Manager -> Backup -> Backup a Database and then not scheduling the backup - worked fine. Some Microsoft knowledge base article pointed me to OSQL. OSQL does give us something to go by:

The bigger database (around 250 MiB):

C:\MSSQL7\BACKUP>osql -UmyUsername -PmyPassword -n -Q "BACKUP DATABASE INTERACTIER TO DISK = 'C:\MSSQL7\BACKUP\INTERACTIER.BAK' WITH INIT, NOUNLOAD, NAME = 'backup INTERACTIER', NOSKIP, STATS = 10, NOFORMAT"
[Microsoft][ODBC SQL Server Driver][Shared Memory]ConnectionRead
(WrapperRead()).
[Microsoft][ODBC SQL Server Driver][Shared Memory]General network error. Check
your network documentation.
10 percent backed up.

The smaller database (around 1 MiB):

C:\MSSQL7\BACKUP>osql -UmyUsername -PmyPassword -n -Q "BACKUP DATABASE model TO DISK = 'C:\MSSQL7\BACKUP\model.BAK' WITH INIT, NOUNLOAD, NAME = 'backup model', NOSKIP, STATS = 10, NOFORMAT"
99 percent backed up.
Processed 96 pages for database 'model', file 'modeldev' on file 1.
100 percent backed up.
Processed 1 pages for database 'model', file 'modellog' on file 1.
Backup or restore operation successfully processed 97 pages in 0.670 seconds
(1.175 MB/sec).

A bit of googling and some trial and error later gives me the following solution:

• In the SQL Server Client Network Utility “Named Pipes”, “TCP/IP” and “Multiprotocol” were enabled. Remove all but “TCP/IP”.
• “Enable shared memory protocol” was disabled. Enable it.
• Restart the server (not sure if this is needed). Now the OSQL command should work. Restart the SQL Server Enterprise Manager. Now database backup jobs, including the large ones, should work. Hit Run Job and press F5 to check the status.

I don't know why backups by hand did succeed before this, though. But hey, everything works now.

2007-01-09 - libapr / read performance

God's Gift to C is how Reg developer describes libapr, the Apache Portable Runtime. Now, the Basic Class Library stuff is really a treat. A standard reusable array type with automatic resource management and easy string operations are just a few of the niceties. If you start using the Portability Layer on the other hand, you might find that the performance may be less than you bargained for.

Obviously, if you're reading configuration files or other tiny tasks, this is not a problem. When you're doing disk I/O for half the time in your application, however, you'll notice that the portable file stream implementation is far from optimized.

Look at the following two application timings that use apr_file_read_full and fread(3) respectively:

$ /usr/bin/time ./libapr_apr_file_read_full data >/dev/null
0.34user 0.66system 0:01.01elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+439minor)pagefaults 0swaps

$ /usr/bin/time ./fread data >/dev/null
0.33user 0.03system 0:00.37elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+438minor)pagefaults 0swaps

I'm guessing that the read(2) function is on 0xffffe410 in the following qprof(1) (profiler) output:

$ qprof ./libapr_apr_file_read_full larger_data
qprof: /tmp/libapr_apr_file_read_full: 1502 samples, 1502 counts
main                                                             16     (  1%)
libc.so.6(__read)                                                4      (  0%)
libapr-0.so.0                                                    9      (  1%)
libapr-0.so.0(apr_file_read_full)                                39     (  3%)
libapr-0.so.0(apr_file_read)                                     22     (  1%)
[0xffffe401]                                                     2      (  0%)
[0xffffe405]                                                     1      (  0%)
[0xffffe410]                                                     1399   ( 93%)
[0xffffe411]                                                     8      (  1%)
[0xffffe412]                                                     1      (  0%)
[0xffffe413]                                                     1      (  0%)

$ qprof ./fread larger_data
qprof: /tmp/fread: 130 samples, 130 counts
main                                                             3      (  2%)
libc.so.6                                                        8      (  6%)
libc.so.6(_IO_fread)                                             46     ( 35%)
libc.so.6(_IO_sgetn)                                             8      (  6%)
libc.so.6(memcpy)                                                22     ( 17%)
[0xffffe410]                                                     43     ( 33%)

As you can see from the huge difference libapr may give you a large performance penalty. Do use libapr; it'll save you time. Do use a profiler as well; it'll save the users of your application time ;-)