Notes to self, 2012

2012-12-12 - canon / mf8350 / driver hell

Building Canon MF8350Cdn (and other) CUPS drivers for Linux Debian and/or Ubuntu on amd64 is still a pain in the behind.

Problems encountered during the installation, include:

  • Regular make installed stuff in different places but forgot many parts.
    Solution: use dpkg-buildpackage
  • The libtool copied wasn't able to build for shared libs.
    Solution: remove --enable-shared/--disable-shared command line options.
  • amd64 had to be added to the architecture targets.
  • A shell script had to lose a bashism.

The right steps

  • Download and extract:
    Linux_UFRII_PrinterDriver_V250_us_EN.tar.gz (download)
  • Change dir into:
    Linux_UFRII_PrinterDriver_V250_us_EN/Sources
  • Extract the two tar balls found there:
    cndrvcups-common-2.50-1.tar.gz
    cndrvcups-lb-2.50-1.tar.gz
  • Patch both with this patch:
    canon_mf8350-wjd.patch (view)
  • Go into both directories and run:
    dpkg-buildpackage
    (* See Update below!)
  • There should now be 4 deb packages in the Sources dir:
    cndrvcups-common_2.50-1_amd64.deb (download)
    cndrvcups-lipslx_2.50-1_amd64.deb (download)
    cndrvcups-ufr2-uk_2.50-1_amd64.deb (download)
    cndrvcups-ufr2-us_2.50-1_amd64.deb (download)
  • Install all, except one of the UK or US versions.
  • Manually run ldconfig

Finally, when installing the printer, you'll have to decide which driver you want. I chose the “Canon MF8300C Series UFRII LT ver.2.5 (color, 2-sided printing)” driver because it is the closest match to our “MF8350Cdn”.

This took way too much time — like some things do. Useful debug information was obtained using the LogLevel debug option in /etc/cups/cupsd.conf.

Observe that these packages require the ia32-libs (* See Update below!) to be installed.
The reason for that is that not everything in Sources is in source form :-(

Update 2013-03-31

Glen Whitney of the National Museum of Mathematics points out that he had to install the cndrvcups-common produced by [this] procedure before [he] could compile the cndrvcups-lb stuff.” Thanks for the feedback!

Update 2013-08-19

Installing all of the ia32-libs can be a bit much. In fact, you only need these: libc6:i386 libp11-kit0 libxml2:i386 zlib1g:i386

Update 2013-09-18

Is buftool.h missing? Look more closely at Update 2013-03-31.

2012-10-25 - easy / certificate generation / testing

If I'm going to be requesting SSL certificates more often, I'd better automate the process a bit.

The result:
easycert.sh (view)

Possible invocation styles:

$ easycert.sh -h
Usage: easycert.sh -c NL -l Groningen -o OSSO\ B.V. -e info@osso.nl osso.nl
Usage: easycert.sh osso.nl "/C=NL/L=Groningen/O=OSSO B.V./CN=osso.nl/"
Usage: easycert.sh -T www.osso.nl 443

Generating a key and certificate:

$ easycert.sh -o "My Company" mycompany.com
Subject: /C=NL/L=Groningen/O=My Company/CN=mycompany.com/emailAddress=info@osso.nl
Enter to proceed...
Generating RSA private key, 4096 bit long modulus
..................................................................................................................................................++
..................++
e is 65537 (0x10001)
mycompany_com-2012.csr

The same, but in a non-interactive fashion:

$ easycert.sh mycompany.com "/C=NL/L=Groningen/O=My Company/CN=mycompany.com/emailAddress=info@osso.nl" </dev/null 2>&0
mycompany_com-2012.csr
$ ls -l mycompany_com-2012.*
-rw-r--r-- 1 walter walter 1704 2012-10-25 12:14 mycompany_com-2012.csr
-rw-r--r-- 1 walter walter 3243 2012-10-25 12:14 mycompany_com-2012.key

Testing SSL configuration on the server:

$ easycert.sh mail.osso.nl 993 -T
The list below should be logically ordered,
and end with a self-signed root certificate:

Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.osso.nl
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---

Now all it needs is a CGI front-end so the customers can generate their own CSRs while leaving the keys privately on the server. No wait.. I added that already. Just run the script as a cgi script and you'll get the files in /tmp.

Update 2013-05-28

Sending the last self-signed certificate is generally only unnecessary overhead. In the example above, the third and final certificate in the chain may be skipped.

And see the SSL Labs SSL analyzer for a helpful report about the state of your HTTPS server.

Common Apache2 configs you may need to add:

# Fight BEAST attack
SSLHonorCipherOrder On
SSLCipherSuite ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH

Common NginX config:

ssl_protocols        SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers          ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!DH:!EDH;
ssl_prefer_server_ciphers on;

2012-08-17 - setuid / seteuid / uid / euid

So, what is the difference between uid and euid and the setuid and seteuid calls?

Hao Chen, David Wagner and Drew Dean wrote an excellent paper called Setuid Demystified. It explains all the ins and outs. To answer the question, we need only parts of the article.

Let the quoting begin.

Each process has three user IDs: the real user ID (real uid, or ruid), the effective user ID (effective uid, or euid), and the saved user ID (saved uid, or suid). The real uid identifies the owner of the process [i.e. the user launching the application], the effective uid is used in most access control decisions, and the saved uid stores a previous user ID so that it can be restored later.

[... A setuid binary starts with the file-owner in the euid and suid. See the examples below. ...]

Similarly, a process has three group IDs: the real group ID, the effective group ID, and the saved group ID. In most cases, the properties of the group IDs parallel the properties of their user ID counterparts.

[... And, linux has two more for filesystem access: fsuid/fsgid. If not set explicitly, they follow the euid/egid. ...]

As System V and BSD influenced each other, both systems implemented setuid, seteuid, and setreuid, although with different semantics. None of these system calls, however, allowed the direct manipulation of the saved uid (although it could be modified indirectly through setuid and setreuid). Therefore, some modern Unix systems introduced a new call, setresuid, to allow the modification of each of the three user IDs directly.

[...] setresuid has the clearest semantics among the four uid-setting system calls. The permission check for setresuid() is intuitive and common to all OSs: for the setresuid() system call to be allowed, either the euid of the process must be root, or each of the three parameters must be equal to one of the three user IDs of the process.

Requirements:
  (euid == 0)
    ||
  (newruid in (ruid, euid, suid) &&
   neweuid in (ruid, euid, suid) &&
   newsuid in (ruid, euid, suid))

[...] seteuid has also a clear semantics. It sets the effective uid while leaving the real uid and saved uid unchanged.

Requirements on Linux/Solaris:
  (euid == 0)
  ||
  (neweuid in (ruid, euid, suid))
Requirements on FreeBSD:
  (euid == 0)
  ||
  (neweuid in (ruid, suid))

[...] Although setuid is the only uid-setting system call standardized in POSIX 1003.1-1988, it is also the most confusing one.

Requirements on Linux/Solaris:
  (euid == 0)
  ||
  (newuid in (ruid, suid))
Requirements on FreeBSD:
  (euid == 0)
  ||
  (newuid in (ruid, euid, suid))

[...] Second, the action of setuid differs not only among different operating systems but also between privileged and unprivileged processes.

Behaviour on Linux/Solaris:
  (euid == 0) ==>
    (ruid:=newuid, euid:=newuid, suid:=newuid)
  (anything else) ==>
    (euid:=newuid)
Behaviour on FreeBSD:
  (ruid:=newuid, euid:=newuid, suid:=newuid)

Clear? Almost. seteuid seems to make sense, but the setuid behaviour seems rather awkward.

Maybe some examples on Linux help.

#define _GNU_SOURCE
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#define call(fun) errno = 0; fun; perror(#fun)
int main(int argc, char **argv) {
  uid_t ruid = -1, euid = -1, suid = -1;
  getresuid(&ruid, &euid, &suid);
  printf("> ruid=%d, euid=%d, suid=%d\n", ruid, euid, suid);
  call(setuid(1000));
  getresuid(&ruid, &euid, &suid);
  printf("> ruid=%d, euid=%d, suid=%d\n", ruid, euid, suid);
  return 0;
}
$ gcc example.c ; sudo chown root ./a.out ; sudo chmod 4755 ./a.out
$ ls -ln a.out 
-rwsr-xr-x 1 0 1000 8716 2012-08-17 17:01 a.out
$ ./a.out 
> ruid=1000, euid=0, suid=0
setuid(1000): Success
> ruid=1000, euid=1000, suid=1000

You were effectively root and you justed dropped those permissions. You should've used seteuid if you wanted them back.

$ sudo ./a.out
> ruid=0, euid=0, suid=0
setuid(1000): Success
> ruid=1000, euid=1000, suid=1000

Same as before.

Lets throw a third user in the mix.

$ sudo chown nobody ./a.out ; sudo chmod 4755 ./a.out 
$ ls -ln a.out 
-rwsr-xr-x 1 65534 1000 8716 2012-08-17 17:33 a.out
$ ./a.out 
> ruid=1000, euid=65534, suid=65534
setuid(1000): Success
> ruid=1000, euid=1000, suid=65534

We're really just doing seteuid here.

Lastly, the case where the euid isn't 0 and newuid isn't in real, effective or saved:

$ sudo ./a.out 
> ruid=0, euid=65534, suid=65534
setuid(1000): Operation not permitted
> ruid=0, euid=65534, suid=65534

Yes, that didn't satisfy the requirements as mentioned above. But we can seteuid(0) and then seteuid(1000) to get euid to 1000.

$ sudo ./a.out2
> ruid=0, euid=65534, suid=65534
seteuid(0): Success
seteuid(1000): Success
> ruid=0, euid=1000, suid=65534

Hope this helps ;-)

If you're interested in the source of the awkward behaviour, read the entire article and/or one of their sources which goes into the cause much deeper: Chris Torek's “setuid mess”

That should also explain why there are three uids and not just two. After all, two should be enough (effective and another one to switch back to), along with a pointer to the effective one: the early solution to temporarily drop powers was setreuid, a method by which two uids could be swapped. However, this swapping was cumbersome and was finally (skipping a few steps) replaced by setresuid.

2012-07-16 - postfix / submission / smtpd_client_restrictions / sleep

After tweaking my postfix configuration, I apparently broke submission on port 587. Every time I connected, I immediately got:

554 5.7.1 <my.host.name[1.2.3.4]>: Client host rejected: Access denied

That's strange. Postfix is supposed to reject unauthenticated clients only in master.cf:

submission inet  n       -       n       -       -       smtpd 
  -o smtpd_tls_security_level=encrypt 
  -o smtpd_tls_auth_only=yes 
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

But if it rejects me at connect time, I don't have a chance to identify myself.

The cause of the problem turned out to be this: smtpd_delay_reject = no

To combat spam, I use the sleep parameter. Many bots give up within a short amount of time after connecting — or they write commands without waiting for feedback (pipelining) — so waiting a bit helps a lot.

smtpd_client_restrictions
...
       sleep seconds
              Pause for the specified number of seconds and proceed with the next restriction in the list, if any. This may stop zombie mail when used as:
              /etc/postfix/main.cf:
                  smtpd_client_restrictions =
                      sleep 1, reject_unauth_pipelining
                  smtpd_delay_reject = no

(Some people will call this waiting bad. However, no MTA has trouble with a little wait, and this is FAR better than graylisting which ensures that the recipient has to wait entire hours, instead of just seconds, for their mail.)

The smtpd_delay_reject = no causes the sleep to actually get executed at connect time. If we skip this, we're already half-way through the mail-sending before any sleep occurs.

For submission/587, I used the default authenticated clients only config:

smtpd_sasl_auth_enable (default: no)
...
       To reject all SMTP connections from unauthenticated clients, specify "smtpd_delay_reject = yes" (which is the default) and use:

           smtpd_client_restrictions = permit_sasl_authenticated, reject

You'll notice the conflicting options.

The fix: replace smtpd_client_restrictions with smtpd_recipient_restrictions. Now the same is authenticated check is performed, but first after we've had a chance to identify ourselves.

2012-07-09 - new ipython / old django

IPython after version 0.10 is not friends with older Django (e.g. 1.1.x) versions anymore.

    shell = IPython.Shell.IPShell(argv=[])
AttributeError: 'module' object has no attribute 'Shell'

This is fixed in newer Django's, but this isn't backported.

Here, a patch. (Nothing more than a diff between the old and the new Django version.)

--- django/core/management/commands/shell.py  2012-03-28 16:10:28.000000000 +0200
+++ django/core/management/commands/shell.py  2012-01-24 10:27:50.405338739 +0100
@@ -8,9 +8,38 @@
             help='Tells Django to use plain Python, not IPython.'),
     )
     help = "Runs a Python interactive interpreter. Tries to use IPython, if it's available."
-
+    shells = ['ipython', 'bpython']
     requires_model_validation = False
 
+    def ipython(self):
+        try:
+            from IPython.frontend.terminal.embed import TerminalInteractiveShell
+            shell = TerminalInteractiveShell()
+            shell.mainloop()
+        except ImportError:
+            # IPython < 0.11
+            # Explicitly pass an empty list as arguments, because otherwise
+            # IPython would use sys.argv from this script.
+            try:
+                from IPython.Shell import IPShell
+                shell = IPShell(argv=[])
+                shell.mainloop()
+            except ImportError:
+                # IPython not found at all, raise ImportError
+                raise
+
+    def bpython(self):
+        import bpython
+        bpython.embed()
+
+    def run_shell(self):
+        for shell in self.shells:
+            try:
+                return getattr(self, shell)()
+            except ImportError:
+                pass
+        raise ImportError
+
     def handle_noargs(self, **options):
         # XXX: (Temporary) workaround for ticket #1796: force early loading of all
         # models from installed apps.
@@ -23,11 +52,7 @@
             if use_plain:
                 # Don't bother loading IPython, because the user wants plain Python.
                 raise ImportError
-            import IPython
-            # Explicitly pass an empty list as arguments, because otherwise IPython
-            # would use sys.argv from this script.
-            shell = IPython.Shell.IPShell(argv=[])
-            shell.mainloop()
+            self.run_shell()
         except ImportError:
             import code
             # Set up a dictionary to serve as the environment for the shell, so

And another patch to get the classic mode to work here too. (The TerminalInteractiveShell() would require a config=load_default_config() from IPython.frontend.terminal.ipapp.)

--- django/core/management/commands/shell.py  2012-07-09 10:30:12.423405004 +0200
+++ django/core/management/commands/shell.py  2012-07-09 10:45:23.391960205 +0200
@@ -13,9 +13,8 @@
 
     def ipython(self):
         try:
-            from IPython.frontend.terminal.embed import TerminalInteractiveShell
-            shell = TerminalInteractiveShell()
-            shell.mainloop()
+            from IPython import embed
+            embed()
         except ImportError:
             # IPython < 0.11
             # Explicitly pass an empty list as arguments, because otherwise

2012-06-21 - serialize json date / microsoft extension

Bertrand Le Roy describes how Microsoft added a Date object extension to JSON in a compatible fashion to implement serialization and serialization of timezone agnostic datetimes.

Our current approach is using a small loophole in the JSON specs. In a JSON string literal, you may (or may not) escape some characters. Among those characters, weirdly enough, there is the slash character ('/'). This is weird because there actually is no reason that I can think of why you'd want to do that. We've used it to our benefit to disambiguate a string from a date literal.

The new format is "\/Date(1198908717056)\/" where the number is again the number of milliseconds since January 1st 1970 UTC.

I actually kind of like the hack. So, here is an implementation for Python: jsonext.py (view)

Before:

>>> import datetime, json
>>> encoded = json.dumps(datetime.date.today())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
...
  File "/usr/lib/python2.6/json/encoder.py", line 344, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.date(2012, 6, 21) is not JSON serializable

After:

>>> import datetime, json, jsonext
>>> encoded = json.dumps(datetime.date.today(), cls=jsonext.ExtendedJSONEncoder)
>>> decoded = json.loads(encoded, cls=jsonext.ExtendedJSONDecoder)
>>> print repr(decoded)
datetime.datetime(2012, 6, 21, 0, 0)

Update 2012-07-09

The v1 version of jsonext did not work with the newer json libs for python 2.7. I've updated it to v2 to support both libs. Unfortunately the new style is a giant hack. Apologies in advance.

2012-06-14 - gigaset n300a / respect srv

Does the Siemens Gigaset N300A handle SRV records?

Yes it does.. but..

Let's look at a bit of Gigaset DNS traffic:

09:24:26.059850 IP gigaset.local.32978 > nameserver.local.53: 50512+ NAPTR? gigaset.voip.example.com. (28)
09:24:26.061552 IP nameserver.local.53 > gigaset.local.32978: 50512 0/1/0 (87)
09:24:26.063894 IP gigaset.local.32978 > nameserver.local.53: 25738+ SRV? _sip._udp.gigaset.voip.example.com. (38)
09:24:26.064445 IP nameserver.local.53 > gigaset.local.32978: 25738 2/3/2 SRV proxy1.voip.example.com.:5060 10 0, SRV proxy2.voip.example.com.:5060 20 0 (231)
09:24:26.066939 IP gigaset.local.32978 > nameserver.local.53: 22676+ SRV? _sip._tcp.gigaset.voip.example.com. (38)
09:24:26.067290 IP nameserver.local.53 > gigaset.local.32978: 22676 2/3/2 SRV proxy2.voip.example.com.:5060 40 0, SRV proxy1.voip.example.com.:5060 30 0 (231)
09:24:26.070633 IP gigaset.local.32978 > nameserver.local.53: 11475+ A? gigaset.voip.example.com. (28)
09:24:26.070934 IP nameserver.local.53 > gigaset.local.32978: 11475 2/3/0 A 1.2.3.5, A 1.2.3.4 (133)

It's doing an NAPTR query. Ok. I don't have those.
Next, it's looking up the _sip._udp and _sip._tcp SRV records. Good.
Next, it's looking up the A record. Why?

09:24:26.076373 IP gigaset.local.5060 > proxy2.voip.example.com.6060: SIP, length: 495
09:24:26.077905 IP proxy2.voip.example.com.6060 > gigaset.local.5060: SIP, length: 497
09:24:26.099485 IP gigaset.local.5060 > proxy2.voip.example.com.6060: SIP, length: 685
09:24:26.102665 IP proxy2.voip.example.com.6060 > gigaset.local.5060: SIP, length: 473

What? Did it does connect to proxy2? Yes.. the first one it got from the A record. That's wrong. Why did it look up the SRV records if it wasn't going to use them?

It gets worse. There's an option to have it send NAT keepalives (4 NUL bytes), which works fine. But the keepalive host gets out of sync with the registration server; causing NAT keepalives to be completely useless!

It sends keepalives to proxy1 while registering to proxy2. Now inbound connections from the proxy server (proxy2) where we register will be rejected by the NAT-router. The keepalives have been punching the wrong hole the whole time.

This is a known problem, which is the reason why we're using SRV records here in the first place. The record priority ensures that all traffic is sent to proxy1 unless it is down, in which case everything goes to proxy2.

However, it turns out that you can convince the Gigaset to use the SRV records. And it's a simple as clearing out the ports. The two port input fields shown here should be blank:

Gigaset N300A SIP settings

Now things work like expected. (At least with firmware 42.051 (420510000000 / V42.00).)

Suggestion to Siemens: put a notice on there that you shan't fill in the port if you want things to work properly.

Update 2013-03-21

RFC 3263 (Locating SIP Servers) 4.2 writes (in June 2002):

If the TARGET was not a numeric IP address, but a port is present in
the URI, the client performs an A or AAAA record lookup of the domain
name.

That might explain why they did what they did. But it still isn't intuitive. And it's still wrong that they query both.

2012-06-13 - thunderbird / mailing list / reply

How do you reply to a mailing list post when you do not have the mail in your INBOX? With Thunderbird it is easy enough, as long as you know how.

How it works

Mail threads are matched by comparing the In-Reply-To header with the Message-ID.

Here's an example from an Asterisk project reviewboard mailing:

Date: Fri, 08 Jun 2012 08:08:42 -0000
Message-ID: <20120608080842.8103.32910@hotblack.digium.com>
In-Reply-To: <20120607143847.27705.11556@hotblack.digium.com>
References: <20120607143847.27705.11556@hotblack.digium.com>
Subject: Re: [asterisk-dev] [Code Review] Fix issue of unrecognized inbound
 ACK when Asterisk responds to an INVITE with a 481

Every e-mail message has a globally unique message identifier. You can tell that a mail is a reply to another mail, by comparing the mail's In-Reply-To to the Message-ID of the original.

In the example above, the original message has:
Message-ID: <20120607143847.27705.11556@hotblack.digium.com>

On to the problem at hand: where do I fill out that In-Reply-To in my Thunderbird e-mail client?

The solution: the command line to the rescue!

How you do it

If you wanted to reply to the message above, you'd take the message id — most mailing list web interfaces do show the message id — from that mail, and call thunderbird like this. (Make sure you replace the values where appropriate.)

$ thunderbird 'mailto:asterisk-dev@lists.example.com?In-Reply-To=<20120608080842.8103.32910@hotblack.digium.com>&Cc=yourself@example.com&Subject=Re:%20bla%20bla'

Thunderbird will pop up a Compose window with the Subject and Cc pre-set and, most importantly, it will set the In-Reply-To and References headers for you.

2012-06-06 - python / base85 / ascii85

So python's base64 does not have a b85decode function? Adobe uses it the ASCII-85 encoding in PDF and PostScript files.

Here is a quick and dirty one hacked together. See the wikipedia article for the ASCII-85 (base85) specs.

Prologue; we only need sys to print a warning.

# vim: set ts=8 sw=4 sts=4 et ai:
# Example base85 decoder, Walter Doekes 2012
import sys

Split the data up into 5-character chunks; 5 characters encode 4 octets.

def _chunk(data):
    # TODO: remove white space
    idx = 0
    while len(data) > idx:
        if data[idx] in 'yz':
            next = data[idx]
            idx += 1
            if next == 'z':
                yield '!!!!!', 0 # 4xNUL
            else:
                assert False # FIXME: 4xSPACE
        else:
            next = data[idx:idx+5]
            idx += len(next)
            padding = 5 - len(next)
            if padding:
                next = next.ljust(5, 'u')
            if next >= 's8W-!':
                print >> sys.stderr, 'Garbage in input: %s' % (next,)
                return
            yield next, padding

Translate the 5 characters into a 32-bit number and then to the octets.

def _decode(chunk, padding):
    assert len(chunk) == 5
    num = (((((ord(chunk[0]) - 33)
         * 85 + ord(chunk[1]) - 33)
         * 85 + ord(chunk[2]) - 33)
         * 85 + ord(chunk[3]) - 33)
         * 85 + ord(chunk[4]) - 33)
    return ('%c%c%c%c' % (
        chr(num >> 24),
        chr((num >> 16) & 0xff),
        chr((num >> 8) & 0xff),
        chr(num & 0xff)
    ))[0:4-padding]

Putting it all together.

b85decode = (lambda x: ''.join(_decode(*i) for i in _chunk(x)))

Optional; some code to test the thing.

input = (r'9jqo^BlbD-BleB1DJ+*+F(f,q/0JhKF<GL>Cj@.4Gp$d7F!,L7@<6@)/0JD'
         r'EF<G%<+EV:2F!,O<DJ+*.@<*K0@<6L(Df-\0Ec5e;DffZ(EZee.Bl.9pF"A'
         r'GXBPCsi+DGm>@3BB/F*&OCAfu2/AKYi(DIb:@FD,*)+C]U=@3BN#EcYf8AT'
         r'D3s@q?d$AftVqCh[NqF<G:8+EV:.+Cf>-FD5W8ARlolDIal(DId<j@<?3r@'
         r":F%a+D58'ATD4$Bl@l3De:,-DJs`8ARoFb/0JMK@qB4^F!,R<AKZ&-DfTqB"
         r"G%G>uD.RTpAKYo'+CT/5+Cei#DII?(E,9)oF*2M7/c")

expected = ('Man is distinguished, not only by his reason, but by '
            'this singular passion from other animals, which is a '
            'lust of the mind, that by a perseverance of delight in '
            'the continued and indefatigable generation of knowledge, '
            'exceeds the short vehemence of any carnal pleasure.')

def format72(text):
    '''Not needed at all, but python generators are so cool, I cannot
    resist.'''
    gen = iter(text.split())
    word = gen.next()
    while True:
        yield word
        linelen = len(word)
        while True:
            word = gen.next()
            if linelen + len(word) > 71:
                yield '\n'
                break
            yield ' '
            yield word
            linelen += len(word) + 1
            
decoded = b85decode(input)
print ''.join(format72(decoded))
assert decoded == expected

The output should look like this:

Man is distinguished, not only by his reason, but by this singular
passion from other animals, which is a lust of the mind, that by a
perseverance of delight in the continued and indefatigable generation of
knowledge, exceeds the short vehemence of any carnal pleasure.

2012-06-04 - libreoffice / spreadsheet l10n / date format

For LibreOffice's oocalc on latest Ubuntu (libreoffice-base 1:3.5.2-2ubuntu1 to take the locale settings into account for the date types, the LC_CTYPE needs to be set.

$ LC_CTYPE=en_US.UTF-8 oocalc

This causes a date input of 31-01-2012 to not get parsed as a date.

$ LC_CTYPE=nl_NL.UTF-8 oocalc

This causes the same input of 31-01-2012 to get properly understood as the DD-MM-YYYY format.

That does not make sense. LC_CTYPE should be used for character classification, collation and case conversion. LC_TIME contains the relevant information in d_fmt (%d-%m-%y). Why isn't that used? Or — equally disturbing — how is it getting the right information from the wrong collation files?

And apparently I'm not the only one having libreoffice locale issues.

2012-05-23 - appartement / groningen / starterswoning

TE KOOP! Leuk appartement voor starters of studerende kinderen. Op een groene locatie in de West-Indische buurt in Groningen, ligt een licht en goed onderhouden appartement (65 m²) op de eerste woonlaag. De woning heeft een ruim balkon en het recht van gebruik op een tuin op het westen.

woonkamer en deel eetkamer deel van tuin en balkon

Voor uitgebreide informatie, bekijk de (mobile-friendly) website: starterswoning in groen deel van Groningen

2012-05-14 - ubuntu / sip video / softphone

So, I wanted to test video support with Asterisk. That was easier said than done, because the SIP softphones that ship with Ubuntu don't all do what they promise.

This was done on a setup that works for numerous hardphones and PBXs out there. Looking through the registration list at any given time reveals at least 40+ different user agents and a large multiple of that if you take the different versions into account.

Drawbacks of Ekiga (3.2.6 and 3.3.2)

  • User interface offers no advanced options.
  • Refuses to run on a different local port than 5060 (or at least it used to).
  • It crashes.
  • Hangs during registration for no apparent reason. (Doesn't do registration at all, but does attempt to SUBSCRIBE to local users on the remote proxy a lot in the mean time.)
  • Sends link-local IPv6 addresses in the Contact header, which my provider uses as only contact, resulting in breakage.
  • Re-registers immediately with an Expires: 0 (unregister) and claims that it is registered.
  • Doesn't use the outboundproxy for the registration attempts.

Conclusion: completely unusable for a setup that works with every hardphone out there.

Drawbacks of Telepathy/Empathy (2.30.3 and 3.4.1)

  • It crashes, but less often than Ekiga.
  • The user interface is unintuitive: accounts list available through empathy-accounts binary only; details-of-call pane is truncated so only the first 6 letters of relevant info is visible; setting up a call not possible through the notification area.
  • Call setup failed when the default NAT settings were enabled. Things started working when we disabled all NAT features in the client and relied on the NAT-magic of the remote OpenSIPs and Asterisk.
  • Video support fails when one end does not have a video source.
  • For no apparent reason, it started doing auto-pickup (with video!). That could create embarrassing situations ;-)

Conclusion: less broken than Ekiga, but still unusable.

Drawbacks of Linphone (3.2.1 and 3.3.2)

  • It did manage to calculate the wrong MD5 response for an authentication request once. But since it did that when I was messing about with the hostname in /etc/hosts, I will forgive it.
  • You cannot disable sending video if you have a working video source without disabling the reception of video as well.
  • It works! Even when one of the clients doesn't have a video input source. Oh wait.. this isn't a drawback.
  • It has a sensible (old school) user interface with settings where you expect them! Another one for the plus side.

Conclusion: use Linphone! Thank you Simon Morlat. (Now, if you could just replace the 't' in “standart” [sic] in the About box ;-) )

By the way, the video support in Asterisk did indeed work fine.

2012-05-04 - django / mark_safe / translatables

Look at this snippet of Django code in models.py, and in particular the help_text bit:

from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils.safestring import mark_safe

class MyModel(models.Model):
    my_field = models.CharField(max_length=123,
                                help_text=mark_safe(_('Some <b>help</b> text.')))

For those unfamiliar with Django. A quick run-down:

  • The definition of MyModel creates a mapping between the MyModel class and a underlying app_mymodel table in a database.
  • That table will consist of two columns: id, an automatic integer as primary key (created by default), and my_field, a varchar/text field of at most 123 characters.
  • With minimal effort a HTML form can be generated from this. That form will show my_field as a text input box and near it the text we defined in help_text.
  • The _()-function has the gettext functions run over it so the text can be served in different languages with minimal effort.
  • The mark_safe()-function tells the template rendererer that this output is already safe to use in HTML. Without it, the user would see: Some &lt;b&gt;help&lt;/b&gt; text.

Unfortunately this doesn't do what you would expect.

Let's examine why.

There is a reason why we use the ugettext_lazy wrapper in models.py. This code is executed once at startup / first run, and the language that was selected at that time would be substituted if we used the non-lazy ugettext. The lazy variant makes sure the substitution takes place at the last possible time.

mark_safe forces the translation to happen immediately.

In the best case that means someone else can get the help text served in the wrong language. In the worst case, you get a recursive import when the translation routines attempt to import all INSTALLED_APPS while looking for locale files. Your MyModel might be referenced from one of those apps. The result: recursion and a resulting ImportError.

...
File "someapp/models.py", line 5, in <module>
    class MyModel(models.Model):
File "someapp/models.py", line 6, in <module>
    my_field = models.CharField(max_length=123,
File "django/utils/safestring.py", line 101, in mark_safe
    return SafeUnicode(s)
...
File "django/utils/translation/trans_real.py", line 180, in _fetch
    app = import_module(appname)
File "django/utils/importlib.py", line 35, in import_module
    __import__(name)
...
ImportError: cannot import name MyModel

Lessons learnt: if you're using translations then don't call mark_safe on anything until it's view time.

In this case, we would fix it by adding the mark_safe call to the Form constructor. We know that that is run for every form instantiation, so that's late enough.

class MyModelForm(forms.ModelForm):
    class Meta:
        model = MyModel

    def __init__(self, *args, **kwargs):
        super(MyModelForm, self).__init__(*args, **kwargs)
        self.fields['my_field'].help_text = mark_safe(self.fields['my_field'].help_text)

But suggestions for prettier solutions are welcome.

Update 2013-03-21

The Django 1.4 docs provide the better solution:

from django.utils import six  # Python 3 compatibility
from django.utils.functional import lazy
from django.utils.safestring import mark_safe
from django.utils.translation import ugettext_lazy as _

mark_safe_lazy = lazy(mark_safe, six.text_type)

2012-04-30 - ipython classic mode / precise pangolin

The Ubuntu do-release-upgrade broke my ipython classic mode. The ipython package was upgraded, and apparently the configuration parser was changed.

In bash, I want colors to help me find the beginning and end of output — see this bug report for others agreeing with me that the derogatory comment about "focus should be on the output, not on the prompt" in the skeleton .bashrc is is retared, but I diverge — in ipython, I just want to see the nice >>> blocks that I'm used to and no extra spaces. I.e.: classic mode.

Previously, one could set classic 1 in ~/.ipythonrc. Since the upgrade, one should be able to fix it by editing .config/ipython/profile_default/ipython_config.py.

However, the fixes that the pages on google propose, do not work:

c.TerminalIPythonApp.classic = True

The following does work:

# Force classic mode!
c = get_config()
c.InteractiveShell.cache_size = 0
c.PlainTextFormatter.pprint = False
c.PromptManager.in_template = '>>> '
c.PromptManager.in2_template = '... '
c.PromptManager.out_template = ''
c.InteractiveShell.separate_in = ''
c.InteractiveShell.separate_out = ''
c.InteractiveShell.separate_out2 = ''
c.InteractiveShell.colors = 'NoColor'
c.InteractiveShell.xmode = 'Plain'
#
c.InteractiveShell.confirm_exit = False
#c.IPythonTerminalApp.display_banner = False

These settings are taken from the source where the --classic option is defined, so they should be ok.

Update 2012-06-07

If your django shell does not show the right classic settings, you're probably running an old version.

Here, in the 1.3 branch and higher, the code looks like this:

    def ipython(self):
        try:
            from IPython import embed
            embed()

Previously that was this, and that doesn't work, unless you pass default config to TerminalInteractiveShell from IPython.frontend.terminal.ipapp.load_default_config:

    def ipython(self):
        try:
            from IPython.frontend.terminal.embed import TerminalInteractiveShell
            shell = TerminalInteractiveShell()
            shell.mainloop()

Just upgrade your django and be done.

2012-04-12 - safe_asterisk / init.d

An init.d script to stop and start safe_asterisk started asterisk. If asterisk is not stopped in 5 seconds, it is forcibly killed.

safe_asterisk-init.d (view)

# wget http://wjd.nu/files/2012/04/safe_asterisk-init.d -O/etc/init.d/asterisk ; chmod 755 /etc/init.d/asterisk

Also possibly useful, the changes I made to safe_asterisk on a machine where:

  1. there wasn't a tty left to spam output on,
  2. root is configured in /etc/aliases to a sane destination,
  3. /var/spool/asterisk is the asterisk user homedir anyway, and,
  4. attempting to set maxfiles to the highest value possible, wasn't allowed.
#TTY=9
CONSOLE=no
NOTIFY=root
DUMPDROP=/var/spool/asterisk
MAXFILES=12288
...
run_asterisk >/dev/null 2>&1 &

2012-03-29 - sip / digest calculation

Every one in a while, I see an unexpected 403 response to a SIP client's REGISTER request. Thusfar the digest response calculation has never been wrong, but it feels good to get that check out of the way and move on to other possible causes.

For your enjoyment and mine, a Bourne-shell compatible shell script that calculates (qop-less) Digest authentication responses.

Download: hahacalc.sh (view)

$ hahacalc
Usage:   hahacalc.sh USERNAME REALM METHOD DIGESTURI NONCE [PASSWORD] [COMPARE]
Or:      hahacalc.sh METHOD < autorization_headers
Example: hahacalc.sh 123456789 itsp.com REGISTER sip:sip.itsp.com:6060 \
           4f7406a80000c2f214774a48d11cc5b 9e533caff7a05904c MYPASSWORD \
           f5787ba2624dfdbcf874f46425d65b53
Or:      echo 'Authorization:...' | hahacalc.sh REGISTER
$ hahacalc 123456789 itsp.com REGISTER sip:sip.itsp.com:6060 4f7406a80000c2f214774a48d11cc5b9e533caff7a05904c MYPASSWORD f5787ba2624dfdbcf874f46425d65b53
A1   = 123456789:itsp.com:MYPASSWORD
HA1  = 7f3c958e155df4070204c2aa5293437f
A2   = REGISTER:sip:sip.itsp.com:6060
HA2  = 2d13f8e4f7343dbd44be6ddabf8c2c1d
RESP = f5787ba2624dfdbcf874f46425d65b53

Update 2013-06-19

Updated hahacalc to be able to parse Authorization: headers too. In that mode of operation, you pass only METHOD as argument, you enter the password on the console, and you input the SIP body through stdin.

2012-03-13 - python virtualenv / global site-packages

If you're switching from Ubuntu Oneiric to Ubuntu Precise and you're using python-virtualenv, you might be in for a surprise:

The default access to the global site-packages modules is reversed between virtualenv 1.6.x and 1.7.

When you were used to finding your apt-get installed python modules like python-mysqldb and python-psycopg2 in your new virtualenv environment, now they're suddenly unavailable.

The culprit:

       --no-site-packages
           Ignored (the default).  Don´t give access to the global
           site-packages modules to the virtual environment.

That was not the default before.

The new option to get the old behaviour:

       --system-site-packages
           Give access to the global site-packages modules to the
           virtual environment.

By the way, if you already populated your new virtualenv directory with the 1.7 version, you don't need to recreate it. Removing the no-global-site-packages.txt is enough:

# find /srv/django-env -name no-global-site-packages.txt
/srv/django-env/mysite/lib/python2.7/no-global-site-packages.txt

2012-02-27 - mysql / replicating repair table

From the MySQL 5.1 manual:

15.4.1.16. Replication and REPAIR TABLE
When used on a corrupted or otherwise damaged table, it is possible for the REPAIR TABLE statement to delete rows that cannot be recovered. However, any such modifications of table data performed by this statement are not replicated, which can cause master and slave to lose synchronization. For this reason, in the event that a table on the master becomes damaged and you use REPAIR TABLE to repair it, you should first stop replication (if it is still running) before using REPAIR TABLE, then afterward compare the master's and slave's copies of the table and be prepared to correct any discrepancies manually, before restarting replication.

Sounds like a pain the behind to have to do manually, especially if you have data updates in one of the two Master-Master replicated machines.

But that is not the point with my quote. The point is:
If REPAIR TABLE does nothing to ensure that two copies are identical, then why on earth does it replicate the REPAIR TABLE statement at all?

Because that was all I wanted to know: does REPAIR TABLE get replicated?

Logical answer: no

Actual answer: yes

Beware.. and be prepared to skip a couple of statements.

2012-02-17 - indirect scp / bypass remote firewall rules

Suppose I'm on machine DESKTOP and I want to copy files from server APPLE to server BANANA. DESKTOP has access to both, but firewalls and/or missing ssh keys prevent direct access between APPLE and BANANA.

Regular scp(1) will now fail. It will attempt to do a direct copy and then give up. This is where this indirect scp wrapper (view) comes in:

  • First, it tries to do the direct copy.
  • If that fails, it uses the local machine as an intermediary.

In this example you'll see it fail twice for the two source files and then fall back to using the local machine.

$ scp -r APPLE:example/file1 APPLE:example/somedir BANANA:some_existing_path/
Host key verification failed.
lost connection
Host key verification failed.
lost connection
(falling back to indirect copy...)
file1                                     100%    6     0.0KB/s   00:00
here                                      100%    5     0.0KB/s   00:00
two_files                                 100%   10     0.0KB/s   00:00
(copy from here to destination...)
file1                                     100%    6     0.0KB/s   00:00
here                                      100%    5     0.0KB/s   00:00
two_files                                 100%   10     0.0KB/s   00:00
(cleaning up temporary files...)

For a bit of added security, it uses shred(1) to clean up the local files, if available.

Installation:

# cd /usr/local/bin
# wget http://wjd.nu/files/2012/02/indirect-scp.sh -O indirect-scp
# chmod 755 indirect-scp
# ln -s indirect-scp scp

If you know the direct copy will fail, you can call indirect-scp directly.

2012-01-25 - mysql replication / relay log pos

So, hardware trouble caused a VPS to go down. This VPS was running a MySQL server in a slave setup. Not surprisingly, the unclean shutdown broke succesful slaving.

There are several possibly causes for slave setup breakage. This time it was the local relay log file (mysqld-relay-bin.xxxx) that was out of sync.

SHOW SLAVE STATUS\G looked like this:

...
       Master_Log_File: mysql-bin.001814  <-- remote/master file (IO thread)
   Read_Master_Log_Pos: 33453535          <-- remote/master pos  (IO thread)
        Relay_Log_File: mysqld-relay-bin.001383  <-- local/slave file (SQL thread)
         Relay_Log_Pos: 34918332                 <-- local/slave pos  (SQL thread)
 Relay_Master_Log_File: mysql-bin.001812  <-- remote/master file (SQL thread)
...
            Last_Errno: 1594
            Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
...
   Exec_Master_Log_Pos: 34918187          <-- remote/master pos  (SQL thread)
...

Step one was to find out where we were in the local and on the remote end. Luckily, most queries ran during the failure period were UPDATEs on the same table.

  • Remote position was ok. On the master, /var/log/mysql/mysql-bin.001812 contained these lines:
    # at 34918187
    #120125  1:16:05 server id 1  end_log_pos 34918531 ...
    SET TIMESTAMP=1327450565/*!*/;
    UPDATE mytable ....
    
  • The statements before that had been ran on the slave and this statement hadn't.
  • On the slave, /var/lib/mysql/mysqld-relay-bin.001383 did contain the previous line, but did not contain position 34918332.
  • Looking further, I could see that mysqld-relay-bin.001384 was practically empty, but mysqld-relay-bin.001385 contained already executed statements, and after a bit of browsing there it was too:
    # at 21491
    #120125  1:16:05 server id 1  end_log_pos 34918531 ...
    SET TIMESTAMP=1327450565/*!*/;
    UPDATE mytable ...
    

Good. So we need only move the relay log file pointer a bit to the front.

mysql> CHANGE MASTER TO RELAY_LOG_FILE='mysqld-relay-bin.001385', RELAY_LOG_POS=21491;
ERROR 1380 (HY000): Failed initializing relay log position: Could not find target log during relay log initialization

What? Searching for that error pointed to a document about copying slave data to another slave and about modifying files. Hmm.. modifying files. I can do that too...

# cat /var/lib/mysql/relay-log.info
./mysqld-relay-bin.001383
34918332
mysql-bin.001812
34918187
0

With a little speed — /etc/init.d/mysql stop ; vim /var/lib/mysql/relay-log.info ; /etc/init.d/mysql start — I edited the relay log file and relay log position in relay-log.info by hand.

Voilà! It worked. Slave replication was running again like it should.

2012-01-08 - mencoder / canon / mjpeg

I tend to make few movies with my digital photo camera because they take up so much space. That's a shame, because having a bit of moving image is fun to look at when the kids have grown up.

The reason they take up so much space is simple. The camera is not equipped with fancy encoding algorithms: the video is stored as MJPEG, basically a series of JPEG images joined together. Somehow I should've known this ;-)

==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG)
==========================================================================
Opening audio decoder: [pcm] Uncompressed PCM audio decoder
AUDIO: 11024 Hz, 1 ch, u8, 88.2 kbit/100.00% (ratio: 11024->11024)
Selected audio codec: [pcm] afm: pcm (Uncompressed PCM)
==========================================================================
...
AO: [pulse] 11024Hz 1ch u8 (1 bytes per sample)
...
VO: [xv] 640x480 => 640x480 Packed YUY2 

The audio is already stored in low quality (low sample rate, mono, low precision), but the video encoding can be improved vastly.

$ mencoder -oac copy -ovc x264 -o ${input%.avi}-h264.avi $input

... or, if you need it rotated clockwise:

$ mencoder -oac copy -ovc x264 -vf rotate=1 -o ${input%.avi}-h264.avi $input

This decreased my samples files by 9.5x on average. 32MiB is still way too big for a low quality 2.5 minute video clip, but it's a lot better than 280MiB.

Update 2013-07-13

Sample script to aid in conversion of multiple files:
mjpegtoh264.sh (view)