Notes to self, 2012
2012-05-14 - ubuntu / sip video / softphone
So, I wanted to test video support with Asterisk. That was easier said than done, because the SIP softphones that ship with Ubuntu don't all do what they promise.
This was done on a setup that works for numerous hardphones and PBXs out there. Looking through the registration list at any given time reveals at least 40+ different user agents and a large multiple of that if you take the different versions into account.
Drawbacks of Ekiga (3.2.6 and 3.3.2)
- User interface offers no advanced options.
- Refuses to run on a different local port than 5060 (or at least it used to).
- It crashes.
- Hangs during registration for no apparent reason. (Doesn't do registration at all, but does attempt to SUBSCRIBE to local users on the remote proxy a lot in the mean time.)
- Sends link-local IPv6 addresses in the
Contactheader, which my provider uses as only contact, resulting in breakage. - Re-registers immediately with an
Expires: 0(unregister) and claims that it is registered. - Doesn't use the outboundproxy for the registration attempts.
Conclusion: completely unusable for a setup that works with every hardphone out there.
Drawbacks of Telepathy/Empathy (2.30.3 and 3.4.1)
- It crashes, but less often than Ekiga.
- The user interface is unintuitive: accounts list available through
empathy-accountsbinary only; details-of-call pane is truncated so only the first 6 letters of relevant info is visible; setting up a call not possible through the notification area. - Call setup failed when the default NAT settings were enabled. Things started working when we disabled all NAT features in the client and relied on the NAT-magic of the remote OpenSIPs and Asterisk.
- Video support fails when one end does not have a video source.
- For no apparent reason, it started doing auto-pickup (with video!). That could create embarrassing situations ;-)
Conclusion: less broken than Ekiga, but still unusable.
Drawbacks of Linphone (3.2.1 and 3.3.2)
- It did manage to calculate the wrong MD5 response for an authentication request once. But since it did that when I was messing about with the hostname in /etc/hosts, I will forgive it.
- You cannot disable sending video if you have a working video source without disabling the reception of video as well.
- It works! Even when one of the clients doesn't have a video input source. Oh wait.. this isn't a drawback.
- It has a sensible (old school) user interface with settings where you expect them! Another one for the plus side.
Conclusion: use Linphone! Thank you Simon Morlat. (Now, if you could just replace the 't' in “standart” [sic] in the About box ;-) )
By the way, the video support in Asterisk did indeed work fine.
2012-05-04 - django / mark_safe / translatables
Look at this snippet of Django code in models.py, and in particular
the help_text bit:
from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils.safestring import mark_safe
class MyModel(models.Model):
my_field = models.CharField(max_length=123,
help_text=mark_safe(_('Some <b>help</b> text.')))
For those unfamiliar with Django. A quick run-down:
- The definition of
MyModelcreates a mapping between the MyModel class and a underlyingapp_mymodeltable in a database. - That table will consist of two columns:
id, an automatic integer as primary key (created by default), andmy_field, a varchar/text field of at most 123 characters. - With minimal effort a HTML form can be generated from this. That form will
show
my_fieldas a text input box and near it the text we defined inhelp_text. - The
_()-function has the gettext functions run over it so the text can be served in different languages with minimal effort. - The
mark_safe()-function tells the template rendererer that this output is already safe to use in HTML. Without it, the user would see:Some <b>help</b> text.
Unfortunately this doesn't do what you would expect.
Let's examine why.
There is a reason why we use the ugettext_lazy wrapper in models.py.
This code is executed once at startup / first run, and the language that was selected at that time
would be substituted if we used the non-lazy ugettext. The lazy variant makes
sure the substitution takes place at the last possible time.
mark_safe forces the translation to happen immediately.
In the best case that means someone else can get the help text served in the wrong language.
In the worst case, you get a recursive import when the translation routines attempt to
import all INSTALLED_APPS while looking for locale files. Your MyModel
might be referenced from one of those apps. The result: recursion and a resulting ImportError.
...
File "someapp/models.py", line 5, in <module>
class MyModel(models.Model):
File "someapp/models.py", line 6, in <module>
my_field = models.CharField(max_length=123,
File "django/utils/safestring.py", line 101, in mark_safe
return SafeUnicode(s)
...
File "django/utils/translation/trans_real.py", line 180, in _fetch
app = import_module(appname)
File "django/utils/importlib.py", line 35, in import_module
__import__(name)
...
ImportError: cannot import name MyModel
Lessons learnt: if you're using translations then don't call mark_safe
on anything until it's view time.
In this case, we would fix it by adding the mark_safe call to the Form
constructor. We know that that is run for every form instantiation, so that's late enough.
class MyModelForm(forms.ModelForm):
class Meta:
model = MyModel
def __init__(self, *args, **kwargs):
super(MyModelForm, self).__init__(*args, **kwargs)
self.fields['my_field'].help_text = mark_safe(self.fields['my_field'].help_text)
But suggestions for prettier solutions are welcome.
2012-04-30 - ipython classic mode / precise pangolin
The Ubuntu do-release-upgrade broke my ipython classic mode.
The ipython package was upgraded, and apparently the configuration parser was changed.
In bash, I want colors to help me find the beginning and end of output — see
this bug report
for others agreeing with me that the derogatory comment about "focus should be on the output,
not on the prompt" in the skeleton .bashrc is is retared, but I diverge —
in ipython, I just want to see the nice >>> blocks that I'm used to and
no extra spaces. I.e.: classic mode.
Previously, one could set classic 1 in ~/.ipythonrc. Since the upgrade,
one should be able to fix it by editing .config/ipython/profile_default/ipython_config.py.
However, the fixes that the pages on google propose, do not work:
c.TerminalIPythonApp.classic = False
The following does work:
# Force classic mode! c = get_config() c.InteractiveShell.cache_size = 0 c.PlainTextFormatter.pprint = False c.PromptManager.in_template = '>>> ' c.PromptManager.in2_template = '... ' c.PromptManager.out_template = '' c.InteractiveShell.separate_in = '' c.InteractiveShell.separate_out = '' c.InteractiveShell.separate_out2 = '' c.InteractiveShell.colors = 'NoColor' c.InteractiveShell.xmode = 'Plain' # c.InteractiveShell.confirm_exit = False #c.IPythonTerminalApp.display_banner = False
These settings are taken from the source where the --classic option
is defined, so they should ok.
2012-04-12 - safe_asterisk / init.d
An init.d script to stop and start safe_asterisk started asterisk.
If asterisk is not stopped in 5 seconds, it is forcibly killed.
# wget http://wjd.nu/files/2012/04/safe_asterisk-init.d -O/etc/init.d/asterisk ; chmod 755 /etc/init.d/asterisk
Also possibly useful, the changes I made to safe_asterisk on a
machine where:
- there wasn't a
ttyleft to spam output on, rootis configured in/etc/aliasesto a sane destination,/var/spool/asteriskis the asterisk user homedir anyway, and,- attempting to set maxfiles to the highest value possible, wasn't allowed.
#TTY=9 CONSOLE=no NOTIFY=root DUMPDROP=/var/spool/asterisk MAXFILES=12288 ... run_asterisk >/dev/null 2>&1 &
2012-03-29 - sip / digest calculation
Every one in a while, I see an unexpected 403 response to a SIP
client's REGISTER request. Thusfar the digest
response calculation has never been wrong, but it feels good to
get that check out of the way and move on to other possible
causes.
For your enjoyment and mine, a Bourne-shell compatible shell
script that calculates (qop-less)
Digest authentication
responses.
Download: hahacalc.sh (view)
$ hahacalc Usage: hahacalc USERNAME REALM METHOD DIGESTURI NONCE [PASSWORD] [COMPARE] Example: hahacalc 123456789 itsp.com REGISTER sip:sip.itsp.com:6060 4f7406a80000c2f214774a48d11cc5b9e533caff7a05904c MYPASSWORD f5787ba2624dfdbcf874f46425d65b53
$ hahacalc 123456789 itsp.com REGISTER sip:sip.itsp.com:6060 4f7406a80000c2f214774a48d11cc5b9e533caff7a05904c MYPASSWORD f5787ba2624dfdbcf874f46425d65b53 A1 = 123456789:itsp.com:MYPASSWORD HA1 = 7f3c958e155df4070204c2aa5293437f A2 = REGISTER:sip:sip.itsp.com:6060 HA2 = 2d13f8e4f7343dbd44be6ddabf8c2c1d RESP = f5787ba2624dfdbcf874f46425d65b53
2012-03-13 - python virtualenv / global site-packages
If you're switching from Ubuntu Oneiric to Ubuntu Precise
and you're using python-virtualenv, you might be in for a
surprise:
The default access to the global site-packages modules is reversed between virtualenv 1.6.x and 1.7.
When you were used to finding your apt-get installed python
modules like python-mysqldb and python-psycopg2 in your
new virtualenv environment, now they're suddenly unavailable.
The culprit:
--no-site-packages
Ignored (the default). Don“t give access to the global
site-packages modules to the virtual environment.
That was not the default before.
The new option to get the old behaviour:
--system-site-packages
Give access to the global site-packages modules to the
virtual environment.
By the way, if you already populated your new virtualenv directory
with the 1.7 version, you don't need to recreate it.
Removing the no-global-site-packages.txt is enough:
# find /srv/django-env -name no-global-site-packages.txt /srv/django-env/mysite/lib/python2.7/no-global-site-packages.txt
2012-02-27 - mysql / replicating repair table
From the MySQL 5.1 manual:
15.4.1.16. Replication and REPAIR TABLE
When used on a corrupted or otherwise damaged table, it is possible for the REPAIR TABLE statement to delete rows that cannot be recovered. However, any such modifications of table data performed by this statement are not replicated, which can cause master and slave to lose synchronization. For this reason, in the event that a table on the master becomes damaged and you use REPAIR TABLE to repair it, you should first stop replication (if it is still running) before using REPAIR TABLE, then afterward compare the master's and slave's copies of the table and be prepared to correct any discrepancies manually, before restarting replication.
Sounds like a pain the behind to have to do manually, especially if you have data updates in one of the two Master-Master replicated machines.
But that is not the point with my quote. The point is:
If REPAIR TABLE does nothing to ensure that two copies are identical, then
why on earth does it replicate the REPAIR TABLE statement at all?
Because that was all I wanted to know: does REPAIR TABLE get replicated?
Logical answer: no
Actual answer: yes
Beware.. and be prepared to skip a couple of statements.
2012-02-17 - indirect scp / bypass remote firewall rules
Suppose I'm on machine DESKTOP and I want to copy files from server APPLE to server BANANA. DESKTOP has access to both, but firewalls and/or missing ssh keys prevent direct access between APPLE and BANANA.
Regular scp(1) will now fail. It will attempt to do a direct copy
and then give up. This is where this
indirect scp wrapper
(view) comes in:
- First, it tries to do the direct copy.
- If that fails, it uses the local machine as an intermediary.
In this example you'll see it fail twice for the two source files and then fall back to using the local machine.
$ scp -r APPLE:example/file1 APPLE:example/somedir BANANA:some_existing_path/ Host key verification failed. lost connection Host key verification failed. lost connection (falling back to indirect copy...) file1 100% 6 0.0KB/s 00:00 here 100% 5 0.0KB/s 00:00 two_files 100% 10 0.0KB/s 00:00 (copy from here to destination...) file1 100% 6 0.0KB/s 00:00 here 100% 5 0.0KB/s 00:00 two_files 100% 10 0.0KB/s 00:00 (cleaning up temporary files...)
For a bit of added security, it uses shred(1) to clean up
the local files, if available.
Installation:
# cd /usr/local/bin # wget http://wjd.nu/files/2012/02/indirect-scp.sh -O indirect-scp # chmod 755 indirect-scp # ln -s indirect-scp scp
If you know the direct copy will fail, you can call indirect-scp
directly.
2012-01-25 - mysql replication / relay log pos
So, hardware trouble caused a VPS to go down. This VPS was running a MySQL server in a slave setup. Not surprisingly, the unclean shutdown broke succesful slaving.
There are several possibly causes for slave setup breakage. This time it
was the local relay log file (mysqld-relay-bin.xxxx)
that was out of sync.
SHOW SLAVE STATUS\G looked like this:
...
Master_Log_File: mysql-bin.001814 <-- remote/master file (IO thread)
Read_Master_Log_Pos: 33453535 <-- remote/master pos (IO thread)
Relay_Log_File: mysqld-relay-bin.001383 <-- local/slave file (SQL thread)
Relay_Log_Pos: 34918332 <-- local/slave pos (SQL thread)
Relay_Master_Log_File: mysql-bin.001812 <-- remote/master file (SQL thread)
...
Last_Errno: 1594
Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
...
Exec_Master_Log_Pos: 34918187 <-- remote/master pos (SQL thread)
...
Step one was to find out where we were in the local and on the remote
end. Luckily, most queries ran during the failure period were UPDATEs
on the same table.
- Remote position was ok. On the master,
/var/log/mysql/mysql-bin.001812contained these lines:# at 34918187 #120125 1:16:05 server id 1 end_log_pos 34918531 ... SET TIMESTAMP=1327450565/*!*/; UPDATE mytable ....
- The statements before that had been ran on the slave and this statement hadn't.
- On the slave,
/var/lib/mysql/mysqld-relay-bin.001383did contain the previous line, but did not contain position 34918332. - Looking further, I could see that
mysqld-relay-bin.001384was practically empty, butmysqld-relay-bin.001385contained already executed statements, and after a bit of browsing there it was too:# at 21491 #120125 1:16:05 server id 1 end_log_pos 34918531 ... SET TIMESTAMP=1327450565/*!*/; UPDATE mytable ...
Good. So we need only move the relay log file pointer a bit to the front.
mysql> CHANGE MASTER TO RELAY_LOG_FILE='mysqld-relay-bin.001385', RELAY_LOG_POS=21491; ERROR 1380 (HY000): Failed initializing relay log position: Could not find target log during relay log initialization
What? Searching for that error pointed to a document about copying slave data to another slave and about modifying files. Hmm.. modifying files. I can do that too...
# cat /var/lib/mysql/relay-log.info ./mysqld-relay-bin.001383 34918332 mysql-bin.001812 34918187 0
With a little speed — /etc/init.d/mysql stop ; vim /var/lib/mysql/relay-log.info ; /etc/init.d/mysql start —
I edited the relay log file and relay log position in relay-log.info by hand.
Voilà! It worked. Slave replication was running again like it should.
2012-01-08 - mencoder / canon / mjpeg
I tend to make few movies with my digital photo camera because they take up so much space. That's a shame, because having a bit of moving image is fun to look at when the kids have grown up.
The reason they take up so much space is simple. The camera is not equipped with fancy encoding algorithms: the video is stored as MJPEG, basically a series of JPEG images joined together. Somehow I should've known this ;-)
========================================================================== Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG) ========================================================================== Opening audio decoder: [pcm] Uncompressed PCM audio decoder AUDIO: 11024 Hz, 1 ch, u8, 88.2 kbit/100.00% (ratio: 11024->11024) Selected audio codec: [pcm] afm: pcm (Uncompressed PCM) ========================================================================== ... AO: [pulse] 11024Hz 1ch u8 (1 bytes per sample) ... VO: [xv] 640x480 => 640x480 Packed YUY2
The audio is already stored in low quality (low sample rate, mono, low precision), but the video encoding can be improved vastly.
$ mencoder -oac copy -ovc x264 -o ${input%.avi}-h264.avi $input
... or, if you need it rotated clockwise:
$ mencoder -oac copy -ovc x264 -vf rotate=1 -o ${input%.avi}-h264.avi $input
This decreased my samples files by 9.5x on average. 32MiB is still way too big for a low quality 2.5 minute video clip, but it's a lot better than 280MiB.