Notes to self, 2012
2012-02-17 - indirect scp / bypass remote firewall rules
Suppose I'm on machine DESKTOP and I want to copy files from server APPLE to server BANANA. DESKTOP has access to both, but firewalls and/or missing ssh keys prevent direct access between APPLE and BANANA.
Regular scp(1) will now fail. It will attempt to do a direct copy
and then give up. This is where this
indirect scp wrapper
(view) comes in:
- First, it tries to do the direct copy.
- If that fails, it uses the local machine as an intermediary.
In this example you'll see it fail twice for the two source files and then fall back to using the local machine.
$ scp -r APPLE:example/file1 APPLE:example/somedir BANANA:some_existing_path/ Host key verification failed. lost connection Host key verification failed. lost connection (falling back to indirect copy...) file1 100% 6 0.0KB/s 00:00 here 100% 5 0.0KB/s 00:00 two_files 100% 10 0.0KB/s 00:00 (copy from here to destination...) file1 100% 6 0.0KB/s 00:00 here 100% 5 0.0KB/s 00:00 two_files 100% 10 0.0KB/s 00:00 (cleaning up temporary files...)
For a bit of added security, it uses shred(1) to clean up
the local files, if available.
Installation:
# cd /usr/local/bin # wget http://wjd.nu/files/2012/02/indirect-scp.sh -O indirect-scp # chmod 755 indirect-scp # ln -s indirect-scp scp
If you know the direct copy will fail, you can call indirect-scp
directly.
2012-01-25 - mysql replication / relay log pos
So, hardware trouble caused a VPS to go down. This VPS was running a MySQL server in a slave setup. Not surprisingly, the unclean shutdown broke succesful slaving.
There are several possibly causes for slave setup breakage. This time it
was the local relay log file (mysqld-relay-bin.xxxx)
that was out of sync.
SHOW SLAVE STATUS\G looked like this:
...
Master_Log_File: mysql-bin.001814 <-- remote/master file (IO thread)
Read_Master_Log_Pos: 33453535 <-- remote/master pos (IO thread)
Relay_Log_File: mysqld-relay-bin.001383 <-- local/slave file (SQL thread)
Relay_Log_Pos: 34918332 <-- local/slave pos (SQL thread)
Relay_Master_Log_File: mysql-bin.001812 <-- remote/master file (SQL thread)
...
Last_Errno: 1594
Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
...
Exec_Master_Log_Pos: 34918187 <-- remote/master pos (SQL thread)
...
Step one was to find out where we were in the local and on the remote
end. Luckily, most queries ran during the failure period were UPDATEs
on the same table.
- Remote position was ok. On the master,
/var/log/mysql/mysql-bin.001812contained these lines:# at 34918187 #120125 1:16:05 server id 1 end_log_pos 34918531 ... SET TIMESTAMP=1327450565/*!*/; UPDATE mytable ....
- The statements before that had been ran on the slave and this statement hadn't.
- On the slave,
/var/lib/mysql/mysqld-relay-bin.001383did contain the previous line, but did not contain position 34918332. - Looking further, I could see that
mysqld-relay-bin.001384was practically empty, butmysqld-relay-bin.001385contained already executed statements, and after a bit of browsing there it was too:# at 21491 #120125 1:16:05 server id 1 end_log_pos 34918531 ... SET TIMESTAMP=1327450565/*!*/; UPDATE mytable ...
Good. So we need only move the relay log file pointer a bit to the front.
mysql> CHANGE MASTER TO RELAY_LOG_FILE='mysqld-relay-bin.001385', RELAY_LOG_POS=21491; ERROR 1380 (HY000): Failed initializing relay log position: Could not find target log during relay log initialization
What? Searching for that error pointed to a document about copying slave data to another slave and about modifying files. Hmm.. modifying files. I can do that too...
# cat /var/lib/mysql/relay-log.info ./mysqld-relay-bin.001383 34918332 mysql-bin.001812 34918187 0
With a little speed — /etc/init.d/mysql stop ; vim /var/lib/mysql/relay-log.info ; /etc/init.d/mysql start —
I edited the relay log file and relay log position in relay-log.info by hand.
Voilà! It worked. Slave replication was running again like it should.
2012-01-08 - mencoder / canon / mjpeg
I tend to make few movies with my digital photo camera because they take up so much space. That's a shame, because having a bit of moving image is fun to look at when the kids have grown up.
The reason they take up so much space is simple. The camera is not equipped with fancy encoding algorithms: the video is stored as MJPEG, basically a series of JPEG images joined together. Somehow I should've known this ;-)
========================================================================== Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG) ========================================================================== Opening audio decoder: [pcm] Uncompressed PCM audio decoder AUDIO: 11024 Hz, 1 ch, u8, 88.2 kbit/100.00% (ratio: 11024->11024) Selected audio codec: [pcm] afm: pcm (Uncompressed PCM) ========================================================================== ... AO: [pulse] 11024Hz 1ch u8 (1 bytes per sample) ... VO: [xv] 640x480 => 640x480 Packed YUY2
The audio is already stored in low quality (low sample rate, mono, low precision), but the video encoding can be improved vastly.
$ mencoder -oac copy -ovc x264 -o ${input%.avi}-h264.avi $input
... or, if you need it rotated clockwise:
$ mencoder -oac copy -ovc x264 -vf rotate=1 -o ${input%.avi}-h264.avi $input
This decreased my samples files by 9.5x on average. 32MiB is still way too big for a low quality 2.5 minute video clip, but it's a lot better than 280MiB.
