wjd.nu

2012-02-17 - indirect scp / bypass remote firewall rules

Suppose I'm on machine DESKTOP and I want to copy files from server APPLE to server BANANA. DESKTOP has access to both, but firewalls and/or missing ssh keys prevent direct access between APPLE and BANANA.

Regular scp(1) will now fail. It will attempt to do a direct copy and then give up. This is where this indirect scp wrapper (view) comes in:

• First, it tries to do the direct copy.
• If that fails, it uses the local machine as an intermediary.

In this example you'll see it fail twice for the two source files and then fall back to using the local machine.

$ scp -r APPLE:example/file1 APPLE:example/somedir BANANA:some_existing_path/
Host key verification failed.
lost connection
Host key verification failed.
lost connection
(falling back to indirect copy...)
file1                                     100%    6     0.0KB/s   00:00
here                                      100%    5     0.0KB/s   00:00
two_files                                 100%   10     0.0KB/s   00:00
(copy from here to destination...)
file1                                     100%    6     0.0KB/s   00:00
here                                      100%    5     0.0KB/s   00:00
two_files                                 100%   10     0.0KB/s   00:00
(cleaning up temporary files...)

For a bit of added security, it uses shred(1) to clean up the local files, if available.

Installation:

# cd /usr/local/bin
# wget http://wjd.nu/files/2012/02/indirect-scp.sh -O indirect-scp
# chmod 755 indirect-scp
# ln -s indirect-scp scp

If you know the direct copy will fail, you can call indirect-scp directly.

2012-01-25 - mysql replication / relay log pos

So, hardware trouble caused a VPS to go down. This VPS was running a MySQL server in a slave setup. Not surprisingly, the unclean shutdown broke succesful slaving.

There are several possibly causes for slave setup breakage. This time it was the local relay log file (mysqld-relay-bin.xxxx) that was out of sync.

SHOW SLAVE STATUS\G looked like this:

...
       Master_Log_File: mysql-bin.001814  <-- remote/master file (IO thread)
   Read_Master_Log_Pos: 33453535          <-- remote/master pos  (IO thread)
        Relay_Log_File: mysqld-relay-bin.001383  <-- local/slave file (SQL thread)
         Relay_Log_Pos: 34918332                 <-- local/slave pos  (SQL thread)
 Relay_Master_Log_File: mysql-bin.001812  <-- remote/master file (SQL thread)
...
            Last_Errno: 1594
            Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
...
   Exec_Master_Log_Pos: 34918187          <-- remote/master pos  (SQL thread)
...

Step one was to find out where we were in the local and on the remote end. Luckily, most queries ran during the failure period were UPDATEs on the same table.

• Remote position was ok. On the master, /var/log/mysql/mysql-bin.001812 contained these lines:

  # at 34918187
  #120125  1:16:05 server id 1  end_log_pos 34918531 ...
  SET TIMESTAMP=1327450565/*!*/;
  UPDATE mytable ....
  

• The statements before that had been ran on the slave and this statement hadn't.
• On the slave, /var/lib/mysql/mysqld-relay-bin.001383 did contain the previous line, but did not contain position 34918332.
• Looking further, I could see that mysqld-relay-bin.001384 was practically empty, but mysqld-relay-bin.001385 contained already executed statements, and after a bit of browsing there it was too:

  # at 21491
  #120125  1:16:05 server id 1  end_log_pos 34918531 ...
  SET TIMESTAMP=1327450565/*!*/;
  UPDATE mytable ...
  

Good. So we need only move the relay log file pointer a bit to the front.

mysql> CHANGE MASTER TO RELAY_LOG_FILE='mysqld-relay-bin.001385', RELAY_LOG_POS=21491;
ERROR 1380 (HY000): Failed initializing relay log position: Could not find target log during relay log initialization

What? Searching for that error pointed to a document about copying slave data to another slave and about modifying files. Hmm.. modifying files. I can do that too...

# cat /var/lib/mysql/relay-log.info
./mysqld-relay-bin.001383
34918332
mysql-bin.001812
34918187
0

With a little speed — /etc/init.d/mysql stop ; vim /var/lib/mysql/relay-log.info ; /etc/init.d/mysql start — I edited the relay log file and relay log position in relay-log.info by hand.

Voilà! It worked. Slave replication was running again like it should.

2012-01-08 - mencoder / canon / mjpeg

I tend to make few movies with my digital photo camera because they take up so much space. That's a shame, because having a bit of moving image is fun to look at when the kids have grown up.

The reason they take up so much space is simple. The camera is not equipped with fancy encoding algorithms: the video is stored as MJPEG, basically a series of JPEG images joined together. Somehow I should've known this ;-)

==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG)
==========================================================================
Opening audio decoder: [pcm] Uncompressed PCM audio decoder
AUDIO: 11024 Hz, 1 ch, u8, 88.2 kbit/100.00% (ratio: 11024->11024)
Selected audio codec: [pcm] afm: pcm (Uncompressed PCM)
==========================================================================
...
AO: [pulse] 11024Hz 1ch u8 (1 bytes per sample)
...
VO: [xv] 640x480 => 640x480 Packed YUY2 

The audio is already stored in low quality (low sample rate, mono, low precision), but the video encoding can be improved vastly.

$ mencoder -oac copy -ovc x264 -o ${input%.avi}-h264.avi $input

... or, if you need it rotated clockwise:

$ mencoder -oac copy -ovc x264 -vf rotate=1 -o ${input%.avi}-h264.avi $input

This decreased my samples files by 9.5x on average. 32MiB is still way too big for a low quality 2.5 minute video clip, but it's a lot better than 280MiB.