Downloading RTMP Flash media with rtmpdump

EDIT: These instructions are for verson 1.9 of rtmpdump! Newer versions make this process easier but also make some of this how-to obsolete. But I've modified the process below so that you can still use them for the legacy 1.9 version.

Today I attended a lecture/demo session on "hacking" various media sites like YouTube, Hulu, last.fm. Essentially it was a practical demonstration of the cat and mouse game hackers play with content pedlars on the internet. It was held at Cyberpipe, a nifty club in Ljubljana, Slovenia where I study.

One of the tools mentioned was rtmpdump which is apparently made by the same guys as the legendary mplayer, mencoder, and ffmpeg.

I decided to compile it myself from the most recent development version. If you're lazy and brave enough to trust compiled binaries and packages from third parties, go to LinuxCentre and install flvstreamer for your operating system. These programs were once one and so far they use the same commands. Read: they work the same.

These instrucitons work on Ubuntu 9.10 (karmic) but please be aware things change rapidly so the following might not work forever.

First of all install the tools needed for accessing the code and compilation.

sudo apt-get install build-essential gcc make subversion libssl0.9.8 libssl-dev libssl0.9.8

Now check out the latest code from the Subversion repository:

svn co svn://svn.mplayerhq.hu/rtmpdump/branches/1.x rtmpdump-1.9

If you prefer a stable version, run this instead:

wget http://rtmpdump.mplayerhq.hu/rtmpdump-1.9.tgz
tar xfz rtmpdump-1.9.tgz

I'm sure the impatient are looking forward to completing this ASAP so here's the final step in setting up rtmpdump on your computer:

cd rtmpdump-1.9
make linux

The program now lies in the same folder and you can only run it by telling your shell (like bash) exactly where it is. If you're in the same folder, running it is as simple as ./rtmpdump but ./ is only short for the current folder and it gets expanded to (in my case) /home/jasa/rtmpdump/rtmpdump.

Great! Now we need a media site which uses a Adobe's RTMP server. I think I won't get in trouble if I use VideoLectures.net as an example. :)

Lets say I'm interested in probability and statistics. And I want to watch these lectures offline, on a train or wherever.

Unfortunately this isn't a straightforward process and quite a bit of work is required. It is derived from a mailing list thread.

I need to see the source code of the web page above, not the rendered output of the web browser. In Firefox press Ctrl+U. Now I need some hard data to pass to rtmpdump. Search for a javascript section which launches the flash video player. For the statistics video this is the interesting section:

var flashvars = {
    
      streamer: "rtmp://oxy.videolectures.net/video",
      file: "2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01.flv",
    
    height: '288',
    autostart: "true",
    bufferlength: '5',
  
  
    image: "http://media.videolectures.net/play.png",
  
    id: "FlvPlayer" // last line, no colon ',' !
  };
  swfobject.embedSWF("http://media.videolectures.net/jw-player/player.swf", "video_embed", "384", "307", "9.0.0", "http://media.videolectures.net/swfobject/expressInstall.swf", flashvars, params, attributes);

Generally searching (Ctrl+F) for flashvars should get you near the required data. The line swfobject.embedSWF(...) causes the video player to load inside your browser and display the correct media. This media rests on a different server than the webpage and I will use rtmpdump to connect to that server instead of the flash player. Now Adobe has put some roadblocks into the process but lucky for me I have a flying car. :)

I'll just give you the final command now and explain later!

./rtmpdump -r rtmp://oxy.videolectures.net/video/ -y 2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01 -a video -s http://media.videolectures.net/jw-player/player.swf -w ffa4f0c469cfbe1f449ec42462e8c3ba16600f5a4b311980bb626893ca81f388 -x 53910 -o test.flv

The -r switch requires an argument which is the URL of the media server and is found in the variable streamer in the JavaScript source code above.
The -y switch needs the playpath and that is found in the JavaScript variable file, (minus the extension .flv or .mp4).
The -a switch is the name of the used player and usually automatically inferred from the URL. Defining it manually works by copying the part after the server name in the streamer.
The -s switch defines the flash video player which normally connects to the media server. In the example it is the first argument of the function embedSWF.

Here's where things get even more complicated. The media server wants some extra data about this player, specifically its sha256 hash-sum and size in bytes. So lets get them:

wget http://media.videolectures.net/jw-player/player.swf
sha256sum player.swf
ls -l player.swf

Supply the sha256sum to switch -w and the file size to the -x switch.

Anything else? Yeah, I need to specify where to save the video with -o.

Run the longest command ever and get yourself a beer (you've earned it)! :)

I admit that doing this for every video gets time consuming but unfortunately it is a procedure specific to every site (look at the mailing list thread link again to see different javascript). But for the same site there's very little extra work.

Hope you lasted this long, be seeing ya!

Automatic Adobe Flash on amd64

Image via WikipediaUPDATE: Adobe has closed down their webpage which hosts the 64-bit version of Flash for Linux so the method described below no longer works! :(


Yes, apparently I have had a one month lurch. Things have been hectic as the new school year started and I couldn't really believe that almost a month has passed since my last effort!



The state of Adobe's Flash Player on Linux is a sad story, even more so for the next architecture PCs are migrating to (AMD64). The simple truth is that the modern web is unusable sans Flash for the common web surfer.
While I and others would like to see it displaced by open standards, such as the forthcoming HTML5 with the video and canvas elements, that is still far on the horizon and Adobe isn't exactly idling.


Okay, we Linux users are a pretty small part of the global pie, but we're arguably more technically savvy than the average computer drone. The 32-bit (x86) version of Flash player can easily be installed from Adobe's site but the amd64 flavour is still an exercise in bash. The software is still deemed to be of alpha quality so perhaps that is the reason why a more friendly install procedure isn't available. Luckily, Linux geeks like Romeo-Adrian Cioaba are friendly. How quickly a need is scratched now that Ubuntu has gained so much traction!


By modifying his script I have managed to make a version that will automatically download and install the latest version available from Adobe Labs. Be warned: I make no warranties that this procedure will work indefinitely or that it may not harm your computer. But the code is available for you to analyse so you may convince yourself of its correctness. Best way to learn is probably to execute each line by hand to see what happens. The manuals for the commands are your friends. :) Also, know that whenever you wish to install the version that comes with ubuntu, you need to run just the commands in the first 17 lines. Remember that, write it down somewhere, like in ~/FLASH_UPGRADE.txt

#!/bin/bash
# Script  created by
# Romeo-Adrian Cioaba romeo.cioaba@spotonearth.com
# Jaša Bartelj jasa.bartelj@gmail.com

echo "Stopping any Firefox that might be running."
sudo killall -9 firefox

echo "Removing any other flash plugin previously installed."
sudo apt-get remove -y --purge flashplugin-nonfree gnash gnash-common mozilla-pl
ugin-gnash swfdec-mozilla libflashsupport nspluginwrapper
sudo rm -f /usr/lib/mozilla/plugins/*flash*
sudo rm -f ~/.mozilla/plugins/*flash*
sudo rm -f /usr/lib/firefox/plugins/*flash*
sudo rm -f /usr/lib/firefox-addons/plugins/*flash*
sudo rm -rfd /usr/lib/nspluginwrapper

echo "Installing Flash Player 10."
cd /tmp
# EDIT: when releasing flash 10.1 adobe changed the site's address.
# I want permalinks! :(  --Jasa B.
#wget http://labs.adobe.com/downloads/flashplayer10.html
wget -q http://labs.adobe.com/downloads/flashplayer10_64bit.html
wget `cat flashplayer10_64bit.html | egrep -o "http:.*"|cut -d\" -f1|grep linux-x86_64.so.tar.gz`
ARCHIVE=`ls libflashplayer-*.linux-x86_64.so.tar.gz`
echo "Version is $ARCHIVE."
tar zxvf $ARCHIVE
sudo cp libflashplayer.so /usr/lib/mozilla/plugins/ 

echo "Linking the libraries so Firefox and apps built on XULRunner can find it."
sudo ln -sf /usr/lib/mozilla/plugins/libflashplayer.so /usr/lib/firefox-addons/plugins/
sudo ln -sf /usr/lib/mozilla/plugins/libflashplayer.so  /usr/lib/xulrunner-addons/plugins/

# now doing some cleaning up:
sudo rm -rf flashplayer10.html
sudo rm -rf libflashplayer.so
sudo rm -rf $ARCHIVE

The most complicated part is really just snipping the HTML for the URL of the archive. Please, enjoy and comment.

The mighty PDF

Image via Wikipedia

PDF documents are a perfect export target in many applications because they're portable and look the same everywhere. So perfect in fact that there are virtual printers (like PDFCreator for Windows and integrated ones in Linux/*BSDs) which transform any printable file to a PDF.

But they are difficult when you want to change them or combine them.

I've had a folder sitting on my otherwise unburdened desktop for a year. It contained a bunch of scanned images which I wanted to aggregate into a nice and shiny single PDF file. Yeah. My google-fu had some slow reflexes this time and the first tries with FPDF, a PHP module, were unsuccessful. The images were in the correct format their manual demanded but I could only helplessly bang my head away as the scripts failed with errors of unsupported formats.

Sometimes my determination falters at moments like this and I put off trying to resolve the problem. But I had to push!

My google-fu got some well-deserved rest and found what I was looking for - a solution using GhostScript. By the way - this solution works for me on Ubuntu Linux but elsewhere YMMV.

Here's a script I ended up using (warning: It overwrites the original images.):

#!/bin/bash
for filename in *
do
    file -bi "$filename" | grep -q "image"
    if [ $? -eq 0 ]
    then
        echo "  Converting $filename"
        #Reduce image size by half 
        convert $filename -resize 50% $filename
        #Convert image to single-page PDF 
        convert $filename ${filename%.*}.pdf
    fi
done
gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=combined.pdf -dBATCH *.pdf

The bash script relies on ImageMagick (the convert command) and GhostScript (the gs command). Best thing is to copy it to a file in the folder with the images destined to comprise the PDF, marking the file as executable and running it! The final PDF will contain all the single-paged PDFs sorted in lexical order.

For the intricacies of GhostScript seen above I will let the devil speak for itself:

$ gs -h
GPL Ghostscript 8.64 (2009-02-03)
Copyright (C) 2009 Artifex Software, Inc.  All rights reserved.
Usage: gs [switches] [file1.ps file2.ps ...]
Most frequently used switches: (you can use # in place of =)
 -dNOPAUSE           no pause after page   | -q       `quiet', fewer messages
 -g<width>x<height>  page size in pixels   | -r<res>  pixels/inch resolution
 -sDEVICE=<devname>  select device         | -dBATCH  exit after last file
 -sOutputFile=<file> select output file: - for stdout, |command for pipe,
                                         embed %d or %ld for page #
Input formats: PostScript PostScriptLevel1 PostScriptLevel2 PostScriptLevel3 PDF

Vanity

Seeing my first post published was accompanied by a healthy dose of feeling accomplishment. But soon I found that the script code left something to be desired.

Prettiness!

After scouring the now humongous blogosphere I found people to relate to. But that's usually easy on the Internet. So after applying google-fu to the problem I found SyntaxHighlighter! Using it on Blogger requires a few simple steps which others have explained but I always try to do it my way so I learn something and memorize the process.

The gist of it is adding more javascript to your blog's HTML template. I appended my <head> section with this:

<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/> 
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/> 
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js'></script> 
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js'></script> 
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js'></script> 
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js'></script> 
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js'></script> 
<script type='text/javascript'> 
 SyntaxHighlighter.config.bloggerMode = true;
 SyntaxHighlighter.all();
</script>

The javascripts parse any <pre> blocks in your blog post and display the code in a nice way. The default style clashes a bit with my blog template but I might fix that. What good is dabbling in vanity if it doesn't achieve its purpose? :)

So let me try Hello World again:

#!/bin/bash
echo 'Hello, world!'
if [ $? -eq 0 ]
then
    ./comment > /dev/null
fi

:P

First post

Image by oskay via Flickr

Time for the first entry.

I could never come up with a good plan on what to write about or perhaps I just didn't want to write strictly about one subject. Oh how the times have changed.

Here I am, typing away precious moments trying to convey a message of what this blog will be about in an ambiguous natural language.

Let me give you a hint:

#!/bin/bash

echo 'Hello, world!'

if [ $? -eq 0 ]

then

    ./comment

fi

Hopefully that didn't drive you away with WARP 9.9! :) Why don't you subscribe to the feed and be notified of changes automatically.