Revisiting rtmpdump [updated!]

Image via Wikipedia

EDIT 2012-05-15: Updated guide so it uses the "new" git code repository! Thanks BecomingJulie!

Time for an updated guide - I told you previously that things change quickly! :)
The new major release 2.x of rtmpdump brings new tools which make things much easier. How much? Well downloading shows of The Colbert Report, The Daily Show and episodes from South Park Studios is now almost a breeze!
What is this gamechanger you say? It's the new rtmpsrv program which comes with rtmpdump 2. It does the guessing game for us and gives a copy-pastable command. In short you have to position it between your browser and the internet and allow it to perform essentially a man-in-the-middle attack on the handshake of the proprietary protocol.
Compilation on GNU/Linux (for other system please read the README file):

sudo apt-get install build-essential git zlib1g-dev libssl-dev

git clone git://git.ffmpeg.org/rtmpdump
cd rtmpdump
make SHARED= SYS=posix

The program is now compiled for use on your system! Now the harder part - you need root access to configure the iptables firewall which should be a part of any modern Linux distro. This is the part which will reroute the RTMP communication of the flash plugin in your browser through a local port:

sudo iptables -t nat -A OUTPUT -p tcp --dport 1935 -j REDIRECT

Excellent, now start the program rtmpsrv which will listen on the local port:

./rtmpsrv

Browse to a media site and wait for rtmpsrv to write out a command in the console. If you see multiple chapters on the media site, try skipping to another one. I'll describe why in a minute.
OK! You should see at least one long string starting with rtmpdump. rtmpsrv did its job so close the browser tab and stop rtmpsrv with a Control+C signal. Ending rtmpsrv seems to be a problem. The cleaning up process seems to go nowhere at times but sending several Ctrl+C events usually produces a segfault. :/
Also delete the firewall redirect so that you can access the media server directly:

sudo iptables -t nat -D OUTPUT -p tcp --dport 1935 -j REDIRECT

If you're in luck taht is it and you only need to run the output of rtmpsrv, like this:

./rtmpdump -r "rtmpe://cp9950.edgefcs.net:1935/ondemand?ovpfv=1.1" -a "ondemand?ovpfv=1.1" -f "LNX 10,0,42,34" -W "http://media.mtvnservices.com/player/release/?v=4.2.2-8" -t "rtmpe://cp9950.edgefcs.net:1935/ondemand?ovpfv=1.1" -p "http://www.colbertnation.com/full-episodes/thu-january-14-2010-kathleen-sebelius" -C Z: -y "mp4:mtvnorigin/gsp.comedystor/com/colbert/season_06/episode_008/cr_06008_act1_768x432_1720.mp4" -o output.flv

The problem with Comedy Central shows are commercial breaks which cut the show into different separate chapters called acts (see the end of command above). This gives us a bit more work but it's so worth it avoiding the horrible full screen flash experience! Smooth full-screen playback has arrived, Randall! :)
This particular episode of The Colbert Report has 4 acts - all you need to do is change the actX in the command and the output filename must be different for each act too. Ending up with 4 differnet videos is kind of lame but manageable. If you know of a FLOSS stitching solution, please comment! (Avidemux doesn't wanna, PiTiVi wants to transcode, haven't tried LiVES yet).
SouthParStudios.com has a different gotcha - the first clip is the intro which is the same for all shows. Only the following acts are show specific. I managed to get a second command from rtmpsrv by seeking in the browser as mentioned above. Behold:

./rtmpsrv
RTMP Server v2.1c
(c) 2010 Andrej Stepanchuk, Howard Chu; license: GPL

Streaming on rtmp://0.0.0.0:1935
WARNING: Trying different position for client digest!


rtmpdump -r "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -a "ondemand?ovpfv=1.1" -f "LNX 10,0,42,34" -W "http://media.mtvnservices.com/player/release/?v=4.2.1" -t "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -p "http://www.southparkstudios.com/episodes/251890" -C Z: -y "mp4:mtvnorigin/gsp.comedystor/com/sp/extras/intros/PromoIntros/HDintroPromoXX_768x432_750.mp4" -o output.flv


rtmpdump -r "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -a "ondemand?ovpfv=1.1" -f "LNX 10,0,42,34" -W "http://media.mtvnservices.com/player/release/?v=4.2.1" -t "rtmpe://cp40493.edgefcs.net:1935/ondemand?ovpfv=1.1" -p "http://www.southparkstudios.com/episodes/251890" -y "mp4:mtvnorigin/gsp.comedystor/com/sp/season_13/1313/acts/sp_1313_act1_768x432_750.mp4" -o output.flv

^CCaught signal: 2, cleaning up, just a second...

And we have lift-off! See the act1 above? Incrementing this value will give you all the acts.
I am the first to admit this should be a total overkill for a good viewing experience but the current situation isn't really bearable. While fast modern multi-core desktop processors are capable of much smoother playback my aging Pentium M and a masses of netbooks struggle with this task. And of course (legally) downloading the shows and watching them offline is now possible! I think modern smartphones and media devices should have no problem with playback of these local files but as always YMMV.
Creators of rtmpdump, on behalf of the Linux masses suffering from stuttering full-screen playback with flash, I salute you!

Related articles by Zemanta

• Adobe acts against Flash video stream recorder (slumpedoverkeyboarddead.com)
• Linux iptables an Introduction (brighthub.com)

Downloading RTMP Flash media with rtmpdump

EDIT: These instructions are for verson 1.9 of rtmpdump! Newer versions make this process easier but also make some of this how-to obsolete. But I've modified the process below so that you can still use them for the legacy 1.9 version.

Today I attended a lecture/demo session on "hacking" various media sites like YouTube, Hulu, last.fm. Essentially it was a practical demonstration of the cat and mouse game hackers play with content pedlars on the internet. It was held at Cyberpipe, a nifty club in Ljubljana, Slovenia where I study.

One of the tools mentioned was rtmpdump which is apparently made by the same guys as the legendary mplayer, mencoder, and ffmpeg.

I decided to compile it myself from the most recent development version. If you're lazy and brave enough to trust compiled binaries and packages from third parties, go to LinuxCentre and install flvstreamer for your operating system. These programs were once one and so far they use the same commands. Read: they work the same.

These instrucitons work on Ubuntu 9.10 (karmic) but please be aware things change rapidly so the following might not work forever.

First of all install the tools needed for accessing the code and compilation.

sudo apt-get install build-essential gcc make subversion libssl0.9.8 libssl-dev libssl0.9.8

Now check out the latest code from the Subversion repository:

svn co svn://svn.mplayerhq.hu/rtmpdump/branches/1.x rtmpdump-1.9

If you prefer a stable version, run this instead:

wget http://rtmpdump.mplayerhq.hu/rtmpdump-1.9.tgz
tar xfz rtmpdump-1.9.tgz

I'm sure the impatient are looking forward to completing this ASAP so here's the final step in setting up rtmpdump on your computer:

cd rtmpdump-1.9
make linux

The program now lies in the same folder and you can only run it by telling your shell (like bash) exactly where it is. If you're in the same folder, running it is as simple as ./rtmpdump but ./ is only short for the current folder and it gets expanded to (in my case) /home/jasa/rtmpdump/rtmpdump.

Great! Now we need a media site which uses a Adobe's RTMP server. I think I won't get in trouble if I use VideoLectures.net as an example. :)

Lets say I'm interested in probability and statistics. And I want to watch these lectures offline, on a train or wherever.

Unfortunately this isn't a straightforward process and quite a bit of work is required. It is derived from a mailing list thread.

I need to see the source code of the web page above, not the rendered output of the web browser. In Firefox press Ctrl+U. Now I need some hard data to pass to rtmpdump. Search for a javascript section which launches the flash video player. For the statistics video this is the interesting section:

var flashvars = {
    
      streamer: "rtmp://oxy.videolectures.net/video",
      file: "2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01.flv",
    
    height: '288',
    autostart: "true",
    bufferlength: '5',
  
  
    image: "http://media.videolectures.net/play.png",
  
    id: "FlvPlayer" // last line, no colon ',' !
  };
  swfobject.embedSWF("http://media.videolectures.net/jw-player/player.swf", "video_embed", "384", "307", "9.0.0", "http://media.videolectures.net/swfobject/expressInstall.swf", flashvars, params, attributes);

Generally searching (Ctrl+F) for flashvars should get you near the required data. The line swfobject.embedSWF(...) causes the video player to load inside your browser and display the correct media. This media rests on a different server than the webpage and I will use rtmpdump to connect to that server instead of the flash player. Now Adobe has put some roadblocks into the process but lucky for me I have a flying car. :)

I'll just give you the final command now and explain later!

./rtmpdump -r rtmp://oxy.videolectures.net/video/ -y 2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01 -a video -s http://media.videolectures.net/jw-player/player.swf -w ffa4f0c469cfbe1f449ec42462e8c3ba16600f5a4b311980bb626893ca81f388 -x 53910 -o test.flv

The -r switch requires an argument which is the URL of the media server and is found in the variable streamer in the JavaScript source code above.
The -y switch needs the playpath and that is found in the JavaScript variable file, (minus the extension .flv or .mp4).
The -a switch is the name of the used player and usually automatically inferred from the URL. Defining it manually works by copying the part after the server name in the streamer.
The -s switch defines the flash video player which normally connects to the media server. In the example it is the first argument of the function embedSWF.

Here's where things get even more complicated. The media server wants some extra data about this player, specifically its sha256 hash-sum and size in bytes. So lets get them:

wget http://media.videolectures.net/jw-player/player.swf
sha256sum player.swf
ls -l player.swf

Supply the sha256sum to switch -w and the file size to the -x switch.

Anything else? Yeah, I need to specify where to save the video with -o.

Run the longest command ever and get yourself a beer (you've earned it)! :)

I admit that doing this for every video gets time consuming but unfortunately it is a procedure specific to every site (look at the mailing list thread link again to see different javascript). But for the same site there's very little extra work.

Hope you lasted this long, be seeing ya!