24 November 2009

Downloading RTMP Flash media with rtmpdump

EDIT: These instructions are for verson 1.9 of rtmpdump! Newer versions make this process easier but also make some of this how-to obsolete. But I've modified the process below so that you can still use them for the legacy 1.9 version.

Today I attended a lecture/demo session on "hacking" various media sites like YouTube, Hulu, last.fm. Essentially it was a practical demonstration of the cat and mouse game hackers play with content pedlars on the internet. It was held at Cyberpipe, a nifty club in Ljubljana, Slovenia where I study.

One of the tools mentioned was rtmpdump which is apparently made by the same guys as the legendary mplayer, mencoder, and ffmpeg.

I decided to compile it myself from the most recent development version. If you're lazy and brave enough to trust compiled binaries and packages from third parties, go to LinuxCentre and install flvstreamer for your operating system. These programs were once one and so far they use the same commands. Read: they work the same.

These instrucitons work on Ubuntu 9.10 (karmic) but please be aware things change rapidly so the following might not work forever.

First of all install the tools needed for accessing the code and compilation.
sudo apt-get install build-essential gcc make subversion libssl0.9.8 libssl-dev libssl0.9.8
Now check out the latest code from the Subversion repository:
svn co svn://svn.mplayerhq.hu/rtmpdump/branches/1.x rtmpdump-1.9
If you prefer a stable version, run this instead:
wget http://rtmpdump.mplayerhq.hu/rtmpdump-1.9.tgz
tar xfz rtmpdump-1.9.tgz
I'm sure the impatient are looking forward to completing this ASAP so here's the final step in setting up rtmpdump on your computer:
cd rtmpdump-1.9
make linux
The program now lies in the same folder and you can only run it by telling your shell (like bash) exactly where it is. If you're in the same folder, running it is as simple as ./rtmpdump but ./ is only short for the current folder and it gets expanded to (in my case) /home/jasa/rtmpdump/rtmpdump.

Great! Now we need a media site which uses a Adobe's RTMP server. I think I won't get in trouble if I use VideoLectures.net as an example. :)

Lets say I'm interested in probability and statistics. And I want to watch these lectures offline, on a train or wherever.

Unfortunately this isn't a straightforward process and quite a bit of work is required. It is derived from a mailing list thread.

I need to see the source code of the web page above, not the rendered output of the web browser. In Firefox press Ctrl+U. Now I need some hard data to pass to rtmpdump. Search for a javascript section which launches the flash video player. For the statistics video this is the interesting section:
var flashvars = {
    
      streamer: "rtmp://oxy.videolectures.net/video",
      file: "2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01.flv",
    
    height: '288',
    autostart: "true",
    bufferlength: '5',
  
  
    image: "http://media.videolectures.net/play.png",
  
    id: "FlvPlayer" // last line, no colon ',' !
  };
  swfobject.embedSWF("http://media.videolectures.net/jw-player/player.swf", "video_embed", "384", "307", "9.0.0", "http://media.videolectures.net/swfobject/expressInstall.swf", flashvars, params, attributes);
Generally searching (Ctrl+F) for flashvars should get you near the required data. The line swfobject.embedSWF(...) causes the video player to load inside your browser and display the correct media. This media rests on a different server than the webpage and I will use rtmpdump to connect to that server instead of the flash player. Now Adobe has put some roadblocks into the process but lucky for me I have a flying car. :)

I'll just give you the final command now and explain later!
./rtmpdump -r rtmp://oxy.videolectures.net/video/ -y 2007/pascal/bootcamp07_vilanova/keller_mikaela/bootcamp07_keller_bss_01 -a video -s http://media.videolectures.net/jw-player/player.swf -w ffa4f0c469cfbe1f449ec42462e8c3ba16600f5a4b311980bb626893ca81f388 -x 53910 -o test.flv

The -r switch requires an argument which is the URL of the media server and is found in the variable streamer in the JavaScript source code above.
The -y switch needs the playpath and that is found in the JavaScript variable file, (minus the extension .flv or .mp4).
The -a switch is the name of the used player and usually automatically inferred from the URL. Defining it manually works by copying the part after the server name in the streamer.
The -s switch defines the flash video player which normally connects to the media server. In the example it is the first argument of the function embedSWF.

Here's where things get even more complicated. The media server wants some extra data about this player, specifically its sha256 hash-sum and size in bytes. So lets get them:
wget http://media.videolectures.net/jw-player/player.swf
sha256sum player.swf
ls -l player.swf
Supply the sha256sum to switch -w and the file size to the -x switch.

Anything else? Yeah, I need to specify where to save the video with -o.

Run the longest command ever and get yourself a beer (you've earned it)! :)

I admit that doing this for every video gets time consuming but unfortunately it is a procedure specific to every site (look at the mailing list thread link again to see different javascript). But for the same site there's very little extra work.

Hope you lasted this long, be seeing ya!


Reblog this post [with Zemanta]

11 comments:

  1. i like it, pretty demonstration of rtmpdump...when ill start to use ubuntu, ill surely use it ;) hehehe

    ReplyDelete
  2. Thanks! How about a challenge - find the rtmp for hulu.

    ReplyDelete
  3. i am trying to develop add-in for xbmc to play the videolectures.net and that is how i found your blog. do you have any recommendation on how to form direct link to this flv thing?

    ReplyDelete
  4. @helpdeskdan Sorry, hulu doesn't like users outside the USA. As far as I know I could fool hulu by using an american proxy but at the moment i don't have access to any US machine nor do I have the money to hire a virtual server there.

    @anonymous Looking at many videos I found that all of the ones I looked at have a direct .wmv link (Usually called "Launch in a standalone WM Player"). Looking for that link is the simplest solution as it is served over the good old http and not the proprietary rtmp. Of course I don't know what you prefer - flv or wmv. Both are proprietary but the wmv seems to be much more easily available.

    This also puts into question why to go to all the trouble with rtmpdump if i could have downloaded the video easily with a link. It was something new to me! (also see name of blog) :)

    ReplyDelete
  5. So this is just for Linuxes? Do you know if it works the same for Windows? And also, is it capable of picking up rtmpe streams? Thanks in advance.

    ReplyDelete
  6. @anonymous This is the first link I found which uses free software: http://forum.videohelp.com/topic374948.html
    It's essentially rtmpdump for Windows but it uses network traffic eavesdropping to find out the needed parameters. This might be a good opportunity to try Linux out! You can do that with virtualisation, with software like VirtualBox.

    Of course you could use other, more streamlined solutions like Orbit Downloader but I think there are security risks which make that undesirable.

    ReplyDelete
  7. Thanks! The link really helps. And I'll give Linux a go sometime.

    ReplyDelete
  8. With rtmpdump 2.1 most of this is a lot easier...

    ReplyDelete
  9. @hyc Yep, the new major version makes the guessing game obsolete! I'll soon post an updated guide for some more interesting sites! >:D

    ReplyDelete
  10. Legend. It took me ages to figure this out. Your method is perfect.

    ReplyDelete
  11. thaaaaaaaaaaaaaaaaaaaaanks

    ReplyDelete

Yin & Yang!