Encoding DVD media to AVI for portable media players, under Linux

Introduction

Before we get started let me first tell you what I am running on my little Linux server that is sitting on a shelf downstairs in the basement of my house:

  • OpenSUSE 10.2, kernel 2.6.18.8
  • Mencoder build dev-SVN-r27637-4.1.2-openSUSE

The process and script described here should work on other systems, but if it doesn't I wanted to make sure you understood what I am using.

For my portable media player (the Creative Zen), I decided I wanted to create AVI files, which are based on MPEG-4 encoding. I'm using the xvid video codec and mp3 (lame) audio at a fixed bit rate. If you have a Zen you can do the same. If you have a different media player, it will likely work if your player can handle AVI files -- but you may need to tweak some of the settings for conversion. You can read more about this, below. Even if you want something other than AVI and/or xvid you can dig deeper into the MPlayer/Mencoder documentation to modify the calls to Mencoder accordingly, to suit your needs.


Problem:  How to get my DVD movies onto my media player

For my birthday my wife bought me one of these (ok, I ordered it and she gave it to me):

Photo of a Zen player

Right after I received it I loaded a bunch of music onto it, and then started playing around with the video capabilites it has. I was a bit disappointed to find that I had to rely on a utility provided by Creative that allowed my video files to be converted to a format that the Zen would be able to play. Sure, it was nice enough for Creative to supply the utility but I really wanted to do my own conversions. So I studied the Zen documentation and started thinking about my little Linux server I have running in my basement. Wouldn't it be really neat to rip DVDs to the Linux box, and have it do the conversion to a format that the Zen could play?

Problem was, I had no idea where to start. So I started Googling for "Linux video conversion" and found a few packages that would run on my OpenSUSE 10.2 system. I installed one called Avidemux-gtk, which was a really excellent program -- except I could never solve a nasty A/V sync problem. My ripped DVDs converted to AVI beautifully -- but with a progressively worsening audio mis-match with video over the course of the movie. I must have spent 20 hours trying to get it to work, even posting questions to the developer forums. No luck.

I did find a utility that worked. It was called QVideoConverter. No A/V sync problems! But, it just didn't have enough dials and knobs for my liking. And I figured that since it was just a GUI front-end utility, I should be able to use Mencoder directly. Mencoder is part of the awesome MPlayer package.

Solution:  Mencoder

I studied the Mencoder documentation and looked up every example I could find on the internet showing how to use it. My first (many) attempts gave me the same A/V sync problem. I also had problems even getting my resultant AVIs to play on the Zen, until I figured out that the Zen requires the AVI container to be type-2. I finally found that there is a way to call Mencoder so it creates an OpenDML AVI container, which meets the type-2 spec. Here is an excerpt from the Creative Zen guide to encoding, which shows the target format needed for the Zen:
Zen encoding spec

After many experiments I found the winning combination. The calls to Mencoder boil down to three general steps:

  1. Determine the cropping required for the original content.
  2. Perform a pass-1 conversion, which builds a frame transition table.
  3. Perform the second and final pass, writing out the final AVI file.

My Linux server is headless. It is on my network, primarily performing file serving duties to the XP machines in the rest of the house. If I need to log into the Linux machine I do so via a VNC x-window. So my workflow for converting a DVD to an AVI file that will play on my Zen is this:

  1. Rip the DVD on my XP machine to a folder that is served by the Linux machine. I use a free program called DVD Decrypter, or alternatively DVDFab 5. The latter seems to handle the newer copy protection schemes a bit better, but I don't like the inflexibility it gives on naming the output directory for the VIDEO_TS folder that it creates.

    Note that if you want to stick with a PURE Linux solution for this workflow, instead of using a dreaded Windows application to rip your DVD to .VOB files, you can certainly put your DVD into the drive on your Linux machine and use either vobcopy or dvd::rip, natively under Linux -- or whichever other of your favorite ripping tools you'd like to use. The focus of this guide is not the "ripping" part of this exercise, it is really the "transcoding" part.
  2. Once I have a VIDEO_TS folder that contains all my .VOB files from the ripped DVD, I then log on to the Linux machine, bring up a console window and execute the encode4me script. I usually just drop into SU mode in the console so I don't have to worry about CHMOD'ing the folder or files where my ripped DVD is stored.

    A call to the script may look like this:

    ./encode4me /home/chuck/movies/minority_report/VIDEO_TS

    This tells the script "Go to the folder where the .VOB files are stored and convert all of them into a single contiguous AVI file".
  3. After about 45 minutes I have a nice little AVI file that I can now copy to my Zen. No further conversion is needed.

As an example, the movie Minority Report converts to a 672 meg file using a video bitrate of 512K and a fixed audio bitrate of 128K. You could probably get away with an even lower audio bitrate, but I like to retain as much of the "movie sound" quality as possible. I did experiment with lower video bitrates, but found that the artifacting (especially with dark scenes) was a bit too much. Of course, you can play around with the settings in this script all you want, or make calls to it with parameters on the command line to set any of the conversion settings you wish.

Widescreen?  4:3?  Cropping?

One other issue that I struggled with was how to fit a widescreen movie into the Zen's 4:3 screen. Should I just crop it to 4:3? Should I retain the original aspect ratio? I tried both ways and wasn't happy with either. I found that just cropping to 4:3 caused the movie to be almost unwatchable. Felt like I was viewing it thorugh a door keyhole. Actors that were having a dialog might be completely "off screen" if they were on the far left or right of the scene! Keeping the original widescreen aspect ratio also wasn't too good. The physical size of the Zen made it simply too small.

So, I came up with the idea of including a variable zoom feature in this script. I experimented with various crop settings and found that by telling it to retain at a minimum 85% of the width of the original movie, it was a good compromise between a straight 4:3 crop and no crop at all. If you look at the script, you will see where I calculate a target crop factor by considering the "zlimit" variable that is defaulted to 15% (i.e. retaining 85% of the original width) -- but can be user specified via command line parameter. If you set zlimit to zero it would retain the original widescreen aspect of the movie. You can set it as high as 50%, via the --zoom command line parameter. Any source media that is already in 4:3 format will of course be unaffected by this command.

zoom = 0, original widescreen zoom = 15, default zoom factor of 15% zoom = 50, 50% maximum zoom (ends up as 4:3)
zoom 0% (original widescreen, no zoom) zoom 15% (default) zoom 50% (maximum zoom, forces 4:3 aspect)

The Script:  encode4me

Ok, before anyone pwns me about the coding style in this script, please keep in mind it was my very first attempt (other than a couple of "hello world" tests) to write a BASH script. If you see anything seriously wrong with it I would appreciate feedback. If you want to use it as a larger program (or even front it with a nice GUI), that would be fine with me.

Here is a list of the key variables. Any of them can be changed to a new default value by editing the script -- or you can pass them in via command line parameter. Here is a list of what you can tweak:

  • -o[--output]  <output path/filename>       
  • -s[--screen]  <screen width:screen height> 
  • -v[--vbit]    <video bitrate>              
  • -a[--abit]    <audio bitrate>              
  • -f[--fps]     <frames per second>          
  • -c[--chapter] <chapter range e.g. 1-5>     
  • -t[--title]   <DVD title number>          
  • -z[--zoom]    <zoom limiter, 0 thru 50>   
  • -L[--Loud]    <message level>             

The defaults for all of these can be found by reading the script itself which is shown at the bottom of this page. One note about the framerate: With my Zen, if I used anything but 30fps I ended up with A/V sync issues. YMMV, of course.

If you have a different media player (say for example, an COWON player with a screen size of 480 by 272), you could still use this utility and set the target --screen to 480:272. Here is what a call to the script would look like in this case:

./encode4me /home/chuck/movies/minority_report/VIDEO_TS -s 480:272

As long as the unit can play an AVI file (with video encoded as MPEG-4), you should be fine. You'd also have to make sure you don't exceed the unit's maximum video or audio bitrate -- but all those parameters are changable in the script or via the call on the command line.

Another neat thing you can do is write ANOTHER small script that calls the encode4me script multiple times. You may want to convert a single movie to multiple formats, or you may have a whole bunch of VIDEO_TS folders that you want to convert to AVIs.

Anyway, here's the script. I tried to put a lot of comments in so you can see how it works. If you want to save it to use on your own Linux system you can simply copy and paste it into your favorite editor. If you'd like to send me feedback, my userid is furtherside and I am at yahoo.com


#!/bin/sh
#  Written by:  CLC
#  Date:  11/04/2008
#
#  This script will drive a two-pass encode of a source DVD (or directory of .VOB files) into an AVI
#  file suitable for playing on portable media players.  I have a Creative Zen, which drove me to try
#  to come up with this script after getting tired of running the Zen software on my Windows machine.
#  I wanted a solution for Linux, and the ones out there seemed to have problems with A/V sync on large
#  AVI files.  I finally found the right combination of parameters for calling mencoder which seem to
#  avoid the A/V sync problem, and give a good quality and small size file.  You can experiment with
#  the parameters here, but I found the defaults to be suitable for almost all source material I wished
#  to transfer to my Zen.  I always first start by ripping my DVD contents from my Windows machine into
#  a directory shared with my Linux machine, using a utility like DVDdecrypter, or something similar.
#  Since my Linux machine doesn't have a DVD drive, I have not tried going straight from DVD to output
#  AVI file, although that should work fine (just specify //DVD:1, or whatever your drive device handle
#  is, as the input file.
#
#  Although I put this together for my Zen, it should work (with some parameter changes, like the 
#  screen dimensions for example) for many other portable media players.  Just tweak the parameters
#  according to your device's specs.
#
#  The resulting AVI is stored in an OpenDML spec container (otherwise known as a type-2 AVI).
#
#  Overall steps:
#     1. call mencoder with the 'cropdetect' option, to find the actual video location on the screen,
#        and to avoid having black bars above and below the picture when changing the aspect ratio to
#        match the portable media player.
#     2. call mencoder for a first-pass encode.
#     3. call mencoder for the second-pass encode, and create the output AVI
#
#  One odd thing I ran into was the call with 'cropdetect'.  I had to play around with the threshold
#  value to get it to accurately locate the "real" video portion on the screen.  However, I could not
#  trust the x/y values it returned for the center of the image.  For widescreen movies it seemed to be
#  shifting the crop box to the right -- so the resulting AVI seemed quite a bit off-center.
#  So I keep the size values (width and height) returned from cropdetect, but I'm forcing it to use the
#  center of the screen instead of the new x/y center values it returned.


echo
scale=320:240                                           # default screen dimensions (for Zen)
vbitrate=512                                            # default video bit rate
abitrate=128                                            # default audio bit rate
fps=30                                                  # default frames per second
chapterspec=""                                          # default chapter range
title=1                                                 # default title
msglvl=1                                                # mencoder messages level (quiet)
zlimit=15                                               # default percentage width loss when cropping
path=" "                                                # input path (directory or device)
output=" "                                              # output path (filename)
default_out=/encode4me.avi                              # default output file if none specified
keymult=7                                               # key frame interval multiplier (by FPS)
language=en                                             # default DVD langauge selection

debug=0                                                 # debug flag

# Process the command line parameters.

if [ $# = 0 ]; then
   echo "No arguments were provided.  encode4me -h to see options."
   echo
   exit 0
fi

while [ $# -gt 0 ]; do    # Until we run out of parameters
  case "$1" in
    -h|--help)
      echo "usage: encode4me -i[--input]   <input directory or device>  (no default, and -i is optional)"
      echo "                 -o[--output]  <output path/filename>       def: {input_path}${default_out}"
      echo "                 -s[--screen]  <screen width:screen height> def: ${scale}"
      echo "                 -v[--vbit]    <video bitrate>              def: ${vbitrate}"
      echo "                 -a[--abit]    <audio bitrate>              def: ${abitrate}"
      echo "                 -f[--fps]     <frames per second>          def: ${fps}"
      echo "                 -c[--chapter] <chapter range e.g. 1-5>     def: all chapters"
      echo "                 -t[--title]   <DVD title number>           def: title ${title}"
      echo "                 -z[--zoom]    <zoom limiter, 0 thru 50>    def: ${zlimit}"
      echo "                 -L[--Loud]    <message level>              def: off (quiet)"
      echo
      echo "example: encode4me /home/chuck/videos_NTSC/godfather/VIDEO_TS"
      echo "example: encode4me /home/chuck/videos_PAL/godfather/VIDEO_TS -v 800 -a 96 -f 25"
      echo "example: encode4me /dev/cddrom/ -t 2 -o /home/mary/videos/irobot.avi"
      echo "example: encode4me -c 1-2 -v 700 -a 160 /home/chuck/videos/rush/VIDEO_TS -o /home/mary/videos/rush.avi"
      echo
      exit 0
      ;;
    -i|--input)
      path="$2"                                         # input path (directory) or device
      shift
      ;;
    -o|--output)
      output="$2"                                       # output path (filename)
      shift
      ;;
    -s|--screen)
      scale="$2"                                        # get desired screen dimensions
      shift
      ;;
    -v|--vbit)
      vbitrate="$2"                                     # get desired video bitrate
      shift
      ;;
    -a|--abit)
      abitrate="$2"                                     # get desired audio bitrate
      shift
      ;;
    -f|--fps)
      fps="$2"                                          # get desired frames per second
      shift
      ;;
     -t|--title)
       title="$2"                                       # get desired title number (or range)
       shift
       ;;
    -c|--chapter)
      chapterspec="-chapter $2"                         # a range of chapters to process
      shift                                             # default is to process entire volume
      ;;
    -z|--zoom)
      zlimit="$2"                                       # zoom loss (percentage) limit
      shift
      ;;
    -L|--Loud)
      msglvl=5                                          # increase the processing messages
      ;;
    * )                                                 # default case, no option selected
      path="$1"                                         # so must just be input file/device
      ;;
  esac
  shift                                                 # Check next set of parameters.
done

if [ "$path" = " " ]; then                              # no input path or device was given
  echo "No input directory or device was specified.  Use the -i command to specify one."
  echo
  exit 0
fi

if ! [[ -d $path || -b $path || -e $path ]] ; then      # check for device or directory
  echo "Input device is not a directory, a DVD drive or a file."
  echo
  exit 0
fi

if [ "$output" = " " ]; then                            # no output path was given
  output=${path%/*}                                     # so use input path as basis
  output=${output}${default_out}                        # and use default filename
fi

echo "test" >> $output                                  # attempt to write to file
if ! [ -w $output ] ; then                              # check to see if file is writeable
  echo "Output file invalid or lacking write permission to directory."
  echo
  rm -f $output                                         # just in case, erase output file
  exit 0
fi

sdim=(`echo $scale | tr ':' ' '`)                       # split screen dimensions into parts
screen_width=${sdim[0]}                                 # get the width component
screen_height=${sdim[1]}                                # get the height component
aspect=$(echo "scale=6; $screen_width/$screen_height" | bc)  # calculate the aspect ratio

maxkey=$(echo "scale=1; $fps*$keymult" | bc)            # calculate maximum key frame interval
maxkey=(`echo $maxkey | tr '.' ' '`)                    # parse results into integer & decimal
maxkey=${maxkey[0]}                                     # strip off non-integer portion

croptemp=${output%/*}/"cropdetect.out"                  # create temp file for crop detection

rm -f divx2pass.log $output $croptemp                   # erase temp and output files


if [ $debug -ne 0 ] ; then
  echo "Path      :" $path
  echo "Chapter   :" $chapterspec
  echo "Output    :" $output
  echo "Abitrate  :" $abitrate
  echo "Vbitrate  :" $vbitrate
  echo "Scale     :" $scale
  echo "FPS       :" $fps
  echo "Aspect    :" $aspect
  echo "Max KeyInt:" $maxkey
fi

echo
echo "Analyzing media, to determine bounding box for cropping..."
mencoder -alang $language -dvd-device $path "dvd://${title}" -chapter 2-2 \
  -o $output -oac  mp3lame -lameopts cbr:"br=${abitrate}" -ovc xvid \
  -xvidencopts "bitrate=${vbitrate}":"max_key_interval=${maxkey}" -vc ffmpeg12 \
  -vf cropdetect=16:2 2>&1 tr '\015' '\012' | grep 'vf crop' | head -1500 > ${croptemp}

crop=`tail -1 ${croptemp} | awk -F\( '{print $3}' | awk -F\) '{print $1}' | awk -F= '{print $2}'`

cropcomps=(`echo $crop | tr ':' ' '`)                  # parse the crop detection results
width=${cropcomps[0]}                                  # suggested video width
height=${cropcomps[1]}                                 # suggested video height
xpos=${cropcomps[2]}                                   # suggested center x-position
ypos=${cropcomps[3]}                                   # suggested center y-position

newwidth=$(echo "scale=1; $height*$aspect" | bc)       # calculate new width based on aspect
newwidth=(`echo $newwidth | tr '.' ' '`)               # parse results into integer & decimal
newwidth=${newwidth[0]}                                # strip off non-integer portion
#crop=${newwidth}:${height}:${xpos}:${ypos}            # use the center suggestion
crop=${newwidth}:${height}:-1:-1                       # ignore center suggestion

if [ $debug -ne 0 ] ; then
  echo "First crop:" $crop
fi


# I found that some widescreen movies got a pretty nasty crop (width) when getting to the Zen's target
# aspect ratio of 1.33.  It just looked like too much information was being lost from the left and right
# sides of the screen.  People being spoken to now "off screen", in some cases.  So I figured that in
# these cases it would be better to retain more of the width of the movie and insert black bars above
# and below the movie on the Zen screen -- rather than try to force it to a full zoom at the Zen's
# aspect ratio.  This is what the "-z" switch is used for -- it's a zoom limiter.
#
# So, say for example we started with a movie that was 720x480 -- but it was a widescreen movie and
# it's video height was really 356 pixels.  Normally, we would use the 356 height and crop the width
# to 474 pixels in order to get to our target 1.33 aspect ratio.  If we did this, however, we'd be
# losing about 34% of the video information (720-474)/720 = 34%
#
# Instead, if we let the -z switch default to 15 (meaning "cap the video lost in the width of the
# movie to 15%") then we would recalculate the width to be 720*0.85 = 612.  Then we would derive the
# height based on the target aspect ratio (in the Zen's case, 1.33):  612/1.33333 = 459
#
# So, our new crop plan would be 612 by 459, thus retaining 85% of the original video information.
# You can play with values using the -z switch when calling this script, but the default limit of 15%
# seems to give good results.  Note that 4:3 media will be unaffected by this, since it already matches
# the Zen's target aspect ratio.
#
# If you call the script with -z 0 that will mean to not allow any loss of video, and you'll see the
# widescreen movie unaltered (albeit with pretty large black bars above and below) on the Zen's 4:3
# LCD screen.
#
# Note that although this talks about the Zen media player, this code will work for any device as long
# as you tell the script what your device's screen dimensions are (using the -s switch).  None of this
# was written to be "hard coded" for the Zen player.



zloss=$(echo "scale=2; 100*(($width-$newwidth)/$width)" | bc)
zloss=(`echo $zloss | tr '.' ' '`)
if [ $zloss -gt $zlimit ] ; then                       # see if we are discarding too much width
  newwidth=$(echo "scale=4; $width*(1-($zlimit/100))" | bc)
  newheight=$(echo "scale=4; $newwidth/$aspect" | bc)  # calc width limited-loss, and new height
  newwidth=(`echo $newwidth | tr '.' ' '`)
  newheight=(`echo $newheight | tr '.' ' '`)
  crop=${newwidth}:${newheight}:-1:-1                  # build new crop (will have black bands)
  if [ $debug -ne 0 ] ; then
    echo "zloss     :" $zloss
    echo "zlimit    :" $zlimit
    echo "Old width :" $width
    echo "Old height:" $height
    echo "Final crop:" $crop
  fi
fi


# Perform the first pass of encoding.  The video bitrate here is actually ignored, and 
# mencoder will encode at the material's maximum bitrate.  This is so it can build a file
# of frame transition data, which tells the second pass when to expect motion within the
# frame transition, thus giving it a better "hint" at encoding (for enhanced quality)

# I found the following to be important, to avoid A/V desync issues:
#   - keep the parameters to mencoder the same for both pass 1 and pass 2
#   - force the frame rate to an integer value (e.g. 30 FPS for NTSC...not 29.7 !)
#   - crop and scale the video at the same time audio is being encoded, in other words
#       don't demux the audio first then try to lace it back into the A/V stream later
#   - tell mencoder to not drop duplicate frames (via the 'harddup' flag)
#   - tighten up the key frame interval a bit more than the default (7*fps instead of 10*fps)
#   - stick with constant bit rate audio encoding.

echo
echo "Starting first pass..."
mencoder -msglevel "all=${msglvl}" -alang $language -dvd-device $path "dvd://${title}" \
  $chapterspec -o $output -sws 2 -oac mp3lame -lameopts cbr:"br=${abitrate}" \
  -ovc xvid -xvidencopts "bitrate=${vbitrate}":"max_key_interval=${maxkey}":pass=1 -vc ffmpeg12 \
  -vf "crop=${crop}","scale=${scale}",harddup -ofps $fps -force-avi-aspect $aspect

# Now do the second/final pass 

echo
echo "Starting second pass..."
mencoder -msglevel "all=${msglvl}" -alang $language -dvd-device $path "dvd://${title}" \
  $chapterspec -o $output -sws 2 -oac mp3lame -lameopts cbr:"br=${abitrate}" \
  -ovc xvid -xvidencopts "bitrate=${vbitrate}":"max_key_interval=${maxkey}":pass=2 -vc ffmpeg12 \
  -vf "crop=${crop}","scale=${scale}",harddup -ofps $fps -force-avi-aspect $aspect

rm -f divx2pass.log $croptemp

echo
exit 0