UPDATES   09 Jul 2001

Httpd.tcl now sets priority before launching ExtractStream, to avoid stuttering.

ExtractStream has improved audio sync, and is now compatible with the TiVoMADD utility.

Httpd.tcl's "NowShowing" list has been rewritten for better compatibility with more machines, and links now provide the functionality to extract video via simple browser selection.  A glide/gtk GUI application, esdirect, has been added to aid playback/storage/transcoding.

Mplex (the multiplexer on the TiVo) has been replaced by netmplex, running on the PC.

The distribution of executables has been split into parts consistent with the platform where the executables are used:  es-*-Pc contains executables for Linux,  es-*-TiVo contains executables for TiVo.

CVS is now used to track modifications.  To get between-release source code versions:

pc#  cd [the directory where I want to keep this stuff]
pc#  export CVSROOT=:pserver:anonymous@stampede.org:/home/cvs
pc#  export CVS_RSH=ssh
pc#  cvs login

pc#  cvs co tivo

• doc  -- Contains this file.
• extractstream  -- Contains the ExtractStream.c source code.
• hack/bin  -- Contains the scripts and latest binaries for the TiVo side.
• mplex  -- Contains source for the TiVo side "mplex".
• netmplex -- Contains source for the PC side "netmplex".
• usr/bin  -- Contains scripts and latest binaries for the Linux PC side.
• windows.-- Contains files for Windows users.

FORWARD

Understand that this code is in a very raw form (neither easy to use or understand).  It is released in this early stage in the spirit of Open Source, hoping that others might use and modify the code, improve it, and share their changes.

We urge users to understand the difference between "fair use" and copyright violation when using copyrighted video.  The persons writing and distributing this code feel strongly that the code be used only in a legal manner.  It is legal for you to view the video captured on your TiVo (or any video capture device) at any time and using any playback device, BUT, it is not legal to distribute or publicly playback copyrighted material.  If you are unsure of the legality of your intended use, consult a real lawyer; this is just our interpretation.

HISTORY

httpd.tcl is a slight modification of sfr's and TiVoTechie's httpd.tcl.  The only change is to allow direct display of the relevant "NowShowing" information.  TiVoTechie would like it known that his changes only concern the virtual remote capabilities that have no interaction with the MFS database.

INSTALLATION

On the TiVo

If you've installed the tkp_tools, then there's a name conflict: "nc" is used as a variant of "mc", as defined in "/var/hack/root/.bashrc" (sometimes "/hack/root/.bashrc").  You can either rename the definition of "nc" in your ".bashrc", or you can rename "nc" on your system, or: always refer to NetCat by path (i.e. /hack/bin/nc).

If you've already installed "httpd.tcl", you might want to back it up, since the archive contains a new version that may overwrite your version.

These instructions assume a network aware TiVo.  You're going to need to transfer files totheTiVo. NFS, SAMBA ( smbfs.oand smbmount), rsync or an ftp daemon   are your best options.

Download the "es-*-TiVo.tgz" archive here.

Transfer the archive to the TiVo (or, in the case of SAMBA or NFS, make sure the file can be accessed from the TiVo).

The archive is relative "hack/bin"  If your "hack/bin" is on "/var", then "cd /var".  If your "hack/bin" is on "/", then "cd /" (and make sure the root is mounted read/write: "mount -o remount,rw /").  The "doehtml" script, in order to avoid name conflict with the "nc" program (as detailed above), calls "nc" using a full path name; if you're not using "/hack/bin", then edit the doehtml script to properly point to the "nc" program.  Finally, unarchive:

bash-2.02#  cat /var/tmp/mnt/es-0.3-TiVo.tgz | gzip -dc | cpio -i -H tar

bash-2.02#  ExtractStream

On the Linux PC:

The scripts assume  "vlc" from Videolan for mpeg2 playback.

Some of the capabilities described in this document require NetCat:.NetCat: may need to run on both the TiVo and your client PC.  The archive contains "nc" for the TiVo.  You may want to get NetCat for your client OS here.  Using "netmplex", NetCat is not necessary on the PC.

Download the "es-*-Pc.tgz" archive here.

As root user in the "/" directory, unarchive

pc#  tar -xzvf es-0.3-TiVo.tgz

In your Netscape 4.7x browser, define (or modify) the following Mime type ("Edit"->"Preferences..."->"Navigator"->"Applications"):
 

Description: MPEG2 Video
Mime Type:   video/x-mpeg2
Suffixes:    mpv2,mp2v
Application: /usr/bin/esdirect %s

On the Windows PC

"TiVo-VCD.mcf" is used to guide TMPGEnc multiplexing of TiVo audio/video in Windows.   There are many discussion threads that show the correct codecs to use.

Source code

Extract these with:

pc#  tar -xzvf es-0.3-src.tgz

USAGE

httpd.tcl

bash-2.02#  httpd.tcl >/dev/null 2>/dev/null &

I'm not about to explain all the functionality of this file... only what's been added for mpeg extraction.

In the "location" or "URL" entry field of your favorite browser (except:"opera" doesn't like this server), type your TiVo's IP address or machine name.  It should provide you a list of options, including "NowShowing". Select thisoption.

The resulting page should list the programs available on the TiVo, their descriptions, and, most importantly, the tyStream FSID's needed by ExtractStream.  Note that these are hyperlinks... Click on them and vlc will start playing the selected video.

The way this works: Your browser requests the tyStream FSID's from /hack/bin/httpd.tcl, which passes these values to /hack/bin/doehtml, which activates NetCat in "listen" mode for the audio and video streams along with ExtractStream.  Httpd.tcl returns the IP address of the TiVo, and an indication of "Best" or "Basic" quality.  The Mime type is used by the browser to activate "esdirect", giving it the IP address and quality indicator.  "esdirect" activates the necessary applications to process the streams incoming from the TiVo.
 

esdirect

Setup a Netscape 4.76 "mime" type, as directed above.

Point your browser at your TiVo's name or IP address, where httpd.tcl is running, and select the NowShowing option.  The following window appears:

The "Watch" tab is initially selected.  The "Netmplex command line" is setup for a standard system.  The viwer command defaults to "vlc" using SDL in a 300x300 display; this is tailored to a 200-300Mhz CPU running XFree86 3.x. Faster CPU's can use a larger window, XFree86 4.x users might want to use the "xv" plugin, rather than "sdl".  The "while" loop is used to keep the stream going if "vlc" crashes.  The environmental variables "TIVOPIP" and  "BEST" are imported from the TiVo, they don't need to be set in your environment.

The "esdirect" application will read from the pipe coming out of "netmplex", and write to the pipe running the output task.  The progress bars at the bottom will indicate how much data has been read/written.  The extent of this bar is always the next multiple of 350MB... so don't expect it to every reach the end.
 

Select the "Write" tab to write the extracted mpeg2 to a file.  In this case, the "OUTFILE" environmental variable is provided, given that you modify the "Output filename" entry field.  Note that "tee" really writes the file, "esdirect" is just given a copy of the written data, which is only used to update the status bar.
 
 

Select the "Transcode" tab to initiate MPEG2 to MPEG1 transcoding.  Given the nature of transcoding utilities, this is done in multiple passes.  In the first pass,  the data is just written to disk.  Since the TiVo can download a stream much faster than it can be transcoded, I thought it would be best to just write the file, allowing other things to be done with TiVo extraction while the lengthy process of transcoding is active.

The "Temporary filename stub"  is used for the intermediate files and final output.  Different extensions will be tacked onto the end of the stub.   The incantations can assume the "STUB" environmental variable is defined.

Initially, the given "Netcat command" is used, in two seperate pipes, for retrieving data from the TiVo.  "$PORT" will be predefined appropriately for the audio or video streams,  $EXTEN" will be predefined "m2a" for audio, "m2v" for video".  One progress bar will denote video data, the other audio.

When the video pipe completes writing its file, the "Video Transcode Command" is executed.  The MPEG2 video output file is read by esdirect, then output to a pipe running that command.  Note that the command, if all is sucessful, will delete the MPEG2 video file once the MPEG1 video file has been written.

Likewise,  when the audio pipe completes writing its file, the "Audio Transcode Command" is executed.  The 33Khz  output file is read by esdirect, then output to a pipe running that command (that should correct the output frequency to 44Khz).  Note that the command, if all is sucessful, will delete the input audio file once the output file has been written.

While the transcoders are running, the progress bars should show the status of each.  This is functionality currently broken; the status bars don't update at all.

Once both the audio and video files are transcoded, the "Multiplexing command" is executed on the MPEG1 a/v files.  This step could also run your application for creating a VCD image.

Select the "Watch and Write" tab to watch the video while simultaneously writing the extracted mpeg to disk.  This is a combination of the first two tabs... if you understand those, you'll understand this.

Under "File" you can exit the application, start extracting (using the currently selected tab settings), or save your current settings.  Saving your settings saves all your incantations to the file "~/.esdirectrc".  The next time the program is launched, it will use the settings in that file over it's default settings.
 
 

ExtractStream

bash-2.02# ExtractStream

ExtractStream [-v] [-s] [-d] [-p] <fsid>...

ExtractStream takes valid <fsid> numbers of the streams you want to play,
extracts the sectors allocated to that stream and attempts to play the
audio/video information on those sectors through the TiVo codecs.

-v Verbose output (stderr)

-s Dump raw tyStream data to stdout

-d Dump sector scan info to stderr (no mpeg dumped at all)

-p Include PES headers with data

If -s is not set, and AUDIO_OUT or VIDEO_OUT environmental variables
are set, then mpeg output will go to these files instead of the
codecs.  If only  VIDEO_OUT is set,then both audio and video are
written to the file (note PS streams not yet working).

If the DUMP_OUT environmental variable is set, then the raw tyStream
is copied to the named file.  This is in addition to writing to
stdout if -s is set, or to the files named in VIDEO_OUT or AUDIO_OUT
if they're set.

If none of the environmental variables (DUMP_OUT , VIDEO_OUT, or AUDIO_OUT) are set, and none of the command line options: "-s" or"-d"are used, then ExtractStream acts as PlayStream does: writing the mpeg data to the proper devices.  This mode hasn't been tested in a long time, it probably requires the "-p" option, and you have to do something to keep it from competing with MyWorld for mpeg attention -- which is not properly documented for TiVo 2.0.1 systems (it's not necessarily different than killing and restarting MyWorld was in TiVo v1.3, but a cursory look at the process queue would make one think that it may not be as simple as it was). Another reason to read the README file that comes with the old PlayStream..

Extracting MPEG

bash-2.02# export VIDEO_OUT=/mnt/program1.m2v
bash-2.02# export AUDIO_OUT=/mnt/program1.m2a
bash-2.02# ExtractStream 53476 5347753478

Creating an MPEG2 Program Stream from Audio and Video Elemental Streams

Note that there are three "mplex" programs discussed here: two are in the distribution, and the one from the mjpegtools group at sourceforge .
 

Netmplex

Netmplex is married to ExtractStream over your LAN, expecting audio and video streams on ports 1235 and 1236.  It's output can be file based, piped to standard output, or on port 8080 for access to web-enabled playback devices.

Running "netmplex -h" provides the following usage information.

Usage:
  netmplex [options] [remote path]
Options:
  -a addr Set remote IP address
  -B        Best quality
  -h         This text
  -H         Host HTTP services on port 8080
  -l          Attach as listener/server to mpeg streams
  -t num  Toss bits from slow readers, debug option
  -o file    Write output to file instead of stdout
  -v          Verbose info

If you have a viewer that reads over the internet, then use the "-H" (That's a capital "H") option.  Connect your viewer to the current host on port 8080, and video will start streaming.

If "tyStream FSID's" are provided on the command line, along with the TiVo's IP address, then netmplex will directly request "httpd.tcl" on the TiVo to start the "doehtml" script, which starts ExtractStream and the appropriate netcat listeners in "server" mode.  Note that you can also use your browser to perform  this activation with the Mime type previously mentioned.

You can setup the TiVo side as a server (listener) or a client, Likewise, you can setup netmplex as a server (listener) or client.  For example, setup the TiVo as a server for the audio and video ports:

bash-2.02# mkfifo /var/tmp/vid /var/tmp/aud
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# nice -20 /hack/bin/nc -n -l -p 1236 <$VIDEO_OUT &
bash-2.02# nice -20 /hack/bin/nc -n -l -p 1235 <$AUDIO_OUT &
bash-2.02# nice -20 ExtractStream [tyStream FSID's] &

pc# netmplex -a [TiVo's IP address] -o myfile.mpeg2

pc# netmplex -l -o myfile.mpeg2

bash-2.02# mkfifo /var/tmp/vid /var/tmp/aud
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# nice -20 /hack/bin/nc -n -w 3 [PC's IP address] 1236 <$VIDEO_OUT &
bash-2.02# nice -20 /hack/bin/nc -n -w 3 [PC's IP address] 1235 <$AUDIO_OUT &
bash-2.02# nice -20 ExtractStream [tyStream FSID's] &

In Linux, the "mplex" program, available with the Mjpegtools at sourceforge ,works quite well.  Assuming you have extracted the MPEG audio and video elemental streams from the TiVo, the correct incantation is:

pc#  mplex-f 2 -m 2 -o program1.mpeg2 program1.m2[av]

The following instructions should be obsolete.  There has been a reported problem with a truncated audio packet that may not be fixed.  This can be easily fixed from mplex.  If you see the mplex error: "Failed to find start of next stream at...", then, in mplex's inptstrm.c change:

    syncword = (syncword<<(8-AU_start % 8));
    next = getbits( &audio_bs,8-(AU_start % 8) );
    syncword = syncword | next;
    if ( syncword != AUDIO_SYNCWORD )
    {
           mjpeg_warn("Failed to find start of next stream at %lld prev %lld !\n", AU_start/8, prev_offset/8 );
        break;
    }

     do {
        syncword = (syncword<<(8-AU_start % 8));
        next = getbits( &audio_bs,8-(AU_start % 8) );
        syncword = syncword | next;
    } while ( syncword != AUDIO_SYNCWORD );
#if 0
    {
        mjpeg_warn("Failed to find start of next stream at %lld prev %lld !\n", AU_start/8, prev_offset/8 );
        break;
    }
#endif

In Windows,   TMPGEnc will multiplex the audio and video data.  Using TMPGEnc with the "TiVo-VCD.mcf"   file will create a  cleaned-up cropped (removal of left edgegreen vertical line and top closed caption data) VCD compatible MPEG file.

Playing an MPEG2 Program Stream

pc#  vlc-V sdl -v program1.mpeg2

Transcoding MPEG2 Elemental and Program Streams toMPEG1

The Mjpegtools at sourceforge contain a transcode script, that requires mpeg2dec, plus some patches to the output drivers provided by the mjpegtools.

I've changed the transcode to two separate files, one for audio, and one for video.  The video script looks like:

cat $1.m2v |  mpeg2dec -o YUVh | mpeg2enc-s -r 24  -o $1.m1v

mpg123 -s -r 44100 --wav -$1.m2a | toolame -p 2 -b 224 - $1.m1a

Again, use mplex to make aVCD conformant MPEG1program stream:

pc#  mplex -f 1 -m 1 -o program1.mpeg1 program1.m1[av]

pc#  vcdimager--cdi-dir=/<where you put it>/cdi/ --cue-file=program1.cue \
            --bin-file=program1.bin \
            --volume-label="My first program"  program1.mpeg1

pc#  cdrdao write --device /dev/sg2 --speed 8 program1.cue

Other MPEG Extraction Methods

First, on the TiVo, make some named pipes:

bash-2.02# mkfifo /var/tmp/aud
bash-2.02# mkfifo/var/tmp/vid

bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# ExtractStream 53476 53477 53478

bash-2.02# nc -n -w 3 192.168.0.1 1235 </var/tmp/vid &
bash-2.02# nc -n-w 3 192.168.0.1 1236 </var/tmp/aud&

On your Linux PC, capture the data:

pc#  nc -l -p 1236 >program1.m2a &
pc#  nc -l -p 1235 >program1.m2v &

Actually, there is a reason for this.  The "nc"'s running on the PC side, rather than writing a file, can be placed into players or transcoders.

For example, the receiving "nc"'s on the PC might start a player:

pc#  nc -l-p 1236 | mpeg2dec &
pc#  nc -l -p 1235 | mpg123 &

Likewise, you could be transcoding on the PC:

pc#  nc -l -p 1236 | mpg123 -s -r 44100 - | \
    toolame -p 2 -b 224 - program1.m1a
pc#  nc -l -p 1235 | \
   mpeg2dec -o YUVh | \
   mpeg2enc -s  -r 24  -o program1.m1v &

Extracting Raw tyStreams

The "-s" command line option in ExtractStream causes the program to not split audio and video streams,  and to merely output the raw tyStream to "stdout" (standard output).  You can pipe the data directly across the net using a SAMBA or NFS mounted partition,as in:

bash-2.02# ExtractStream -s 53476 53477 53478 >/mnt/program1.rawtivo

pc#  nc-l -p 1234 >/mnt/program1.rawtivo

bash-2.02# ExtractStream -s 498441 498445  | nc -n -w 3 192.168.0.1 1234

MPLEX on the TiVo

This means: it can be used to directly generate an mpeg2 file, or playback video from the TiVo without creating any file at all.

The precompiled TiVo version is in the distribution  "ExtractStream-0.1.tgz", the source, "mplex.tgz",  is also available.  When compiling on the TiVo, make sure to use the "-msoft-float" option.

This program has performance problems on the TiVo.  It needs to be profiled.  It probably has to do with it's use of floating point numbers, when integers would do.  For example:

(int)ceil((double)foo/8.0)

(foo+7)/8

• It can't process higher bitrates (better than "BASIC" quality) fast enough (in "real time").  This is not a problem if you're just writing an mpeg2 file, only if you're trying to playback in realtime.
• The bitrates have been hardcoded for "BASIC" quality.  Mplex needs three command line arguments to overcome this: the buffer size for video, the buffer size for audio, and the video bitrate.  Refer to the "COMPBITRATE_MAX" case in multiplex.cpp.
• ExtractStream too buffers data as a function of mplex's input buffers (this doesn't keep it from working for other bitrates).  It too should take audio and video buffer sizes as command line arguments.
• Even multiplexing basic quality causes stuttering on the TiVo's TV output (it doesn't effect "mplex"'s output).

Creating an MPEG2 Program Stream with the obsolete "mplex"

Our mplex can run on either a Linux workstation (you need to compile it from source), or your TiVo (in the distribution).  You won't get TiVo stuttering if you run it on your workstation.  Here's an example running our "mplex" on a Linux workstation:

pc#  mkfifo vid
pc#  mkfifo aud
pc#  export VIDEO_OUT=./vid
pc#  export AUDIO_OUT=./aud
pc#  nc -l -p 1238 >$VIDEO_OUT &
pc#  nc -l -p 1239 >$AUDIO_OUT &

pc#  mplex > foo.mpeg2

pc#  mplex | (while true; do vlc -V sdl -v /dev/stdin; done)

On the TiVo, start the extraction

bash-2.02# mkfifo /var/tmp/vid
bash-2.02# mkfifo /var/tmp/aud
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# nc -w 3 -n <workstation IP address>  1238 <$AUDIO_OUT
bash-2.02# nc -w 3 -n <workstation IP address>  1239 <$VIDEO_OUT
bash-2.02# ExtractStream <tyStream FSID's>

If you're willing to allow stuttering on your TiVo, then you can run "mplex" there.  If you want realtime video playback on your workstation, run:
 

pc#  nc -l -p 1255 | (while true; do vlc --width=200 --height=200 -V sdl -v /dev/stdin; done)

bash-2.02# mkfifo /var/tmp/vid
bash-2.02# mkfifo /var/tmp/aud
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# mplex | nc -n -w 3 <workstation IP address> 1255 &
bash-2.02# ExtractStream <tyStream FSID's>

bash-2.02#  mplex | nc -n -w 3 <workstation IP address> 1255 &

bash-2.02# mplex >foo.mpeg2

Known Bugs