.tivo file format (split from: Decrypting TiVo ToGo)
Posted on January 7th, 2009 by cfz
/* All elements are in big-endian format */
typedef struct tivo_stream_header_s {
char filetype[4]; /* the string 'TiVo' */
uint16_t dummy_0004;
uint16_t dummy_0006;
uint16_t dummy_0008;
uint32_t mpeg_offset; /* 0-based offset of MPEG stream */
uint16_t chunks; /* Number of metadata chunks */
struct tivo_stream_chunk_s *chunk;
} __attribute__((packed)) tivo_stream_header;
#define TIVO_CHUNK_XML 0
#define TIVO_CHUNK_BLOB 1
typedef struct tivo_stream_chunk_s {
uint32_t chunk_size; /* Size of chunk */
uint32_t data_size; /* Length of the payload */
uint16_t id; /* Chunk ID */
uint16_t type; /* Subtype */
const uint8_t *data;
} __attribute__((packed)) tivo_stream_chunk;
The 'BLOB' types appear to be either (a) compressed or (b) encrypted, as they have very high entropy. (ie, can't be compressed). The 'BLOB' sizes seem to get longer for longer .tivo streams.
Chunk Type XML has the following text:
TiVo takes violations seriously. This file may contain
content protected by laws. If you are not the owner or rights
holder of such content, you should be aware that if you distribute this
file or otherwise make it available to anyone else, you may be violating the
intellectual property rights of the owners of the content contained in this
file. If you are not the owner or rights holder of such content, and if you
are a TiVo customer, and if you distribute this file or otherwise make it
available to anyone else, you may be violating the service agreement.
We may permanently discontinue your TiVo service as a result.
01/12/2005 11:22:01
... hex string that is 224 hex digits long ....
... hex string that is 32 hex digits long...
For example, a stream off of my machine has:
TiVo Stream:
filetype: TiVo
dummy_0004: 0x0004 // Possibly format version?
dummy_0006: 0x0001 // Possibly format minor version?
dummy_0008: 0x0000
mpeg_offset: 0x00003000
chunks: 0x0003
Chunk 0:
chunk_size: 0x00000474
data_size: 0x00000463
id: 0x0003
type: 0x0000
Chunk 1:
chunk_size: 0x00000f54
data_size: 0x00000f44
id: 0x0001
type: 0x0001
Chunk 2:
chunk_size: 0x00001434
data_size: 0x00001422
id: 0x0002
type: 0x0001
I couldn't find any references to .ty stream format on the web, but for those that do know, does this look similar?
BTW- I suggest throwing REC and IDA at the dll. I downloaded the Windows debugger and am using it for the first time tonight and am pulling keys. I think it's the only reasonable way to make progress. Especially if the keys aren't becoming as obvious as hoped. :'(
BTW- Warewolf - most of this work is rather mundane. We need more eyes and more man-hours tinkering and gathering information.
IMHO, the hacks to date posted elsewhere are all things which simply use the existing dll. It's a matter of time that the new dll will be released and make them not work. And then, a Tivo update which will make the old dll not work with new .tivo files.
Hey Alphawolf, I'm not sure what you mean, but I'm sure the encryption is done on the tivo. As cool as it would be to hack the box to make it send unencrypted files, it would be MUCH better if we figured out how to not void the warranty by doing the decryption on the PC/Mac/Linux side. (btw- from what I read, most of the speed issue seems to be bandwidth related and not cpu, and the encryption doesn't seem to add much [if any] bloat to the files.)
Quick start for folks who want to jump in and poke around: Get REC (http://www.backerstreet.com/rec/rec.htm), get IDA Pro [the demo is fine] (http://www.datarescue.com/), and windbg (http://www.microsoft.com/whdc/ddk/debugging/). Take a look around, read what's been posted, set some breakpoints (note: rec and ida show addresses like 4xxxxx, drop the 4 and add the result to the base dll load address for the proper break point in windbg).
Anyways, we need more people gathering information, testing, and thinking... Jump in! YOU can make a difference. ;')
Over the weekend, I plan to use this to help decode the 'Tivo Sharing Keys' in the registry, and I'll describe my results (if any) on Monday.
Thanks for the research - we're getting one step closer every day.
Thanks! The colors came out a little too girly/pastelly for my taste, but I wanted them to be light enough to make the text readable when printed out. ;')
BTW- I did a little tracing on the sharing keys decrypt stuff earlier this week. It looks like the dll uses the CryptUnprotectData function to look at the registry entries. If you put in the wrong password, it goes so far as to call CryptUnprotectData, which generates an error, which then triggers the 'wrong password' in the user dialog. So, it seems it is this function that's providing the test for whether the pass is right or not. Here's Microsoft's documentation on the function:
CryptUnprotectData (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcesecurity5/html/wce50lrfcryptunprotectdata.asp)
Is there a way to do this, and if so, what software do I need to install and run? I'd be happy to scriptify the results and post...
Eventhough the files are encrypted, we should have the decryption key (the MAK?). I think it is 4 bytes long. The salt (file specific) is 16 bytes long. So that's a 20 byte decryption key. The files are easily parsed to find the encrypted chunks. Pull a chunk out, decrypt with different methods using the key (it might also take some experimentation to figure out if it is MAK padded with salt, the other way around, or something else?) using different cipher methods to see what works.(?)
It's difficult to find the current status on the effort to decrypt these things other than using round-about methods in a Windoze-only environment. Any pointers to current info, or is this it? :')
-Phil
I think download speed is fixed (by the series 2) and is related directly to encoding quality. The higher the quality, the higher the download speed.
As for the decoding and DLL use, I stopped posting because I realized that reverse engineering the DLL is useless: they can change it and the scrambling mechanism at any time. Instead, I started using srcfilter (http://outflux.net/software/pkgs/srcfilter/), which will use whatever DLL you give it. Under Windows, this tool just works: .TiVo file gets converted to the raw MPEG2, using the Windows Way. They can't ever change this method because it is the very mechanism Windows uses to play the files back.
Under Linux, you need to be using bleeding-edge Wine (outta CVS) and you have to set the "TiVoToGo Media" registry entry by hand. This is produced by running the plain text of "tivo:MAK" through CryptProtectData (under Wine) with a single NULL byte (an "empty" password) for the pOptionalEntropy. Once the output of this is in the registry item, the DLL works under Linux too. I'm sure someone can cook up a quick tool to do the CryptProtectData call.
tystream info:
http://dvd-create.sourceforge.net/tystudio/tystream.shtml
http://alt.org/forum/index.php?t=msg&th=15&start=0&rid=0
http://alt.org/forum/index.php?t=msg&th=90&start=0&rid=0
tydemux/tystudio sources: http://dvd-create.sourceforge.net/tystudio/download.shtml
I was hoping to avoid the lba24/monte setup that's needed with S2_unscramble.
Thank you.
Well, mostly what jamie said as from what I understand the ttg transfers are much slower than mfs_ftp transfers which can go as high as 4 megabytes per second. Not only that, but if you patched tivoapp, thats one less inconveniene to deal with as theres no need to decrypt on the PC side.
Fun fun fun.
placed on the web the EXACT SAME SOURCE CODE that TiVo used in their stream cypher.
http://www.qualcomm.com.au/Turing.html
Time for the fun to begin!
Poking around with the dll some more, I noticed that stream keys are changing a lot (breakpoints hit several times when playing a file), but once the file begins to play, the keys don't seem to change.
I've spent some time today creating an initial draft of a function flow chart. I've tried to spend a lot of time eliminating things which aren't interesting (hard to do since we don't know really what's interesting yet). The file is a mere 80K, but there is a huge amount of detail and not a lot of helpful hints in it that show what the functions do.
I can give a brief walkthrough, though:
First of all, the very top (sub_4075F8) is Mpeg2Lib::TivoMpegDecrypt.
In rough quarters of the document from left to right, the left is final cleanup and setup code for the dll. Pretty uninteresting to us.
The second fourth which is a little below the first is the Turing code, playback stream decode, and block_IV decrypt. Probably the most interesting! In this quarter sub_42549C is the function which has the Turing key debug code. The cluster to the right of it (sub_4257F0) is where we find the Playback stream key debug code. Lastly, just to the right of it is sub_425A1C which has the reference to BLOCK_IV.
Third fourth of the document and a little in the middle is the file access and registry routines. Reference to Tivo meta-data is on the right side of this quarter.
The last quarter is the password dialog box. Also are the references to CryptUnprotectData and CryptProtectData. Ignore the widows on the very upper right. Those were supposed to be deleted.
- Something to note. The software tries to lay out all the functions in such a way as to keep functions that call each other close to each other. Notice how far away the CryptProtect functions are from the Turing fuctions, yet close to the registry, file access, and password dialog functions?
Can't be done. But the monte isn't so bad, and FYI I've monted from the lba48-aware 3.1.5 killhdinitrd'd kernel to the 3.1.x S2_Unscramble kernel several times now without issue.
I spaced it out a bit vertically, removed a vast majority of the uninteresting functions, color-coded by memory location (similar colors mean similar location in the lib which I would infer to mean the same source file), and numbered the edges (connecting lines) in the order that they are referenced in the calling function. The first edge (the first call) is unnumbered to help unclutter things. Notice that this graph has no information on flow control on the edges (if's, loops, etc.). Lastly, I added some nodes with string references that use gray edges to show where interesting strings are found.
It still begs to be printed out on several pieces of paper and taped together to be understood, but at least now you should be able to understand and follow most things without being overwhelmed (as much ;') and can see where edges go without them being lost.
BTW-This was created from a 'dot' file which in turn was produced by a Perl script that parses the .gdl file from IDA. Post a reply if you'd like to see either of these.
A few choice/random things found in that file:
TivoMpegFileStream *
Mpeg2Lib::TivoMpegDecrypt *
Mpeg2Lib::TivoMediaFile
Mpeg2Lib::TivoMpegDecrypt
_STL::auto_ptr
_STL::auto_ptr
MediaPasswordDialog
ResourceString *
MediaAccessKey *
MediaAccessKey
CryptoLib::Cypher
CryptoLib::Data
MpegProgramStreamDecrypt
Mpeg2Lib::PcMpeg
Mpeg
MpegTuring
ref
TuringFast *
TuringRef *
TuringTable *
Turing *
Turing
TuringTable
TuringRef
TuringFast
ref
ref
Q@YY]
[YY]
ref
ref
8luY
MpegData *
ref
MpegData
MpegBlowfish *
MpegTuring::CipherStream *
Borland C++ - 2002 Borland Corporation
Incorrect media access key or data file is damaged.
SoftwareTiVoDirectShowVolume
TiVo PCM boost filter
Multiplier
TiVoToGo Metadata
TiVoToGo Media
TiVoToGo Transfer
tivo:TiVo DVR:
tivo:transfer:
basic_string
MediaPasswordDialog
Error
Password incorrect
SrccryptTuring.C
Turing::Create
hexkey should start with 0x or 0X
Key length exceeds 448 bits
Hexadecimal key contains non-hex digit %c
Hexadecimal key contains non-even number of digits
Residue error, out of zeros at offset %ld
MpegBlowfish::Initialize(%s)
BLOWFISH KEY=0x
%02X
MpegBlowfish::Seek(%lld)
TURING KEY=0x
%02X
PLAYBACK_STREAM_KEY[0x%02x]=0x
%02X
BLOCK_IV[0x%02x,%ld]=0x
invalid video offset, not a multiple of 1024 (%ld)
invalid video offset
invalid key %s
no key to
*** TiVo File header, videoOffset=%ld, count=%d, flags=0x%x
*** 0x%04x: TiVo Public (%d bytes)
*** 0x%04x: TiVo Private (%d bytes)
*** 0x%04x: License Agreement (%d bytes)
*** 0x%04x: Unknown (%d bytes)decrypt meta data
%012lld: adler32 checksum mismatch 0x%lx != 0x%lx
%012lld: 0x%02x skip=%d
%012lld: no key was provided
%012lld: 0x%02x block=%ld/%ld/%d, flags=0x%02x, cs=0x%lx/0x%lx
%012lld: 0x%02x data=%d
SoftwareTiVoSharingKeys
As for the decoding and DLL use, I stopped posting because I realized that reverse engineering the DLL is useless: they can change it and the scrambling mechanism at any time. Instead, I started using srcfilter (http://outflux.net/software/pkgs/srcfilter/), which will use whatever DLL you give it. Under Windows, this tool just works: .TiVo file gets converted to the raw MPEG2, using the Windows Way. They can't ever change this method because it is the very mechanism Windows uses to play the files back.
Under Linux, you need to be using bleeding-edge Wine (outta CVS) and you have to set the "TiVoToGo Media" registry entry by hand. This is produced by running the plain text of "tivo:MAK" through CryptProtectData (under Wine) with a single NULL byte (an "empty" password) for the pOptionalEntropy. Once the output of this is in the registry item, the DLL works under Linux too. I'm sure someone can cook up a quick tool to do the CryptProtectData call.
placed on the web the EXACT SAME SOURCE CODE that TiVo used in their stream cypher.
http://www.qualcomm.com.au/Turing.html
Time for the fun to begin!
Yeah, I noticed that too. Although it's not obvious to me that it is the same exact source code. How did you determine that?
BTW- I needed to comment out the endian.h and byteorder.h includes and make dummy functions (which do nothing but pass values through) for __be32_to_cpu and __be16_to_cpu to make it work on MacOS-X. Namely, I put this
#ifdef __ppc__
int32_t __be32_to_cpu(int32_t data) { return data; }
int16_t __be16_to_cpu(int16_t data) { return data; }
#endif
in near the top of tivo.c.
It gives identical output as my x86 Linux box.
It shouldn't be too hard to determine if the decryption works. The first chunk is smallish and of a special type (0). Must be the meta info, which is probably largely text. Perhaps all XML.
Post any progress you make! :')
srcfilter $Revision: 1.7 $
http://outflux.net/software/pkgs/srcfilter/
2005-2006, Kees Cook
This program is licensed under the GNU General Public License
http://www.gnu.org/copyleft/gpl.html
Loading 'SpongeBob.tivo' ...
trace:crypt:CryptUnprotectData called
trace:crypt:report pPromptStruct: 0x61ec2c
trace:crypt:report cbSize: 0x10
trace:crypt:report dwPromptFlags: 0x0
trace:crypt:report hwndApp: (nil)
trace:crypt:report szPrompt: (nil)
trace:crypt:report dwFlags: 0x0000
trace:crypt:report pDataIn cbData: 256
trace:crypt:report pDataIn pbData @ 0x630bd8:01,00,00,00,d0,8c,9d,df,01,15,d1,11...
trace:crypt:report pOptionalEntropy cbData: 50
trace:crypt:report pOptionalEntropy pbData @ 0x630a9c:7b,42,30,41,37,39,36,31,33
,2d,44,32,32,32,2d,34,37,44,30,2d,38,39,41,43,2d,37,31,30,42,31,44,41,44,34,43,4
6,36,7d,00,2a,52,d6,9c,10,c7,01,00,00,00,00
trace:crypt:report "{B0A79613-D222-47D0-89AC-710B1DAD4CF6}x00*Rxd6x9cx10x
c7x01x00x00x00x00"
trace:crypt:CryptUnprotectData ppszDataDescr: 0x61ec3c
trace:crypt:unserialize called
trace:crypt:valid_protect_data called
err:crypt:valid_protect_data info0 magic value not matched !
err:crypt:valid_protect_data unrecognized CryptProtectData block
fixme:crypt:CryptUnprotectData CryptUnprotectData received a DATA_BLOB that seem
s to have NOT been generated by Wine. Please enable tracing ('export WINEDEBUG=
crypt') to see details.
trace:crypt:free_protect_data called
trace:crypt:CryptUnprotectData returning FAIL
trace:crypt:CryptUnprotectData called
trace:crypt:report pPromptStruct: 0x61ec2c
trace:crypt:report cbSize: 0x10
trace:crypt:report dwPromptFlags: 0x0
trace:crypt:report hwndApp: (nil)
trace:crypt:report szPrompt: (nil)
trace:crypt:report dwFlags: 0x0000
trace:crypt:report pDataIn cbData: 256
trace:crypt:report pDataIn pbData @ 0x630bd8:01,00,00,00
...
Warning: DS_Filter() could not Load source file.
(DLL=C:Program FilesCommon FilesTiVo SharedDirectShowTiVoDirectShowFilter
.dll, r=0x80071771)
Failed
where the "01,00,00,00,d0,8c,9d,df,01,15,d1,11..." is the value of the registry entry.
Any idea what these errors mean? Could it be that the value of the registry on the windows machine is different from what it should be on the linux box?
Thanks for any help.
Interesting. I did a memory search using WinHex after 2 calls to CryptUnprotectData (it seems to be called a total of 3 times when beginning to play a video) and I also see the "tivo:####.." for each. The first time it seems to be on the registry entry "TivoToGo Metadata", as I can see the registry data sitting there in memory (and no password prompted yet). The result is as Tok sees it, 16-byte ascii hex.
Then the DLL asks for the password and then it loads the "TivoToGo Media" registry key and decrypts that. There, the result is "tivo:MAK" (with MAK of course being my MAK, which is in ascii and base-10, just as it is shown by the tivo on the TV).
I haven't pinned down the third access. It doesn't seem to be to a registry key, but I'm not sure. Perhaps the Metadata or MAK is used with it to unlock something else?
So has the action moved somewhere's else? Or has it been abandoned? I just got a new series 2 DT and I got the TivoToGo installed, and I'm looking for the easiest (maybe cheapest) way to burn watchable dvd's of shows... I know tivo refers to the sonic deal, and I'm trying to figure out if TyTools will do what I'm after, but this seems the most promising.
http://www.apecity.com/tivo/
Under preparation, I used his zip file of xmuxer, and not the proper website, and it works great. I've got MPEG2 files coming out of my pc now, which is fabulous... Thanks again.
edit: Also, in case anyone finds it useful, this is the page that started me out (and I think linked me to the above page). Props to all these guys, fo shizzle...
http://www.zatznotfunny.com/ttg.htm
Over the weekend, I plan to use this to help decode the 'Tivo Sharing Keys' in the registry, and I'll describe my results (if any) on Monday.
Thanks for the research - we're getting one step closer every day.
https://
Not much activity for a while. But, maybe I can infuse a little bit of info and conversation to keep things rolling.
With TOK's trampoline code, I'm now going to monitor the Turing crypto functions, to see what values they are being passed. With luck, it'll be some derivative of the MAK.Try monitoring the function at 417E6B. It's what is producing the keys (both Turing and Stream, I think). It is called from 4 different place; see the PDF I posted. The call which winds up in the Turing function which has the debug stuff converts the key to ascii (a proper "0x###.." string) and then the later function turns it back to binary. Clearly some mish-mash glueing of code from different sources. Anyways, it seems to have 3 arguments (not nessesarily in this order): input-pointer, input byte count, and output-ptr. I'm not sure if the output is twice the number of bytes as the input or what. Looking at it with REC (http://www.backerstreet.com/rec/rec.htm) makes my head hurt from all the shifts and such. BTW- REC has some bugs in it when it comes to strings and references, so take it's output with a grain of salt.
weird. The one that decrypts to "tivo:
I hope some other folks jump in!
Is this correct?
Do stand alones store the files as .tivo?
I have an HR10-250, which obviously is not stand alone.
I have a couple of killer HDnet concerts recorded before enabling unscrambling in the box.
Went the graphedit route and realized it works with .tivo files only.
HR10-250
3.1.5f-01-2-357
Still though, it would be nice not need to muck with the innerds of the Tivo to read its files, even if it's slowing the xfer down and making it a hassle to decrypt on the PC. Creating something for the typical end user to use to play the files on anything, anywhere (like with VLC) without violating warranties would be cool. Anything much past that might be self defeating for most folks
Anyways, sorry to drag this thread further off-topic! Ack! Perhaps we can take this to another thread if we want to chat more about this to reduce the noise here.
1) The 'Metadata' key is probably just a constant, specific key for the .tivo metadata chunks.
2) The 'Media' key is the MAK. This is fantasticlly cool, because it means that the MAK is directly related to decoding the MPEG2 streams.
With TOK's trampoline code, I'm now going to monitor the Turing crypto
functions, to see what values they are being passed. With luck, it'll be
some derivative of the MAK.
#If you have any other info about this subject , Please add it free.# |