WimTiVoServer changes to use FFProbe

WimTiVoServer was originally written using the libraries that FFMPEG is based on to retrieve details about video files. I had downloaded the packages from http://ffmpeg.zeranoe.com/builds/ and used the DLLs for the library calls. In other program I’m building related to FFMPEG I am updating FFMPEG on a regular basis. Maintaining the correct link path any time I came back for a minor adjustment to WimTiVoServer became more of an effort than I wanted to deal with, so I investigated what else was available.

My solution has been to use FFProbe, which is distributed with FFmpeg. I am using the spawning a child process and capturing the standard output. I read the results of my command and put it into a IStream memory stream object, which I then use the IXmlReader object to parse the XML for the items I’m looking for.

The command line I’m using for FFProbe is ffprobe.exe -show_streams -show_format -print_format xml INPUT. An example of the output it produces is:

<?xml version="1.0" encoding="UTF-8"?>
<ffprobe>
    <streams>
        <stream index="0" codec_name="ac3" codec_long_name="ATSC A/52A (AC-3)" codec_type="audio" codec_time_base="1/48000" codec_tag_string="[0][0][0][0]" codec_tag="0x0000" sample_fmt="fltp" sample_rate="48000" channels="6" bits_per_sample="0" dmix_mode="-1" ltrt_cmixlev="-1.000000" ltrt_surmixlev="-1.000000" loro_cmixlev="-1.000000" loro_surmixlev="-1.000000" id="0x27" r_frame_rate="0/0" avg_frame_rate="0/0" time_base="1/10000000" start_pts="22054844" start_time="2.205484" duration_ts="19133694951" duration="1913.369495" bit_rate="384000">
            <disposition default="0" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="0"/>
        </stream>
        <stream index="1" codec_name="ac3" codec_long_name="ATSC A/52A (AC-3)" codec_type="audio" codec_time_base="1/48000" codec_tag_string="[0][0][0][0]" codec_tag="0x0000" sample_fmt="fltp" sample_rate="48000" channels="2" bits_per_sample="0" dmix_mode="-1" ltrt_cmixlev="-1.000000" ltrt_surmixlev="-1.000000" loro_cmixlev="-1.000000" loro_surmixlev="-1.000000" id="0x28" r_frame_rate="0/0" avg_frame_rate="0/0" time_base="1/10000000" start_pts="23039510" start_time="2.303951" bit_rate="192000">
            <disposition default="0" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="0"/>
        </stream>
        <stream index="2" codec_name="mpeg2video" codec_long_name="MPEG-2 video" profile="Main" codec_type="video" codec_time_base="1001/120000" codec_tag_string="[0][0][0][0]" codec_tag="0x0000" width="1280" height="720" has_b_frames="1" sample_aspect_ratio="1:1" display_aspect_ratio="16:9" pix_fmt="yuv420p" level="4" timecode="00:00:00:00" id="0x29" r_frame_rate="60000/1001" avg_frame_rate="60000/1001" time_base="1/10000000" start_pts="31875510" start_time="3.187551">
            <disposition default="0" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="0"/>
        </stream>
        <stream index="3" codec_type="subtitle" codec_time_base="1/10000000" codec_tag_string="[0][0][0][0]" codec_tag="0x0000" id="0x2a" r_frame_rate="0/0" avg_frame_rate="0/0" time_base="1/10000000" start_pts="32209177" start_time="3.220918">
            <disposition default="0" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="0"/>
        </stream>
        <stream index="4" codec_name="mjpeg" codec_long_name="MJPEG (Motion JPEG)" codec_type="video" codec_time_base="1/90000" codec_tag_string="[0][0][0][0]" codec_tag="0x0000" width="200" height="113" has_b_frames="0" sample_aspect_ratio="1:1" display_aspect_ratio="200:113" pix_fmt="yuvj420p" level="-99" id="0xffffffff" r_frame_rate="90000/1" avg_frame_rate="0/0" time_base="1/90000" start_pts="198494" start_time="2.205489" duration_ts="172203255" duration="1913.369500">
            <disposition default="0" dub="0" original="0" comment="0" lyrics="0" karaoke="0" forced="0" hearing_impaired="0" visual_impaired="0" clean_effects="0" attached_pic="1"/>
            <tag key="title" value="TV Thumbnail"/>
        </stream>
    </streams>

    <format filename="d:\Recorded TV\Archer_FXPHD_2013_02_28_22_00_00.wtv" nb_streams="5" nb_programs="0" format_name="wtv" format_long_name="Windows Television (WTV)" start_time="2.205484" duration="1913.369495" size="1956642816" bit_rate="8180930" probe_score="100">
        <tag key="WM/MediaClassPrimaryID" value="db9830bd-3ab3-4fab-8a371a995f7ff74"/>
        <tag key="WM/MediaClassSecondaryID" value="ba7f258a-62f7-47a9-b21f4651c42a000"/>
        <tag key="Title" value="Archer"/>
        <tag key="WM/SubTitle" value="Live and Let Dine"/>
        <tag key="WM/SubTitleDescription" value="Archer, Lana and Cyril go undercover in celebrity chef Lance Casteau&apos;s hellish kitchen."/>
        <tag key="genre" value="Comedy;General;Series"/>
        <tag key="WM/OriginalReleaseTime" value="0"/>
        <tag key="language" value="en-us"/>
        <tag key="WM/MediaCredits" value="H. Jon Benjamin/Jessica Walter/Aisha Tyler/George Coe/Chris Parnell/Judy Greer;;;Anthony Bourdain"/>
        <tag key="service_provider" value="FXPHD"/>
        <tag key="service_name" value="FX HD (Pacific)"/>
        <tag key="WM/MediaNetworkAffiliation" value="Satellite"/>
        <tag key="WM/MediaOriginalChannel" value="728"/>
        <tag key="WM/MediaOriginalChannelSubNumber" value="0"/>
        <tag key="WM/MediaOriginalBroadcastDateTime" value="2013-02-28T08:00:00Z"/>
        <tag key="WM/MediaOriginalRunTime" value="19144791872"/>
        <tag key="WM/MediaIsStereo" value="false"/>
        <tag key="WM/MediaIsRepeat" value="false"/>
        <tag key="WM/MediaIsLive" value="false"/>
        <tag key="WM/MediaIsTape" value="false"/>
        <tag key="WM/MediaIsDelay" value="false"/>
        <tag key="WM/MediaIsSubtitled" value="false"/>
        <tag key="WM/MediaIsMovie" value="false"/>
        <tag key="WM/MediaIsPremiere" value="false"/>
        <tag key="WM/MediaIsFinale" value="false"/>
        <tag key="WM/MediaIsSAP" value="false"/>
        <tag key="WM/MediaIsSport" value="false"/>
        <tag key="WM/Provider" value="MediaCenterDefault"/>
        <tag key="WM/VideoClosedCaptioning" value="false"/>
        <tag key="WM/WMRVEncodeTime" value="2013-03-01 06:00:05"/>
        <tag key="WM/WMRVSeriesUID" value="!MCSeries!225842780"/>
        <tag key="WM/WMRVServiceID" value="!MCService!188913961"/>
        <tag key="WM/WMRVProgramID" value="!MCProgram!285145704"/>
        <tag key="WM/WMRVRequestID" value="0"/>
        <tag key="WM/WMRVScheduleItemID" value="0"/>
        <tag key="WM/WMRVQuality" value="0"/>
        <tag key="WM/WMRVOriginalSoftPrePadding" value="300"/>
        <tag key="WM/WMRVOriginalSoftPostPadding" value="120"/>
        <tag key="WM/WMRVHardPrePadding" value="-300"/>
        <tag key="WM/WMRVHardPostPadding" value="0"/>
        <tag key="WM/WMRVATSCContent" value="true"/>
        <tag key="WM/WMRVDTVContent" value="true"/>
        <tag key="WM/WMRVHDContent" value="true"/>
        <tag key="Duration" value="19151788198"/>
        <tag key="WM/WMRVEndTime" value="2013-03-01 06:32:00"/>
        <tag key="WM/WMRVBitrate" value="8.173201"/>
        <tag key="WM/WMRVKeepUntil" value="-1"/>
        <tag key="WM/WMRVActualSoftPrePadding" value="294"/>
        <tag key="WM/WMRVActualSoftPostPadding" value="120"/>
        <tag key="WM/WMRVContentProtected" value="true"/>
        <tag key="WM/WMRVContentProtectedPercent" value="99"/>
        <tag key="WM/WMRVExpirationSpan" value="9223372036854775807"/>
        <tag key="WM/WMRVInBandRatingSystem" value="255"/>
        <tag key="WM/WMRVInBandRatingLevel" value="255"/>
        <tag key="WM/WMRVInBandRatingAttributes" value="0"/>
        <tag key="WM/WMRVWatched" value="false"/>
        <tag key="WM/MediaThumbWidth" value="352"/>
        <tag key="WM/MediaThumbHeight" value="198"/>
        <tag key="WM/MediaThumbStride" value="1056"/>
        <tag key="WM/MediaThumbRet" value="0"/>
        <tag key="WM/MediaThumbRatingSystem" value="9"/>
        <tag key="WM/MediaThumbRatingLevel" value="17"/>
        <tag key="WM/MediaThumbRatingAttributes" value="0"/>
        <tag key="WM/MediaThumbAspectRatioX" value="16"/>
        <tag key="WM/MediaThumbAspectRatioY" value="9"/>
        <tag key="WM/MediaThumbTimeStamp" value="4647772712253334203"/>
    </format>
</ffprobe>

I am parsing the XML and keeping track of only the first video stream details and the first audio stream details, and then looking for some specific items in the metadata tags. I store the information and return it to the TiVo as information when it’s requesting a list of what programs are available to transfer and then when I transfer the file itself.

An interesting side effect of moving to using XML from using the libraries is that the XML created by FFProbe handles extended characters that are not in the ASCII character set. Because I’m using the XML Parser that works with Unicode by default, it takes care of the characters properly. When I was using the libraries, I was looping on AVDictionaryEntry values and doing comparisons with char values.

Here is the code that I’m currently using. It’s not the prettiest code but it gets the job done and runs quickly enough.

void cTiVoFile::PopulateFromFFProbe(void)
{
	static const CString csFFProbePath(FindEXEFromPath(_T("ffprobe.exe")));
	if (!csFFProbePath.IsEmpty())
	{
		// Set the bInheritHandle flag so pipe handles are inherited. 
		SECURITY_ATTRIBUTES saAttr;  
		saAttr.nLength = sizeof(SECURITY_ATTRIBUTES); 
		saAttr.bInheritHandle = TRUE; 
		saAttr.lpSecurityDescriptor = NULL; 

		// Create a pipe for the child process's STDOUT. 
		HANDLE g_hChildStd_OUT_Rd = NULL;
		HANDLE g_hChildStd_OUT_Wr = NULL;
		if ( ! CreatePipe(&g_hChildStd_OUT_Rd, &g_hChildStd_OUT_Wr, &saAttr, 0x800000) ) 
			std::cout << "[" << getTimeISO8601() << "] "  << __FUNCTION__ << "\t ERROR: StdoutRd CreatePipe" << endl;
		else
		{
			// Ensure the read handle to the pipe for STDOUT is not inherited.
			if ( ! SetHandleInformation(g_hChildStd_OUT_Rd, HANDLE_FLAG_INHERIT, 0) )
				std::cout << "[" << getTimeISO8601() << "] "  << __FUNCTION__ << "\t ERROR: Stdout SetHandleInformation" << endl;
			else
			{
				// Create a child process that uses the previously created pipes for STDIN and STDOUT.
				// Set up members of the PROCESS_INFORMATION structure.  
				PROCESS_INFORMATION piProcInfo; 
				ZeroMemory( &piProcInfo, sizeof(PROCESS_INFORMATION) );
 
				// Set up members of the STARTUPINFO structure. 
				// This structure specifies the STDIN and STDOUT handles for redirection.
				STARTUPINFO siStartInfo;
				ZeroMemory( &siStartInfo, sizeof(STARTUPINFO) );
				siStartInfo.cb = sizeof(STARTUPINFO); 
				siStartInfo.hStdError = GetStdHandle(STD_ERROR_HANDLE);
				siStartInfo.hStdInput = GetStdHandle(STD_INPUT_HANDLE);
				siStartInfo.hStdOutput = g_hChildStd_OUT_Wr;
				siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
 
				CString csCommandLine(QuoteFileName(csFFProbePath));
				csCommandLine.Append(_T(" -show_streams -show_format -print_format xml "));
				csCommandLine.Append(QuoteFileName(m_csPathName));

				TRACE(_T("CreateProcess: %s\n"), csCommandLine.GetString());
				// Create the child process.
				if (CreateProcess(NULL, 
					(LPTSTR) csCommandLine.GetString(),     // command line 
					NULL,          // process security attributes 
					NULL,          // primary thread security attributes 
					TRUE,          // handles are inherited 
					0,             // creation flags 
					NULL,          // use parent's environment 
					NULL,          // use parent's current directory 
					&siStartInfo,  // STARTUPINFO pointer 
					&piProcInfo))  // receives PROCESS_INFORMATION 
				{
					CloseHandle(g_hChildStd_OUT_Wr);	// If I don't do this, then the parent will never exit!
					CComPtr<IStream> spMemoryStreamOne(::SHCreateMemStream(NULL, 0));
					if (spMemoryStreamOne != NULL)
					{
						const int RAWDataBuffSize = 0x1000;	// 0x1000 is 4k
						char * RAWDataBuff = new char[RAWDataBuffSize];
						for (;;)
						{
							DWORD dwRead = 0;
							BOOL bSuccess = ReadFile(g_hChildStd_OUT_Rd, RAWDataBuff, RAWDataBuffSize, &dwRead, NULL);
							if( (!bSuccess) || (dwRead == 0)) break;
							ULONG cbWritten;
							spMemoryStreamOne->Write(RAWDataBuff, dwRead, &cbWritten);
						} 
						delete[] RAWDataBuff;
						// reposition back to beginning of stream
						LARGE_INTEGER position;
						position.QuadPart = 0;
						spMemoryStreamOne->Seek(position, STREAM_SEEK_SET, NULL);
						HRESULT hr = S_OK;
						CComPtr<IXmlReader> pReader; 
						if (SUCCEEDED(hr = CreateXmlReader(__uuidof(IXmlReader), (void**) &pReader, NULL))) 
						{
							if (SUCCEEDED(hr = pReader->SetProperty(XmlReaderProperty_DtdProcessing, DtdProcessing_Prohibit))) 
							{
								if (SUCCEEDED(hr = pReader->SetInput(spMemoryStreamOne))) 
								{
									int indentlevel = 0;
									XmlNodeType nodeType; 
									const WCHAR* pwszLocalName;
									const WCHAR* pwszValue;
									CString csLocalName;
									bool bIsFormat = false;
									bool bVideoStreamInfoNeeded = true;
									bool bAudioStreamInfoNeeded = true;

									//read until there are no more nodes 
									while (S_OK == (hr = pReader->Read(&nodeType))) 
									{
										if (nodeType == XmlNodeType_Element)
										{
											if (SUCCEEDED(hr = pReader->GetLocalName(&pwszLocalName, NULL)))
											{
												csLocalName = CString(pwszLocalName);
												if ((bVideoStreamInfoNeeded || bAudioStreamInfoNeeded) && !csLocalName.Compare(_T("stream")))
												{
													CString cs_codec_name;
													CString cs_codec_type;
													CString cs_codec_time_base;
													CString cs_width;
													CString cs_height;
													CString cs_duration;
													while (S_OK == pReader->MoveToNextAttribute())
													{
														if (SUCCEEDED(hr = pReader->GetLocalName(&pwszLocalName, NULL)))
															if (SUCCEEDED(hr = pReader->GetValue(&pwszValue, NULL)))
														{
															csLocalName = CString(pwszLocalName);
															if (!csLocalName.Compare(_T("codec_name")))
																cs_codec_name = CString(pwszValue);
															else if (!csLocalName.Compare(_T("codec_type")))
																cs_codec_type = CString(pwszValue);
															else if (!csLocalName.Compare(_T("codec_time_base")))
																cs_codec_time_base = CString(pwszValue);
															else if (!csLocalName.Compare(_T("width")))
																cs_width = CString(pwszValue);
															else if (!csLocalName.Compare(_T("height")))
																cs_height = CString(pwszValue);
															else if (!csLocalName.Compare(_T("duration")))
																cs_duration = CString(pwszValue);
														}
													}
													if (!cs_codec_type.Compare(_T("video")))
													{
														bVideoStreamInfoNeeded = false;
														if (!cs_codec_name.Compare(_T("mpeg2video")))
															m_VideoCompatible = true;
														m_SourceFormat = cs_codec_type + CString(_T("/")) + cs_codec_name;
														int width = 0;
														std::wstringstream ss;
														ss << cs_width.GetString();
														ss >> width;
														if (width >= 1280)
															m_VideoHighDefinition = true;
														double duration = 0;
														ss = std::wstringstream();
														ss << cs_duration.GetString();
														ss >> duration;
																												m_Duration = duration * 1000 + 5;													}
													else if (!cs_codec_type.Compare(_T("audio")))
													{
														bAudioStreamInfoNeeded = false;
														if (!cs_codec_name.Compare(_T("ac3")))
															m_AudioCompatible = true;
													}	
												}
												else if (!csLocalName.Compare(_T("format")))
												{
													bIsFormat = true;
													const CString ccs_duration(_T("duration"));
													while (S_OK == pReader->MoveToNextAttribute())
													{
														if (SUCCEEDED(hr = pReader->GetLocalName(&pwszLocalName, NULL)))
															if (SUCCEEDED(hr = pReader->GetValue(&pwszValue, NULL)))
														{
															if (!ccs_duration.Compare(pwszLocalName))
															{
																double duration = 0;
																std::wstringstream ss;
																ss << pwszValue;
																ss >> duration;
																m_Duration = duration * 1000 + 5;
															}
														}
													}
												}
												// Here's where I need to dig deeper.
												else if (bIsFormat && (!csLocalName.Compare(_T("tag"))))
												{
													CString csAttributeKey;
													CString csAttributeValue;
													while (S_OK == pReader->MoveToNextAttribute())
													{
														if (SUCCEEDED(hr = pReader->GetLocalName(&pwszLocalName, NULL)))
															if (SUCCEEDED(hr = pReader->GetValue(&pwszValue, NULL)))
														{
															if (!CString(_T("key")).Compare(pwszLocalName))
																csAttributeKey = CString(pwszValue);
															else if (!CString(_T("value")).Compare(pwszLocalName))
																csAttributeValue = CString(pwszValue);
														}
													}
													if (!csAttributeKey.CompareNoCase(_T("title")))
														m_Title = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("episode_id")))
														m_EpisodeTitle = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("description")))
														m_Description = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("WM/SubTitle")))
														m_EpisodeTitle = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("WM/SubTitleDescription")))
														m_Description = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("genre")))
														m_vProgramGenre = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("service_provider")))
														m_SourceStation = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("WM/MediaOriginalChannel")))
														m_SourceChannel = csAttributeValue;
													else if (!csAttributeKey.CompareNoCase(_T("WM/MediaCredits")))
													{
														m_vActor = csAttributeValue;
														while (0 < m_vActor.Replace(_T(";;"),_T(";")));
														while (0 < m_vActor.Replace(_T("//"),_T("/")));
													}
													else if (!csAttributeKey.CompareNoCase(_T("WM/WMRVEncodeTime")))
													{
														CTime OriginalBroadcastDate = ISO8601totime(std::string(CStringA(csAttributeValue).GetString()));
														if (OriginalBroadcastDate > 0)
															m_CaptureDate = OriginalBroadcastDate;
													}
													else if (!csAttributeKey.CompareNoCase(_T("WM/MediaOriginalBroadcastDateTime")))
													{
														CTime OriginalBroadcastDate = ISO8601totime(std::string(CStringA(csAttributeValue).GetString()));
														if (OriginalBroadcastDate > 0)
															m_CaptureDate = OriginalBroadcastDate;
													}
																										m_Description.Trim();
												}
											}
										}
										else if (nodeType == XmlNodeType_EndElement)
										{
											if (SUCCEEDED(hr = pReader->GetLocalName(&pwszLocalName, NULL)))
												if (!CString(pwszLocalName).Compare(_T("format")))
													bIsFormat = false;
										}
									}
								}
							}
						}
					}
					// Close handles to the child process and its primary thread.
					// Some applications might keep these handles to monitor the status
					// of the child process, for example. 
					CloseHandle(piProcInfo.hProcess);
					CloseHandle(piProcInfo.hThread);
				}
			}
			CloseHandle(g_hChildStd_OUT_Rd);
		}
	}
}

WimTiVoServer Processes

I have a TiVoHD, and so have written my program specifically to support the features of the TiVoSeries3 and TiVoHD. The newer TiVoPremier units support more codecs and faster transfer speeds, and the earlier Series2 units do not support the 720p or 1080i resolutions that I allow to be transferred to the TiVo. My initial TiVo server serves files from specific shares on the server, \\server\TiVo\* and \\server\Videos\*, which are specified in the registry key and does not have a user interface to modify them. I recursively parse the shares for files FFMPEG reports as video files and present them as a flattened container sorted by date.
[HKEY_LOCAL_MACHINE\SOFTWARE\WimsWorld\WimTiVoServer]
"Container"="//server/TiVo/*;//server/Videos/*"

The TiVo Server operates using the HTTP protocol. It can also operate using SSL/HTTPS. The TiVo itself uses HTTPS for all of the XML transfers, but transfers its videos using HTTP. The videos the TiVo transfers are encrypted using the Media Access Key (MAK) specific to each TiVo. Videos transferred to the TiVo do not need to be encrypted.

There are multiple ways that your server can announce itself to the TiVo. I’m using the simplest to develop, which is broadcasting a specially formatted UDP packet every minute on the local network to port number 2190. An example packet looks like:
tivoconnect=1
method=broadcast
platform=pc/win-nt
machine=FREDS-PC
identity={D936E980-79E3-11D6-A84A-00045A43EEE7}
services=FooService:1234,BarService:4321

Here is a current broadcast from my server indicating that it will serve TiVo compatible media via the HTTP protocol on port 56667. My program follows the tivo recommendation of using a TCP port supplied by the OS, and attempting to use the same port on subsequent restarts of the server. It also generates the GUID and attempts to save and reuse it.
tivoconnect=1
method=broadcast
platform=pc/WinNT:6.2.9200
machine=Acid
identity={FF121976-2D7B-4682-A1DA-464510DCACEB}
services=TiVoMediaServer:56667/http
swversion=20130604103659

After the TiVo has received a broadcast with the service it is interested in, it will make a request of the server to get the top level containers. http://{machine}/TiVoConnect?Command=QueryContainer&Container=/ The server responds with the XML describing the top level containers.

Here’s an example of the code produced by my server, which is named Acid and has a single top level container with the same name. URLs embedded in the XML can be either relative or absolute. I’ve chosen to always use relative URLs because that way I don’t have to embed the TCP port I’m serving data from in the XML.

<?xml version="1.0" encoding="UTF-8"?>
<TiVoContainer xmlns="http://www.tivo.com/developer/calypso-protocol-1.6/">
  <Details>
    <Title>Acid</Title>
    <ContentType>x-tivo-container/tivo-server</ContentType>
    <SourceFormat>x-tivo-container/folder</SourceFormat>
    <TotalItems>1</TotalItems>
  </Details>
  <Item>
    <Details>
      <Title>Acid</Title>
      <ContentType>x-tivo-container/tivo-videos</ContentType>
      <SourceFormat>x-tivo-container/folder</SourceFormat>
    </Details>
    <Links>
      <Content>
        <Url>/TiVoConnect?Command=QueryContainer&amp;Container=%2FTiVoNowPlaying</Url>
        <ContentType>x-tivo-container/tivo-videos</ContentType>
      </Content>
    </Links>
  </Item>
  <ItemStart>0</ItemStart>
  <ItemCount>1</ItemCount>
</TiVoContainer>

When a user goes to the Now Playing screen on the TiVo, the top level containers should show up in the list at the bottom. Selecting the container will cause the TiVo to request the particular container from the server via its content URL. http://{machine}/TiVoConnect?Command=QueryContainer&Container=/Foo

That container may contain TiVoItems which have URLS for both details and content. The TiVo will only request the content URL after it’s successfully requested the URL for the details, which was something that it took a while for me to understand.

Here is an example of the output when the TiVo requests the container with the URL http://acid:56667/TiVoConnect?Command=QueryContainer&Container=/TiVoNowPlaying&ItemCount=1 :

<?xml version="1.0" encoding="UTF-8"?>
<TiVoContainer xmlns="http://www.tivo.com/developer/calypso-protocol-1.6/">
  <ItemStart>0</ItemStart>
  <ItemCount>1</ItemCount>
  <Details>
    <Title>Acid</Title>
    <ContentType>x-tivo-container/folder</ContentType>
    <SourceFormat>x-tivo-container/folder</SourceFormat>
    <TotalItems>2116</TotalItems>
  </Details>
  <Item>
    <Details>
      <Title>ShowTitle</Title>
      <ContentType>video/x-tivo-mpeg</ContentType>
      <SourceFormat>video/h264</SourceFormat>
      <SourceSize>1805804000</SourceSize>
      <Duration>1289860</Duration>
      <CaptureDate>0x51b0ff42</CaptureDate>
    </Details>
    <Links>
      <Content>
        <ContentType>video/x-tivo-mpeg</ContentType>
        <Url>/TiVoConnect/TivoNowPlaying///Acid/TiVo/ShowTitle.mp4</Url>
      </Content>
      <CustomIcon>
        <ContentType>image/*</ContentType>
        <AcceptsParams>No</AcceptsParams>
        <Url>urn:tivo:image:save-until-i-delete-recording</Url>
      </CustomIcon>
      <TiVoVideoDetails>
        <ContentType>text/xml</ContentType>
        <AcceptsParams>No</AcceptsParams>
        <Url>/TiVoConnect?Command=TVBusQuery&amp;Url=/TiVoConnect/TivoNowPlaying///Acid/TiVo/ShowTitle.mp4</Url>
      </TiVoVideoDetails>
    </Links>
  </Item>
</TiVoContainer>

When I select “ShowTitle” in the tivo onscreen interface, it will request this same URL, with the anchoritem set to the content URL. It will then display the details about the show on the TV screen. After I select that I want the show to be transferred it first requests an undocumented URL, my server responds that the URL was not found, then the TiVo requests the TiVoVideoDetails URL, and I respond with an XML chunk that I couldn’t find documentation for anywhere and had to fake together from looking at the output of TiVoDecode and pyTiVo. The really ugly bit of the XML is in the initial element declaration. I wasn’t able to make the XML be accepted by the TiVo if it didn’t have the full namespace declarations, even though the hosts listed are not accessable.

<?xml version="1.0" encoding="UTF-8"?>
<TvBusMarshalledStruct:TvBusEnvelope xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xmlns:TvBusMarshalledStruct="http://tivo.com/developer/xml/idl/TvBusMarshalledStruct" xmlns:TvPgdRecording="http://tivo.com/developer/xml/idl/TvPgdRecording" xmlns:TvBusDuration="http://tivo.com/developer/xml/idl/TvBusDuration" xmlns:TvPgdShowing="http://tivo.com/developer/xml/idl/TvPgdShowing" xmlns:TvDbShowingBit="http://tivo.com/developer/xml/idl/TvDbShowingBit" xmlns:TvBusDateTime="http://tivo.com/developer/xml/idl/TvBusDateTime" xmlns:TvPgdProgram="http://tivo.com/developer/xml/idl/TvPgdProgram" xmlns:TvDbColorCode="http://tivo.com/developer/xml/idl/TvDbColorCode" xmlns:TvPgdSeries="http://tivo.com/developer/xml/idl/TvPgdSeries" xmlns:TvDbShowType="http://tivo.com/developer/xml/idl/TvDbShowType" xmlns:TvPgdBookmark="http://tivo.com/developer/xml/idl/TvPgdBookmark" xmlns:TvPgdChannel="http://tivo.com/developer/xml/idl/TvPgdChannel" xmlns:TvDbBitstreamFormat="http://tivo.com/developer/xml/idl/TvDbBitstreamFormat" xs:schemaLocation="http://tivo.com/developer/xml/idl/TvBusMarshalledStruct TvBusMarshalledStruct.xsd http://tivo.com/developer/xml/idl/TvPgdRecording TvPgdRecording.xsd http://tivo.com/developer/xml/idl/TvBusDuration TvBusDuration.xsd http://tivo.com/developer/xml/idl/TvPgdShowing TvPgdShowing.xsd http://tivo.com/developer/xml/idl/TvDbShowingBit TvDbShowingBit.xsd http://tivo.com/developer/xml/idl/TvBusDateTime TvBusDateTime.xsd http://tivo.com/developer/xml/idl/TvPgdProgram TvPgdProgram.xsd http://tivo.com/developer/xml/idl/TvDbColorCode TvDbColorCode.xsd http://tivo.com/developer/xml/idl/TvPgdSeries TvPgdSeries.xsd http://tivo.com/developer/xml/idl/TvDbShowType TvDbShowType.xsd http://tivo.com/developer/xml/idl/TvPgdBookmark TvPgdBookmark.xsd http://tivo.com/developer/xml/idl/TvPgdChannel TvPgdChannel.xsd http://tivo.com/developer/xml/idl/TvDbBitstreamFormat TvDbBitstreamFormat.xsd" xs:type="TvPgdRecording:TvPgdRecording">
  <recordedDuration>PT21M29S</recordedDuration>
  <vActualShowing />
  <vBookmark />
  <recordingQuality value="75">HIGH</recordingQuality>
  <showing>
    <showingBits value="0" />
    <time>2013-06-06:21:29:38Z</time>
    <duration>PT21M29S</duration>
    <program>
      <vActor />
      <vAdvisory />
      <vChoreographer />
      <colorCode value="4">COLOR</colorCode>
      <description></description>
      <vDirector />
      <episodeTitle></episodeTitle>
      <vExecProducer />
      <vProgramGenre />
      <vGuestStar />
      <vHost />
      <isEpisode>false</isEpisode>
      <originalAirDate>2013-06-06:21:29:38Z</originalAirDate>
      <vProducer />
      <series>
        <isEpisodic>false</isEpisodic>
        <vSeriesGenre />
        <seriesTitle>ShowTitle</seriesTitle>
      </series>
      <showType value="5">SERIES</showType>
      <title>ShowTitle</title>
      <vWriter />
    </program>
    <channel>
      <displayMajorNumber />
      <displayMinorNumber />
      <callsign />
    </channel>
  </showing>
  <startTime>2013-06-06:21:29:38Z</startTime>
  <stopTime>2013-06-06:21:51:07Z</stopTime>
</TvBusMarshalledStruct:TvBusEnvelope>

After the TvBusMarshalledStruct XML has been transferred to the TiVo the tivo will finally request the content url from the original container listing. When the content is sent, it should be sent using HTTP Chunked Encoding.

My current ffmpeg command line is fairly simple, and mostly copied from my reading of the pyTiVo source code. I make a couple of simple decisions based on the source file codecs. If the video is mpeg2video or the audio is ac3 I use the “-vcodec copy” or “-acodec copy” command,
otherwise I specify those as codecs. I also include the options “-b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -f vob” which I need to spend more time learning about.

One of the gotchas that I spent a significant amount of time debugging was that the SourceSize element in the Item listing is significant. It appears that the tivo pre-allocates space for the file as it starts the transfer and if the transfer is proceeding significantly larger than the reported source size the tivo will close its end of the http connection. I have not verified importance in any internal part of the TvBusMarshalledStruct. I got it working, and have been adding support from metadata as available but that’s it currently.

It’s interesting to me that the transfer speed seems completely governed by the speed of the TiVo processor and what the TiVo is doing. The maximum speed I’ve been able to transfer a file to the TiVo is about 16Mb/s. That speed was achieved while the TiVo was not recording any shows and had a 480p program displayed on the screen in a paused state. If I’m playing a 1080i or 720p program the transfer rate is usually much closer to 7Mb/s. This means that if I’m transferring a program that was recorded in HD on my windows media center from my HDHomeRunPrime, I generally cannot watch in real time because it transfers slightly slower than the data rate of the video and audio.

WimTiVoServer

With the lack of movement on TiVo Desktop support for Windows 8, and several other issues I’ve had I decided to write my own implementation of a TiVo Server. I’d first investigated what else was out there. The primary server is pyTiVo, a python and FFMPEG based tivo server. It has had significant work into it, and supports both push and pull methods for transferring files to TiVos. My issue with it is that it requires the installation of python, which is one more environment that I don’t really want to install and maintain on my windows server if I don’t need to.

I used to run a TiVo Publisher Add-In for my windows home server. http://durfee.net/software/2007/07/tivo-publisher-for-whs.html It hasn’t been updated in over 5 years, and didn’t handle newer codecs or install on newer servers. I’ve upgraded my server to a much newer server and would like a similar service to run on my server, so that files on the server can be directly transcoded to the TiVo without requiring a running desktop machine, or a logged in console.

In the decision to write my own software, I was reminded that the reason I’ve not done it in the past is that the information on the TiVo protocol is so hard to come by, plus it has some confusing issues.  The TiVo Server protocol is referred to as TiVo Home Media Option, or HMO Server.  The documents from TiVo are sometimes difficult to be found, but a copy is located at http://www.tivocommunity.com/tivo-vb/showthread.php?p=5834238#post5834238

The fact that TiVo doesn’t host the documents in a standardized location indicates something about the withering support for the developer community by the company.

After starting to type some of this information up, I’ve realized how much information I’ve collected and distilled down to a small running program. Because of that I plan on breaking my thoughts into several posts.