New FFMPEG install on BeagleBone Black

There was a 1.0 release of FFMPEG that came out in December of 2012. It moved the bar forward significantly in working with video formats. The Angstrom distribution is still including a 0.8 version which doesn’t even have direct support for h264.

My solution was to use “git” to download the latest code image for both ffmpeg and libx264, and build the libraries. The interesting complications I had were removing the old libraries from the machine, because they caused ffmpeg not to build correctly. Because I do not plan on using thwe machine interactively, I don’t care that I had to remove gimp. I don’t know what the other dependencies I removed may be related to.

I repeatedly issued the command “opkg remove –force-removal-of-dependent-packages libav” until no packages were removed.

I setup a usb drive partition, and a second user. As the second user on that drive partition I issued the following commands.

git clone git://source.ffmpeg.org/ffmpeg.git
git clone git://git.videolan.org/x264.git
cd x264/
./configure --enable-static --enable-shared
date ; make ; date

I switched to the root user, changed to the x264/ directory and

make install
ldconfig

back to the second user

cd ../ffmpeg/
./configure --enable-gpl --enable-libx26
date ; make ; date

and as root

make install
ldconfig

I had a problem because the default installation for libx264 puts the libraries in /usr/local/lib. I had to add that line to the file /etc/ld.so.conf before running ffmpeg.

root@beaglebone:/usr/local/lib# ffmpeg -?
ffmpeg: error while loading shared libraries: libx264.so.133: cannot open shared object file: No such file or directory

I ran the make commands surrounded by the date commands so that I’d be able to get an estimate on time required. With the BeagleBone running at 1GHz, libx264 gets built in slightly less than 10 minutes and FFMPEG takes 1 hour and 40 minutes.

I was left with an ffmpeg that I wanted to use for my video streaming project.

ffmpeg version N-54212-g034b31d Copyright (c) 2000-2013 the FFmpeg developers
  built on Jun 27 2013 04:05:20 with gcc 4.7.3 (Linaro GCC 4.7-2013.02-01) 20130205 (prerelease)
  configuration: --enable-gpl --enable-libx264
  libavutil      52. 37.101 / 52. 37.101
  libavcodec     55. 17.100 / 55. 17.100
  libavformat    55. 10.100 / 55. 10.100
  libavdevice    55.  2.100 / 55.  2.100
  libavfilter     3. 77.101 /  3. 77.101
  libswscale      2.  3.100 /  2.  3.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

BeagleBoneBlack WiFi configuration problems

I got a BeagleBone Black last week. It seems to be a nice system, with a 1GHz ARM7 processor, ethernet, USB, HDMI and plenty of expansion possibilities, all for $45.

It’s shipping configuration runs the Angstrom Linux distribution. I’ve worked with Angstrom in the past on a Beagleboard used for an embedded application. It seems that the newer Angstrom is using ConnMan 1.4 as the network manager. The version that’s available at the ConnMan site, 1.7, has a command line configuration tool, while the version included in the distribution does not.

I’ve not been able to figure out how to enable wireless networking. Plugging in an ethernet cable just works. Plugging in the USB WiFi adapter gives me a wlan0 device, and the MAC matches that printed on the device.

I’m not able to issue the command “ifup wlan0” without an error. It looks like the connman settings file should enable WiFi by default.

root@beaglebone:~# ifup wlan0
ifup: can't open '/etc/network/interfaces': No such file or directory
root@beaglebone:~# ifconfig wlan0 up
ifconfig: SIOCSIFFLAGS: No such file or directory
root@beaglebone:~# lsusb
Bus 001 Device 002: ID 04b4:6560 Cypress Semiconductor Corp. CY7C65640 USB-2.0 "TetraHub"
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 13b1:002f Linksys AE1000 v1 802.11n [Ralink RT3572]
root@beaglebone:~# ls -alFR /var/lib/connman/
/var/lib/connman/:
total 16
drwxr-xr-x  3 root root 4096 Jan  1  2000 ./
drwxr-xr-x 17 root root 4096 Jan  1  2000 ../
drwx------  2 root root 4096 Jan  1  2000 ethernet_c8a030a62b80_cable/
-rw-------  1 root root   68 Jan  1  2000 settings

/var/lib/connman/ethernet_c8a030a62b80_cable:
total 16
drwx------ 2 root root 4096 Jan  1  2000 ./
drwxr-xr-x 3 root root 4096 Jan  1  2000 ../
-rw------- 1 root root 4096 Jan  1  2000 data
-rw------- 1 root root  186 Jan  1  2000 settings
root@beaglebone:~# cat /var/lib/connman/settings 
[global]
OfflineMode=false

[Wired]
Enable=true

[WiFi]
Enable=true

I went so far as to connect to an HDMI monitor with keyboard and mouse and was able to see the graphical connection manager. I tried both with and without having the ethernet cable plugged in, but was not able to click on the “Enable” button on the wireless networks dialog.

BeagleBoneWiFi

I believe that I need to do something with wpa_supplicant to get the password properly accepted on my machine, but I’m more interested in getting the wireless up and running than worrying about getting the security set.

I included the lsusb command in my code listing before because it lists the WiFi device that I’m using. I’m powering the device by a standalone power supply, not via USB, so I believe that I should have enough power to run the wifi. This connection has it connected via an external hub, but I’ve had the same results when connected directly to the board.

WimTiVoServer Processes

I have a TiVoHD, and so have written my program specifically to support the features of the TiVoSeries3 and TiVoHD. The newer TiVoPremier units support more codecs and faster transfer speeds, and the earlier Series2 units do not support the 720p or 1080i resolutions that I allow to be transferred to the TiVo. My initial TiVo server serves files from specific shares on the server, \\server\TiVo\* and \\server\Videos\*, which are specified in the registry key and does not have a user interface to modify them. I recursively parse the shares for files FFMPEG reports as video files and present them as a flattened container sorted by date.
[HKEY_LOCAL_MACHINE\SOFTWARE\WimsWorld\WimTiVoServer]
"Container"="//server/TiVo/*;//server/Videos/*"

The TiVo Server operates using the HTTP protocol. It can also operate using SSL/HTTPS. The TiVo itself uses HTTPS for all of the XML transfers, but transfers its videos using HTTP. The videos the TiVo transfers are encrypted using the Media Access Key (MAK) specific to each TiVo. Videos transferred to the TiVo do not need to be encrypted.

There are multiple ways that your server can announce itself to the TiVo. I’m using the simplest to develop, which is broadcasting a specially formatted UDP packet every minute on the local network to port number 2190. An example packet looks like:
tivoconnect=1
method=broadcast
platform=pc/win-nt
machine=FREDS-PC
identity={D936E980-79E3-11D6-A84A-00045A43EEE7}
services=FooService:1234,BarService:4321

Here is a current broadcast from my server indicating that it will serve TiVo compatible media via the HTTP protocol on port 56667. My program follows the tivo recommendation of using a TCP port supplied by the OS, and attempting to use the same port on subsequent restarts of the server. It also generates the GUID and attempts to save and reuse it.
tivoconnect=1
method=broadcast
platform=pc/WinNT:6.2.9200
machine=Acid
identity={FF121976-2D7B-4682-A1DA-464510DCACEB}
services=TiVoMediaServer:56667/http
swversion=20130604103659

After the TiVo has received a broadcast with the service it is interested in, it will make a request of the server to get the top level containers. http://{machine}/TiVoConnect?Command=QueryContainer&Container=/ The server responds with the XML describing the top level containers.

Here’s an example of the code produced by my server, which is named Acid and has a single top level container with the same name. URLs embedded in the XML can be either relative or absolute. I’ve chosen to always use relative URLs because that way I don’t have to embed the TCP port I’m serving data from in the XML.

<?xml version="1.0" encoding="UTF-8"?>
<TiVoContainer xmlns="http://www.tivo.com/developer/calypso-protocol-1.6/">
  <Details>
    <Title>Acid</Title>
    <ContentType>x-tivo-container/tivo-server</ContentType>
    <SourceFormat>x-tivo-container/folder</SourceFormat>
    <TotalItems>1</TotalItems>
  </Details>
  <Item>
    <Details>
      <Title>Acid</Title>
      <ContentType>x-tivo-container/tivo-videos</ContentType>
      <SourceFormat>x-tivo-container/folder</SourceFormat>
    </Details>
    <Links>
      <Content>
        <Url>/TiVoConnect?Command=QueryContainer&amp;Container=%2FTiVoNowPlaying</Url>
        <ContentType>x-tivo-container/tivo-videos</ContentType>
      </Content>
    </Links>
  </Item>
  <ItemStart>0</ItemStart>
  <ItemCount>1</ItemCount>
</TiVoContainer>

When a user goes to the Now Playing screen on the TiVo, the top level containers should show up in the list at the bottom. Selecting the container will cause the TiVo to request the particular container from the server via its content URL. http://{machine}/TiVoConnect?Command=QueryContainer&Container=/Foo

That container may contain TiVoItems which have URLS for both details and content. The TiVo will only request the content URL after it’s successfully requested the URL for the details, which was something that it took a while for me to understand.

Here is an example of the output when the TiVo requests the container with the URL http://acid:56667/TiVoConnect?Command=QueryContainer&Container=/TiVoNowPlaying&ItemCount=1 :

<?xml version="1.0" encoding="UTF-8"?>
<TiVoContainer xmlns="http://www.tivo.com/developer/calypso-protocol-1.6/">
  <ItemStart>0</ItemStart>
  <ItemCount>1</ItemCount>
  <Details>
    <Title>Acid</Title>
    <ContentType>x-tivo-container/folder</ContentType>
    <SourceFormat>x-tivo-container/folder</SourceFormat>
    <TotalItems>2116</TotalItems>
  </Details>
  <Item>
    <Details>
      <Title>ShowTitle</Title>
      <ContentType>video/x-tivo-mpeg</ContentType>
      <SourceFormat>video/h264</SourceFormat>
      <SourceSize>1805804000</SourceSize>
      <Duration>1289860</Duration>
      <CaptureDate>0x51b0ff42</CaptureDate>
    </Details>
    <Links>
      <Content>
        <ContentType>video/x-tivo-mpeg</ContentType>
        <Url>/TiVoConnect/TivoNowPlaying///Acid/TiVo/ShowTitle.mp4</Url>
      </Content>
      <CustomIcon>
        <ContentType>image/*</ContentType>
        <AcceptsParams>No</AcceptsParams>
        <Url>urn:tivo:image:save-until-i-delete-recording</Url>
      </CustomIcon>
      <TiVoVideoDetails>
        <ContentType>text/xml</ContentType>
        <AcceptsParams>No</AcceptsParams>
        <Url>/TiVoConnect?Command=TVBusQuery&amp;Url=/TiVoConnect/TivoNowPlaying///Acid/TiVo/ShowTitle.mp4</Url>
      </TiVoVideoDetails>
    </Links>
  </Item>
</TiVoContainer>

When I select “ShowTitle” in the tivo onscreen interface, it will request this same URL, with the anchoritem set to the content URL. It will then display the details about the show on the TV screen. After I select that I want the show to be transferred it first requests an undocumented URL, my server responds that the URL was not found, then the TiVo requests the TiVoVideoDetails URL, and I respond with an XML chunk that I couldn’t find documentation for anywhere and had to fake together from looking at the output of TiVoDecode and pyTiVo. The really ugly bit of the XML is in the initial element declaration. I wasn’t able to make the XML be accepted by the TiVo if it didn’t have the full namespace declarations, even though the hosts listed are not accessable.

<?xml version="1.0" encoding="UTF-8"?>
<TvBusMarshalledStruct:TvBusEnvelope xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xmlns:TvBusMarshalledStruct="http://tivo.com/developer/xml/idl/TvBusMarshalledStruct" xmlns:TvPgdRecording="http://tivo.com/developer/xml/idl/TvPgdRecording" xmlns:TvBusDuration="http://tivo.com/developer/xml/idl/TvBusDuration" xmlns:TvPgdShowing="http://tivo.com/developer/xml/idl/TvPgdShowing" xmlns:TvDbShowingBit="http://tivo.com/developer/xml/idl/TvDbShowingBit" xmlns:TvBusDateTime="http://tivo.com/developer/xml/idl/TvBusDateTime" xmlns:TvPgdProgram="http://tivo.com/developer/xml/idl/TvPgdProgram" xmlns:TvDbColorCode="http://tivo.com/developer/xml/idl/TvDbColorCode" xmlns:TvPgdSeries="http://tivo.com/developer/xml/idl/TvPgdSeries" xmlns:TvDbShowType="http://tivo.com/developer/xml/idl/TvDbShowType" xmlns:TvPgdBookmark="http://tivo.com/developer/xml/idl/TvPgdBookmark" xmlns:TvPgdChannel="http://tivo.com/developer/xml/idl/TvPgdChannel" xmlns:TvDbBitstreamFormat="http://tivo.com/developer/xml/idl/TvDbBitstreamFormat" xs:schemaLocation="http://tivo.com/developer/xml/idl/TvBusMarshalledStruct TvBusMarshalledStruct.xsd http://tivo.com/developer/xml/idl/TvPgdRecording TvPgdRecording.xsd http://tivo.com/developer/xml/idl/TvBusDuration TvBusDuration.xsd http://tivo.com/developer/xml/idl/TvPgdShowing TvPgdShowing.xsd http://tivo.com/developer/xml/idl/TvDbShowingBit TvDbShowingBit.xsd http://tivo.com/developer/xml/idl/TvBusDateTime TvBusDateTime.xsd http://tivo.com/developer/xml/idl/TvPgdProgram TvPgdProgram.xsd http://tivo.com/developer/xml/idl/TvDbColorCode TvDbColorCode.xsd http://tivo.com/developer/xml/idl/TvPgdSeries TvPgdSeries.xsd http://tivo.com/developer/xml/idl/TvDbShowType TvDbShowType.xsd http://tivo.com/developer/xml/idl/TvPgdBookmark TvPgdBookmark.xsd http://tivo.com/developer/xml/idl/TvPgdChannel TvPgdChannel.xsd http://tivo.com/developer/xml/idl/TvDbBitstreamFormat TvDbBitstreamFormat.xsd" xs:type="TvPgdRecording:TvPgdRecording">
  <recordedDuration>PT21M29S</recordedDuration>
  <vActualShowing />
  <vBookmark />
  <recordingQuality value="75">HIGH</recordingQuality>
  <showing>
    <showingBits value="0" />
    <time>2013-06-06:21:29:38Z</time>
    <duration>PT21M29S</duration>
    <program>
      <vActor />
      <vAdvisory />
      <vChoreographer />
      <colorCode value="4">COLOR</colorCode>
      <description></description>
      <vDirector />
      <episodeTitle></episodeTitle>
      <vExecProducer />
      <vProgramGenre />
      <vGuestStar />
      <vHost />
      <isEpisode>false</isEpisode>
      <originalAirDate>2013-06-06:21:29:38Z</originalAirDate>
      <vProducer />
      <series>
        <isEpisodic>false</isEpisodic>
        <vSeriesGenre />
        <seriesTitle>ShowTitle</seriesTitle>
      </series>
      <showType value="5">SERIES</showType>
      <title>ShowTitle</title>
      <vWriter />
    </program>
    <channel>
      <displayMajorNumber />
      <displayMinorNumber />
      <callsign />
    </channel>
  </showing>
  <startTime>2013-06-06:21:29:38Z</startTime>
  <stopTime>2013-06-06:21:51:07Z</stopTime>
</TvBusMarshalledStruct:TvBusEnvelope>

After the TvBusMarshalledStruct XML has been transferred to the TiVo the tivo will finally request the content url from the original container listing. When the content is sent, it should be sent using HTTP Chunked Encoding.

My current ffmpeg command line is fairly simple, and mostly copied from my reading of the pyTiVo source code. I make a couple of simple decisions based on the source file codecs. If the video is mpeg2video or the audio is ac3 I use the “-vcodec copy” or “-acodec copy” command,
otherwise I specify those as codecs. I also include the options “-b:v 16384k -maxrate 30000k -bufsize 4096k -ab 448k -ar 48000 -f vob” which I need to spend more time learning about.

One of the gotchas that I spent a significant amount of time debugging was that the SourceSize element in the Item listing is significant. It appears that the tivo pre-allocates space for the file as it starts the transfer and if the transfer is proceeding significantly larger than the reported source size the tivo will close its end of the http connection. I have not verified importance in any internal part of the TvBusMarshalledStruct. I got it working, and have been adding support from metadata as available but that’s it currently.

It’s interesting to me that the transfer speed seems completely governed by the speed of the TiVo processor and what the TiVo is doing. The maximum speed I’ve been able to transfer a file to the TiVo is about 16Mb/s. That speed was achieved while the TiVo was not recording any shows and had a 480p program displayed on the screen in a paused state. If I’m playing a 1080i or 720p program the transfer rate is usually much closer to 7Mb/s. This means that if I’m transferring a program that was recorded in HD on my windows media center from my HDHomeRunPrime, I generally cannot watch in real time because it transfers slightly slower than the data rate of the video and audio.

WimTiVoServer reasons for existence.

I had multiple reasons for writing this program.

I wanted to work with the XMLLite library for XML processing which Microsoft has included as part of their operating system for several years. http://msdn.microsoft.com/en-us/library/windows/desktop/ms752838(v=vs.85).aspx I’ve used XML as my data storage and transfer medium for years but most of the time have manually processed the XML as opposed to using a prebuilt library. My reasoning behind my own parsing and generation was that I was building cross platform solutions that ran on embedded systems working with windows programs. It was easier to build the same code to run on both platforms and assure compatibility than to deal with the overhead and maintenance of third party libraries. I’ve still not formed a full opinion on using the library for reading XML, but it certainly makes creating XML easier by making sure tags are properly closed and nested, and the option of auto indentation in the formatting is nice.

I wanted experience using FFMPEG and its associated video processing libraries. With the release of FFMPEG v1 late last year it became much more capable in dealing with all of the container formats and encoding types that I was interested in, including the WTV containers that Windows Media Center uses and the MKV containers that are common on the internet for high definition files. In my functioning version of my server, I’m using the libraries directly linked into the program to parse the media files for metadata, but spawning a full copy of FFMPEG to do the required transcoding to send the proper format to the TiVo. I’m considering migrating entirely to the spawned FFMPEG process to simplify licensing if I want to make my program publicly available. It would also simplify future support for codecs and containers that FFMPEG may support if my service itself didn’t need to be relinked with updated libraries.

I’ve been frustrated with the state of the TiVo Desktop software provided by the TiVo company. It was designed so that it plugs into the windows video codec stack for video transcoding, as well as the apple protocol stack for Bonjour protocol support. Both of those lead to complications when upgrading programs. Apple regularly releases updates to iTunes. Whenever an update to iTunes needed to be installed, it caused problems because the TiVo service was running and keeping some of the apple services locked in use. It essentially required a full uninstall and reinstall of iTunes every time I needed to update it, with several machine reboots in that process.  Somehow I’d managed to get a codec installed on my machine that allowed the TiVo desktop to support the MKV container. Duplicating that installation on a new machine or server was not something I wanted to attempt. Since the modern FFMPEG supports MKV, I get that support without manipulating a video codec stack in windows.

The TiVo Desktop software only runs as a user process, when a user is logged in. Files are not served from a background service. This is an issue when I’d like to run the process on a headless server. There are ways around the issue, but my program solves it simply by being a pure service. The TiVo Desktop both announces itself via the UDP beacon process on port 2190 and also listens for other servers on the same port. Because my program is purely a server I saw no reason to listen for incoming beacons, and so I do not tie up the UDP port for receiving. This allows secondary programs to be started at the users discretion to listen for TiVo processes on the network.

WimTiVoServer

With the lack of movement on TiVo Desktop support for Windows 8, and several other issues I’ve had I decided to write my own implementation of a TiVo Server. I’d first investigated what else was out there. The primary server is pyTiVo, a python and FFMPEG based tivo server. It has had significant work into it, and supports both push and pull methods for transferring files to TiVos. My issue with it is that it requires the installation of python, which is one more environment that I don’t really want to install and maintain on my windows server if I don’t need to.

I used to run a TiVo Publisher Add-In for my windows home server. http://durfee.net/software/2007/07/tivo-publisher-for-whs.html It hasn’t been updated in over 5 years, and didn’t handle newer codecs or install on newer servers. I’ve upgraded my server to a much newer server and would like a similar service to run on my server, so that files on the server can be directly transcoded to the TiVo without requiring a running desktop machine, or a logged in console.

In the decision to write my own software, I was reminded that the reason I’ve not done it in the past is that the information on the TiVo protocol is so hard to come by, plus it has some confusing issues.  The TiVo Server protocol is referred to as TiVo Home Media Option, or HMO Server.  The documents from TiVo are sometimes difficult to be found, but a copy is located at http://www.tivocommunity.com/tivo-vb/showthread.php?p=5834238#post5834238

The fact that TiVo doesn’t host the documents in a standardized location indicates something about the withering support for the developer community by the company.

After starting to type some of this information up, I’ve realized how much information I’ve collected and distilled down to a small running program. Because of that I plan on breaking my thoughts into several posts.