Monthly Archives: August 2013

Researching DVD Subtitle Format

by

I am attempting to stream webcam video from a BeagleBoneBlack to other computers over ethernet. I want to add an overlay with details about the video. I am capturing video from a Logitech C920 webcam, which is doing the hard work of creating the video on H.264 format, using FFMPEG to MUX the video into a network stream. The current video stream runs at 3Mb/s over ethernet, and seems to run at the same bitrate whether I’m sending video 30FPS at 1920×1080, 1280×720, or any other resolution I’ve tried. If I’m running the BBB at 1GHz FFMPEG uses only 3% load on the processor, while at 300MHz it uses 10% load. Either processor speed indicates that I should have plenty of CPU available for creating a subtitle frame a second.

If I transcode the H264 coming from the C920 to h.264 from FFMPEG the BBB CPU is 100% used and I’ve not been able to get over 5 FPS. This has led me to the idea of adding a second stream with much more compressible data and requiring the client computer to know how to enable subtitles.

My understanding of DVD Subtitles is that they are stored as image overlays. The images seem to be 3 color plus transparency, with the color indexed. They are RLE (Run Length Encoded) images but don’t seem to conform to any standard that would be created by an image library such as OpenCV.

The most useful links I’ve come across related to the DVD subtitles are these three:

Using FFMPEG to examine at a video that was ripped from a DVD into an MKV file with several subtitle layers shows the following:

Stream #0:0(eng): Video: mpeg2video (Main), yuv420p, 720x480 [SAR 32:27 DAR 16:9], SAR 186:157 DAR 279:157, 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc
Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s (default)
Metadata:
  title           : 3/2+1
Stream #0:2(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 384 kb/s
Metadata:
  title           : 3/2+1
Stream #0:3(spa): Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s
Metadata:
  title           : 2/0
Stream #0:4(fre): Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s
Metadata:
  title           : 2/0
Stream #0:5(eng): Subtitle: dvd_subtitle (default)
Stream #0:6(spa): Subtitle: dvd_subtitle
Stream #0:7(eng): Subtitle: dvd_subtitle
Stream #0:8(spa): Subtitle: dvd_subtitle
Stream #0:9(fre): Subtitle: dvd_subtitle

All of the descriptions of creating subtitle tracks are directly related to creating textual subtitles using tools that are wonderful for mainstream movie content but not what I want to do. e.g.

I’ve not figured out how to create my own subtitle stream and am still looking for information on that. I’ve not figured out what parameters may need to be passed to FFMPEG to indicate that I’m passing in a subtitle track. I’ve not figured out if there’s a way in FFMPEG to indicate that the subtitles should be on by default, or forced subtitles, while still keeping them as a separate stream.

It doesn’t help that the DVD subtitle files seem to use the STL extension and that same extension is used for the input files for many 3D Printers.

BeagleBoneBlack Webcam using a V4L2, FFMPEG and a FIFO

by

In my previous post I was starting FFMPEG with the command to read from standard input using a pipe. That’s a great method when you have a single input you want to supply to FFMPEG, but my ultimate goal is to supply multiple streams to FFMPEG and have it multiplex them into a single transport stream.

I believe that my end solution for this is going to be using several named pipes, which are First In First Out devices created by the command mkfifo() in linux.

The following code snippet functions nearly identically to the snippet in my previous post except that it creates a FIFO in the /tmp/ directory and tells FFMPEG to read from the FIFO instead of from stdin. 

if (0 != mkfifo("/tmp/Wim_C920_FFMPEG", S_IRWXU))
	std::cerr << "[" << getTimeISO8601() << "] FiFo /tmp/Wim_C920_FFMPEG NOT created Successfully" << std::endl;
else
	std::cerr << "[" << getTimeISO8601() << "] FiFo /tmp/Wim_C920_FFMPEG created Successfully" << std::endl;
/* Attempt to fork */
pid_t pid = fork();
if(pid == -1)
{
	fprintf(stderr,"Fork error! Exiting.\n");  /* something went wrong */
	exit(1);        
}
else if (pid == 0)
{
	/* A zero PID indicates that this is the child process */
	/* Replace the child fork with a new process */
	if(execlp("ffmpeg", "ffmpeg", "-i", "/tmp/Wim_C920_FFMPEG", "-vcodec", "copy", "-f", "rtp", "rtp://239.8.8.8:8090/", "-metadata", "title=\"Wims BoneCam\"", "-vcodec", "copy", OutputFilename.c_str(), NULL) == -1)
	{
		std::cerr << "execlp Error! Exiting." << std::endl;
		exit(1);
	}
}
else
{
	/* A positive (non-negative) PID indicates the parent process */
	open_device();
	init_device();
	start_capturing();
	//mainloop();
	int pipe_fd = open("/tmp/Wim_C920_FFMPEG", O_WRONLY);
	if (pipe_fd <= 0)
		std::cerr << "[" << getTimeISO8601() << "] FiFo NOT opened Successfully" << std::endl;
	else
	{
		std::cerr << "[" << getTimeISO8601() << "] FiFo opened Successfully" << std::endl;
		time_t CurrentTime;
		time(&CurrentTime);
		time_t FileStartTime = CurrentTime;
		while ((difftime(CurrentTime, FileStartTime) < (60 * 60)) && (bRun))
		{
			fd_set fds;
			struct timeval tv;
			int r;

			FD_ZERO(&fds);
			FD_SET(fd, &fds);

			/* Timeout. */
			tv.tv_sec = 2;
			tv.tv_usec = 0;

			r = select(fd + 1, &fds, NULL, NULL, &tv);
			if (-1 == r) 
			{
				if (EINTR == errno)
					continue;
				errno_exit("select");
			}
			if (0 == r) 
			{
				fprintf(stderr, "select timeout\n");
				exit(EXIT_FAILURE);
			}
			struct v4l2_buffer buf;
			WIMLABEL:
			CLEAR(buf);
			buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
			buf.memory = V4L2_MEMORY_MMAP;

			if (-1 == xioctl(fd, VIDIOC_DQBUF, &buf)) 
			{
				switch (errno) 
				{
				case EAGAIN:
					goto WIMLABEL;
				case EIO:
					/* Could ignore EIO, see spec. */
					/* fall through */
				default:
					errno_exit("VIDIOC_DQBUF");
				}
			}
			else
			{
				assert(buf.index < n_buffers);
				write(pipe_fd, buffers[buf.index].start, buf.bytesused);
			}
			if (-1 == xioctl(fd, VIDIOC_QBUF, &buf))
					errno_exit("VIDIOC_QBUF");
			time(&CurrentTime);
		}
		stop_capturing();
		uninit_device();
		close_device();
		close(pipe_fd);		/* Close side of pipe I'm writing to, to get the child to recognize it's gone away */
		std::cerr << "\n[" << getTimeISO8601() << "] Pipe Closed, Waiting for FFMPEG to exit" << std::endl;
	}
	int ffmpeg_exit_status;
	wait(&ffmpeg_exit_status);				/* Wait for child process to end */
	std::cerr << "[" << getTimeISO8601() << "] FFMPEG exited with a  " << ffmpeg_exit_status << " value" << std::endl;
	if (0 == remove("/tmp/Wim_C920_FFMPEG"))
		std::cerr << "[" << getTimeISO8601() << "] FiFo /tmp/Wim_C920_FFMPEG removed Successfully" << std::endl;
	else
		std::cerr << "[" << getTimeISO8601() << "] FiFo /tmp/Wim_C920_FFMPEG NOT removed Successfully" << std::endl;
}

An interesting side effect is that ffmpeg now can read from stdin for arbitrary commands, as if it had been run from the linux command line, and so the "q" key will cause it to quit running. I've not fully investigated this behavior to see what happens to the parent program when the fifo consumer suddenly quits. Initially, it seems that the fifo supplier quits as well.

A bad thing about going with named pipes is that there should be a way of ensuring that the names used are unique and don't conflict with anything else on the system. They should also be cleaned up at the end of the process. My current method has very little error correction and should not be used as an example beyond to get started.

BeagleBoneBlack Webcam using a V4L2, FFMPEG and a Pipe

by

I’ve been working on streaming video from my BeagleBoneBlack (BBB) over WiFi and also keeping a copy of the video on the local flash. My starting point for capturing the video using Video For Linux 2 (V4L2) came from demonstration code from Derek Molloy’s site. He has some very well done instructional videos that are much more accessible and complete than most of what I’ve put together.

I am doing most of my development in C/C++ simply because it’s my strongest language. I’m doing the development on the BBB under linux which means that some of the skills I’m using are fairly rusty.

Because I wanted to have the program pulling the video from the camera in charge of the entire process, including managing which local output files are created, and making sure that there is disk storage space, I decided to use pipe(), fork(), and exec(), to manage ffmpeg as a child process of my program.

I started with the same demonstration program from V4L2 and then modified it so that instead of writing out to stdout, it’s writing to a file descriptor I supply. This all happens inside of the primary loop that generates the outputfilename for the ffmpeg command line and makes sure that there is enough free storage for an hours worth of video. Here’s the main code snippet. 

int	pipefd[2];		/* This holds the fd for the input & output of the pipe */
/* Setup communication pipeline first */
if(pipe(pipefd))
{
	std::cerr << "[" << getTimeISO8601() << "] Pipe error! Exiting." << std::endl;
	exit(1);
}
/* Attempt to fork */
pid_t pid = fork();
if(pid == -1)
{
	std::cerr << "[" << getTimeISO8601() << "] Fork error! Exiting." << std::endl; /* something went wrong */
	exit(1);        
}
else if (pid == 0)
{
	/* A zero PID indicates that this is the child process */
	dup2(pipefd[0], 0);	/* Replace stdin with the in side of the pipe */
	close(pipefd[1]);	/* Close unused side of pipe (out side) */
	/* Replace the child fork with a new process */
	if(execlp("ffmpeg", "ffmpeg", "-i", "-", "-vcodec", "copy", "-f", "rtp", "rtp://239.8.8.8:8090/", "-metadata", "title=\"Wims BoneCam\"", "-vcodec", "copy", OutputFilename.c_str(), NULL) == -1)
	{
		std::cerr << "[" << getTimeISO8601() << "] execlp Error! Exiting." << std::endl;
		exit(1);
	}
}
else
{
	/* A positive (non-negative) PID indicates the parent process */
	close(pipefd[0]);		/* Close unused side of pipe (in side) */
	open_device();
	init_device();
	start_capturing();
	//mainloop();

	time_t CurrentTime;
	time(&CurrentTime);
	time_t FileStartTime = CurrentTime;
	while ((difftime(CurrentTime, FileStartTime) < (60 * 60)) && (bRun))
	//unsigned int count = frame_count;
	//while ((count-- > 0) && (bRun))
	{
		fd_set fds;
		struct timeval tv;
		int r;

		FD_ZERO(&fds);
		FD_SET(fd, &fds);

		/* Timeout. */
		tv.tv_sec = 2;
		tv.tv_usec = 0;

		r = select(fd + 1, &fds, NULL, NULL, &tv);
		if (-1 == r) 
		{
			if (EINTR == errno)
				continue;
			errno_exit("select");
		}
		if (0 == r) 
		{
			fprintf(stderr, "select timeout\n");
			exit(EXIT_FAILURE);
		}
		struct v4l2_buffer buf;
		WIMLABEL:
		CLEAR(buf);
		buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
		buf.memory = V4L2_MEMORY_MMAP;

		if (-1 == xioctl(fd, VIDIOC_DQBUF, &buf)) 
		{
			switch (errno) 
			{
			case EAGAIN:
				goto WIMLABEL;
			case EIO:
				/* Could ignore EIO, see spec. */
				/* fall through */
			default:
				errno_exit("VIDIOC_DQBUF");
			}
		}
		else
		{
			assert(buf.index < n_buffers);
			write(pipefd[1], buffers[buf.index].start, buf.bytesused);
		}
		if (-1 == xioctl(fd, VIDIOC_QBUF, &buf))
				errno_exit("VIDIOC_QBUF");
		time(&CurrentTime);
	}

	stop_capturing();
	uninit_device();
	close_device();
	close(pipefd[1]);		/* Close side of pipe I'm writing to, to get the child to recognize it's gone away */
	std::cerr << "\n[" << getTimeISO8601() << "] Pipe Closed, Waiting for FFMPEG to exit" << std::endl;
	int ffmpeg_exit_status;
	wait(&ffmpeg_exit_status);				/* Wait for child process to end */
	std::cerr << "[" << getTimeISO8601() << "] FFMPEG exited with a  " << ffmpeg_exit_status << " value" << std::endl;
}

You can see the calls to open_device(), init_device(), and start_capturing() that I didn’t change at all. My loop runs for a particular amount of time instead of number of frames. I flattened the call to mainloop() into this routine in a very quick and dirty fashion to make sure I had access to the file descriptor to write the data from the camera to the input for ffmpeg.

ffmpeg is started with the execlp() command. The parameters -i – tell ffmpeg to read it’s input from standard input, which in this case I’ve pointed to be the output end of the pipe I created.