Webcam on BeagleBoardBlack using OpenCV

I’ve been working with my BBB and Logitech C920 webcam trying to stream video at low latency for some time and have not yet managed to get the latency under 2 seconds.

As a side project I wanted to use the BBB to create a time lapse video, capturing a picture a second, and then later stitching all of the pictures into a video using ffmpeg.

I’m using OpenCV for the first time. I’m really only using it for the capture/save and to draw some text and lines onto the image, which probably makes OpenCV significant overkill.

My C++ code for the process is:

#include <iostream> // for standard I/O
#include <string>   // for strings
#include <iomanip>  // for controlling float print precision
#include <sstream>  // string to number conversion
#include <unistd.h> // for sleep
#include<opencv2/opencv.hpp>
using namespace std;
using namespace cv;

std::string timeToISO8601(const time_t & TheTime)
{
	std::ostringstream ISOTime;
	struct tm * UTC = gmtime(&TheTime);
	ISOTime.fill('0');
	ISOTime << UTC->tm_year+1900 << "-";
	ISOTime.width(2);
	ISOTime << UTC->tm_mon+1 << "-";
	ISOTime.width(2);
	ISOTime << UTC->tm_mday << "T";
	ISOTime.width(2);
	ISOTime << UTC->tm_hour << ":";
	ISOTime.width(2);
	ISOTime << UTC->tm_min << ":";
	ISOTime.width(2);
	ISOTime << UTC->tm_sec;
	ISOTime << "Z";
	return(ISOTime.str());
}
std::string getTimeISO8601(void)
{
	time_t timer;
	time(&timer);
	return(timeToISO8601(timer));
}

int main()
{
    VideoCapture capture(-1);	// Using -1 tells OpenCV to grab whatever camera is available.
    if(!capture.isOpened()){
	    std::cout << "Failed to connect to the camera." << std::endl;
		return(1);
    }
    capture.set(CAP_PROP_FRAME_WIDTH,1920);
    capture.set(CAP_PROP_FRAME_HEIGHT,1080);
    //capture.set(CAP_PROP_FRAME_WIDTH,2304);	// This should be possible for still images, but not for 30fps video.
    //capture.set(CAP_PROP_FRAME_HEIGHT,1536);

	for (int OutputFolderNum = 100;	OutputFolderNum < 1000; OutputFolderNum++)
		for (int OutputImageNum = 1; OutputImageNum < 10000; OutputImageNum++)
		{
			Mat C920Image;
		    capture >> C920Image;
			if(!C920Image.empty())
			{
				std::ostringstream OutputFilename;
				OutputFilename.fill('0');
				OutputFilename << "/media/BONEBOOT/DCIM/";
				OutputFilename.width(3);
				OutputFilename << OutputFolderNum;
				OutputFilename << "WIMBO/img_";
				OutputFilename.width(4);
				OutputFilename << OutputImageNum;
				OutputFilename << ".jpg";

				line(C920Image, Point(0, C920Image.rows/2), Point(C920Image.cols, C920Image.rows/2), Scalar(255, 255, 255, 32)); // Horizontal line at center
				line(C920Image, Point(C920Image.cols/2, 0), Point(C920Image.cols/2, C920Image.rows), Scalar(255, 255, 255, 32)); // Vertical line at center

				circle(C920Image, Point(C920Image.cols/2, C920Image.rows/2), 240, Scalar(255, 255, 255, 32)); // Circles based at center
				putText(C920Image, "10", Point((C920Image.cols/2 + 240), (C920Image.rows/2)), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(0, 0, 255));
				circle(C920Image, Point(C920Image.cols/2, C920Image.rows/2), 495, Scalar(255, 255, 255, 32)); // Circles based at center
				putText(C920Image, "20", Point((C920Image.cols/2 + 495), (C920Image.rows/2)), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(0, 0, 255));
				circle(C920Image, Point(C920Image.cols/2, C920Image.rows/2), 785, Scalar(255, 255, 255, 32)); // Circles based at center
				putText(C920Image, "30", Point((C920Image.cols/2 + 785), (C920Image.rows/2)), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(0, 0, 255));
				circle(C920Image, Point(C920Image.cols/2, C920Image.rows/2), 1141, Scalar(255, 255, 255, 32)); // Circles based at center
				putText(C920Image, "40", Point((C920Image.cols/2 + 1141), (C920Image.rows/2)), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(0, 0, 255));

				string DateTimeText = "WimsWorld.com " + getTimeISO8601();
				int baseline=0;
				Size textSize = getTextSize(DateTimeText, FONT_HERSHEY_SIMPLEX, 1, 1, &baseline);
				putText(C920Image, DateTimeText, Point((C920Image.cols - textSize.width), (C920Image.rows - baseline)), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(0, 0, 255));
				imwrite(OutputFilename.str(), C920Image);
				std::cout << DateTimeText << " Wrote File : " << OutputFilename.str() << std::endl;
			}
			std::cout << getTimeISO8601() << "\r" << std::flush;
			sleep(1);
		}
    return 0;
}

I compile it on the BBB with the command:

g++ -O2 `pkg-config --cflags --libs opencv` TimeLapse.cpp -o TimeLapse

I’ve got a bug in that I don’t automatically create the directory structure that I’m saving files into. That’s in the to-do list.

I had been interested in the angle of view on the C920 and found it defined on the Logitech support site that the “Diagonal Field of View (FOV) for the Logitech C920 is 78°”. Unfortunately I was not able to understand if that varied based on the resolution being used. I’m currently using the resolution of 1920×1080, but for stills the camera can capture up to 2304×1536.

I did the geometry math to figure out that 10° off center would be a radius of 240, 20° off center would be a radius of 495, and 30° off center would be a radius of 785. Remembering SOHCAHTOA as Some Old Hags Can’t Always Hide Their Old Age from 9th grade math class came in useful. Using 1920×1080 and 78°angle, my diagonal radius (opposite) works out at 1101 and angle of 39° for tangent, allowing me to calculate my eye height of 1360 = (1101/Tan(39°)). Once I had my eye height I could calculate the radius of circles at any angle by Radius = Tan(Angle) * EyeHeight.

I wanted the circles and angles of vision for my streaming video application and decided that seeing them drawn on the images created here would be helpful, along with both the horizontal and vertical center lines.

The thing I’m not happy with is that the application seems to be running between 30% and 60% of the CPU load on the BBB. When I stream video from the C920 using the native H.264 output the C920 can produce, I was only using about 3% of the BBB CPU. I’ve commented out my drawing code, and verified that the CPU load is primarily related to acquiring the image from the capture device and saving it out to a jpeg file. The lines and text drawing produce minimal incremental CPU. I want to keep the CPU load as low as possible because I’m powering this device from a battery and want it to have as long a runtime as possible.

I believe that the OpenCV library is opening the capture device in a movie streaming mode, and it’s using more CPU interpreting the stream as it’s coming in than the method I was using for streaming to a file. I’ve not yet figured out if there’s a way to define what mode OpenCV acquires the image from the camera.

I was trying to draw the lines and circles with some alpha transparency, but it seems that my underlying image is not the right number of channels and so the lines are being drawn fully opaque.

When the capture opens, it outputs several instances of the same error “VIDIOC_QUERYMENU: Invalid argument” that I’ve not figured out what they mean, or stopped procucing.

I am working on a 32GB flash card, partitioned into two 16GB filesystems. The first is Fat32, has a simple uEnv.txt file in the root allowing the BBB onboard flash to be used, and following the Design rules for Camera File systems standard for the image naming. It allows me to take out the card put it in a PC and it’s recognized just like a normal camera memory card.

Contents of uEnv.txt:

mmcdev=1
bootpart=1:2
mmcroot=/dev/mmcblk1p2
optargs=quiet

The camera seems to be focusing on the building across the street instead of West Seattle.

View from 1200 Western Ave, 13th Floor Elevator Room

1200 Western Ave, 13th Floor Elevator Room

Advertisements

3 thoughts on “Webcam on BeagleBoardBlack using OpenCV

  1. Out of curiosity, did you ever get the h.264 stream working with the C920 and opencv. I am working on this currently and am running into various issues.

    • I’ve not made any progress with using OpenCV and h.264 recently.

      I’m able to stream using the v4l2 interface and ffmpeg. I only needed OpenCV when I intended to modify or analyze my video on the BBB.

      There has been a good discussion on the BeagleBoard email list recently about using webcams and OpenCV. It lead to the understanding that for video manipulation on the BBB using mjpeg from the camera was significantly easier to deal with.

      Quoting: MJPEG uses intraframe compression whereas H.264 uses interframe compression. Put simply, each MJPEG frame gets compressed individually as a JPEG image and each compressed frame is independent of all others. H.264 on the other hand, uses interframe prediction to \take advantage from temporal redundancy between neighboring frames to achieve higher compression rates”. [6] While this is good for compressing video streams meant to be viewed as a continuous stream, it is not well-suited for embedded vision applications since H.264 decompression is CPU intensive and can exhibit decompression artifacts and lag when there is a lot of motion in the video.

      The bandwidth to stream mjpeg from the C920 works out around 30Mbps while streaming h264 is only about 3Mbps. In the lab I’ve got plenty of bandwidth over my 5GHz link, but I don’t know what I may get in the field yet. I’ve alternated between unicast and broadcast on my wired network. broadcast on the shared media of wireless might be a really bad thing.

      • This is what have been seeing as well. It seems that it is not specifically suggested to go with h.264 and may be better to just stick with mjpeg for processing. If recording, then using h.264 is a good idea. I found the same results in decompression and have since decided to stick with mjpeg.

        I was also curious about what kind of quality processing you are seeing when running the C920 in opencv? I am using a quad core ODroid-U2 and it seems to be doing very well. I am currently testing out various resolutions and rates to see how high I can get. I was curious how the BBB compared.

        When you do full processing (grabbing full fps), have you noticed that opencv is not allowing you to use the full capability of the camera. For example, 1280×1024 with mjpeg encoder, opencv seems to sit around 15fps, even on my desktop. I can not seem to set the value above this.

        It’s nice to see someone else using this camera in an embedded solution such as the BBB.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s