Odd Wildcard Matching in Windows 10

I recently ran into an odd behavior of more files matching a pattern than I expected. I’d used exiftool to modify the dates on files my GoPro produced. It creates backup files of the original images when it modifies the tags. Here’s the command I ran.

exiftool.exe -r "-AllDates+=4:7:6 17:40:00" -ext jpg f:\GoPro\20170807

Now I had about 4000 files with the .JPG extension and another 4000 files with a .JPG_original extension.

I ran my program that parses the directory structure and turns all those images into a time lapse movie, and it seemed to be including both the file extensions, making a very disjointed movie.

I loaded my source code in the debugger and it seemed to be doing a findfirst / findnext specifically looking for .JPG files, and not some other extension, but it was definitely retrieving files both with .JPG and .JPG_original extensions.

I then ran a couple of commands at the windows command prompt and was surprised to find the same results there.

dir F:\GoPro\20170807\372GOPRO\G*.JPG /p
dir F:\GoPro\20170807\372GOPRO\G???????.JPG /p

Each command returned both the JPG and JPG_original files.

dir F:\GoPro\20170807\372GOPRO\G*.JPG_original /p

returned just the JPG_original files.

dir F:\GoPro\20170807\372GOPRO\G??????.JPG /p

had one less question mark and correctly returned no files.

This is all unexpected behavior, though I’m glad to see that it was consistent with the operating system and not something specific to the C runtime. I’d love an explanation of what’s going on.

DJI Battery Monitoring

DJI charges a significant amount of money for their batteries and calls them Smart Batteries. I’ve seen the statistics reported of how many times the battery has been power cycled, as well as details of how much flight time is available. This was a new and interesting feature to me.

Screenshot_20160502-075201

I had left the drone in the back of my car overnight. The temperature had probably gotten into the high 30s, and was still in the mid 50s with the sun shining. The warning message “Battery Temperature Too Low. Warm battery to at least 15 degrees Celcius before flying” came up on my screen and would not let me initiate a takeoff.

I have used batteries in cold climates in the past. I know how temperature affects both current output from batteries and future usability. I’m impressed that DJI has built in this feature to their firmware.

Flashing ESCs on Hobbylord BumbleBee

I bought a Bumblebee Quad from a local hobby shop a few months ago, and when I finally got around to trying to build it with a proper autopilot found that it’s ESCs used a protocol called UltraPWM that is a very uncommon protocol.

I came across this page, https://github.com/sim-/tgy/issues/13 , which leads me to believe that I should be able to flash the ESCs with a simonk tgy firmware and use the hardware I already have.

I came across a cable from hobbyking that is designed to make contact with the surface mounted atmel device and allow programming without any soldering or desoldering. http://hobbyking.com/hobbyking/store/__27195__Atmel_Atmega_Socket_Firmware_Flashing_Tool.html It was designed to be used with an atmel programming device that they also sell http://hobbyking.com/hobbyking/store/__27990__USBasp_AVR_Programming_Device_for_ATMEL_proccessors.html and so I thought I’d be good to go. The cable cost $20 while the programmer cost $4, but not needing to solder anything was a very positive solution for me.

USBasp AVR Programming Device for ATMEL proccessors

USBasp AVR Programming Device

Atmel Atmega Socket Firmware Flashing Tool

Atmel Atmega Socket Firmware Flashing Tool

What I didn’t recognize until it all arrived was that Hobbyking has updated the USB Programmer to use a 6 conductor connector, but not updated their programming cable from the 10 conductor cable. The message boards on hobbyking discuss the change, and have pinout descriptions, but it’s been very frustrating because getting the parts to do the correct wiring has not been as simple as plug and play.

Atmega contact points

Atmega contact points

Atmega contact points

Atmega contact points

Cable Pinout Description

Cable Pinout Description

This has been extremely frustrating to me as the parts I ordered were billed as no soldering required, but could not be simply plugged into each other.

File a flight plan before takeoff

Always think through what you are going to do before you take off.

Broken XHeli

Broken Quadcopter

I had decided that I wanted to try the Follow-Me option in AndroPilot. Turned on my Taranis transmitter, but then left it sitting on the bench with the throttle down, and all the switches in default positions. I connected the 3DR radio to my Google Nexus 7 running AndoPilot. I hit the button on the tablet to arm the drone, and took it off entirely from the tablet. I was flying using the virtual sticks on the screen of the tablet, which was not extremely intuitive because there’s no tactile feedback on the tablet. I’d raised the drone up to about 40 feet, and was trying to find the button to attempt follow me.  It didn’t appear to be moving towards me, or doing anything very predictable. My friend near the bench asked what it took to take control from the RC transmitter, and I said just start flipping switches.  I continued to fly for about another 30 seconds when I asked him to flip return to launch. (I’ve got that set on a switch of it’s own, using channel 7 of the transmitter.)  He was getting concerned because I’d drifted halfway to the trees, and putting the copter in the trees 40 feet up was likely to be a total loss. He flipped the flight mode switch going to loiter. The copter stopped and tumbled out of the sky.

Afterwards he pointed out that before a flight we really needed to discuss what was supposed to happen, and what he could do to recover before the flight begins.

The reason the copter fell out of the sky was that he changed the flight mode without moving the throttle from zero, so when the copter took over it dropped the throttle and fell out of the sky. If he’d just changed the RTL switch things probably would have been just fine. My transmitter is set differently from his. I’ve got all 8 channels configured, with channel 7 being it’s own switch that only deals with the RTL command. He’s got an older firmware and uses RTL as one of his flight modes.

I asked for one thing to happen, and he did something else. He didn’t know exactly what I wanted, and by the time I needed it I wasn’t in a position to explain it quickly. The broken parts can be replaced for $15 from the original place I bought my X, or I can use another supplier and get other parts. I’ve bought an entirely second frame already with nothing mounted on it for only $12 so I might go that route instead of ordering more parts, or I may move parts to run on my new BumbleBee platform.

Media Center Content Protection Error

I’ve noticed that I sometimes get errors in windows media center reporting that programs could not be recorded. This generally happens when I’ve left media center running full screen on one of my monitors overnight…

MediaCenterErrorThe strange thing is that if the media center application is not running, or is running in a window, the recording of the programs seems to work properly. I only get these errors coming up when the application is full screen on one of my monitors.

 

Time-lapse videos from GoPro

In November 2013 I purchased a GoPro HERO3+ Black Edition to play with video recording, but almost immediately became enthralled with taking sequences of photos over long time periods.

The GoPro can be configured to take a picture every 0.5, 1, 2, 5, 10, 30, or 60 seconds. I’ve found that I like taking pictures every 2 seconds, and then converting them to video at 30fps (Frames per second) which gets me an easy to convert time scale. 1 second of video came from 1 minute of photos, 1 minute of video came from 1 hour of photos.

GoPro has a freely available software package to edit videos, as well as creating videos from sequences of images. Because of my past familiarity with FFMPEG I wanted a more scriptable solution for creating videos from thousands of photos.

https://trac.ffmpeg.org/wiki/Create%20a%20video%20slideshow%20from%20images has nice instructions for creating videos from sequences of images using FFMPEG. What it glosses over is that the first image in the sequence needs to be numbered zero or one. Another complication in the process is that the GoPro uses the standard camera file format where no more than 1000 images will be stored in a single directory. This means that with the 1800 images created in a single hour, at least two directories will hold the source images. An interesting issue I ran across is that sometimes the GoPro will skip a number in its image sequence, especially when it has just moved to the next directory in sequence. This is why I had to write my program using directory listings as opposed to simply looking for known files.

The standard GoPro battery will record just about two hours worth of photos. If the GoPro is connected to an external power supply, you can be limited only by the amount of storage space.

Here’s yesterday morning’s weather changing in Seattle.

Here’s a comparison of cropping vs compressing the video. I took this video on a flight from Seattle to Pullman last weekend. You can see much more of the landscape in the compressed version, and see that the top of the propeller leaves the frame in the cropped version.

Compressed:


Cropped:

I’ve written a program that takes three parameters, copies all of the images to a temporary location with an acceptable filename sequence, runs FFMPEG to create a video, then deletes the temporary images. The GoPro is configured to take full resolution still frames, 4000×3000, and I convert those to a 1080p video format using FFMPEG. Because the aspect ratio is different, and the GoPro uses a fish eye lens to begin with, both vertical and horizontal distortion shows up. I run FFMPEG twice, once creating a compressed video and a second time creating a cropped video. This allows me to chose which level of distortion I prefer after the fact.

The three parameters are the video name, the first image in the sequence, and the last image in the sequence. I am currently doing very little error checking. I’m presenting this code here, just to document what I’ve done so far. If you find this useful, please let me know.

Here’s some helper functions I regularly use.

using namespace std;

/////////////////////////////////////////////////////////////////////////////
CString FindEXEFromPath(const CString & csEXE)
{
	CString csFullPath;
	CFileFind finder;
	if (finder.FindFile(csEXE))
	{
		finder.FindNextFile();
		csFullPath = finder.GetFilePath();
		finder.Close();
	}
	else
	{
		TCHAR filename[MAX_PATH];
		unsigned long buffersize = sizeof(filename) / sizeof(TCHAR);
		// Get the file name that we are running from.
		GetModuleFileName(AfxGetResourceHandle(), filename, buffersize );
		PathRemoveFileSpec(filename);
		PathAppend(filename, csEXE);
		if (finder.FindFile(filename))
		{
			finder.FindNextFile();
			csFullPath = finder.GetFilePath();
			finder.Close();
		}
		else
		{
			CString csPATH;
			csPATH.GetEnvironmentVariable(_T("PATH"));
			int iStart = 0;
			CString csToken(csPATH.Tokenize(_T(";"), iStart));
			while (csToken != _T(""))
			{
				if (csToken.Right(1) != _T("\\"))
					csToken.AppendChar(_T('\\'));
				csToken.Append(csEXE);
				if (finder.FindFile(csToken))
				{
					finder.FindNextFile();
					csFullPath = finder.GetFilePath();
					finder.Close();
					break;
				}
				csToken = csPATH.Tokenize(_T(";"), iStart);
			}
		}
	}
	return(csFullPath);
}
/////////////////////////////////////////////////////////////////////////////
static const CString QuoteFileName(const CString & Original)
{
	CString csQuotedString(Original);
	if (csQuotedString.Find(_T(" ")) >= 0)
	{
		csQuotedString.Insert(0,_T('"'));
		csQuotedString.AppendChar(_T('"'));
	}
	return(csQuotedString);
}
/////////////////////////////////////////////////////////////////////////////
std::string timeToISO8601(const time_t & TheTime)
{
	std::ostringstream ISOTime;
	struct tm UTC;// = gmtime(&timer);
	if (0 == gmtime_s(&UTC, &TheTime))
	{
		ISOTime.fill('0');
		ISOTime << UTC.tm_year+1900 << "-";
		ISOTime.width(2);
		ISOTime << UTC.tm_mon+1 << "-";
		ISOTime.width(2);
		ISOTime << UTC.tm_mday << "T";
		ISOTime.width(2);
		ISOTime << UTC.tm_hour << ":";
		ISOTime.width(2);
		ISOTime << UTC.tm_min << ":";
		ISOTime.width(2);
		ISOTime << UTC.tm_sec;
	}
	return(ISOTime.str());
}
std::wstring getTimeISO8601(void)
{
	time_t timer;
	time(&timer);
	std::string isostring(timeToISO8601(timer));
	std::wstring rval;
	rval.assign(isostring.begin(), isostring.end());
	
	return(rval);
}
/////////////////////////////////////////////////////////////////////////////

Here’s a routine I found useful to parse the standard camera file system naming format.

/////////////////////////////////////////////////////////////////////////////
bool SplitImagePath(
	CString csSrcPath,
	CString & DestParentDir,
	int & DestChildNum,
	CString & DestChildSuffix,
	CString & DestFilePrefix,
	int & DestFileNumDigits,
	int & DestFileNum,
	CString & DestFileExt
	)
{
	bool rval = true;
	DestFileExt.Empty();
	while (csSrcPath[csSrcPath.GetLength()-1] != _T('.'))
	{
		DestFileExt.Insert(0, csSrcPath[csSrcPath.GetLength()-1]);
		csSrcPath.Truncate(csSrcPath.GetLength()-1);
	}
	csSrcPath.Truncate(csSrcPath.GetLength()-1); // get rid of dot

	CString csDestFileNum;
	DestFileNumDigits = 0;
	while (iswdigit(csSrcPath[csSrcPath.GetLength()-1]))
	{
		csDestFileNum.Insert(0, csSrcPath[csSrcPath.GetLength()-1]);
		DestFileNumDigits++;
		csSrcPath.Truncate(csSrcPath.GetLength()-1);
	}
	DestFileNum = _wtoi(csDestFileNum.GetString());

	DestFilePrefix.Empty();
	while (iswalpha(csSrcPath[csSrcPath.GetLength()-1]))
	{
		DestFilePrefix.Insert(0, csSrcPath[csSrcPath.GetLength()-1]);
		csSrcPath.Truncate(csSrcPath.GetLength()-1);
	}
	csSrcPath.Truncate(csSrcPath.GetLength()-1); // get rid of backslash

	DestChildSuffix.Empty();
	while (iswalpha(csSrcPath[csSrcPath.GetLength()-1]))
	{
		DestChildSuffix.Insert(0, csSrcPath[csSrcPath.GetLength()-1]);
		csSrcPath.Truncate(csSrcPath.GetLength()-1);
	}

	CString csDestChildNum;
	while (iswdigit(csSrcPath[csSrcPath.GetLength()-1]))
	{
		csDestChildNum.Insert(0, csSrcPath[csSrcPath.GetLength()-1]);
		csSrcPath.Truncate(csSrcPath.GetLength()-1);
	}
	DestChildNum = _wtoi(csDestChildNum.GetString());

	DestParentDir = csSrcPath;
	return(rval);
}
/////////////////////////////////////////////////////////////////////////////

And here’s the main program.

/////////////////////////////////////////////////////////////////////////////
int _tmain(int argc, TCHAR* argv[], TCHAR* envp[])
{
	int nRetCode = 0;

	HMODULE hModule = ::GetModuleHandle(NULL);

	if (hModule != NULL)
	{
		// initialize MFC and print and error on failure
		if (!AfxWinInit(hModule, NULL, ::GetCommandLine(), 0))
		{
			// TODO: change error code to suit your needs
			_tprintf(_T("Fatal Error: MFC initialization failed\n"));
			nRetCode = 1;
		}
		else
		{
			CString csFFMPEGPath(FindEXEFromPath(_T("ffmpeg.exe")));
			CString csFirstFileName;
			CString csLastFileName;
			CString csVideoName;

			if (argc != 4)
			{
				std::wcout << "command Line Format:" << std::endl;
				std::wcout << "\t" << argv[0] << " VideoName PathToFirstFile.jpg PathToLastFile.jpg" << std::endl;
			}
			else
			{
				csVideoName = CString(argv[1]);
				csFirstFileName = CString(argv[2]);
				csLastFileName = CString(argv[3]);

				int DirNumFirst = 0;
				int DirNumLast = 0;
				int FileNumFirst = 0;
				int FileNumLast = 0;
				CString csFinderStringFormat;

				CString DestParentDir;
				CString DestChildSuffix;
				CString DestFilePrefix;
				CString DestFileExt;
				int DestFileNumDigits;
				SplitImagePath(csFirstFileName, DestParentDir, DirNumFirst, DestChildSuffix, DestFilePrefix, DestFileNumDigits, FileNumFirst, DestFileExt);
				csFinderStringFormat.Format(_T("%s%%03d%s\\%s*.%s"), DestParentDir.GetString(), DestChildSuffix.GetString(), DestFilePrefix.GetString(), DestFileExt.GetString());
				SplitImagePath(csLastFileName, DestParentDir, DirNumLast, DestChildSuffix, DestFilePrefix, DestFileNumDigits, FileNumLast, DestFileExt);

				std::vector<CString> SourceImageList;
				int DirNum = DirNumFirst;
				int FileNum = FileNumFirst;
				do 
				{
					CString csFinderString;
					csFinderString.Format(csFinderStringFormat, DirNum);
					CFileFind finder;
					BOOL bWorking = finder.FindFile(csFinderString.GetString());
					while (bWorking)
					{
						bWorking = finder.FindNextFile();
						SplitImagePath(finder.GetFilePath(), DestParentDir, DirNum, DestChildSuffix, DestFilePrefix, DestFileNumDigits, FileNum, DestFileExt);
						if ((FileNum >= FileNumFirst) && (FileNum <= FileNumLast))
							SourceImageList.push_back(finder.GetFilePath());
					}
					finder.Close();
					DirNum++;
				} while (DirNum <= DirNumLast);

				std::wcout << "[" << getTimeISO8601() << "] " << "First File: " << csFirstFileName.GetString() << std::endl;
				std::wcout << "[" << getTimeISO8601() << "] " << "Last File:  " << csLastFileName.GetString() << std::endl;
				std::wcout << "[" << getTimeISO8601() << "] " << "Total Files: " << SourceImageList.size() << std::endl;

				TCHAR szPath[MAX_PATH] = _T("");
				SHGetFolderPath(NULL, CSIDL_MYVIDEO, NULL, 0, szPath);
				PathAddBackslash(szPath);
				CString csImageDirectory(szPath);
				csImageDirectory.Append(csVideoName);
				if (CreateDirectory(csImageDirectory, NULL))
				{
					int OutFileIndex = 0;
					for (auto SourceFile = SourceImageList.begin(); SourceFile != SourceImageList.end(); SourceFile++)
					{
						CString OutFilePath(csImageDirectory);
						OutFilePath.AppendFormat(_T("\\Wim%05d.JPG"), OutFileIndex++);
						std::wcout << "[" << getTimeISO8601() << "] " << "CopyFile " << SourceFile->GetString() << " to " << OutFilePath.GetString() << "\r";
						CopyFile(SourceFile->GetString(), OutFilePath, TRUE);
					}
					std::wcout << "\n";

					CString csImagePathSpec(csImageDirectory); csImagePathSpec.Append(_T("\\Wim%05d.JPG"));
					CString csVideoFullPath(csImageDirectory); csVideoFullPath.Append(_T(".mp4"));
					if (csFFMPEGPath.GetLength() > 0)
					{
						csVideoFullPath = csImageDirectory + _T("-1080p-cropped.mp4");
						std::wcout << "[" << getTimeISO8601() << "] " << csFFMPEGPath.GetString() << " -i " << QuoteFileName(csImagePathSpec).GetString() << " -y " << QuoteFileName(csVideoFullPath).GetString() << std::endl;
						if (-1 == _tspawnlp(_P_WAIT, csFFMPEGPath.GetString(), csFFMPEGPath.GetString(), 
							#ifdef _DEBUG
							_T("-report"),
							#endif
							_T("-i"), QuoteFileName(csImagePathSpec).GetString(),
							_T("-vf"), _T("crop=in_w:3/4*in_h"),
							// _T("-vf"), _T("rotate=PI"), // Us this to rotate the movie if we forgot to put the GoPro in upside down mode.
							_T("-s"), _T("1920x1080"),
							_T("-y"), // Cause it to overwrite exiting output files
							QuoteFileName(csVideoFullPath).GetString(), NULL))
							std::wcout << "[" << getTimeISO8601() << "]  _tspawnlp failed: " /* << _sys_errlist[errno] */ << std::endl;
						csVideoFullPath = csImageDirectory + _T("-1080p-compressed.mp4");
						std::wcout << "[" << getTimeISO8601() << "] " << csFFMPEGPath.GetString() << " -i " << QuoteFileName(csImagePathSpec).GetString() << " -y " << QuoteFileName(csVideoFullPath).GetString() << std::endl;
						if (-1 == _tspawnlp(_P_WAIT, csFFMPEGPath.GetString(), csFFMPEGPath.GetString(), 
							#ifdef _DEBUG
							_T("-report"),
							#endif
							_T("-i"), QuoteFileName(csImagePathSpec).GetString(),
							// _T("-vf"), _T("rotate=PI"), // Us this to rotate the movie if we forgot to put the GoPro in upside down mode.
							_T("-s"), _T("1920x1080"),
							_T("-y"), // Cause it to overwrite exiting output files
							QuoteFileName(csVideoFullPath).GetString(), NULL))
							std::wcout << "[" << getTimeISO8601() << "]  _tspawnlp failed: " /* << _sys_errlist[errno] */ << std::endl;
					}
					do 
					{
						CString OutFilePath(csImageDirectory);
						OutFilePath.AppendFormat(_T("\\Wim%05d.JPG"), --OutFileIndex);
						std::wcout << "[" << getTimeISO8601() << "] " << "DeleteFile " << OutFilePath.GetString() << "\r";
						DeleteFile(OutFilePath);
					} while (OutFileIndex > 0);
					std::wcout << "\n[" << getTimeISO8601() << "] " << "RemoveDirectory " << csImageDirectory.GetString() << std::endl;
					RemoveDirectory(csImageDirectory);
				}
			}
		}
	}
	else
	{
		_tprintf(_T("Fatal Error: GetModuleHandle failed\n"));
		nRetCode = 1;
	}
	return nRetCode;
}

Old Humor: Jesus & Satan Programming Contest

I’m cleaning up old text files on my machine and came across this story..

Jesus and Satan were having an ongoing argument about who was better on the computer. They had been going at it for days, and frankly God was tired of hearing all the bickering.

Finally fed up, God said, “THAT’s IT! I have had enough and I am going to set up a test that will run for two hours, and from those results, I will judge who does the better job.”

So Satan and Jesus sat down at the keyboards and typed away.
They moused.
They faxed.
They e-mailed
They e-mailed with attachments
They downloaded.
They did spreadsheets.
They wrote reports.
They created labels and cards.
They created charts and graphs.
They did some genealogy reports.
They did every job known to man.

Jesus worked with heavenly efficiency and Satan was faster then hell! Then, ten minutes before their time was up, lightning suddenly flashed across the sky, thunder rolled, rain poured and, of course, the power went off!

Satan stared at his blank screen and screamed every curse word known in the underworld. Jesus just sighed!

Finally the electricity came back on, and each of them restarted. Satan searching frantically, screaming……. “It’s gone! It’s ALL GONE! “I lost everything when the power went out!

Meanwhile, Jesus quietly started printing out all of his files from the past two hours of work. Satan observed this and became irate. “Wait!” he screamed…. “That’s not fair! He cheated! How come he has all his work and I don’t have any?”

God just shrugged and said, “Jesus Saves.”

I don’t know the origin. If you can legitimately claim it, I’ll definitely add an attribution.