Part 4 of ROAV Dash Cam C1 Pro

I’d figured out that a 128 GB micro sd card would yield about 16 hours of video on the ROAV Dashcam which was good enough that I could make most of my daily long drives without overwriting anything.

I had a PNY 128 GB drive that I’d been using in my GoPro so I put it into the ROAV Dashcam. The ROAV requires the card be formatted as FAT32 and not the newer exFAT, but the dashcam was able to do the formatting.

After a couple of months of use, I got a strange error on the ROAV saying that the memry card was corrupted and it needed reformatting. Before I did that, I put the card into my computer and tried to copy all the data off it. I ran into an issue with some files being corrupted, and then found I was not able to reformat the card in the computer, as it was somehow being reported as read only.

Because there is no physical write protect slider on a micro SD card, I fired up diskpart and issued the commands that should clear the write protect status on a drive.

It appeared to work, but then checking the status after issuing the command attributes clear disk readonly showed that no changes had really happened.

I’d been frustrated trying to figure this out for the past week. Then I came across an article in comp.risks that described exactly this sort of issue.

Micro SD cards silently switching to read-only when they’re “too old”

Mon, 16 Jul 2018 23:38:44 +0200

The 64G Patriot micro SD I had been using in my cell phone from mid 2014
just decided to turn itself into a read-only memory card.  From what I read,
it most likely reached its maximum number of uses, as it happens at least
with some Samsung cards too.  It would be to protect the card from losing
all its data, after its cells were erased "too many times" (limit number
depending on the card, and appearing to be in the order of 10-100k).  And
according to Internet forums, and card reviews on Amazon, it looks like it's
getting more and more common!

A very bad point is that there were no error messages at all.  I added music
files before a trip, but I had none of the new files available later so at
first I thought I didn't do it correctly (even if the transfer was fine, it
could for example have been to my card backup on an hard drive instead of
going to the actual card).  Then, despite the pictures still being taken
correctly by my phone (browsing was OK, able to delete the bad ones...), I
lost all of the new ones when my phone rebooted. So they were only in a
cache memory somewhere, but nowhere on the SD card (not found by deep
recovery tools either).  More fun, the older ones I deleted came back during
the same reboot...

I understand it would be bothering to have an error message at each card
access, but at least I would have known to change the card and would not
have lost 3 days of pictures!  So beware...

I’ve written low level code dealing with flash memory in the past, so I understand that there are only so many rewrite times each sector can handle. I’m also familiar with the differences in file systems between FAT32, exFAT and NTFS. I don’t believe MicroSD cards have any sort of wear leveling algorithms in them that a full fledged SSD has between the flash and the controller. The fact that ROAV puts their constantly changing data two levels deep in the directory structure at least means the data for the root directory isn’t the constantly erased and overwritten sector, but it does mean that the directory structure is getting re-written each time a new file is created. They probably aren’t writing the directory data to a new sector each time, but just overwriting the old location, possibly accelerating the death of the microsd card. because the cluster size on a 128GB Fat32 partition is fairly large, at least 32KB, it holds a lot of directory entries in a single sector.

I had a 200 GB card that I thought I’d replace the failed drive with. The ROAV attempted to format it but reported that it couldn’t. I’m thinking that they simply cannot handle a drive bigger than 128 GB.

Since I had a second 128GB card, I put it into the dashcam. I’ll be interested in seeing how long it takes for it to report as non-functional.

 

Part 3 of ROAV Dash Cam C1 Pro

After getting the video files from my dashcam strung together at high speed I realized I was just as interested in the GPS data that was stored in .info files alongside each .mp4 file.

Some digging around in forums led me to believe that the files were fairly simple comma separated files with the headers working out to be Datetime, Latitude, Longitude, FixType, SatCount, Altitude, SpeedKph, Heading, AccelerometerX, AccelerometerY, AccelerometerZ. The Datetime field is it’s own special format, but easy enough to interpret.

Because I’ve written a GPS data app in the past, I’d learned how to write Keyhole Markup Language (KML) files, as well as learning to use ZIP routines to package them into KMZ files. My original code had hard coded the KML tags because I didn’t want to rely on an external library requirement on a limited platform. In re-using the code I updated to use the XMLLite API. Microsoft may be discontinuing support for this API as well, but at least now it’s still included with current operating systems. The advantage of using the XML API to create the XML is knowing that all of the tags are properly and consistently formed and closed. A secondary feature was that it made it much simpler to explore different data formatting options for the folders in the KML structure.

I worked on the basic idea that the most interesting data from the source files was based on speed and altitude. Then I broke the data into segments by the day, so if I store multiple days worth of files the kml will automatically have reasonable breaks in it. I create a Placemark that includes a LineString with all the GPS coordinates for the day. I calculate the distance by multiplying the reported speed by the time between points.

The calculated distance isn’t as accurate as I’d like, but the code I had from years ago didn’t seem to get more accurate distances. My web references from years ago no longer work, which is the frustrating thing about pointing to documentation on the web. The code I’d used years before was used in a bicycle gps program, which meant that the speeds were slower and the distance traveled between points may have been smaller. The data the ROAV is writing to the log file may not have as many significant digits as the raw data available from the the GPS chip itself.

I create three more Placemarks for each day, each with a simple Point defined. Max Speed, Max Altitude, and Min Altitude. Each of those data points is selected from a simple scan of the input data.

By creating a large KML file and then converting it to a KMZ, it becomes a manageable size. KMZ files load into google earth significantly faster then KML files. One of the interesting things that you can do in Google Earth is display the elevation profile of a path.

On this day you can see that I started at just under 4500 feet, drove over 6000 feet, then down to 1733 feet before ending the day near 2500 feet. When playing with the desktop app, you can drag a marker along the path and it will coordinate a marker along the elevation profile.

I’ve added this code to my ROAV-Concat program, as well as parameters that can tell the program to not output KML or MP4. this allows me to run the program and just generate the KMZ or MP4 file, though beyond during debugging I can’t think of a reason I’d not want both files generated.

I’m hoping that understanding all of this data will allow me to generate text to overlay the video with GPS data beyond what was embedded by the original dashcam.

Part 2 of FFMPEG and ROAV Dash Cam C1 Pro

While writing my software to concatenate and speed up the video files from my ROAV Dashcam I ran into an interesting issue with FFMPEG.

The -filter_complex option seems to stop parsing it’s parameters somewhere above 960 characters on the command line. I didn’t narrow down the exact point, or go digging in the FFMPEG source code to find the size. I expect this is an arbitrary buffer size in FFMPEG. I may contribute to the source code since it fails with no explanation, even when generating a report file. Learning the FFMPEG source structure in itself is a large task, meaning that I’ve not found time to do anything beyond find a workaround.

My workaround was to recognize when the command size will get long and fall back to using the -f concat option with a temporary file listing the input files instead of using the complex filtergraph.

The advantage of the complex filtergraph is twofold. It does not require a secondary input file or any cleanup. It can deal with input files that change resolution if necessary.

Here’s an example of the complex filtergraph:

ffmpeg.exe -report -i 2018_0705_101335_006.MP4 -i 2018_0705_101635_007.MP4 -i 2018_0705_101935_008.MP4 -i 2018_0705_102235_009.MP4 -i 2018_0705_102535_010.MP4 -i 2018_0705_102835_011.MP4 -i 2018_0705_103135_012.MP4 -i 2018_0705_103435_013.MP4 -i 2018_0705_103613_014A.MP4 -i 2018_0705_103615_015A.MP4 -i 2018_0705_104449_016A.MP4 -i 2018_0705_104750_017A.MP4 -i 2018_0705_105050_018A.MP4 -i 2018_0705_105350_019A.MP4 -i 2018_0705_105650_020A.MP4 -i 2018_0705_105950_021A.MP4 -i 2018_0705_110250_022A.MP4 -i 2018_0705_110550_023A.MP4 -i 2018_0705_110850_024A.MP4 -i 2018_0705_143415_025.MP4 -i 2018_0705_143716_026.MP4 -i 2018_0705_144016_027.MP4 -i 2018_0705_144316_028.MP4 -i 2018_0705_144616_029.MP4 -i 2018_0705_144916_030.MP4 -i 2018_0705_145216_031.MP4 -i 2018_0705_145516_032.MP4 -i 2018_0705_145816_033.MP4 -i 2018_0705_150116_034.MP4 -i 2018_0705_150416_035.MP4 -i 2018_0705_150716_036.MP4 -i 2018_0705_151016_037.MP4 -i 2018_0705_151316_038.MP4 -i 2018_0705_151616_039.MP4 -i 2018_0705_151916_040.MP4 -i 2018_0705_152216_041.MP4 -i 2018_0705_152516_042.MP4 -i 2018_0705_152816_043.MP4 -i 2018_0705_153116_044.MP4 -i 2018_0705_153416_045.MP4 -i 2018_0705_153716_046.MP4 -i 2018_0705_154016_047.MP4 -i 2018_0705_154316_048.MP4 -i 2018_0705_154616_049.MP4 -i 2018_0705_154916_050.MP4 -i 2018_0705_155216_051.MP4 -i 2018_0705_155516_052.MP4 -i 2018_0705_155816_053.MP4 -i 2018_0705_160116_054.MP4 -i 2018_0705_160416_055.MP4 -i 2018_0705_160716_056.MP4 -i 2018_0705_161016_057.MP4 -i 2018_0705_161316_058.MP4 -i 2018_0705_161616_059.MP4 -i 2018_0705_161916_060.MP4 -i 2018_0705_162216_061.MP4 -i 2018_0705_162516_062.MP4 -i 2018_0705_162816_063.MP4 -i 2018_0705_163116_064.MP4 -i 2018_0705_163416_065.MP4 -i 2018_0705_163716_066.MP4 -i 2018_0705_164016_067.MP4 -i 2018_0705_164316_068.MP4 -i 2018_0705_164616_069.MP4 -i 2018_0705_164916_070.MP4 -i 2018_0705_165215_071.MP4 -i 2018_0705_165516_072.MP4 -i 2018_0705_165815_073.MP4 -i 2018_0705_170115_074.MP4 -i 2018_0705_170415_075.MP4 -i 2018_0705_170715_076.MP4 -i 2018_0705_171015_077.MP4 -i 2018_0705_171315_078.MP4 -i 2018_0705_171615_079.MP4 -i 2018_0705_171915_080.MP4 -i 2018_0705_172216_081.MP4 -i 2018_0705_172515_082.MP4 -i 2018_0705_172815_083.MP4 -i 2018_0705_173115_084.MP4 -i 2018_0705_173415_085.MP4 -i 2018_0705_173715_086.MP4 -i 2018_0705_174015_087.MP4 -i 2018_0705_174315_088.MP4 -i 2018_0705_174615_089.MP4 -i 2018_0705_174915_090.MP4 -i 2018_0705_175215_091.MP4 -i 2018_0705_175515_092.MP4 -i 2018_0705_175815_093.MP4 -i 2018_0705_180115_094.MP4 -i 2018_0705_180415_095.MP4 -i 2018_0705_180715_096.MP4 -i 2018_0705_181015_097.MP4 -i 2018_0705_181315_098.MP4 -i 2018_0705_181615_099.MP4 -i 2018_0705_181915_100.MP4 -i 2018_0705_182215_101.MP4 -i 2018_0705_182515_102.MP4 -i 2018_0705_182815_103.MP4 -i 2018_0705_183115_104.MP4 -i 2018_0705_183415_105.MP4 -i 2018_0705_183715_106.MP4 -i 2018_0705_184015_107.MP4 -i 2018_0705_184315_108.MP4 -i 2018_0705_184615_109.MP4 -i 2018_0705_184915_110.MP4 -i 2018_0705_185215_111.MP4 -i 2018_0705_185515_112.MP4 -i 2018_0705_185815_113.MP4 -i 2018_0705_190115_114.MP4 -i 2018_0705_190415_115.MP4 -filter_complex [0:v][1:v][2:v][3:v][4:v][5:v][6:v][7:v][8:v][9:v][10:v][11:v][12:v][13:v][14:v][15:v][16:v][17:v][18:v][19:v][20:v][21:v][22:v][23:v][24:v][25:v][26:v][27:v][28:v][29:v][30:v][31:v][32:v][33:v][34:v][35:v][36:v][37:v][38:v][39:v][40:v][41:v][42:v][43:v][44:v][45:v][46:v][47:v][48:v][49:v][50:v][51:v][52:v][53:v][54:v][55:v][56:v][57:v][58:v][59:v][60:v][61:v][62:v][63:v][64:v][65:v][66:v][67:v][68:v][69:v][70:v][71:v][72:v][73:v][74:v][75:v][76:v][77:v][78:v][79:v][80:v][81:v][82:v][83:v][84:v][85:v][86:v][87:v][88:v][89:v][90:v][91:v][92:v][93:v][94:v][95:v][96:v][97:v][98:v][99:v][100:v][101:v][102:v][103:v][104:v][105:v][106:v][107:v][108:v][109:v]concat=n=110:v=1[v];[v]setpts=(1/60)*PTS,drawtext=fontfile=C\\:/WINDOWS/Fonts/consola.ttf:fontcolor=white:fontsize=80:y=main_h-text_h-50:x=50:text=WimsWorld[o] -map [o] -c:v libx265 -crf 23 -preset veryfast -movflags +faststart -bf 2 -g 15 -pix_fmt yuv420p -y “Output.mp4”

Here’s an example of the command where all the input files are defined in the temporary file:

ffmpeg.exe -report -f concat -safe 0 -i C:\Users\Wim\AppData\Local\Temp\Wim4BD3.tmp -vf setpts=(1/60)*PTS,drawtext=fontfile=C\\:/WINDOWS/Fonts/consola.ttf:fontcolor=white:fontsize=80:y=main_h-text_h-50:x=50:text=WimsWorld -an -c:v libx265 -crf 23 -preset veryfast -movflags +faststart -bf 2 -g 15 -pix_fmt yuv420p -y “Output.mp4”