GPS output format

[crwper 7]

Hi All--

It's been a while since I posted on this topic. I've been working on the skydiver's GPS, and am perhaps three months away from what I'd call a marketable product. Key stats are:

• 2" x 2" x 0.5"
• 4 Hz u-blox NEO-5Q GPS receiver
• Report glide ratio, horizontal velocity, or vertical velocity as audible tones through earphones
• USB interface for configuration, charging, and reading logged data
• Open firmware, so sufficiently nerdy users can play.

At the moment, the unit logs to a CSV file, which can be read when it is plugged into a USB port. The CSV file includes time, latitude, longitude, elevation, and 3D velocity. The next prototype will include temperature and pressure as well, which will also be used to improve the accuracy of calculations.

From my viewpoint, the CSV is a compact, versatile format. However, in testing by others, it's become apparent that there might be a problem for some users, who are unable to use Excel to do number crunching.

My questions, then, are:

• Is there a preferred output format which would be most easily readable by third-party software?
• What third-party software is available to do this kind of number crunching?

I'd like to retain all the measured data in the log, which excludes some of the less flexible formats. My preference would be simply to write a CSV file, and then have the number crunching done by software after the fact. This keeps the firmware simple, and the device as versatile as possible.

Any thoughts?

Michael

[redfox 0]

I would prefer a format which has structure and is extensible.
A simple format for this is JSON, which I can recommend.

Cordially, Herwig

[phoenixlpr 0]

Quote

* Is there a preferred output format which would be most easily readable by third-party software?
* What third-party software is available to do this kind of number crunching?

NMEA
Its a well known format and supported by lots of apps.

[crwper 7]

Herwig: I've used something similar to JSON for the configuration file, but Im not sure JSON has any advantage for logging the data itself. The first few lines of the CSV file might look something like this:

time,lat,lon,elev,vel_n,vel_e,vel_d

59518750,510166666,-1140166666,1048000,0,0,0

59519000,510166666,-1140166666,1048000,0,0,0

59519250,510166666,-1140166666,1048000,0,0,0

This is fairly easy to parse for a human reader, and doesn't get much easier for a computer. If more information needs to be added--say, when I add a barometer to the device--I just add columns on the right, with an appropriate label at the top. The equivalent JSON records will look something like this:

[

{

"time": "59518750",

"lat": "510166666",

"lon": "-1140166666",

"elev": "1048000",

"vel_n": "0",

"vel_e": "0",

"vel_d": "0"

},

{

"time": "59519000",

"lat": "510166666",

"lon": "-1140166666",

"elev": "1048000",

"vel_n": "0",

"vel_e": "0",

"vel_d": "0"

},

{

"time": "59519250",

"lat": "510166666",

"lon": "-1140166666",

"elev": "1048000",

"vel_n": "0",

"vel_e": "0",

"vel_d": "0"

}

]

To my eye, this is harder to parse and scan quickly, takes more space on disk, and isn't really more extensible than CSV.

Finally, I'm pretty sure software support is better for CSV than for JSON. Surely any program that can read tabular data as a JSON file can also read it as CSV. Am I wrong about this?

Michael

[crwper 7]

phoenixlpr: There are a couple of related problems with NMEA:

• The records generally contain information we are not actually interested in, e.g., "time since last DGPS update" in the venerable GPGGA message.
• Unfortunately, NMEA has no support for 3D velocity. In particular, there is no record that contains vertical speed.

At the root of it, NMEA is a standard for communication between nautical instruments. It's been adopted by GPS manufacturers as well, but because of its heritage, it's not really a great fit.

Basically, even assuming that we could shoehorn our current 3D PVT data into the NMEA format, there's no guarantee that we could find similar workarounds for data that might be available in the future. The fact that I even have to use words like "workaround" in talking about the NMEA format turns me completely off of it.

Does the software you guys are using right now include any support for CSV? What kinds of analysis do you do on data after a jump?

Michael

[redfox 0]

Hi Michael,

this forum may be a little bit off-topic,
but I play since years with such toys to get feedback during flight.

Played with STM Cortex M3, Segger embOS, emWin and USB.

Output in JSON is parsed in fractions of a second on modern Computers.
Can be done in C, C++, Java, Matlab, ...
I used Lua on the PC and Mac side.

You may add or change some fields later, e.g. barometric data,
then it is nice if you can still work with older software.
Adding a field in the beginning to define the schema or version also helps...

Akustik feedback during flight is hard...
We have to wait for cheap and leightweight optics ;-)

Or are we concentrating too much on performance?
At last, flying should be fun!

Cordially,

Herwig

[crwper 7]

Herwig: I agree wholeheartedly, flying should be, first and foremost, fun.

This is one of the reasons I've avoided visual feedback, until now. My next prototype (waiting on boards) includes the ability to output to a 4-digit 7-segment LED display. However, I suspect this will be more distracting than audible feedback. Furthermore, I suspect audible feedback gives a more intuitive sense of change, whereas visual feedback requires us to remember what the number was a moment ago.

Ultimately, I think the best solution is the simplest one. For the most recent round of development, my goal has been to build kind of an "iPod" for glide ratio--something you don't have to think too hard about, just put in an earphone and go. This leaves more attention for flying, and generally, I think, removes the focus from absolute numbers. Of course, if you want absolute numbers, you can view the logs later. But I don't think "in the air" is the time to do this.

The tones I am producing right now have been surprisingly helpful. There are two aspects to the tone I produce:

• The tone's pitch is determined by the value being measured--for example, glide ratio.
• The tone's rate is determined by the change in value, with a minimum rate of 1 Hz. The device tries to ensure that one tone is produced for each semitone change in pitch.

So, if you're flying well, you hear a regular high-pitched tone. If you make a change, you hear a momentary acceleration of the tone's rate while the pitch changes, then it settles into a steady rate at the new pitch. The effect is quite intuitive.

Interestingly, in testing, we've actually noticed the effect of this feedback as a 5-10 second wobble in the glide ratio. Without earphones in, the wobble does not occur. With feedback, the user decreases vertical speed, but at the cost of horizontal speed. This results in improved glide angle, but the situation is temporary. Eventually, the loss of horizontal speed catches up, vertical speed once again increases, and the cycle continues.

So, one challenge in using the device is learning not to chase these small bumps, but instead to fly steadily.

It's my hope that this device will be something someone can "play" with in freefall, rather than getting hung up on maximizing the pitch moment to moment.

Michael

[crwper 7]

I thought it might be fun to include an example of the feedback I was talking about. In the attached plot, the left vertical axis shows glide angle times 100, and the right vertical axis shows speed in cm/s. The horizontal axis is time in ms. The three lines are glide angle, horizontal, and vertical speed.

In testing, the horizontal speed seems to have almost no lag. There is a small lag in the vertical speed, which may contribute to this effect.

I'm pretty excited to see what kind of numbers I get from the barometer, in the next prototype. The chip itself will give an accuracy of about 0.25 m, but it remains to be seen how noisy that is when it's mounted to the back of someone's head.

Michael

P.S. I hope others don't consider this to be off-topic. I've built this thing entirely with wingsuit flight in mind, so it was the first place I thought to post. If anyone feels differently, please let me know.

[BrianM 1]

Quote

From my viewpoint, the CSV is a compact, versatile format. However, in testing by others, it's become apparent that there might be a problem for some users, who are unable to use Excel to do number crunching.

CSV is not an Excel format. CSV has been around forever and is supported by just about every spreadsheet out there (not to mention lots of databases and programming languages).

Shouldn't be a problem for non-Excel users. For example, I use OpenOffice on Linux, it will read CSV files just fine.

"It's amazing what you can learn while you're not talking." - Skydivesg

[phoenixlpr 0]

If you want to work with other GPS software you might have to speak their language. NMEA is simple and even a microcontroller can produce that. Anyway its quite straitforward to just dump "raw" GPS data.

Quote

$GPGGA,095107.000,6010.070400,N,02457.347040,E,1,5,3.33,-23.506,M,0.0,M,,*7B
$GPGLL,6010.070400,N,2457.347040,E,095107.000,A,A*67
$GPRMC,095107.000,A,6010.070400,N,02457.347040,E,2.146,0.00,031009,,,A*5A
$GPVTG,0.00,T,,M,2.146,N,3.974,K,A*35
$GPZDA,095107.000,03,10,2009,,*55

Quote

Unfortunately, NMEA has no support for 3D velocity. In particular, there is no record that contains vertical speed.

So?

1. why to log something if that can be cumputed from the rest of the data?
2. you can use your own extention if you wish.

BTW CSV is not a format, just a way to dump some data.

[redfox 0]

The idea is to use more than GPS.
GPS is not very precise for direct feedback during flight.

So you may add barometric data and data from acceleration sensors too.

Without barometric data, some Smartphones could deliver that :-)

There are a lot of ideas out here...
Let's play and look what works best :-)

Herwig

[crwper 7]

Quote

Anyway its quite straitforward to just dump "raw" GPS data.

My raw GPS data isn't in NMEA--it's in the u-blox UBX binary format. The reason for this choice is the fact that NMEA does not support vertical speed.

Quote

1. why to log something if that can be cumputed from the rest of the data?

Vertical speed as reported by a GPS is actually computed using Doppler measurements on the incoming signal--not by finite differencing on altitudes. Vertical speed derived from Doppler measurements is a lot more accurate than finite differencing, so I think it's important to write it out.

Quote

2. you can use your own extention if you wish.

True, but I am also trying to look forward. If I write my own extension, what are the chances that other software will ever support it? Pretty much zilch. If I write data in an open, extensible format, these chances are much higher, because, as software evolves, other software will surely have the ability to read generic columnar data.

Quote

BTW CSV is not a format, just a way to dump some data.

Actually, strictly speaking, CSV is a well specified format. Things get a bit hairy as soon as you want to write a field that includes a comma, like "Hello, world", and the format specifies how this should be done, for example.

Michael

[crwper 7]

Quote

Without barometric data, some Smartphones could deliver that :-)

One problem I ran into, actually, trying to use Bluetooth GPSes, is that everyone focuses entirely on NMEA output. Some of these units have proprietary binary formats which include vertical speed, but on the rest, you'd need to use finite differencing on altitude, and that's rubbish.

Finally, after a lot of experimenting, I decided it might be easier just to build the GPS into the unit. This also eliminates the need to "pair" the two.

Quote

There are a lot of ideas out here...
Let's play and look what works best :-)

That may be best. I was hoping to learn a bit more about the software people are using, but that doesn't seem to be getting anywhere.

Are people just not using any software? Are Google Earth and Tracking Derby the only tools available to someone who doesn't want to write their own?

Michael

[vidiot 0]

Michael,

a) GPX is the widely accepted standard for recording GPS information.

b) Paralog

Klaus

My Logbook

[phoenixlpr 0]

Quote

Vertical speed as reported by a GPS is actually computed using Doppler measurements on the incoming signal--not by finite differencing on altitudes. Vertical speed derived from Doppler measurements is a lot more accurate than finite differencing, so I think it's important to write it out.

Than write it out.
http://www8.garmin.com/support/pdf/NMEA_0183.pdf

Quote

Actually, strictly speaking, CSV is a well specified format.

Sure like something,something else...
Anyway NMEA is a special case of CSV.

[phoenixlpr 0]

Quote

Michael,

a) GPX is the widely accepted standard for recording GPS information.

b) Paralog

Klaus

Microcontrollers and XML do not mix too well.

[phoenixlpr 0]

You can output some text format of your own and you can have a converter running on a smartphone or a computer using java, python, whatever...

[vidiot 0]

I totally agree with regards to parsing XML, but totally disagree with regards to writing XML.

My Logbook

[redfox 0]

If I understand you correctly, it is for logging...

If you look at GPX, you have just added the velocity fields, and GPX is XML. KML for Google Earth is based on XML too...

JSON is simpler if you just want to exchange data...

But don't worry too much - if I really want to get the data,
parsing XML, JSON or CSV is pretty easy.
Don't switch to a proprietary binary format ;-)

Have fun working on this - and enjoy flying!

Herwig

[phoenixlpr 0]

Quote

I totally agree with regards to parsing XML, but totally disagree with regards to writing XML.

Consider an unreliable data channel for download.
What do you if you have your connection broken with XML in the middle? Broken XML document?
With NMEA just a broken sentence....

[crwper 7]

Quote

Consider an unreliable data channel for download.

phoenixlpr: In this case, the "download" channel, a USB connection, is quite reliable, so I don't think we need worry too much about response to errors. Generating a GPX file would probably be easy enough, but I'm concerned that a format like XML or JSON is really built more for heterogeneous records, as opposed to a homogeneous stream of data.

One thing which could be a larger concern is file size. At the moment, the largest flash chip I can get is 8 MB. It would be nice to use microSD, but they are just too big, and in any case, the best I can manage is a full speed USB connection, so transfers would take all night.

I actually built several prototypes around the Garmin GPS 10, specifically because it uses a proprietary NMEA sentence for 3D velocity. However, I get an uneasy feeling about shoehorning my data into a format which happens to match it in some places, but not in others. I am aware that NMEA remains a virtually universal format, but can't quite bring myself to use it as the primary method of storage for a device which I hope will grow in unexpected directions.

Klaus: I've actually never played with Paralog before, but will take a look a it. It would be lovely if there were already avenues for analyzing and compiling the data. The best case scenario for me would be third-party software which offered solid analysis of the data output by the GPS.

Herwig: No chance of my going to a proprietary binary format. I want to keep this thing as open as possible, in order to encourage innovation. Part of that, to me, is ensuring that the data recorded by the device belongs to the user, not to someone else.

Michael

[bofh 0]

Quote

I am aware that NMEA remains a virtually universal format, but can't quite bring myself to use it as the primary method of storage for a device which I hope will grow in unexpected directions.

Why store it in the same format as you export it in?

[crwper 7]

Quote

Quote

I am aware that NMEA remains a virtually universal format, but can't quite bring myself to use it as the primary method of storage for a device which I hope will grow in unexpected directions.

Why store it in the same format as you export it in?

I'm not sure I understand what you're asking. Right now, I receive data from several onboard devices in whatever format they use. The data is used to generate a tone, and then the relevant parts are written to flash memory in some format.

In this context, I'd have said "storing" and "exporting" both refer to the same step--i.e., writing the data to flash memory.

What do you have in mind?

Michael

[SethInMI 145]

I am guessing that the process of moving the data from the device onboard flash to the PC via USB is transparent to the device software, that is, the flash mounts as a standard usb disk and files can be read and written without having to execute custom device code.

I think the questioner assumed a more complex transfer scheme, where the device software had to participate and therefore could do format conversion on the fly, taking raw binary data from the flash memory and streaming it to the PC via USB as CSV formatted text.

It's flare not flair, brakes not breaks, bridle not bridal, "could NOT care less" not "could care less".

[crwper 7]

Gotcha. That's exactly right. The device mounts as a USB disk, so what's written to disk is exactly what the user sees.

Michael