Converting NMEA stream to geopackage (or equivalent)

[gdt]

I have an NMEA logfile from an RTK receiver, e.g. Sparkfun RTK Express, with a variety of sentences. I'd like to process that to have a table (in the geopackage sense, but I mean it logically) with the following fields, for use with qgis:

• datetime
• geometry as "3D MEASURED POINT", with HAE in height, and horizontal error as M
• fixtype (as 1,2,4,5 probably, but could be tokens)
• perhaps horizontal and vertical errors, and DOPs
• perhaps something else, but basically as a property of a particular fix

The point of error as M is to align with tracks recorded in QFIeld, when QFIeld is configured to put error in M, and the larger point is that data points with errors larger than 0.010 m (with an F9P) are slightly suspect, and errors greater than 0.020 m are really suspect. I already symbolize the error when viewing, so I can decide how to deal with new evidence in adjusting features for micromapping.

This is tricky because one needs information from three sentences together, basically inverting the receiver's process in creating separate sentences. G*GGA has time, position (including HAE backed out from orthometric height and geoid separation) and fixmode. But it lacks the date, which is only available in G*RMC, otherwise not so useful. And it lacks the error estimates, which are found in G*GST.

My current thinking, not yet validated by running code, is to use gnssstreamer to convert to JSON, and then to write a small python program to accumulate sentences for a particular epoch, merging position and error estimate and getting the right date. And then to use gdal methods to write a geopackage.

I am curious if anyone knows of existing Free Software command-line code to do what I want, or if not, if my approach makes sense.

[semuadmin]

Couple of clarifications:

1. Are these all pre-existing NMEA data logs or is the intention to create new logs?
2. If new, does the data have to be NMEA? If you're using a u-blox receiver like that incorporated in the RTK-Express, could you not use UBX data for the same purpose - a single UBX NAV-PVT sentence may, for example, give you most if not all of what you need?

I doubt you'll find an off-the-shelf CLI tool that will give you exactly what you're after, but If you're comfortable with Python, it would be relatively simple to create a script based around pyubx2 to iterate through the data logs, identity the relevant message identities and data attributes and accumulate the data into a structure (or perhaps a geopackage-compatible SQLite or spatialite database - sqlite3 is a native Python library) which could be imported into QGIS. I'm not entirely clear why it would be beneficial to convert to JSON first, unless you have some specific transformation in mind which is already implemented in JSON.

If you can clarify how you would be looking to visualise the data in QGIS - perhaps a visual mock-up - we may be able to assist further with the scripting, packaging or import processes.

QGIS itself can utilise Python scripts directly (though this would require some familiarity with its API), and there are numerous data analysis plugins available (many of which leverage GDAL and/or geopackage under the covers). It's highly likely that one such plugin may offer a reasonably close approach. It may be worth asking this question in one of the QGIS support forums.

[semuadmin]

Geopandas might offer an alternative Python-based approach e.g. https://mapscaping.com/reading-and-writing-geopackage-in-python/

[gdt]

For now, I'm thinking about NMEA logs which are already generated, and also by other RTK receivers (while I don't have any, SparkFun is selling both Septentrio Mosaic and LG290 devices). I'm really looking to extract information from existing NMEA, rather than designing a toolchain with workflow modifications. For example, these receivers log NMEA to uSD by themselves, and I think QFIeld can write the received NMEA to a logfile. As for within qgis, while I'm intending to look at these with qgis, as an old-school command-line type, I really want to produce a dataset which can be visualized, but also could be used in other ways (that I can't articulate now).

It's normal in qgis to add a track (as a Line String or Multi Line String). Basically I want to do that, and then symbolize the vertices by color (e.g. red for autonomous, orange for differential, green for RTK float and blue for RTK fixed), and then sized by error estimate, perhaps even to scale. I expect to iterate/tweak that and thus am aiming for a dataset with a geometry, fixmode, and error estimtes etc., so that I can make symbolization choices later. Also diffAge is useful, but mostly it seems when falling back to differential and then losing WAAS.

My use of JSON first is because gnssstreamer already reads NMEA and outputs JSON. Thus I can ignore NMEA parsing and just deal with the semantics, being grateful for what you've published. I've so far written a small python program to just deal with GGA and output the fix mode ("quality") and diffAge fields, and that's already been helpful. Now I'm going through examples of writing geopackages with the ogr module, and this is all likely to be reasonably straightforward. I don't really need help with that; it's more that I was wondering if someone had already done this, or could tell me that I'm confused and why. I do realize that I could write a program which wraps your libraries in the way that gnssstreamer does, and also OGR, and skip the intermediate JSON, but it seems conceptually simpler for me to use the intermediate representation, without much real cost.

Already, I've analyzed a track (just GGA, so no error estimates), seeing how fixmode changes, and how the position jumps when changing from differential to RTK float (more than the datum shift). Overall it seems deep woods canopy was tougher than expected (this is a lot about trail mapping in consevation land).

Thanks for the pointers (I should learn pandas some day) and the implicit comment that I'm not way off base.

[semuadmin]

For now, I'm thinking about NMEA logs which are already generated, and also by other RTK receivers (while I don't have any, SparkFun is selling both Septentrio Mosaic and LG290 devices). I'm really looking to extract information from existing NMEA, rather than designing a toolchain with workflow modifications. For example, these receivers log NMEA to uSD by themselves, and I think QFIeld can write the received NMEA to a logfile. As for within qgis, while I'm intending to look at these with qgis, as an old-school command-line type, I really want to produce a dataset which can be visualized, but also could be used in other ways (that I can't articulate now).

Fair enough

It's normal in qgis to add a track (as a Line String or Multi Line String). Basically I want to do that, and then symbolize the vertices by color (e.g. red for autonomous, orange for differential, green for RTK float and blue for RTK fixed), and then sized by error estimate, perhaps even to scale. I expect to iterate/tweak that and thus am aiming for a dataset with a geometry, fixmode, and error estimtes etc., so that I can make symbolization choices later. Also diffAge is useful, but mostly it seems when falling back to differential and then losing WAAS.

My use of JSON first is because gnssstreamer already reads NMEA and outputs JSON. Thus I can ignore NMEA parsing and just deal with the semantics, being grateful for what you've published. I've so far written a small python program to just deal with GGA and output the fix mode ("quality") and diffAge fields, and that's already been helpful. Now I'm going through examples of writing geopackages with the ogr module, and this is all likely to be reasonably straightforward. I don't really need help with that; it's more that I was wondering if someone had already done this, or could tell me that I'm confused and why. I do realize that I could write a program which wraps your libraries in the way that gnssstreamer does, and also OGR, and skip the intermediate JSON, but it seems conceptually simpler for me to use the intermediate representation, without much real cost.

Already, I've analyzed a track (just GGA, so no error estimates), seeing how fixmode changes, and how the position jumps when changing from differential to RTK float (more than the datum shift). Overall it seems deep woods canopy was tougher than expected (this is a lot about trail mapping in consevation land).

Honestly, if you're already commited to using one or other of the SEMU Python GNSS tools for parsing and QGIS for visualising/symbolising what you describe, I would be inclined to at least consider something alone these lines:

1. create a simple spatialite database layer and table using QGIS's browser, incorporating the spatial (lat/lon/hae?) and non-spatial (fix/corr type, pdop, hacc, etc.) fields that you're after. This would only need to be done once.

2. create what would be a relatively simple Python script which uses pyubx2 to iterate through one or more raw NMEA data logs and sqlite3 (with the spatialite extension, which will already be installed with QGIS e.g. "C:\Program Files\QGIS 3.44.1\bin\mod_spatialite.dll") to write the parsed attributes to the database.

   (I may even have something in my archives you could reuse)

This might afford a very flexible workflow which leverages the native data analysis and visualisation capabilities of QGIS while requiring only basic familiarity with pyubx2 and sqlite3 SQL syntax. Once in the spatialite database, the world is your oyster.

Thanks for the pointers (I should learn pandas some day) and the implicit comment that I'm not way off base.

geopandas is a hugely powerful tool but probably massive overkill for your particular use case.

[gdt]

Thanks. I've already written code to create a geopackage file with a trackpoints layer, with the schema you describe, ish. More or less the same except I'm using a geopackage not spatialite but I realize they are very close. I may look into using pyubx directly vs the json representation from gnssstreamer, but so far that isn't causing me difficulties and I'm making minor improvements iteratively. Right now I'm working on merging the multiple sentences. I hope to clean this up and publish it.

[semuadmin]

For what it's worth, here's something I knocked up a while back as a demonstrator...

https://github.com/semuconsulting/pyubx2/blob/master/examples/ubx_spatialite.py