Feature request: geologic compass

[r-gibs]

As a geologist I measure the orientation of geologic features in the field as part of a standard geologic mapping workflow. Currently, I measure geologic features with a compass or separate app and manually fill in the attributes of a point "structure" layer. It would be nice if the tablet sensors (magnetometer and gyroscope) could be used to measure the orientation of structural features and record:

• dip direction / trend (numeric field from 0-360, azimuth relative to north)
• dip / plunge (numeric field from 0-90, angle from a horizontal plane)

It would be helpful to have a functionality that allows me to take these measurements from within the attribute form using the tablet or phone sensors. I suggest an additional button / action on the attribute form of this layer that opens a dialogue and asks to align the phone along a planar feature and select "take measurement" once aligned, data can then be written to dip direction and dip attributes. Linear geologic features are measured differently than planar features, a "planar or linear" option could be included in the dialogue or a separate button / action could be used for linear features.

This feature should also prompt the user to recalibrate the compass when needed and specify if a magnetic declination was applied.

In an advanced version this dialogue should show a visualization of the structural measurement with a compass circle and dip / dip direction symbol along with numeric dip / dip direction attributes (see Rockd compass visualization screenshot below).

[label-actions]

This has been accepted as an enhancement request. Good job 👍 Enhancement requests are very valuable to make QField better. They sometimes attract others and just happen magically. More often however they need someone to jump in and make them come true.

[noah-clay]

this is great, thank you. I (and everyone I convert from ESRI and/or paper) use QField daily for geologic mapping and surveying, and this would be a very useful feature.

[keinadmin]

dont want to be a wisenheimer, hope this could be useful: https://x-io.co.uk/open-source-imu-and-ahrs-algorithms/ and https://ahrs.readthedocs.io/en/latest/

[jfmoyen]

I would also appreciate very much this feature (which would be a good reason for me to switch from FieldMove to QField). Not sure this is a very constructive comment but if it helps to show user support...

[jfmoyen]

Hi r-gibs,
Elaborating on the "specs" for this feature...

Linear geologic features are measured differently than planar features, a "planar or linear" option could be included in the dialogue or a separate button / action could be used for linear features.

It's actually only partially true, and totally more complex. As we both know a plane is perfectly defined in terms of dip direction / dip, which is indeed a line. But there are differences, both in measurement and in notation.

In terms of taking the reading, when you measure a line you need to align some reference of your device (probably the long axis) with the linear feature. I presume the device then records its orientation (from the compass), and its inclination (from the tilt-meter) along the relevant axis, which are your trend and plunge. So if you have access to sensors, that's trivial.
When you measure a plane however there is an extra step involved (as my second year students are struggling with these days !!), which is identify the dip direction ("line of maximum slope"), and then you can align the device to this line and call your reading a plane (this is effectively what you do with your regular compass). With a device, a more clever way would be to use the tilt captor, find out in which direction (related to its axis) the device is tilting (presumably by combining X and Y components of tilt), and call that a dip direction. Well, not massively difficult either.

In some cases it makes sense to link the line and the plane: think of a fault with slickenslides (probably recorded as pitch in this case), or a foliation+lineation is a SL fabric rock (your will probably record the L as trend/plunge). In these cases the L and the S should be measured concurrently, to ensure that the line belongs to the plane geometrically (again, a concept my 2nd years struggle to grasp...).

In terms of recording the data, now there are different conventions, which would be useful to implement for different users and/or use cases:

• Planes can be recorded either as strike/dip or dip direction/dip. This is not a very big deal, it's a simple matter of adding or removing 90° (modulo 360). We'll probably stick to right hand rule here, as dip + quadrant (i.e. N0135/20 rhr -> N045/20 NW) is really a can of worms on computer
• Lines can be recorded as trend/plunge (needed on map), but also as pitch on a plane (faults), in which case you want a plane first ! The trig to convert from TP to pitch is a bit involved (because modulo 360 and also because you have two lines of similar pitch on a plane).

If you are curious, I'm attaching my current database that demonstrates these constrains and options on the "observations" layer. Have a look at the form, and what happens if you create a new observation. As you say, what it lacks now is a "fill from compass" button at the bottom of the form (blue placeholder...).

demo_geology_database.zip

(by the way, the form is live - see what happens if you type strike or dip - but I'm not sure I understand why it works, as I have the warnings that I cannot use fields in default values. I'm happy like that but I just hope it's not just implementation specific and will go away with the next update).

[keinadmin]

Hi jfmoyen,
i think the qfield compass feature needs two save buttons. one for "save plane" and one "save linear". As you mentioned one needs a reference line on the device - but only for measuring lineaments. because for measuring planes the orientation of the device is not relevant since the device is lying parallel on the plane. i have attached a code experiment which (hopefully correctly) measures two things:

1. the orientation of planar features independend of the orientation of the device parallel to the plane (ly your device on a plane and rotate it, the dipdir/dip values should not change)

and 2) the angle of the x axis ("GRAVITY_X_ANGLE" in the script). The only thing one has to do when measuring lineaments is aligning one side of the phone/tablet on the linear to be measured, and rotating the device on this axis until the value = 0, meaning the device is aligned horizontally on the short axis. When aligned this way the values for dipdir and dip are correct for that linear feature. This is exactly the same procedure like measuring a linear feature with a breithaupt compass.

the question of the used notation is independent from all that, this can be calculated from the measured values afterwards.

what would be more important is to save the date / time / location to every measurement to execute true north corrections afterwards. The necessary correction should of coure be build in the app but the uncorrected raw data should be saved too. Here the used correction values must be saved also for reproducibility.

Attached my experiment script for illustration the above written for use with a device with magnetometer and gravity sensor. install the app "droidscript", remane the attached .txt to .js and load that file and run it. Consider the script as a naive experiment for illustration purpose only. this script is not be usable in the field because a lot of useful things are not implemented (sensor drift compensation, no gyro info implemented etc etc).
Of cource the values for the lineament could also be calculated (the horizontal alignment would not be necessary) but i have not implemented this. But one would need the "save plane" and "save linear" anyway.
Test_25_geocompass.txt

[jfmoyen]

Hi keinadmin (Lukas ?),

Thanks for the script, it is quite nice indeed. I would perhaps take a slightly different approach, by combining gravity_x and gravity_y (if I read your code properly) to get the orientation of the gravity vector relative to the device (= i.e., the pitch of the device) and do a bit of trig to extract the dip dir out of that. In my view this has the benefit (i) of allowing to measure a plane just by laying the device on the surface, with no need to orient it further (lazy !) and (ii) more critically perhaps, to measure a line and a plane without moving the device, thereby ensuring that geometric consistency (i.e. the line is guaranteed to belong to the plane, which is a desirable feature when you do e.g. paleostress on faults).

This highlights, however, a possible key problem here: we are thinking of an interface that suits our specific needs and ways of recording field data. Now, one great feature of Qfield is that we are not tied to one way, we can organize our project the way we want it. I do not think QField developers will be wanting to develop something very specific to one use-case (nor should they). So we do not need so much an interface or way to organize the data - we need tools to allow everybody to develop one (well, everybody with some understanding of coding etc., granted, but frankly I doubt anybody who doesn't will be using the attribute form editor, default values and such like anyway).

Let's face it, a geological compass is a niche, unlikely to be a priority. On the other hand, a "low level" solution may be easier to implement (without the hassle of writing the GUI), and useful for a larger user base. Of course the down-side is that it would leave more work to the end user to customize and integrate it into his/her workflow.

I can see a couple of ways to enlarge "our" problem (a geological compass) to make it general enough to be worth developing - QField developers, it would be great if you could jump in the conversation at that stage and tell us what you think is feasible, realistic, desirable etc. !

1. We can already access quite a lot of GPS information with various @ tags, for instance QField honors things like @position_direction etc. (cf. https://docs.qfield.org/how-to/gnss/?h=gnss). Could it be possible to have similar @ tags for all sensors in the device? Here we think of @gravity_x and @tilt, but there could also be a use for all other sensors such as pressure, humidity, ambient light, noise level etc.

Armed with that, it would be not to difficult to add a default value to your table, along the lines of "strike" = atan(@gravity_x/@gravity_y) [I did not check the trig, but it should be something like that]. So the moment you'd create a point, by default QField would populate it with the current compass readings (you can always overwrite the defaults, remember...).

Interestingly, this may be a valuable option for other use cases. A bee keeper, for instance, may be interested in monitoring the noise level around the hives and use @noise_level in a table !

2. More boldly, or perhaps more simply actually, the possibility to add custom scripts to a QField entry form would be great. Perhaps it would mean something like a @script tag, which would then fire a user-defined script; or perhaps a "script" widget type (much in the same way we already have a date/time widget, colour picker widget, etc) that would also allow to fire a user script and collect the return value. Of course the scripts should be able to access device's sensors (QML can with QSensors, droidscript can in js, etc). I have no idea on which language should be supported : QML ? js ? python ?

Then, everybody would be able to develop scripts suiting the individual use cases. keinadmin's example being a sound basis for one.

So in summary, my two suggestions/feature requests would be

1. introduce @tags returning the values of all sensors (not only GNSS)
2. introduce a new type of widget, a "script widget", that can launch a script and return arbitrary values (the script itself being able to read sensors).

Best,

Jeff

PS- Brunton or Breithaupt? ;-)

[keinadmin]

Hi Jeff,
as mentioned my code is an unfinished experiment. i have only implemented a method where one can measure planes without taking care about the orientation of the device. This was also a while ago (2016). I then stopped further effords in this when i went deeper into the matter and discovered madgwicks algorithm and the technical things one has to consider when making an electronic compass (see my first post above). To me it was a little frustrating to have the ahrs module in python but nothing in js waht i could use with droidscript. since i am no programmer at all translating the ahrs part into js was beyond my skills. To conclude: making a good geologic compass with a device is no trivial task. This task should be done by people knowing their trade :-)
Breithaupt (the old one from the "DDR") and only Clar notation (superior due to simplicity)
Regards Lukas

[jfmoyen]

Oh, python and js are not that different, porting the code shouldn't be too difficult (famous last words).

Returning to the topic, how much of the complexities re. magnetic field, rotation etc. are left to be dealt with by the user, and how much are already managed at device level ? When your device returns, say, an orientation (and droidscript gives an example of this, by the way: CreateSensor("Orientation") ), has all the pre-processing been done already behind the scenes ?

[keinadmin]

when implemented properly everything runs behind the scenes. the goal is find good algorithms for calculating the "orientation" value. By the way, i even dont't know if madgwicks algorithm is perfect for this - i assume. Perhaps there are better solutions?
PS: "Freiberger" not "Breithaupt" - i left geology a time ago....

[keinadmin]

The goal is to get good orientation values for a variety of devices out there and the variety of build in sensors. You have lots of different stuff and you have to find a way to deal with that properly - eg as mentioned above the sensor drift is a proplem one has to deal with. it needs some technical skill and understanding of electronics to find a good solution.

[jfmoyen]

Yes, I see your point. I was hoping, naively it would appear, that the correction algorithms were dealt with at lower level, in such a way that the user would be served a "reliable" value (i.e. one already corrected for drift, and so on) through whatever API (s)he is using... Apparently such is not the case, which adds another layer of complexity to the whole story !

[keinadmin]

If you choose a good algorithm the things should run behind the scenes. this thing must deliver good orientation values. the point is, that one could calculate this himself with the raw (low level) sensor data in qfield as an intermediate level. The goal is only HOW to calculate these values. This approach must be watertight and bulletproof and must work best for a variety of devices - and this on the whole earth! As mentioned this is no trivial task and needs understanding in the behaviour of electronic sensors and lots more. On TOP of that we then can calculate our orientation(z) x gravity (z) or whatever we want with highlevel operations also in qfield.
Recognizing the technical complexity was the point when i dissatisfied stopped my droidscript geocompass experiment back then. But now good technicians and programmers have attended to this task - a reason for joy! I would be interested what the developers think about all this.

PS:
-) If you want you can fly over a paper that shows the technical background (eg https://www.researchgate.net/publication/342824423_Fast_AHRS_Filter_for_Accelerometer_Magnetometer_and_Gyroscope_Combination_with_Separated_Sensor_Corrections/fulltext/5f07d3a245851550509a1393/Fast-AHRS-Filter-for-Accelerometer-Magnetometer-and-Gyroscope-Combination-with-Separated-Sensor-Corrections.pdf?origin=publication_detail). But keep in mind that in most of this kind of publications the technical data of the used sensors are well known and mostly they use good sensors (good resolution, little drift, temperature compensation etc.). in our case we dont't know the sensors - most devices also have only low cost / low quality comsumer grade sensors :-(

-) Have also a look at the ahrs library if you want to https://ahrs.readthedocs.io/en/latest/index.html#

-) We still have our good old analog compasses and dont have to deal with all the above stuff if we dont want to :-)

I hope this helps in some way
Best Regards

[jfmoyen]

Interesting reading indeed, many complexities I was woefully unaware of ! I digged a little bit further along this line. It seems to me that Android is already doing the AHRS calculations before serving the value to the user:
https://developer.android.com/guide/topics/sensors/sensors_position
"The system computes the orientation angles by using a device's geomagnetic field sensor in combination with the device's accelerometer. Using these two hardware sensors, the system provides data for the following three orientation angles (...) " --> while not explicitely said, it seems to fit the bill.

So you should be downstream of this point the moment you call the API. That is, if you call orientation sensor, apparently a virtual sensor combining the input from two physical ones. You can still call the magnetometer and accelerometer and do the calculation yourself.

I have no experience with Apple but apparently it is also doing that, probably even better if anything.

However:
http://www.geo.cornell.edu/geology/faculty/RWA/programs/submitted_ms.pdf

... suggesting, amongst other things, that one of the issue is noise and sensor fluctuation, and that you want to average your reading for a period of time.

[dsilas]

Just another voice chiming in to say that I too would be interested in these features. Of course since most qfield users aren't geologists I don't expect qfield to implement a geological compass, but the idea (described above) of being able to get as @ tags some of these orientation values from the system (hopefully after android/iOS has already cleaned them up in some way) or other phone sensor values in project fields would be great for using the phone as a geological compass or using it to record topographic shielding, etc. I

I guess the reliability of the sensors in smartphones is variable (c.f https://doi.org/10.1016/j.jsg.2017.02.015) but nevertheless it would be useful especially when you don't have enough geological compasses to go round or not all the participants are trained to use geological compasses.

[jfmoyen]

I certainly would not use a smartphone for any "serious" structural work - something that would require precise measurements, good statistics, and so on. However I find it perfectly satisfying for regional mapping, students field camps or my own field work when I'm basically after a foliation attitude more than exact paleostress, for instance. It's also great when hiking with the kids, to take a quick reading of something you see on the way without actually stopping and pulling your notebook out :-)

Lastly - and this is where Qfield comes into the picture- it saves one rather tedious step, copying your notebook into some computer format to go to GIS. Saving the measurements directly in numeric form is actually great.

Still, even then, I would not get out without my compass. 10% of the time perhaps, I get a completely bogus reading, for no immediately obvious reason (they can range from a nearby powerline, to me having kept my hammer in the same hand, to me "locking" the measurement at the wrong time, to compass calibration....).

Returning to the topic of new features, the more I think about it the more convinced I am that the right way would be to allow QField to access all sensors on the device (perhaps with a @tag, same as GNSS info). Since all sensors have a similar API and are all accessed by the same QSensor library, it would probably not be more difficult to implement all than one - and it would bring functionalities to more than geologists.

[keinadmin]

You are very strict with your requirements Jeff. Besides the fact i know a couple of geologists who even don't make a north correction (strict south ahead) i would be fully stisfied if the thing has an accuracy of +-5°. Besides the fact that the size of the measuring area (phone vs tablet) has a big impact of the measuring result, geologic planes are usually never exactly planar in space and undulate/vary from bigger to smaller scale (at least where i live).
I think one could use eg fieldmove / clino app very good for efficiently measuring planar features in the field to get reproducable results. a friend told me that they tested smartphone compasses on a student field trip for structural geology and the results were not bad at all - also compared with a analog device, even the instructor was positive impressed.
Besides that disturbing effects like powerlines or enrichment of magnetic minerals interfere with every magnetic sensor, analog or electronic. Being concentrated at field work and checking every measurement for plausibility is certainly of big importance too.

[jfmoyen]

You are very strict with your requirements

Well, it's actually the opposite - I'm not a structural geologist, I'm perfectly aware of what I'm doing is not on par with proper structural work, and in any case I don't have enough interest in structures to take the trouble to assess how reliable my measurements are - so I rather err on the side of caution.

Having said that, the paper I was citing a few days ago seems to conclude that a (good) smartphone produces results that are not worse than a compass. And yes I agree, in real life 1° accuracy is not serious, natural variation, irregular surface and what next mean that you are probably measuring within 5° (another reason why I was not too bothered by the compass issues we were discussing earlier in this thread).

Which is my whole point I guess. With a mid-range Android smartphone and an app not fully optimized (as we may get in QField), you can probably not expect more than 5-10 ° accuracy, for all the reasons we have been discussing. If this acceptable for your purpose - great ! Go for it by all means. But if you require something more precise, shop somewhere else...

PS- try ignoring North correction when you are working in Southern Africa -- Mag North is no less than 22° from True North ! That can seriously mess up your results...

[keinadmin]

I am aware of the problems and the possible traps. By the way thanks for the Allmendinger et.al. manuskript . I've read it some time ago - a profound analysis and summary. I am very curious about the solution the qfield developers will implement :-)

[mbernasocchi]

I am very curious about the solution the qfield developers will implement :-)
thanks all for the interesting discussion, please note that, as mentioned above, a feature request will not just happen. Unless someone stepsup with a pull request, someone sponsors the work or there is a general interest that leads to a [sort of] crowdfunding effort, we can't guarantee that it will be implemented.
for more information, check out https://www.opengis.ch/custom-development

[jfmoyen]

Hi Marco, I'm perfectly aware of that. Still the discussion has been instructive in that it helped to delineate what was feasible/desirable/realistic (and also prompted me to explore what was already there).

To keep the discussion on topic, I'd like to make several suggestions/requests/comments (feedback welcome !):

1. Would it be possible for QField developers to comment on the technical solutions we proposed (e.g. new @ tags, scripts, QML widgets etc.) ? What is technically unfeasible or undesirable, what is doable, what is simple or quasi-implemented ?

2. I fully understand that a geological compass is a very restricted use case, unlikely to be of broad interest and therefore unlikely to ever be a priority for developers. Hence, my suggestion to generalize the problem and give access to all sensors (using QSensors, that is apparently already loaded in QField ?). It would then be user's responsibility to integrate the readings in their database (and, in our case, to convert orientation to strike/dip, etc - but also in the case of, say, environment, to make use of a sound intensity reading or an illumination reading). Would that be a broad enough application to be more palatable to QField developers ?

3. Following a different tack, if there are enough geologists around, we may pool our efforts and either (a) code what we need or (b) see if we can leverage enough money to sponsor the development of this feature. If this something you are willing to consider, I'm happy to coordinate our efforts, contact me directly (my email is found in my profile).

Regarding option (a) - I can code some but I have no experience with android deployment, so I can help but hardly do it on my own (not in a time compatible with a full time job, anyway...). I've never done QML, too, but it seems simple enough.
Regarding option (b) - well, at some point we will need a ballpark estimate from Marco and co. to see if we are talking thousands, tens of thousands or hundreds of thousands ;-) (@mbernasocchi ??)

[jbell6]

An estimate would be great - I'd happily donate to a crowdfunding campaign and spread the word - we might be able to persuade some oil/mining companies or universities to donate as well and speed up the process.

[r-gibs]

I contacted QField about feature development several years ago and they thought estimated hundreds to thousands of euros (depending on format). An external sensor format might be the best option (like a separate app that QField grabs the strike/dip from).
I couldn't fully support the development myself, but started this feature request to see if there was interest / financial commitment from other parties.
@m-kuhn it seems like @jfmoyen is willing to coordinate efforts and fundraising for this feature.

[m-kuhn]

Nice. Please shoot us a mail to [email protected]

[davenquinn]

I am a maintainer of Rockd, the app mentioned at the top here. Our compass works quite well and, since Rockd is a geology-focused app, it's a little more realistic for us to maintain a well-integrated compass than leaving it to the QField team. We've indeed spent hundreds to low thousands of dollars keeping the compass feature working in the last few years (as part of broader update costs).

One path forward would potentially be to allow orientations collected on our structural compass to be easily exported to QField. We'd be open to making modifications to allow better cooperative mapping between apps.

[nirvn]

@davenquinn , we have implemented sensor handling capability to QField's upcoming 2.8 and QGIS 3.32. The three raw sensor types implemented ATM are: TCP socket, UDP multicast socket, and serial port (for windows & linux, not as interesting for mobile devices).

It would be great to test out your geological compass app streaming values through TCP or UDP socket and caught by QField. Let me know if you need some help.

[jbell6]

The new version of QField (3.3) appears to allow plugins - hopefully this means a geological compass plugin can be made reasonably easily.