Add postprocessing_info to device

[viktorsmari]

Closes #179

• Add postprocessing_info to the db as a jsonb
• Allow postprocessing_info in the controller
• Do we need to manually do a JSON.parse on the controller or will rails handle it?
  In mqtt_messages_handler.rb we do this:
  device.update hardware_info: JSON.parse(message)
• Add the examples below in https://developer.smartcitizen.me/#update-a-device

Example PATCH via curl

As a string:
curl -i -X PATCH <url> -d "postprocessing_info=some data here"

Or a JSON:
curl -i -X PATCH <url> -H "Content-Type: application/json" -d '{"postprocessing_info":"bbbca"}'

Nested JSON:
'{"postprocessing_info":{"some":"data2", "other":"data"}}'

Using the data

In rails console, this is stored as:

Device.last.postprocessing_info
=> {"some"=>"data2", "other"=>"data"}

Allowing nested objects

In order to allow posting nested objects, we need to specify each object on the device_controller

postprocessing_info: [:some, :other]

https://stackoverflow.com/questions/16549382/how-to-permit-an-array-with-strong-parameters

[pral2a]

@oscgonfer could you review this?

Maybe you could post an example payload to verify it works

[oscgonfer]

Hey,

I was testing this now with the following:

curl -i -X PATCH 'https://api.smartcitizen.me/v0/devices/13238/' -H "Content-Type: application/json"  -H 'Authorization: Bearer -----------REDACTED-------------------' --data-binary '{"postprocessing_info":{"hardware_id":"SCS20008"}}'

HTTP/2 200
date: Wed, 28 Oct 2020 15:39:07 GMT
...

Without success, even with the 200:

curl 'https://api.smartcitizen.me/v0/devices/13238/' -X GET | grep 'postprocessing_info'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10336    0 10336    0     0  21137      0 --:--:-- --:--:-- --:--:-- 21137

-_-

Am I doing anything wrong?

[viktorsmari]

@oscgonfer this has not been merged yet - still a work in progress.

We need a bit more context / examples on what info we need to save in the database, and how we want to structure it.

For example if we want to create a few new columns (how many, what they should be called) in a table that we can sort/index on - or if we just "dump" a long string into a single column.

If we talk about the requirements in #179 we can figure out what makes sense.

[pral2a]

@oscgonfer I think what @viktorsmari needs to know is simply:

A. We want to store a single level json

That is easier to integrate in to the Rails flow and we can easily filter properties inside or via the REST interface with a bit of configuration on Ransak. (more info on Ransak integration) i.e.

{
   "key": value,
   "key": value,
   "key": value
}

B. We want to store a full json object

That is easy to integrate if we consider it an string, no search or filter available, at least by default. We can always implement a custom filter or mapping for a specific property i.e.

{
   "keyA": value,
   "keyB": {
       "keyB1": value,
       "keyB2": value,
       "keyB3": value
     },
   "keyC": value
}

@viktorsmari I'm explaining that to @oscgonfer as the purpose of using JSON is we don't know yet how the model might grow. However, I don't think option B is bad for now, the same I think option A can perfectly support the model @oscgonfer has in mind.

[oscgonfer]

Hi,

Right now, what I am aiming at is option A. Actually, it would be a single level json with a multiple properties, that will lately be mapped to the IDs here: https://github.com/fablabbcn/smartcitizen-data/blob/master/hardware/hardware.json

As @pral2a says, I am not sure atm how the data model will grow, but I rather have the largest amount of information on the framework side, since it's the one that ultimately will make the numbers. Nevertheless, If you think that hardware.json could be or should be fully stored in the postprocessing_info, option B would be best, as this will evolve and should be adaptable to multiple sensor blueprints (2x, 4x, 6x, ... sensors)

[viktorsmari]

With option B we need to be sure that when updating only keyB2 (in the example above) that the other keys wont be removed.

Every time we want to update keyB2 we have to post all the keys (I think - I need to double check) - We won't be able to simply update a part of the object. Since it is just a string, and not a row in a database.

Imagine we do a POST with keyB2: 125 and it would remove the rest of the data (key A,B,C) and only input the data we posted. So the poster needs to know the whole object, or fetch it first, edit parts of it, and the post the whole object / string to the API.

This would be the perfect use case for a 'nosql' database like Mongodb, where you don't care about the schema and just put in whatever you want.
But we are using an SQL database (Postgres) so we need to think a bit how to structure the data.

Maybe it's easier to talk on a call to sync faster?

[pral2a]

@viktorsmari I think option A is perfectly fine as @oscgonfer already stated.

The postprocessing_info will look something like:

{
	"updated_at": "2020-10-29T04:35:23Z",
	"template_url": "https://github.com/fablabbcn/smartcitizen-data/blob/master/hardware/hardware.json",
	"template_version_hash": "434a1d245c3f115e99988a4ae3ed6751bb9f31c3",
	"latest_postprocessing": "2020-10-29T08:35:23Z"
}

Also, we always plan to update all the properties at once. The application reads the postprocessing_info blob, performs some mutations and writes the full blob at once.

The only time we might need to access the properties individually will be from rails inside. Imagine the case we want to add a system_tags to devices where postprocessing_info and latest_postprocessing exists, and the date is within the last 24h, in this case, we might append a system_tags like postprocessed. Keep in mind it is just a random example.

[oscgonfer]

Fully agree with @pral2a latest comment. Option A it is.