Error produced when selecting checkbox in DFG checklist

[DMorgan94]

Description / Beschreibung

When clicking on x tick box (data) *.asc , for the question "Which data types in terms of data formats (e.g. image data, text data or measurement data) arise in your project?" in the DFG checklist, the following message appears:

"This field could not be saved, since somebody else did so while you were editing. You will need to reload the page to make changes, but your input will be overwritten."

and is accompanied by an error in the console log:

I believe this might also be occurring in other places in the DFG checklist as well, in the absence of any saving indicated by the error message produced. Upon refreshing the page the selected response is not saved, meaning that users cannot select that particular option.

It could be related to issue #815 , as this was identified within the same setup.

Expected behaviour / Erwartetes Verhalten

The software should save the response and not produce the aforementioned error.

Steps to reproduce / Schritte zum Reproduzieren

1. Go to the DFG checklist DMP
2. Go to section: Which data types in terms of data formats (e.g. image data, text data or measurement data) arise in your project?
3. Select (data) *.asc
4. See error

Context / Kontext

OS: Debian 12
RDMO version: 2.1.1
General setup: apache2 2.4.57, Python 3.11.2, WSGI setup with Shibboleth

[jochenklar]

This is a new feature which should prevent inputs by several users at once. (We had nothing to prevent that before.)

I assume that only you edited the page in this case? Does it work if you reload the page? Did you have bad internet when this occured?

[DMorgan94]

Hi, thanks for the quick response!

I first selected multiple options and then I just tried to select that option in isolation by itself - if that is what you mean by edit the page? (apologies if I am misunderstanding). The error occurs under both circumstances. The error also still occurs when I reload the page and when the internet connection is stable.

Let me know if you need any more information. Thanks again for your help.

[MyPyDavid]

I've had something similar like this when testing out an older existing project in 2.1.2. I thought there was something wrong with the project or catalog because I could not generally reproduce this error.

In my case in an existing project, for a question with checkboxes, there were already Values stored in the Project but they didn't show up as checked checkboxes. I deleted the Values (via admin) and the error didn't occur anymore.
Could there be something going on with stored Values for questions with checkboxes in relation to this feature? @jochenklar

I thought that maybe the checkboxes were referring to the same Attribute but this wasn't the case..

[jochenklar]

Maybe I missed something here: https://github.com/rdmorganiser/rdmo/blob/main/rdmo/projects/validators.py#L32. From the code, I don't think values are used for the validator, but not displayed on the page.

[jochenklar]

@DMorgan94 can you check in the network screen of the browser what the error message for the POST request, which returns 400 is?

[DMorgan94]

Thanks both for the help, here is a screenshot of the network screen showing the error message for the post request:

[jochenklar]

Is an attribute set for the question? You can check that in the management interface. Otherwise can you make a screenshot of the same request, but the json payload it sends?

[MyPyDavid]

In https://github.com/rdmorganiser/rdmo/blob/main/rdmo/projects/validators.py#L32 I get serializer.instance is None but serializer.context['view'].request.data.get('changed') is True for a certain checkbox question. Maybe it gets the wrong collection_index?

[DMorgan94]

Morning @jochenklar and @MyPyDavid, here are the screenshots of the attribute set for this question (which I retrieved from the management interface) and the screenshot of the json payload which it tries to send upon selecting the checkbox:

[jochenklar]

Hey @DMorgan94 , thanks for the update. Are those 2 option sets in this question? (Which would be fine, but maybe this causes the bug.)

[DMorgan94]

Hi @jochenklar, from what I can see there is only 1 option set within this question.

Thanks again for the support!

[jochenklar]

Ok, so I tried to reproduce the problem here and it works as expected. It could be that there are older values (i.e. user inputs) stored in the database which cause problems. You could go to /admin/projects/value/ on your instance (if you have access there) and search for project/dataset/format. Then you see all answers for all projects. You can sort and look for your project (or search) and check if all rows have distinct set_prefix, set_index and set_collection. As a quick fix you could also delete all values for this project (!) and check in the interview if the problem persists.

[DMorgan94]

Morning @jochenklar, thanks for your previous response. I looked into what you proposed and I think you are correct, but I'm not sure. In questions where I was getting the aforementioned error (e.g., What measures are being adopted to ensure high data quality?), I can see that there is already a previously stored response despite that section of the DMP. For that question there are identical Set_Index and Collection_Index values, one referring to a previous snapshot of the plan and another presumably referring to the current version:

I did not however see this for the original question which prompted my post here (see above).

Nevertheless as you suggested I deleted all values for the project in our development instance and the problem no longer persists, which is great. So perhaps this was indeed the issue after all. What do you think? How can we avoid this happening in future?

Thanks again for all of your support, I really appreciate it.

[jochenklar]

It could be, that the value for one checkbox was stored twice before the update and it did not cause any problems. There is a check now so, it should not happen again. Please tell us if the problem pops up again. (I hope not)

[MyPyDavid]

So in previous versions it could happen that a value for one checkbox would be stored twice? This depended then on the catalog/question?

As an instance admin, how can I find all of these "duplicate" values in the db, so that I could check all potentially affected projects in the instance? Think that this should be addressed before deploying on an instance with a lot existing projects.

[MyPyDavid]

I could reproduce this bug or behaviour.
Only after switching the Catalag in a Project (and answering a checkbox question with the same Attribute) I got the "conflict" validation error again.
In a new Project it doesn't appear.
For switching a Catalog in a current Project we might need some signals or something (also for updating the progress for example)?

[MyPyDavid]

I'm trying to figure this bug out, but am not sure where the source is..
In the interview, after I've answered a checkbox question and switched catalog. I get the following JSON:

Values in current question by attributes from
http://localhost:8000/api/v1/projects/projects/12/values/?attribute=194&attribute=193&attribute=286&attribute=276
For 193 only:

{
	"1": {
		"id": 529,
		"created": "2024-02-05T16:27:30.108609+01:00",
		"updated": "2024-02-05T16:27:30.108613+01:00",
		"attribute": 193,
		"attribute_uri": "https://rdmorganiser.github.io/terms/domain/project/dataset/format",
		"set_prefix": "",
		"set_index": 0,
		"set_collection": true,
		"collection_index": 1,
		"text": "",
		"option": 145,
		"option_uri": "https://rdmorganiser.github.io/terms/options/file_type/GR",
		"file_name": null,
		"file_url": null,
		"value_type": "text",
		"unit": "",
		"external_id": ""
	}
}

Clicking the checkbox then gives:

POST Error 400 from

"conflict": ["An existing value for this attribute/set_prefix/set_index/collection_index was found."]

http://localhost:8000/api/v1/projects/projects/12/values/
With request json.

{
	"additional_input": {},
	"attribute": 193,
	"changed": true,
	"collection_index": 1,
	"errors": [],
	"external_id": "",
	"file": null,
	"option": 444,
	"project": 12,
	"removed": false,
	"selected": "444",
	"set_collection": true,
	"set_index": "0",
	"set_prefix": "",
	"text": "",
	"unit": "",
	"value_type": "option"
}

The only difference seems to be the option. The checkbox is also not checked even though there is a value for that attribute/set_prefix/set_index/collection_index.
What should be the behaviour for this @jochenklar? How to handle values that have similar attribute/set_prefix/set_index/collection_index but have different options?
I guess adding and checking for the option in ValueConflictValidator would also fix this bug right?

[jochenklar]

Ahh! Thanks for investigating. So there is a checkbox checked in one catalog, but it is not available in the other. So the user tries to create a new value for this attribute/set_prefix/set_index/collection_index, but there is already one in the database. I need to think about this.

[MyPyDavid]

yes, but the combination attribute/set_prefix/set_index/collection_index/option is different ;)

[jochenklar]

I wonder if there is a case where this should be (in-)validated in this way. Maybe with a collection of select fields? Somebody POSTs a value with option A and then somebody POSTs one with option B.