Review design of data entry forms to make mandatory vs. optional fields clearer

[emmajclegg]

Task

• Implement changes in formbuilder #1572
• Fix document link form in result and indicator #1599

Context:

Follows discussion about the result element in #1542

User story: As a user, I want data entry forms to be simple and intuitive, with mandatory elements clearly marked, so that it is easy for me to input my data.

There are currently collapsed optional fields for the element document-link, although we don't think this format will work to copy across the rest of IATI Publisher.

Key user issues:

1. Mandatory asterisks are used at the level of data fields within sub-elements, with no mandatory vs. optional information for the sub-element itself:

2. There are an overwhelming number of data entry boxes in many forms (especially transaction, result, indicator and period)

Can we improve the design of data entry forms further by:

• removing unnecessary green lines and borders to reduce clutter in the interface (see example screenshot below)
• clearly label optional data elements as "optional", or collapse areas of the Standard that are very rarely used (ODS can advise what to do for each element, and the example below shows how "optional" can be added to text labels)
• in the case of nested elements (e.g. adding a document-link within result), the user shouldn't see all of the nested document-link data fields in the result data entry form, unless they have chosen to expand this element - e.g. with a "Add a document" option. This will vastly simplify the forms by reducing the number of data entry boxes the user is first presented with.

This is a (very rough) example of how much cleaner the form could look without unnecessary borders:

If we use the result element as an example,

• type, title and indicator are mandatory
• document-link can be collapsed, with the user given the option to "add a document"
• reference, description and aggregation-status can be kept on screen and labelled as "optional"
• both type and aggregation-status could be better presented as multiple choice buttons (as in AidStream simplified UI)

It would help us to think beyond rigidly presenting the technical IATI Standard to people, and more about what users expect to see from a data entry form, in language that makes sense to them.

Once we agree on a design, we can create separate GitHub issues for implementing across data forms in IATI Publisher, in an order that makes sense.

[emmajclegg]

@praweshsth - if there's free capacity for front end design work this month, this issue has been separated out from #1542 and is high priority to work on.

We increasingly get support queries related to problems interpreting the data entry forms, and I'm conscious these are going to get scrutinised more as we start raising awareness of IATI Publisher over the next few months. This also affects more users than Excel imports so we can discuss prioritisation further on our Tuesday call.

[emmajclegg]

For info - I've added a second screenshot to the description above (under "key user issues") to emphasise the two levels of mandatory vs. optional fields that we're dealing with, and that are causing user confusion. Only one (the nested level) is currently clear in the data entry forms.

Any questions, please let me know.

[BibhaT]

@emmajclegg Asmina will share the design with you, Momik will come up with the time estimation on Monday.

@PG-Momik @asminashrestha

[asminashrestha]

Hi @emmajclegg We have come up with the first draft for the form redesign. View link We’ve simplified the form by removing lines that previously cluttered the layout and minimizing the nesting of elements. To enhance the user experience, we’ve made the fields collapsible, ensuring the form looks clean and organized. This way, users can focus on entering only the necessary information, expanding fields only when needed. Also, we have added a sticky sidebar so that user can use it to open and move to the need fields as necessary.

[emmajclegg]

Thanks all.

@asminashrestha - the design's looking better. I'm not sure how the mandatory asterisks ( * ) are working in this design but making mandatory vs. optional fields intuitive remains the main thing i'm trying to address.

In this result example, @/type and the Title sub-element are the only things that are mandatory, so can we make sure this is clear - e.g. by adding an "optional" label on the collapsed fields highlighted:

I'm not sure how much the sidebar will be needed for navigation given the new collapsed format - I'll ask my team for quick thoughts on Monday. Either way, the sidebar should probably use the plain English element names to match (rather than the technical IATI schema terms)

[PG-Momik]

Hi @emmajclegg,

Just a quick note on this user story from a development perspective. All forms in IATI Publisher are rendered via the form builder, which uses a common mechanism across all forms. This means any design changes we implement for activity results (as shown in the example) will affect all other activity elements and sub-elements as well. Unfortunately, we wont be able to isolate elements and implement changes to them one after next (Example: We cant begin with result, then indicator, then period then activity sector, .. etc).

With this in mind, my rough estimation is that implementing the changes will take approximately 50-60 hours, plus an additional 2-4 days for QA, depending on the scope of changes once the designs are finalized.

cc: @BibhaT @Sanilblank

[emmajclegg]

Hi @PG-Momik

All forms in IATI Publisher are rendered via the form builder, which uses a common mechanism across all forms. This means any design changes we implement for activity results (as shown in the example) will affect all other activity elements and sub-elements as well.

This makes sense. The important thing will be to make sure that the form design we agree on works well for both simple IATI elements and the more complicated data entry forms like activity transaction and result

With this in mind, my rough estimation is that implementing the changes will take approximately 50-60 hours, plus an additional 2-4 days for QA, depending on the scope of changes once the designs are finalized.

Can you provide more detail here on the steps required for implementation? This time estimate is higher than one of our developers would expect.

In particular - how has your core component library been set up to reuse common elements across data entry forms in IATI Publisher? A number of issues on the project board now involve reuse of IATI schema rules too - mainly, interpretation of what data is mandatory vs. optional. I would hope this schema logic is maintained in one place in IATI Publisher and reused, rather than being hard-coded in multiple places within data entry forms and processing?

If helpful, I can ask advice from one of the ODS developers on how we best implement changes.

---

@asminashrestha - if helpful for feedback, my team agree that the new form sidebar design might confuse more than help, and does not seem necessary for navigation in forms that will be shorter in length (due to the collapsed sub-elements):

If removing this sidebar completely, or keeping the sub-element menu that's currently in IATI Publisher (below), saves time on development work then please can we take that approach.

[PG-Momik]

@emmajclegg I've created an issue for the tasks for this issue based on the design. Implement changes in formbuilder. I've estimated a higher for this issue because of my previous experiences with form builder. Form builder is the most complex module in IATI publisher. The initial team that worked with the form builder are no longer here. The inner working of formbuilder is quite frankly "tribal knowledge". Overtime we've made small changes on form builder (freeze fields/elememts, collapsable, default value highlight), these small changes have always taken longer than what we'd expected.

In particular - how has your core component library been set up to reuse common elements across data entry forms in IATI Publisher

I dont completely understand what you mean by this. Basically forms can be broken down into:

1. How form is defined:

• We maintain json echemas that basically define what and how a form SHOULD look like (basically control how form is rendered). elementJsonSchema.json contains the definition of all activity element form and children form, organisationJsonSchema.json for organization level form.

2. How form is rendered: (Form builder)

• Files inside Builder and Form will basically render the forms based on the definitions in json schema.

3. Form Validation:

• Forms are validated against interpretations of IATI standard on request level. i.e request file. Example if you fill Activity title with 2 narrative with same language, it will throw a validation error. This logic is handled in TitleRequest.php

4. Behind the scene logic (completeness, mandatory v optional):

• Calculations like core complete calculation, element completion status, deprecation code usage are usually done once the form is saved. The logic for this is handled by service files in https://github.com/younginnovations/iatipublisher/tree/main/app/IATI/Services.

• These service file either user json schema or have common functions within them to do necessary calculation. Example to calculate if activity sector is complete or not. ElementCompleteService.php reads the JSON file for rules needed to check what what attributes and subelements are mandatory. It then fetches activity sector data from database and checks if the elements and attributes (needed for completion, defined in JSON file) is complete or not. The logic to check if complete or not is found in service files and not in JSON files.

This is a very simple breakdown of how forms are rendered and handled.

cc: @Sanilblank

[emmajclegg]

Thanks @PG-Momik for the detail here and on the call earlier today. It's useful for us to have a basic understanding of how these functionalities work in IATI Publisher.

I had another look at the form design you presented in Figma (cc' @asminashrestha):

• One question - will the open vs. collapsed field approach (as shown in the Initial State view) also be applied to nested elements? For example, in the result form, if the user expands the optional "Document Link" sub-element, will they see "Title" and "Category" open in the interface, then "Description"/"Language"/"Document Date" as collapsed? This would be the ideal to continue to guide users to mandatory sub-elements first.

I've seen the implementation steps here and have no further questions for now.

[asminashrestha]

@emmajclegg We won’t be applying the collapsible approach to nested elements. Hiding too much information within collapsible sections could indeed lead to confusion. You're right about distinguishing between mandatory and optional elements, but we believe the use of an asterisk (*) is sufficient to indicate required fields. Nesting optional elements could make the form more complex and add unnecessary confusion.

[BibhaT]

Hi @emmajclegg , depending on whether or not your concern has been addressed by the above comment, should we move ahead with the development?

[emmajclegg]

We won’t be applying the collapsible approach to nested elements. Hiding too much information within collapsible sections could indeed lead to confusion. You're right about distinguishing between mandatory and optional elements, but we believe the use of an asterisk (*) is sufficient to indicate required fields.

Having spoken to my support colleagues, we still don't think this is clear enough unfortunately. document-link is the nested element we've needed to support users with the most (in terms of making sure they fill in mandatory fields), so there is evidence behind asking for the following:

1. We would still like optional sub-elements, even within nested elements, to be collapsed when the user first views an element within the data entry form.

Looking at the example of "Category" and "Description" from Document Link below, the mandatory asterisk next to Category is a good start, but it still looks like the Description narrative and language fields need completing when they don't:

2. We would like the word "optional" clearly marked on the sub-elements it applies to.

As soon as the user starts expanding optional elements, similar to above, we can see that it becomes less clear what is optional. A label like below would retain that clarity even with the form fully expanded:

Any questions, let me know.

With these changes applied, I'm happy to move ahead with the development.

[emmajclegg]

For further user context @asminashrestha @BibhaT - the critical errors in this publisher's data (https://validator.iatistandard.org/report/fowdk-activities) are representative of what us in support have to troubleshoot daily.

It is far too common to see critical errors in document-link and result, as in this case, which is why this data entry form work is so important. We want to guide, rather than just show, users what is mandatory to improve the data currently being published.

[asminashrestha]

Hi @emmajclegg We've made the requested changes. Please take a look and let us know if you have any feedback. Updated Form

[emmajclegg]

Thank you @asminashrestha - the changes look good so go ahead and start on development

[BibhaT]

Thanks @emmajclegg we'll start the development from today.

[Sanam-07]

• Issue 1 : We can give the equal width to all the container containing the text like the figma design. Based on discussions, pill buttons will not require changes
  Actual Design :
  
  Figma Design :
  

• Issue 2 : The First letter of the title , description ,document-link and reference should be capital letter just like the figma design.
  Actual :
  
  Figma :
  

• Issue 3 : When the user goes inside the result or try to edit the result then the path should be defined above the page as highlighted in the screenshot.
  
  Figma Design :
  

• Issue 4 : The placeholder seems missing in the narrative field of title section.
  

• Issue 5 : When the user encounters an error in the location form while saving, then the error message should be displayed without collapsing the error section.
  Video.webm
  

• Issue 6 : An error message occurs when the user try to open the document-link form in activity level.

• Issue 7: There are two "Add additional mailing address" button inside the mailing address of contact-info form . (activity level) INTENDED BEHAVIOUR
  

• Issue 8: The error message gets displayed when the user try to open the transaction form.
  

• Issue 9 : When the user navigates to the activity from the path in the add period form then the page not found error gets displayed.
  
  video1.webm

• Issue 10 : When the user navigates to the period list, indicator list, from the path in the period form then the page not found error gets displayed.
  issue10.webm

[PG-Momik]

@emmajclegg This has been deployed to staging. Forms in both activity level and organisation level have been update.

Major Changes:

1. Simplify nested forms + consistent borders
2. Implement collapse functionality to every elements (Previously only in document link)
3. Added required / optional label on form's element header.
4. Changed positioning and text of delete parent button
5. Added Highlight on delete parent button hover
6. Breadcrumbs on edit form.

Please have a look and let me know if any changes are required.

cc: @Sanam-07

[emmajclegg]

Thanks @PG-Momik - this is looking good. I've gone through element-by-element with things to check below.

1. Check mandatory vs. optional labelling

ACTIVITY DATA:

• conditions sub-element "condition" is showing as mandatory when it's optional
• planned-disbursement sub-element "period-end" is showing as mandatory when it's optional
• budget sub-element "value" is showing as optional when it's mandatory

ORGANISATION DATA:

• recipient-region-budget attribute "status" is showing as mandatory when it's optional

2. Remove yellow "core" rings against sub-elements
(these are confusing at the sub-element level as mandatory fields are already marked with a red asterisk).

The following elements are affected:
ACTIVITY DATA: budget, planned-disbursement, transaction
ORGANISATION DATA: total-budget, recipient-organisation-budget, recipient-region-budget, recipient-country-budget

Example for budget:

3. No space between element names and the "(Optional)" label in cases where there's no help text:

After completing this issue, it'll be much easier for us to review remaining data completeness alerts across the tool - I've created a new issue for this (#1597), likely for November work.

Any questions, let me know.

[PG-Momik]

@emmajclegg Changes you requested have been deployed to staging. Since Core elements ring is displayed via a generic function, core elements ring has been removed for all sub-elements.
Let me know if you wanted them removed specifically for :
ACTIVITY DATA: budget, planned-disbursement, transaction
ORGANISATION DATA: total-budget, recipient-organisation-budget, recipient-region-budget, recipient-country-budget

[emmajclegg]

Thanks @PG-Momik. The element below still needs checking but the other changes look good:

ORGANISATION DATA: recipient-region-budget attribute "status" is showing as mandatory when it's optional

When we implement these new data entry forms, is there anything to be aware of in terms of impact on existing users' data in the interface? As far as I can see, the main thing would be that data they've entered in optional sub-elements will now appear hidden to them (within the new collapsed sub-element format).

Also, for awareness - I was showing the data entry forms to support colleagues today and we discussed the positioning of help text being inaccessible (i.e. the yellow box in the screenshot below). I have already created an issue on the backlog (#1531) about reviewing it, likely over the next two months, but wanted to mention here in case there's overlap with this form builder work.

[PG-Momik]

@emmajclegg No there wont be any data level changes for existing data. As you mentioned, even if users have filled optional sub elements, they will be rendered in collapsed state.

[emmajclegg]

I'm happy for these data entry form changes to be implemented

[PG-Momik]

Changes have been deployed to production.

[emmajclegg]

@PG-Momik - I'm reopening this issue because of a potential bug related to document-link:

CASE 1 (correct behaviour): For document-link as an activity element (example), the mandatory vs. optional fields of the data entry form are correct:

CASE 2 (incorrect): For document-link as a sub-element within a result (example), "category" is marked optional when it should be mandatory:

CASE 3 (incorrect): For document-link as a sub-element within an indicator (example) or period target/actual (example), both "title" and "category" are marked optional when they should be mandatory:

We know users already don't complete "document-link" well so can the data entry forms in cases 2 & 3 be fixed to match case 1 as soon as possible please (to avoid users getting even more confused)

Any questions let me know.

[PG-Momik]

@emmajclegg This has been fixed and deployed to all environments.

[emmajclegg]

Great - thank you @PG-Momik