[EPIC] Import/Export: ✨ Improve Error Formats in JSON /validate Responses

[fmontes]

Problem Statement

The current validation response structure in our system is unclear and inconsistent. This complexity leads to increased development costs and potential errors as it mixes data types, includes unnecessary information, and lacks a standardized error format. Simplifying and standardizing the structure would improve reliability and ease of integration.

Objectives

• Redesign the validation response object to ensure consistent data structures.
• Introduce standardized error and warning messages with unique codes.
• Eliminate redundant data and logically group related data within the response.

User Story

As a developer, I want the validation responses to be standardized and clearly structured, so that I can easily integrate and manage them in my applications.

Detailed Requirements

Here is an example of the proposal response.

{
  "results": {
    "file": {
      "totalRows": 4,
      "parsedRows": 3,
      "headers": {
        "valid": ["title"],
        "invalid": ["contentHost"],
        "missing": ["image", "blogs"]
      }
    },
    "data": {
      "processed": {
        "valid": 3,
        "invalid": 0
      },
      "summary": {
        "created": 3,
        "updated": 0,
        "contentType": "Fake Content"
      }
    },
    "warnings": [
      {
        "code": "INVALID_HEADER",
        "field": "contentHost",
        "message": "Header doesn't match any Content Type field; column will be ignored"
      },
      {
        "code": "MISSING_KEY_FIELD",
        "message": "No key fields specified, may result in duplicate content"
      },
      {
        "code": "INCOMPLETE_HEADERS",
        "message": "Not all Content Type fields match file headers. Some fields may be empty"
      }
    ],
    "errors": [
      {
        "code": "INVALID_FILE_TYPE",
        "message": "File type is not supported"
      },
      {
        "code": "INVALID_IMAGE_PATH",
        "row": 2,
        "field": "image",
        "value": "/invalid/path.jpg",
        "message": "Image path not found in Site Browser"
      },
      {
        "code": "INVALID_RELATIONSHIP",
        "row": 3,
        "field": "blogs",
        "value": "invalid-blog-id",
        "message": "Blog with ID 'invalid-blog-id' not found"
      },
      {
        "code": "REQUIRED_FIELD_MISSING",
        "row": 4,
        "field": "title",
        "message": "Required field 'title' is missing"
      }
    ]
  }
}

The response is a JSON object with a main results object containing:

• File processing information
• Data processing summary
• Warnings
• Errors

File Information (results.file)

Must include:

• totalRows: Total number of rows in the uploaded file
• parsedRows: Number of successfully parsed rows
• headers: Object containing:
  • valid: Array of valid headers matching content type fields
  • invalid: Array of headers that don't match any fields
  • missing: Array of required fields not present in headers

Data Processing (results.data)

Must include:

• processed: Object with counts of valid/invalid records
• summary: Object containing:
  • Record creation/update counts
  • Content type information

Warning System (results.warnings)

Each warning must include:

• code: Standardized warning code (e.g., INVALID_HEADER, MISSING_KEY_FIELD)
• field: Affected field name (when applicable)
• message: Human-readable description

Common warning codes to implement:

• INVALID_HEADER: When headers don't match content type fields
• MISSING_KEY_FIELD: When unique identifiers are missing
• INCOMPLETE_HEADERS: When required fields are missing from headers
• There are probably more that I don't know about

Error System (results.errors)

Each error must include:

• code: Standardized error code
• message: Detailed error description
• row: Row number where error occurred (when applicable)
• field: Field where error occurred (when applicable)
• value: Invalid value causing the error (when applicable)

Common error codes to implement:

• INVALID_FILE_TYPE: For unsupported file formats
• INVALID_IMAGE_PATH: When image references are invalid
• INVALID_RELATIONSHIP: When referenced entities don't exist
• REQUIRED_FIELD_MISSING: When mandatory fields are empty
• There are probably more that I don't know about

Importance

1. Debugging: Detailed response structure helps identify issues quickly
2. User Experience: Clear error messages help content editors fix issues independently
3. Validation: Comprehensive checks ensure data integrity
4. Maintenance: Standardized error codes make error handling consistent

Development Guidelines

1. Maintain consistent error/warning code naming conventions
2. Include row numbers whenever possible for easier debugging
3. Provide specific field names in error messages
4. Include invalid values in error messages when applicable
5. Keep messages human-readable and actionable

Acceptance Criteria

Preview Give feedback

1. Validation responses use consistent data structures for all types of data.
2. Error and warning messages in the responses include unique identification codes.
3. Each part of the JSON response (file details, data processing results, warnings, and errors) is logically grouped and clearly differentiated.
4. Redundant information is removed from the validation responses.
5. Responses are backward compatible with previous versions or include adaptational guidelines for easy integration.
6. External Links
7. [Placeholder for links to previous discussions, design documents, and related tickets on architectural decisions]

Technical Details

The response structure reform relies on standardized JSON formats.

• Consideration of how these changes may affect existing systems, particularly those heavily dependent on the specific format of error messages and codes.

Potential Challenges

• Detailed definition and documentation of new error and warning codes to ensure they cover all conceivable scenarios.
• Ensuring that the new structure caters to both machine readability and human usability.

Tasks

Preview Give feedback

1. Refactor Header Validation in ImportUtil to Use Structured Response Format #31236
   Merged QA : Passed Internal Release : 25.02.07 Team : Scout Type : Task +5
2. Refactor Line Processing in ImportUtil to Use Structured Response Format #31237
   OKR : Core Features Priority : 2 High QA : Passed Internal Release : 25.02.12 Team : Scout Type : Task +6
3. Migrate ImportContentletsProcessor to Use New Import Result Format While Maintaining Legacy Support #31339
   Merged Note to QA OKR : Core Features Priority : 2 High QA : Passed Internal Release : 25.02.26 Team : Scout Type : Task +8

[github-actions]

PRs:

• Feat (Core): Initial ImportUtil refactor to improve responses. #31183
• Feat (Core): ImportUtil refactor to improve responses - second part #31294