Add merged datasets functionality

[ukanga]

Allows data from two or more forms to be represented as one form/endpoint.

The model will inherit from existing xform model. The difference are:

• It will have its schema be a created from a combination of N+ xforms
• It will not accept submissions

Requirements

• Add functionality in forms endpoint for creating merged dataset (may not be required)
• Generate Schema from existing xforms
• Query data from child xforms
• Use existing endpoints (projects, forms, data) to access/modify the form
• Allows exports
• Integrations
  • Webhooks
  • Google sheets
  • Rapidpro
• HXL support
• Will reflect changes (updates/deletions) in the parent datasets

Aha! Link: https://ona.aha.io/features/PROD-552

[urbanslug]

Implementation plan

Merged datasets

To do:

• Add a MergedXForm model

  • Subclass the XForm model
  • Extend it with an `xforms` field

• Add MergedXFormViewset

  • Add a merged-dataset endpoint api/v1/merged-datasets

• Add MergedXFormSerializer

  • POST Create - takes {"xforms": ["list-of-xform-urls"], "name": "form-name-string", "project": "url-to-parent-project"}
  • GET Retreive a user's merged datasets - list and detail (data endpoint)
  • Expose an endpoint to retrieve merged dataset JSON and XML
  • Prevent PATCH & PUT Update - renaming etc
  • GET should also contain num_of_submissions which should be the sum of all the submissions from all the merged forms

• Validate that: (serializer validation and model before save)

  • name is a required field
  • project is a required field
  • A merged XForm is made of at least two xforms

• Add documentation on:

  • Creating a merged dataset
  • Retrieving all merged datasets
  • Retrieving a merged dataset
  • Deleting a merged dataset
  • Retrieving data from a merged dataset
  • How data in parent xforms differs from and affects the merged xform
  • Exporting data from a merged dataset

• Add permissions and authourization

  • Inherit permissions from the project

• Data

  • Expose a merged-dataset data endpoint to give data for specific merged dataset merged-dataset/pk/data
  • Preventing submissions to the merged xform
  • Exports

[urbanslug]

ManyToManyField v OneToMany for xforms in MergedXForm model.

---

OneToMany field

In this case django would create an merged xform relation with a foreign key field to reference a specific xform. It would however not be able to store more then one xform id. This means a merged xform would only reference a single xform. However, what we want is for a single merged xform to reference an arbitrary number of xforms.

xform table

id	other	xform	columns
267	data	data	data
377	data	data	data
487	data	data	data

merged xform table

id	xform_id_foreign_key
1	267
2	377
3	478

ManyToMany field

A many to many field creates another table that references both the xforms and merged xforms table

xform table

id	other	xform	columns
267	data	data	data
377	data	data	data
487	data	data	data

merged xform table

id
1
2
3

logger merged xform table

id	merged_xform	xform
1	1	267
2	1	377
3	2	267
4	2	487

This way when fetching the xforms for a merged dataset, we select the records where the merged xfrom value matches the id of the merged xform and get all the xform values for those records.

[ukanga]

closed by #1082