add support for merged datasets #1082

ukanga · 2017-08-06T03:39:51Z

Implements merged datasets as described in #1057

ivermac · 2017-08-07T06:43:18Z

Hey @ukanga I was testing out the the last_submission_time on merged dataset and it seems not to be returned. On further investigation, it seems a merged dataset object is created before setting the last_submission_time variable - which I believe was what this function was meant to do. We could add a utility function that returns the last submission time outside the serializer class (borrowed this snippet from what is in the get_last_submission_time function):

def get_last_submission_time(xforms):
    values = [
        x.last_submission_time
        for x in xforms
        if x.last_submission_time
    ]
    if values:
        return sorted(values, reverse=True)[0]

Then inside the create serializer function, we could set the last_submission_time in the validated_data variable:

...
validated_data['last_submission_time'] = get_last_submission_time(
            validated_data['xforms']
        )

The above snippets seem to work for me but if there is a better implementation, that would be great.

ukanga · 2017-08-07T07:13:06Z

The idea was that it will be calculated on the fly. I will check on it.

ukanga · 2017-08-07T08:19:15Z

@ivermac I have made the changes on the forms endpoint, 8692573.

ivermac · 2017-08-07T08:30:42Z

Cool, thanks @ukanga. I'll check it out

UPDATE: works as expected

pld · 2017-08-09T12:18:24Z

docs/merged-datasets.rst

+Merged Datasets
+***************
+
+This endpoint provides access to data from multiple forms Merged datasets should have the same functionality as the forms endpoint with the difference being:


add a period after "from" and before "Merged datasets"

pld · 2017-08-09T12:36:56Z

docs/merged-datasets.rst

+^^^^^^^^^^
+- ``name`` - Name or title of the merged dataset (required)
+- ``project`` -  Project for the merged dataset (required)
+- ``xforms`` -  list of forms to merge (required, at least 2 forms should be provided )


should the "L" in "List" be capitalized? To make it consistent with the lines above, or should the others be lowercased (also, I know it's a nitpick sorry, there's an extra space before the parenthesis at the end of this line)

pld · 2017-08-09T12:38:00Z

docs/merged-datasets.rst

+How data in parent forms differs from and affects the merged xform
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A merged dataset combines data from multiple forms into one. It creates a new form structure from the intersection of the fields from the forms being merged.


I change the final "from" to "in" for clarity, so end of this line changes to "in the forms being merged."

pld · 2017-08-09T18:30:20Z

onadata/apps/api/viewsets/charts_viewset.py

-                    CacheControlMixin,
-                    ETagsMixin,
+# pylint: disable=R0901
+class ChartsViewSet(AnonymousUserPublicFormsMixin, AuthenticateHeaderMixin,


I don't get why to change this esp if it requires adding the pylint flag? I like the previous format

Pylint R0901 has to do with "Too many ancestors".

The formatting at this time was under the mercy of yapf command, I tend to take its recommendations. I should probably look into and check-in yapf's configuration.

pld · 2017-08-09T18:34:21Z

onadata/apps/viewer/models/parsed_instance.py

@@ -171,7 +171,13 @@ def _get_instances(xform, start, end):
    if isinstance(end, datetime.datetime):
        kwargs.update({'date_created__lte': end})

-    instances = xform.instances.filter(**kwargs)
+    instances = xform.instances


this could go in an else

ivermac · 2017-08-18T12:32:40Z

I would like is_merged_dataset field to be added to this line to allow zebra to only list non-merged forms when a user wants to select forms that can be merged i.e. the form listing on the screenshot below should only have normal forms - no dataviews or merged datasets.

ukanga · 2017-08-18T14:56:04Z

@ivermac I will add the field.

ukanga · 2017-08-19T15:26:45Z

This PR now includes service integrations support for merged datasets.

ivermac · 2017-08-21T11:24:58Z

onadata/libs/serializers/merged_xform_serializer.py

+    intersect = set([__ for (__, ___) in intersect])
+
+    merged_xform_dict['children'] = _get_elements(children, intersect)
+    del merged_xform_dict['_xpath']


@ukanga This could be re-written as

... merged_xform_dict.get("_xpath") and merged_xform_dict.pop("_xpath")

On stage and locally, saving a merged dataset triggers the following error

Signed-off-by: Njagi Mwaniki <[email protected]>

…ractViewSet Signed-off-by: Njagi Mwaniki <[email protected]>

Signed-off-by: Njagi Mwaniki <[email protected]>

Ensure that: - `name` is a required field - `project` is a required field - `xforms` is a list of at least 2 xforms & should have unique elements Signed-off-by: Njagi Mwaniki <[email protected]>

Signed-off-by: Njagi Mwaniki <[email protected]>

apply the MergedXForm permissions in roles Signed-off-by: Njagi Mwaniki <[email protected]>

Also, set sms_keyword to match the id_string, this affects sms_id_string Signed-off-by: Njagi Mwaniki <[email protected]>

It is messing up other tests and there is no point updating all fixtures to add the additional fields specific to this one test.

Added get_merged_xform_survey() function that uses the intersection of fields of xforms being merged to create a new survey object with only the intersecting fields.

PyxfformMarkdown is already mixed in TestBase

pld reviewed Aug 9, 2017

View reviewed changes

ukanga force-pushed the merged-datasets branch 2 times, most recently from 16e7429 to 776fdf9 Compare August 17, 2017 04:35

ukanga force-pushed the merged-datasets branch from 776fdf9 to 800bf41 Compare August 19, 2017 03:24

pld approved these changes Aug 20, 2017

View reviewed changes

ivermac reviewed Aug 21, 2017

View reviewed changes

Ona KE code machine added 17 commits August 24, 2017 06:43

NM/UK: correct #egg portion of github hosted packages

bcaed61

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: update dev pip requirements

fe20a3f

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: initial MergedXForm model

64ef600

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: add PyxformTestCase for markdown XLSForm support into TestAbst…

1e11bec

…ractViewSet Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: set or create default user project in tests

0ca1313

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: add MergedXFormSerializer

c8ec7cf

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: initial MergedXFormViewSet with tests

f589347

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: Test merged datasets get list and retrieve

09d9bf1

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: Expose an endpoint to retrieve mewrged dataset JSON and XML

1db4f43

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: Add validation for the MergedXFormSerializer

19e86dd

Ensure that: - `name` is a required field - `project` is a required field - `xforms` is a list of at least 2 xforms & should have unique elements Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: consistent root node "data" for model instance markdown forms

ebf29e4

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: initial implementation of the merged dataset data endpoint

d3f2a55

Signed-off-by: Njagi Mwaniki <[email protected]>

initial api documentation for merged datasets

765d5fd

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: Remove guideline comments

2d0e981

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: use XFormPermissions in MergedXFormViewset permission_classes

8f68ca5

Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: add view_mergedxform permission

461d473

apply the MergedXForm permissions in roles Signed-off-by: Njagi Mwaniki <[email protected]>

NM/UK: set values of user and created_by properties of a MergedXForm

6f64e04

Also, set sms_keyword to match the id_string, this affects sms_id_string Signed-off-by: Njagi Mwaniki <[email protected]>

ukanga added 25 commits August 24, 2017 06:43

last_submission_time for merged datasets at the forms endpoint

258e1b2

refactor get_or_create_opendata() as a helper function

fd8b91e

Data for merged datasets for open datasets

61d2cec

add _xform_id to exports

d24b9d7

return merged datasets data for filtered datasets

b9b817b

pick extra export columns from settings

1ae99f0

refactor test to inlude docstrings

b162c44

include XFORM_ID in mongo to dict

f1f8613

Don't override EXTRA_COLUMNS setting in tests

517ab01

It is messing up other tests and there is no point updating all fixtures to add the additional fields specific to this one test.

code & docs cleanup

aacf6b5

remove ID => _id from ignored columns in CSV exports

1003cf4

Create survey object from merging forms

bfd98d5

Added get_merged_xform_survey() function that uses the intersection of fields of xforms being merged to create a new survey object with only the intersecting fields.

Validate no matching fields for merged datasets

a21c81a

change order of tests

25c0581

add missing import

27efe98

maintain order of questions on merged dataset

4ee161a

Support rest services merged datasets

15cbce7

Update TextIt service to handle merged datasets

a33778a

Test merged datasets with groups

10ba548

Merge forms by xpath and type

fbddf87

Delete _xpath only if present

1d32598

KML export support merged datasets

b6f2eb4

refactor _publish_md to _publish_markdown

9d6deb8

Attachment export support merged datasets

3e91f5c

remove PyxformMarkdown in tests

d3534e3

PyxfformMarkdown is already mixed in TestBase

ukanga force-pushed the merged-datasets branch from 50ad2ae to d3534e3 Compare August 24, 2017 03:44

ukanga merged commit 960250d into master Aug 24, 2017

ukanga deleted the merged-datasets branch August 24, 2017 09:49

This was referenced Aug 28, 2017

Add merged datasets functionality #1057

Closed

Include ID aka _id in exports by default #960

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for merged datasets #1082

add support for merged datasets #1082

ukanga commented Aug 6, 2017 •

edited

Loading

ivermac commented Aug 7, 2017

ukanga commented Aug 7, 2017

ukanga commented Aug 7, 2017

ivermac commented Aug 7, 2017 •

edited

Loading

pld Aug 9, 2017

pld Aug 9, 2017

pld Aug 9, 2017

pld Aug 9, 2017

ukanga Aug 19, 2017

pld Aug 9, 2017

ivermac commented Aug 18, 2017 •

edited

Loading

ukanga commented Aug 18, 2017

ukanga commented Aug 19, 2017

ivermac Aug 21, 2017

add support for merged datasets #1082

add support for merged datasets #1082

Conversation

ukanga commented Aug 6, 2017 • edited Loading

ivermac commented Aug 7, 2017

ukanga commented Aug 7, 2017

ukanga commented Aug 7, 2017

ivermac commented Aug 7, 2017 • edited Loading

pld Aug 9, 2017

Choose a reason for hiding this comment

pld Aug 9, 2017

Choose a reason for hiding this comment

pld Aug 9, 2017

Choose a reason for hiding this comment

pld Aug 9, 2017

Choose a reason for hiding this comment

ukanga Aug 19, 2017

Choose a reason for hiding this comment

pld Aug 9, 2017

Choose a reason for hiding this comment

ivermac commented Aug 18, 2017 • edited Loading

ukanga commented Aug 18, 2017

ukanga commented Aug 19, 2017

ivermac Aug 21, 2017

Choose a reason for hiding this comment

ukanga commented Aug 6, 2017 •

edited

Loading

ivermac commented Aug 7, 2017 •

edited

Loading

ivermac commented Aug 18, 2017 •

edited

Loading