Retrieve updatedAt, version, and keywords when searching through API

[tainguyenbui]

What we want to achieve
To retrieve the latest datasets information such as updatedAt, dataset version, description, authors, name, keywords and dataverse link through search without the need to perform extra requests. (We believe these heavy operations could be reduced.)

Current scenario:
When searching for files contained in the latest datasets, I am able to retrieve a list of results. However, it requires for an iteration through the results in order to get the dataset information based on doi for each of those results in order to retrieve some of the fields above.

Desired scenario:
No extra iterations are necessary

---

Current example of search output

{
                "name": "demo20190123a",
                "type": "dataset",
                "url": "https://doi.org/10.5072/FK2/H1DFMX",
                "global_id": "doi:10.5072/FK2/H1DFMX",
                "description": "demo",
                "published_at": "2019-01-23T16:56:36Z",
                "citationHtml": "Reilly, Grainne, 2019, \"demo20190123a\", <a href=\"https://doi.org/10.5072/FK2/H1DFMX\" target=\"_blank\">https://doi.org/10.5072/FK2/H1DFMX</a>, Demo Dataverse, V1, UNF:6:xR77FQWCFydmL55sRXuD+w== [fileUNF]",
                "identifier_of_dataverse": "demo",
                "name_of_dataverse": "Demo Dataverse",
                "citation": "Reilly, Grainne, 2019, \"demo20190123a\", https://doi.org/10.5072/FK2/H1DFMX, Demo Dataverse, V1, UNF:6:xR77FQWCFydmL55sRXuD+w== [fileUNF]",
                "entity_id": 301815,
                "authors": [
                    "Reilly, Grainne"
                ]
            }

---

An example of what it might be useful

{
        "id": 3066,
        "doi": "doi:10.5072/FK2/ML0NCM",
        "publisher": "Demo Dataverse",
        "storageIdentifier": "file://10.5072/FK2/ML0NCM",
        "versionId": 807,
        "versionState": "RELEASED",
        "createdAt": "2016-05-24T15:32:55Z",
        "updatedAt": "2016-05-24T15:35:07Z",
        "title": "Test GeoConnect",
        "authors": [
            "Admin, Dataverse"
        ],
        "contacts": [
            {
                "fullname": "Admin, Dataverse",
                "email": "[email protected]"
            }
        ],
        "description": {
            "text": "1"
        },
        "subject": [
            "Arts and Humanities"
        ],
        "keywords": [],
        "publication": [],
        "producer": {},
        "relatedMaterial": [],
        "dataSource": [],
        "geographicCoverage": [],
    }

Potentially, having the datasetId as part of the search result could also be beneficial.

What we are currently doing with some of the metadata blocks
We are mapping metadata blocks to a simpler structure in order to reduce complexity at the client side.

Thanks in advance! We are really enjoying working with Dataverse and collaborating with Dataverse developers

[djbrooke]

Thanks @tainguyenbui for suggesting an approach for reducing the number of API requests. We'll take a look at this during sprint planning and may be back with questions.

[tainguyenbui]

I've removed the Files requirement because it could considerably increase the size of the payload and it is usually not that important as that request to retrieve the files information could be done later on, when a user clicks on a dataset. Perhaps, having the number of files could be an interesting property.

[djbrooke]

• We would want to make sure we respect the setting that does not make available email addresses (consider making this the default if it's not already)
• For performance concerns, we'd be OK taking the hit on the indexing side instead of going to the DB every time
• We should take a look if there are other requests for adding additional fields

[pdurbin]

We should take a look if there are other requests for adding additional fields

We might want to take a look at this one: API search with subtree param :add more hierarchy information in result #6354

@tainguyenbui any interest in that? 😄

[tainguyenbui]

@pdurbin that would be an interesting approach 🤔

Definitely interested 😜 although I am a bit worried about "very long list" of extra params requested 😬

[pdurbin]

We should take a look if there are other requests for adding additional fields

To fix #6396 we could add "draft" or "1.1" or whatever, the dataset version. Perhaps also optionally the id from the "datasetversion" table (perhaps triggered by setting "show_entity_ids" to true).

[pdurbin]

I just wanted to note that this issue came up while discussing #6396 during tech hours on Tuesday. It may or may not make sense to make a single pull request for both issues. Time will tell.