Merge branch 'develop' into 7000-mpconfig-fqdn

conf/docker-aio/0prep_deps.sh

@@ -4,9 +4,9 @@ if [ ! -d dv/deps ]; then
 fi
 wdir=`pwd`
 
-if [ ! -e dv/deps/payara-5.2021.6.zip ]; then
+if [ ! -e dv/deps/payara-5.2022.3.zip ]; then
 	echo "payara dependency prep"
-	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.6/payara-5.2021.6.zip  -O dv/deps/payara-5.2021.6.zip
+	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2022.3/payara-5.2022.3.zip  -O dv/deps/payara-5.2022.3.zip
 fi
 
 if [ ! -e dv/deps/solr-8.11.1dv.tgz ]; then

conf/docker-aio/c8.dockerfile

@@ -24,7 +24,7 @@ COPY disableipv6.conf /etc/sysctl.d/
 RUN rm /etc/httpd/conf/*
 COPY httpd.conf /etc/httpd/conf 
 RUN cd /opt ; tar zxf /tmp/dv/deps/solr-8.11.1dv.tgz
-RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.6.zip ; ln -s /opt/payara5 /opt/glassfish4
+RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2022.3.zip ; ln -s /opt/payara5 /opt/glassfish4
 
 # this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
 # this dies under Java 11, do we keep it?

doc/release-notes/8732-date-in-citation-harvested-datasets.md

@@ -0,0 +1,7 @@
+Fix the year displayed in citation for harvested dataset, specialy for oai_dc format.
+
+For normal datasets, the date used is the "citation date" which is by default the publication date (the first release date) (https://guides.dataverse.org/en/latest/api/native-api.html?highlight=citationdate#set-citation-date-field-type-for-a-dataset).
+
+But for a harvested dataset, the distribution date is used instead and this date is not always present in the harvested metadata. With oai_dc format the date tag if used as production date.
+
+Now, the production date is used for harvested dataset in addition to distribution date.

doc/release-notes/8733-oai_dc-date.md

@@ -0,0 +1,4 @@
+For exports and harvesting in `oai_dc` format, if "Production Date" is not set, "Publication Date" is now used instead. This change is reflected in the [Dataverse 4+ Metadata Crosswalk][] linked from the [Appendix][] of the User Guide.
+
+[Dataverse 4+ Metadata Crosswalk]: https://docs.google.com/spreadsheets/d/10Luzti7svVTVKTA-px27oq3RxCUM-QbiTkm8iMd5C54/edit#gid=1901625433&range=K7
+[Appendix]: https://guides.dataverse.org/en/latest/user/appendix.html

doc/sphinx-guides/source/_static/docsdataverse_org.css

@@ -68,7 +68,7 @@ a.headerlink {
 #sidebar.bs-sidenav {
     background-color: #f8d5b8;
 }
-#sidebar.bs-sidenav .nav > li > a:hover, #sidebar.bs-sidenav .nav > li > a:focus {
+#sidebar.bs-sidenav .nav > li > a:hover, #sidebar.bs-sidenav .nav > li > a:focus, #sidebar.bs-sidenav .nav > li > a.current {
     background-color: #fbf4c5;
     border-right: 1px solid #dbd8e0;
     text-decoration: none;

doc/sphinx-guides/source/_static/util/clear_timer.sh

@@ -17,7 +17,7 @@ DV_DIR=${PAYARA_DIR}/glassfish/domains/domain1
 ${PAYARA_DIR}/bin/asadmin stop-domain
 
 rm -rf ${PAYARA_DIR}/${DV_DIR}/generated/
-rm -rf ${PAYARA_DIR}/${DV_DIR}/osgi-cache/felix
+rm -rf ${PAYARA_DIR}/${DV_DIR}/osgi-cache/
 
 # restart the domain (also generates a warning if app server is stopped)
 ${PAYARA_DIR}/bin/asadmin start-domain

doc/sphinx-guides/source/admin/dataverses-datasets.rst

@@ -15,7 +15,7 @@ Dataverse collections have to be empty to delete them. Navigate to the Dataverse
 Move a Dataverse Collection
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Moves a Dataverse collection whose id is passed to a new Dataverse collection whose id is passed. The Dataverse collection alias also may be used instead of the id. If the moved Dataverse collection has a guestbook, template, metadata block, link, or featured Dataverse collection that is not compatible with the destination Dataverse collection, you will be informed and given the option to force the move and remove the association. Only accessible to superusers. ::
+Moves a Dataverse collection whose id is passed to an existing Dataverse collection whose id is passed. The Dataverse collection alias also may be used instead of the id. If the moved Dataverse collection has a guestbook, template, metadata block, link, or featured Dataverse collection that is not compatible with the destination Dataverse collection, you will be informed and given the option to force the move and remove the association. Only accessible to superusers. ::
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/dataverses/$id/move/$destination-id
 

doc/sphinx-guides/source/admin/harvestserver.rst

@@ -26,7 +26,7 @@ The email portion of :ref:`systemEmail` will be visible via OAI-PMH (from the "I
 How does it work?
 -----------------
 
-Only the published, unrestricted datasets in your Dataverse installation can
+Only the published datasets in your Dataverse installation can
 be made harvestable. Remote clients normally keep their records in sync
 through scheduled incremental updates, daily or weekly, thus
 minimizing the load on your server. Note that it is only the metadata
@@ -115,10 +115,10 @@ Some useful examples of search queries to define OAI sets:
 
   ``keywordValue:censorship``
 
-Important: New SOLR schema required!
+Important: New Solr schema required!
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-In order to be able to define OAI sets, your SOLR server must be upgraded with the search schema that came with release 4.5 (or later), and all your local datasets must be re-indexed, once the new schema is installed. 
+In order to be able to define OAI sets, your Solr server must be upgraded with the search schema that came with release 4.5 (or later), and all your local datasets must be re-indexed, once the new schema is installed. 
 
 OAI Set updates
 ---------------

doc/sphinx-guides/source/admin/integrations.rst

@@ -57,7 +57,7 @@ their research results and retain links to imported and exported data. Users
 can organize their data in "Datasets", which can be exported to a Dataverse installation via
 the command-line interface (CLI).
 
-Renku dataset documentation: https://renku-python.readthedocs.io/en/latest/reference/commands.html#module-renku.cli.dataset
+Renku documentation: https://renku-python.readthedocs.io
 
 Flagship deployment of the Renku platform: https://renkulab.io
 

doc/sphinx-guides/source/admin/metadatacustomization.rst

@@ -565,12 +565,14 @@ In general, the external vocabulary support mechanism may be a better choice for
 The specifics of the user interface for entering/selecting a vocabulary term and how that term is then displayed are managed by third-party Javascripts. The initial Javascripts that have been created provide auto-completion, displaying a list of choices that match what the user has typed so far, but other interfaces, such as displaying a tree of options for a hierarchical vocabulary, are possible. 
 Similarly, existing scripts do relatively simple things for displaying a term - showing the term's name in the appropriate language and providing a link to an external URL with more information, but more sophisticated displays are possible.
 
-Scripts supporting use of vocabularies from services supporting the SKOMOS protocol (see https://skosmos.org) and retrieving ORCIDs (from https:/orcid.org) are available https://github.com/gdcc/dataverse-external-vocab-support. (Custom scripts can also be used and community members are encouraged to share new scripts through the dataverse-external-vocab-support repository.)
+Scripts supporting use of vocabularies from services supporting the SKOMOS protocol (see https://skosmos.org) and retrieving ORCIDs (from https://orcid.org) are available https://github.com/gdcc/dataverse-external-vocab-support. (Custom scripts can also be used and community members are encouraged to share new scripts through the dataverse-external-vocab-support repository.)
 
 Configuration involves specifying which fields are to be mapped, whether free-text entries are allowed, which vocabulary(ies) should be used, what languages those vocabulary(ies) are available in, and several service protocol and service instance specific parameters.
 These are all defined in the :ref:`:CVocConf <:CVocConf>` setting as a JSON array. Details about the required elements as well as example JSON arrays are available at https://github.com/gdcc/dataverse-external-vocab-support, along with an example metadata block that can be used for testing.
 The scripts required can be hosted locally or retrieved dynamically from https://gdcc.github.io/ (similar to how dataverse-previewers work).
 
+Please note that in addition to the :ref:`:CVocConf` described above, an alternative is the :ref:`:ControlledVocabularyCustomJavaScript` setting.
+
 Tips from the Dataverse Community
 ---------------------------------
 

doc/sphinx-guides/source/admin/metadataexport.rst

@@ -11,19 +11,35 @@ Publishing a dataset automatically starts a metadata export job, that will run i
 
 A scheduled timer job that runs nightly will attempt to export any published datasets that for whatever reason haven't been exported yet. This timer is activated automatically on the deployment, or restart, of the application. So, again, no need to start or configure it manually. (See the :doc:`timers` section of this Admin Guide for more information.)
 
-Batch exports through the API 
+.. _batch-exports-through-the-api:
+
+Batch Exports Through the API
 -----------------------------
 
-In addition to the automated exports, a Dataverse installation admin can start a batch job through the API. The following 2 API calls are provided: 
+In addition to the automated exports, a Dataverse installation admin can start a batch job through the API. The following four API calls are provided: 
 
 ``curl http://localhost:8080/api/admin/metadata/exportAll``
 
 ``curl http://localhost:8080/api/admin/metadata/reExportAll``
 
-The former will attempt to export all the published, local (non-harvested) datasets that haven't been exported yet. 
-The latter will *force* a re-export of every published, local dataset, regardless of whether it has already been exported or not. 
+``curl http://localhost:8080/api/admin/metadata/clearExportTimestamps``
+
+``curl http://localhost:8080/api/admin/metadata/:persistentId/reExportDataset?persistentId=doi:10.5072/FK2/AAA000``
+
+The first will attempt to export all the published, local (non-harvested) datasets that haven't been exported yet. 
+The second will *force* a re-export of every published, local dataset, regardless of whether it has already been exported or not. 
+
+The first two calls return a status message informing the administrator that the process has been launched (``{"status":"WORKFLOW_IN_PROGRESS"}``). The administrator can check the progress of the process via log files: ``[Payara directory]/glassfish/domains/domain1/logs/export_[time stamp].log``.
+
+Instead of running "reExportAll" the same can be accomplished using "clearExportTimestamps" followed by "exportAll".
+The difference is that when exporting prematurely fails due to some problem, the datasets that did not get exported yet still have the timestamps cleared. A next call to exportAll will skip the datasets already exported and try to export the ones that still need it. 
+Calling clearExportTimestamps should return ``{"status":"OK","data":{"message":"cleared: X"}}`` where "X" is the total number of datasets cleared.
+
+The reExportDataset call gives you the opportunity to *force* a re-export of only a specific dataset and (with some script automation) could allow you the export specific batches of datasets. This might be usefull when handling exporting problems or when reExportAll takes too much time and is overkill. Note that :ref:`export-dataset-metadata-api` is a related API.
+
+reExportDataset can be called with either ``persistentId`` (as shown above, with a DOI) or with the database id of a dataset (as shown below, with "42" as the database id).
 
-These calls return a status message informing the administrator, that the process has been launched (``{"status":"WORKFLOW_IN_PROGRESS"}``). The administrator can check the progress of the process via log files: ``[Payara directory]/glassfish/domains/domain1/logs/export_[time stamp].log``.
+``curl http://localhost:8080/api/admin/metadata/42/reExportDataset``
 
 Note, that creating, modifying, or re-exporting an OAI set will also attempt to export all the unexported datasets found in the set.