Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8394 maven mods #8395

Merged
merged 34 commits into from
Mar 4, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
b023bdb
build: fix wrong XML namespaces in pom.xml
poikilotherm Feb 3, 2022
2e76494
build(parent): add a basic Maven Parent POM #8394
poikilotherm Feb 3, 2022
f102122
build(dataverse): make main pom.xml inherit from new dataverse-parent…
poikilotherm Feb 3, 2022
c70e75c
build(dataverse): make Maven plugin versions controlled by parent #8394
poikilotherm Feb 3, 2022
424400b
chore(jacoco): fix wrongly typed config options of JaCoCo Maven plugi…
poikilotherm Feb 3, 2022
8fb4b9a
build(dataverse): move <pluginRepositories> and <repositories> to par…
poikilotherm Feb 3, 2022
6089cbb
build(dataverse): move common version properties to parent #8394
poikilotherm Feb 3, 2022
0ad680c
build(dataverse): make SolrJ dependency use a version from parent #8394
poikilotherm Feb 3, 2022
c56e901
build(dataverse): move common entries from <dependencyManagement> to …
poikilotherm Feb 3, 2022
477a180
ci(unit): make Github Action for Maven Unit Tests be triggered on POM…
poikilotherm Feb 3, 2022
19b9bec
build(dataverse): move postgresql lib versioning to parent <dependenc…
poikilotherm Feb 7, 2022
66461a9
style(zipdownloader): fix accidental tab char usage
poikilotherm Feb 7, 2022
ea59f8a
build(zipdownloader): make zipdownloader use parent POM #8394
poikilotherm Feb 7, 2022
41e80db
refactor(zipdownloader): replace Maven Assembly Plugin with Spring Bo…
poikilotherm Feb 7, 2022
febc61d
build(deps): re-add SLF4J changes from #8377 in parent #8394
poikilotherm Feb 9, 2022
fbdf5c1
build(modules): move Maven plugin versioning to parent #8394
poikilotherm Feb 9, 2022
452feea
docs(dev): rework dependencies.rst and add parent POM #8394
poikilotherm Feb 10, 2022
0f6c8f5
Merge branch 'develop' into 8394-maven-mods
poikilotherm Feb 10, 2022
b416618
build(zipdownloader): change version number to experimental #8394
poikilotherm Feb 10, 2022
c91c071
refactor(zipdownloader): adapt cgi-bin script to name and version of …
poikilotherm Feb 10, 2022
429ec3f
docs(zipdownloader): refactor slightly for name and version, crosslin…
poikilotherm Feb 10, 2022
3422d13
docs(releases): adapt for parent module #8394
poikilotherm Feb 11, 2022
e9a97eb
build(war): make logging libs runtime scoped #8394
poikilotherm Feb 11, 2022
c98b078
Merge branch 'develop' into 8394-maven-mods
poikilotherm Feb 14, 2022
488a392
doc tweaks for #8394
pdurbin Feb 14, 2022
73213f4
docs(dev): clarify nearest dependency wins #8394
poikilotherm Feb 15, 2022
b63a010
simplify wording #8394
pdurbin Feb 15, 2022
8b30406
clarify dep resolution #8394
pdurbin Feb 15, 2022
0682d19
Merge pull request #524 from IQSS/8394-doc-tweaks
poikilotherm Feb 15, 2022
03e45be
Minor changes to the Zipper installation section
landreev Feb 23, 2022
dacdbd2
another cosmetic fix
landreev Feb 23, 2022
6dcf911
Merge branch 'develop' into 8394-maven-mods
poikilotherm Feb 28, 2022
f2189c5
Merge branch 'develop' into 8394-maven-mods
poikilotherm Mar 4, 2022
09167d5
Revert "chore(jacoco): fix wrongly typed config options of JaCoCo Mav…
pdurbin Mar 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/maven_unit_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,13 @@ on:
push:
paths:
- "**.java"
- "pom.xml"
- "modules/**/pom.xml"
pull_request:
paths:
- "**.java"
- "pom.xml"
- "modules/**/pom.xml"

jobs:
unittest:
Expand Down
262 changes: 222 additions & 40 deletions doc/sphinx-guides/source/developers/dependencies.rst

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions doc/sphinx-guides/source/developers/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Getting Help

If you have any questions at all, please reach out to other developers via the channels listed in https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md such as http://chat.dataverse.org, the `dataverse-dev <https://groups.google.com/forum/#!forum/dataverse-dev>`_ mailing list, `community calls <https://dataverse.org/community-calls>`_, or [email protected].

.. _core-technologies:

Core Technologies
-----------------

Expand Down
3 changes: 2 additions & 1 deletion doc/sphinx-guides/source/developers/making-releases.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Make the following changes in the release branch:

Increment the version number to the milestone (e.g. 4.6.2) in the following two files:

- pom.xml
- modules/dataverse-parent/pom.xml -> ``<properties>`` -> ``<revision>``
- doc/sphinx-guides/source/conf.py (two places)

Add the version being released to the lists in the following two files:
Expand All @@ -31,6 +31,7 @@ Add the version being released to the lists in the following two files:
- scripts/database/releases.txt

Here's an example commit where three of the four files above were updated at once: https://github.com/IQSS/dataverse/commit/99e23f96ec362ac2f524cb5cd80ca375fa13f196
(Note: the version has been moved to a property in parent module since this commit was created.)

2. Check in the Changes Above...
================================
Expand Down
63 changes: 32 additions & 31 deletions doc/sphinx-guides/source/installation/advanced.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,43 +72,44 @@ Once a standardized version of you Custom Terms are registered as a license, an
Optional Components
-------------------

.. _zipdownloader:

Standalone "Zipper" Service Tool
++++++++++++++++++++++++++++++++

As of Dataverse Software 5.0 we offer an experimental optimization for the multi-file, download-as-zip functionality. If this option
(``:CustomZipDownloadServiceUrl``) is enabled, instead of enforcing
the size limit on multi-file zipped downloads (as normally specified
by the option ``:ZipDownloadLimit``), we attempt to serve all the
files that the user requested (that they are authorized to download),
but the request is redirected to a standalone zipper service running
as a cgi-bin executable under Apache. Thus moving these potentially
long-running jobs completely outside the Application Server (Payara);
and preventing worker threads from becoming locked serving them. Since
zipping is also a CPU-intensive task, it is possible to have this
service running on a different host system, freeing the cycles on the
main Application Server. (The system running the service needs to have
access to the database as well as to the storage filesystem, and/or S3
bucket).

Please consult the scripts/zipdownload/README.md in the Dataverse Software 5.0+ source tree for more information.

To install: You can follow the instructions in the file above to build
``ZipDownloadService-v1.0.0.jar``. It will also be available, pre-built as part of the Dataverse Software 5.0 release on GitHub. Copy it, together with the shell
script scripts/zipdownload/cgi-bin/zipdownload to the cgi-bin
directory of the chosen Apache server (/var/www/cgi-bin standard).

Make sure the shell script (zipdownload) is executable, and edit it to configure the
database access credentials. Do note that the executable does not need
access to the entire Dataverse installation database. A security-conscious admin
can create a dedicated database user with access to just one table:
``CUSTOMZIPSERVICEREQUEST``.

You may need to make extra Apache configuration changes to make sure /cgi-bin/zipdownload is accessible from the outside.
For example, if this is the same Apache that's in front of your Dataverse installation Payara instance, you will need to add another pass through statement to your configuration:
As of Dataverse Software 5.0 we offer an **experimental** optimization for the multi-file, download-as-zip functionality.
If this option (``:CustomZipDownloadServiceUrl``) is enabled, instead of enforcing the size limit on multi-file zipped
downloads (as normally specified by the option ``:ZipDownloadLimit``), we attempt to serve all the files that the user
requested (that they are authorized to download), but the request is redirected to a standalone zipper service running
as a cgi-bin executable under Apache. This moves these potentially long-running jobs completely outside the Application Server (Payara), and prevents worker threads from becoming locked serving them. Since zipping is also a CPU-intensive task, it is possible to have
this service running on a different host system, freeing the cycles on the main Application Server. (The system running
the service needs to have access to the database as well as to the storage filesystem, and/or S3 bucket).

Please consult the `README at scripts/zipdownload <https://github.com/IQSS/dataverse/tree/master/scripts/zipdownload>`_
in the Dataverse Software 5.0+ source tree for more information.

To install:

1. Follow the instructions in the file above to build ``zipdownloader-0.0.1.jar``. Please note that the package name and
the version were changed as of the release 5.10, as part of an overall cleanup and reorganization of the project
tree. In the releases 5.0-5.9 it existed under the name ``ZipDownloadService-v1.0.0``. (A pre-built jar file was
distributed under that name as part of the 5.0 release on GitHub. Aside from the name change, there have been no
changes in the functionality of the tool).
2. Copy it, together with the shell script :download:`cgi-bin/zipdownload <../../../../scripts/zipdownload/cgi-bin/zipdownload>`
to the ``cgi-bin`` directory of the chosen Apache server (``/var/www/cgi-bin`` standard).
3. Make sure the shell script (``zipdownload``) is executable, and edit it to configure the database access credentials.
Do note that the executable does not need access to the entire Dataverse installation database. A security-conscious
admin can create a dedicated database user with access to just one table: ``CUSTOMZIPSERVICEREQUEST``.

You may need to make extra Apache configuration changes to make sure ``/cgi-bin/zipdownload`` is accessible from the outside.
For example, if this is the same Apache that's in front of your Dataverse installation Payara instance, you will need to
add another pass through statement to your configuration:

``ProxyPassMatch ^/cgi-bin/zipdownload !``

Test this by accessing it directly at ``<SERVER URL>/cgi-bin/download``. You should get a ``404 No such download job!``. If instead you are getting an "internal server error", this may be an SELinux issue; try ``setenforce Permissive``. If you are getting a generic Dataverse collection "not found" page, review the ``ProxyPassMatch`` rule you have added.
Test this by accessing it directly at ``<SERVER URL>/cgi-bin/download``. You should get a ``404 No such download job!``.
If instead you are getting an "internal server error", this may be an SELinux issue; try ``setenforce Permissive``.
If you are getting a generic Dataverse collection "not found" page, review the ``ProxyPassMatch`` rule you have added.

To activate in your Dataverse installation::

Expand Down
7 changes: 5 additions & 2 deletions doc/sphinx-guides/source/installation/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2380,15 +2380,18 @@ If you don’t want date facets to be sorted chronologically, set:
:CustomZipDownloadServiceUrl
++++++++++++++++++++++++++++

The location of the "Standalone Zipper" service. If this option is specified, the Dataverse installation will be redirecing bulk/mutli-file zip download requests to that location, instead of serving them internally. See the "Advanced" section of the Installation guide for information on how to install the external zipper. (This is still an experimental feature, as of Dataverse Software 5.0).
The location of the "Standalone Zipper" service. If this option is specified, the Dataverse installation will be
redirecing bulk/multi-file zip download requests to that location, instead of serving them internally.
See :ref:`zipdownloader` of the Advanced Installation guide for information on how to install the external zipper.
(This is still an **experimental** feature, as of Dataverse Software 5.0).

To enable redirects to the zipper installed on the same server as the main Dataverse Software application:

``curl -X PUT -d '/cgi-bin/zipdownload' http://localhost:8080/api/admin/settings/:CustomZipDownloadServiceUrl``

To enable redirects to the zipper on a different server:

``curl -X PUT -d 'https://zipper.example.edu/cgi-bin/zipdownload' http://localhost:8080/api/admin/settings/:CustomZipDownloadServiceUrl``
``curl -X PUT -d 'https://zipper.example.edu/cgi-bin/zipdownload' http://localhost:8080/api/admin/settings/:CustomZipDownloadServiceUrl``

:ArchiverClassName
++++++++++++++++++
Expand Down
Loading