Merge pull request #2935 from IQSS/develop

Merge dev out to 2722 branch

doc/sphinx-guides/source/user/dataverse-management.rst

@@ -10,7 +10,7 @@ Once a user creates a dataverse they, by default, become the
 administrator of that dataverse. The dataverse administrator has access
 to manage the settings described in this guide.
 
-Create a Dataverse (within the "root" Dataverse)
+Create a Dataverse (Within the "Root" Dataverse)
 ===================================================
 
 Creating a dataverse is easy but first you must be a registered user (see Create Account).
@@ -49,7 +49,7 @@ where you will be presented with the following editing options:
 General Information
 =====================================================
 
-The General Information page is how you edit the information you filled in while creating your dataverse. If you need to change or add a contact email address, this is the place to do it. Additionally, you can update the metadata elements used for datasets within the dataverse, change which metadafields are hidden, required, or optional, and update the facets you would like displayed for browsing the dataverse. If you plan on using templates, you need to select the metadata fields on the General Information page.
+The General Information page is how you edit the information you filled in while creating your dataverse. If you need to change or add a contact email address, this is the place to do it. Additionally, you can update the metadata elements used for datasets within the dataverse, change which metadata fields are hidden, required, or optional, and update the facets you would like displayed for browsing the dataverse. If you plan on using templates, you need to select the metadata fields on the General Information page.
 
 Tip: The metadata fields you select as required, will appear on the Create Dataset form when someone goes to add a dataset to the dataverse. 
 
@@ -60,7 +60,7 @@ Theme + Widgets
 
 The Theme + Widgets feature provides you with a way to customize the look of your dataverse as well as provide code for you to put on your personal website to have your dataverse appear there. (For adding your dataverse to an OpenScholar site, follow these instructions.)
 
-For Theme, you can decide either to use the customization from the dataverse above yours or upload your own image file. Supported image types are jpeg, tiff, or png and should be no larger than 500kb. The maximum display for an image file in a dataverse's theme is 940 piels wide by 120 pixels high. Additionally, you can select the colors for the header of your dataverse and the text that appears in your dataverse. You can also add a link to your personal website, the website for your organization or institution, your department, journal, etc.
+For Theme, you can decide either to use the customization from the dataverse above yours or upload your own image file. Supported image types are JPEG, TIFF, or PNG and should be no larger than 500 KB. The maximum display size for an image file in a dataverse's theme is 940 pixels wide by 120 pixels high. Additionally, you can select the colors for the header of your dataverse and the text that appears in your dataverse. You can also add a link to your personal website, the website for your organization or institution, your department, journal, etc.
 
 There are two options for Widgets, a Dataverse Search box widget and a Dataverse Listing widget. The Dataverse Search Box will add a search box to your website that when someone enters a search term in and clicks Find, will bring them to Dataverse to see the results. The Dataverse Listing widget will provide a listing of all your dataverses and datasets. When someone clicks on a dataverse or dataset in the widget, it will bring them to your dataverse to see the actual data. Within the Widgets page, you can copy and paste the code for the widget you would like to have on your website. 
 
@@ -85,7 +85,7 @@ By clicking on the Edit Access button, you are able to change the settings allow
 |image4|
 The Edit Access pop up allows you to also select if someone adding a dataset to this dataverse should be allowed to publish it (Curator role) or if the dataset will be submitted to the administrator of this dataverse to be reviewed then published (Contributor role). These Access settings can be changed at any time.
 
-Assigning a role
+Assign Role
 -----------------------
 You can also give access to a Dataverse user to allow them to access an unpublished dataverse as well as other roles. To do this, click on the Assign Roles to Users/Groups button in the Users/Groups section. You can also give multiple users the same role at one time.
 |image5|
@@ -121,7 +121,7 @@ How do you create a guestbook?
 
 #. After creating a dataverse, click on the Edit Dataverse button and select Dataset Guestbook
 #. By default, guestbooks created in the dataverse your dataverse is in, will appear. If you do not want to use or see those guestbooks, uncheck the checkbox that says Include Guestbooks from Root Dataverse.
-#. To create a new guestbook, click the Create Dataset Guestbook button on the rightside of the page. 
+#. To create a new guestbook, click the Create Dataset Guestbook button on the right side of the page. 
 #. Name the guestbook, determine the account information that you would like to be required (all account information fields show when someone downloads a file), and then add Custom Questions (can be required or not required). 
 #. Hit the Create Dataset Guestbook button once you have finished.
 

src/main/java/Bundle.properties

@@ -419,7 +419,7 @@ dataverse.theme.inheritCustomization.title=Check this to use the existing theme.
 dataverse.theme.inheritCustomization.label=Inherit Customization
 dataverse.theme.inheritCustomization.checkbox=Inherit customization from {0}
 dataverse.theme.logo=Logo
-dataverse.theme.logo.tip=Files larger than 500 kb cannot be uploaded. Maximum display for files is 940 pixels wide by 120 pixels high.
+dataverse.theme.logo.tip=Supported image types are JPEG, TIFF, or PNG and should be no larger than 500 KB. The maximum display size for an image file in a dataverse's theme is 940 pixels wide by 120 pixels high.
 dataverse.theme.logo.format=Logo Format
 dataverse.theme.logo.format.selectTab.square=Square
 dataverse.theme.logo.format.selectTab.rectangle=Rectangle

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -1298,6 +1298,22 @@ public String init() {
 
             // Is the Dataset harvested?
             if (dataset.isHarvested()) {
+                // if so, we'll simply forward to the remote URL for the original
+                // source of this harvested dataset:
+                String originalSourceURL = dataset.getRemoteArchiveURL();
+                if (originalSourceURL != null && !originalSourceURL.equals("")) {
+                    logger.fine("redirecting to "+originalSourceURL);
+                    try {
+                        FacesContext.getCurrentInstance().getExternalContext().redirect(originalSourceURL);
+                    } catch (IOException ioex) {
+                        // must be a bad URL...
+                        // we don't need to do anything special here - we'll redirect
+                        // to the local 404 page, below.
+                        logger.warning("failed to issue a redirect to "+originalSourceURL);
+                    }
+                    return originalSourceURL;
+                }
+
                 return "/404.xhtml";
             }
 

src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java

@@ -129,6 +129,13 @@ public Dataset findByGlobalId(String globalId) {
         String identifier = "";
         int index1 = globalId.indexOf(':');
         String nonNullDefaultIfKeyNotFound = ""; 
+        // This is kind of wrong right here: we should not assume that this is *our* DOI - 
+        // it can be somebody else's registered DOI that we harvested. And they can 
+        // have their own separator characters defined - so we should not assume 
+        // that everybody's DOIs will look like ours! 
+        // Also, this separator character gets applied to handles lookups too, below. 
+        // Which is probably wrong too...
+        // -- L.A. 4.2.4
         String separator = settingsService.getValueForKey(SettingsServiceBean.Key.DoiSeparator, nonNullDefaultIfKeyNotFound);        
         int index2 = globalId.indexOf(separator, index1 + 1);
         int index3 = 0;
@@ -142,15 +149,20 @@ public Dataset findByGlobalId(String globalId) {
             logger.info("Error parsing identifier: " + globalId + ". Second separator not found in string");
             return null;
         } else {
-            if (index2 != -1) {
-                authority = globalId.substring(index1 + 1, index2);
-            }
+            authority = globalId.substring(index1 + 1, index2);
         }
         if (protocol.equals("doi")) {
 
             index3 = globalId.indexOf(separator, index2 + 1);
             if (index3 == -1 ) {
-                identifier = globalId.substring(index2 + 1).toUpperCase();
+                // As of now (4.2.4, Feb. 2016) the ICPSR DOIs are the only 
+                // use case where the authority has no "shoulder", so there's only 
+                // 1 slash in the full global id string... hence, we get here. 
+                // Their DOIs also have some lower case characters (for ex., 
+                // 10.3886/ICPSR04599.v1), and that's how are they saved in the 
+                // IQSS production database. So the .toUpperCase() below is 
+                // causing a problem. -- L.A. 
+                identifier = globalId.substring(index2 + 1); //.toUpperCase();
             } else {
                 if (index3 > -1) {
                     authority = globalId.substring(index1 + 1, index3);

src/main/java/edu/harvard/iq/dataverse/api/Access.java

@@ -29,6 +29,7 @@
 import edu.harvard.iq.dataverse.datavariable.VariableServiceBean;
 import edu.harvard.iq.dataverse.export.DDIExportServiceBean;
 import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
+import edu.harvard.iq.dataverse.util.SystemConfig;
 import edu.harvard.iq.dataverse.worldmapauth.WorldMapTokenServiceBean;
 
 import java.util.List;
@@ -93,7 +94,9 @@ public class Access extends AbstractApiBean {
     @EJB
     VariableServiceBean variableService;
     @EJB
-    SettingsServiceBean settingsService; 
+    SettingsServiceBean settingsService;
+    @EJB
+    SystemConfig systemConfig;
     @EJB
     DDIExportServiceBean ddiExportService;
     @EJB
@@ -401,6 +404,13 @@ public ZippedDownloadInstance datafiles(@PathParam("fileIds") String fileIds, @Q
         // create a Download Instance without, without a primary Download Info object:
         ZippedDownloadInstance downloadInstance = new ZippedDownloadInstance();
 
+
+        long zipDownloadSizeLimit = systemConfig.getZipDownloadLimit();
+        if (zipDownloadSizeLimit > 0L) {
+            logger.fine("setting zip download size limit to " + zipDownloadSizeLimit + " bytes.");
+            downloadInstance.setSizeLimit(zipDownloadSizeLimit);
+        }
+
         if (fileIds == null || fileIds.equals("")) {
             throw new WebApplicationException(Response.Status.BAD_REQUEST);
         }

src/main/java/edu/harvard/iq/dataverse/dataaccess/DataFileZipper.java

@@ -38,7 +38,7 @@
  * @author Leonid Andreev
  */
 public class DataFileZipper {
-    public static long DEFAULT_ZIPFILE_LIMIT = 10 * 1024 * 1024; // 10MB (?)
+    public static long DEFAULT_ZIPFILE_LIMIT = 100 * 1024 * 1024; // 100MB
 
     private static final Logger logger = Logger.getLogger(DataFileZipper.class.getCanonicalName());
 
@@ -74,22 +74,22 @@ public void zipFiles(List<DataFile> files, OutputStream outstream, String fileMa
         Iterator iter = files.iterator();
 
         while (iter.hasNext()) {
-            int fileSize = 0;
             DataFile file = (DataFile) iter.next();
 
             DataAccessRequest daReq = new DataAccessRequest();
             DataFileIO accessObject = DataAccess.createDataAccessObject(file, daReq);
 
             if (accessObject != null) {
                 accessObject.open();
+                long fileSize = accessObject.getSize();
 
                 String fileName = accessObject.getFileName();
                 String mimeType = accessObject.getMimeType();
                 if (mimeType == null || mimeType.equals("")) {
                     mimeType = "application/octet-stream";
                 }
 
-                if (sizeTotal < sizeLimit) {
+                if (sizeTotal + fileSize < sizeLimit) {
 
                     Boolean Success = true;
 
@@ -124,11 +124,12 @@ public void zipFiles(List<DataFile> files, OutputStream outstream, String fileMa
                         byte[] data = new byte[8192];
 
                         int i = 0;
+                        long byteSize = 0; 
                         while ((i = instream.read(data)) > 0) {
                             zout.write(data, 0, i);
                             logger.fine("wrote " + i + " bytes;");
 
-                            fileSize += i;
+                            byteSize += i;
                             //zout.flush();
                         }
                         instream.close();
@@ -139,9 +140,12 @@ public void zipFiles(List<DataFile> files, OutputStream outstream, String fileMa
                             fileManifest = fileManifest + zipEntryName + " (" + mimeType + ") " + fileSize + " bytes.\r\n";
                         }
 
-                        if (fileSize > 0) {
+                        if (byteSize > 0) {
                             successList.add(file.getId());
-                            sizeTotal += Long.valueOf(fileSize);
+                            if (fileSize != byteSize) {
+                                logger.warning("File size mismatch: "+fileSize+" in the database, "+byteSize+" on disk.");
+                            }
+                            sizeTotal += Long.valueOf(byteSize);
                         }
                     }
                 } else {

src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java

@@ -154,7 +154,7 @@ public enum Key {
         /** Optionally override http://guides.dataverse.org . */
         GuidesBaseUrl,
         /* zip download size limit */
-        ZipDonwloadLimit,
+        ZipDownloadLimit,
         /* zip upload number of files limit */
         ZipUploadFilesLimit,
         /* Size limits for generating thumbnails on the fly */

src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java

@@ -297,7 +297,7 @@ private String getGuidesLanguage() {
      */
 
     public long getZipDownloadLimit() {
-        String zipLimitOption = settingsService.getValueForKey(SettingsServiceBean.Key.ZipDonwloadLimit);   
+        String zipLimitOption = settingsService.getValueForKey(SettingsServiceBean.Key.ZipDownloadLimit);   
 
         Long zipLimit = null; 
         if (zipLimitOption != null && !zipLimitOption.equals("")) {

src/main/webapp/dataset.xhtml

@@ -255,7 +255,7 @@
                                         <div id="metrics-views" class="metrics-views tab-pane fade small text-center">
                                             #{bundle['metrics.views']} <em>#{bundle['metrics.comingsoon']}</em>
                                         </div>
-                                        <div id="metrics-downloads" class="metrics-downloads tab-pane fade small text-center">
+                                        <div id="metrics-downloads" class="metrics-downloads tab-pane small text-center active in">
                                             <h:outputFormat value="{0} #{bundle['metrics.downloads']}">
                                                 <f:param value="#{guestbookResponseServiceBean.getCountGuestbookResponsesByDatasetId(DatasetPage.dataset.id)}"/>
                                             </h:outputFormat>
@@ -268,17 +268,20 @@
                                         </div>
                                     </div>
                                     <div id="metrics-tabs">
+                                        <!-- DISABLED TOGGLE UNTIL FURTHER DEVELOPMENT ON METRICS IS COMPLETED 
+                                        ADDED style="pointer-events:none;" ATTRIBUTE TO A-HREF ELEMENTS BELOW 
+                                        REMOVED fade CLASS FROM metrics-downloads DIV ABOVE AND ADDED active in -->
                                         <div class="metrics-hover pull-left">
-                                            <a href="#metrics-views" class="metrics-views" data-toggle="tab">&#160;</a>
+                                            <a href="#metrics-views" style="pointer-events:none;" class="metrics-views" data-toggle="tab">&#160;</a>
                                         </div>
                                         <div class="metrics-hover pull-left">
-                                            <a href="#metrics-downloads" class="metrics-downloads first" data-toggle="tab">&#160;</a>
+                                            <a href="#metrics-downloads" style="pointer-events:none;" class="metrics-downloads first" data-toggle="tab">&#160;</a>
                                         </div>
                                         <div class="metrics-hover pull-left">
-                                            <a href="#metrics-citations" class="metrics-citations" data-toggle="tab">&#160;</a>
+                                            <a href="#metrics-citations" style="pointer-events:none;" class="metrics-citations" data-toggle="tab">&#160;</a>
                                         </div>
                                         <div class="metrics-hover pull-left">
-                                            <a href="#metrics-shares" class="metrics-shares" data-toggle="tab">&#160;</a>
+                                            <a href="#metrics-shares" style="pointer-events:none;" class="metrics-shares" data-toggle="tab">&#160;</a>
                                         </div>
                                     </div>
                                 </div>