- Revert deployment tagging changes, since they didn't solve the duplicate workflow trigger problem.
- Force parameter facets based on GCMD keywords to be upper-case.
- Only use short name for sensor facets in which the short name and long name are identical.
- Updating harvesting to harvest storage system and spatial coverage into separate facets, instead of a combined facet_featured facet
- Updating harvesting to include cumulus value
- Adding a new facet that will cover both cumulus and global spatial coverage.
- Updating logging default configuration
- Adding logging functionality to the code, including the ability to specify log file destination and log level for both the file and console output
- Added GitHub Action workflows for continuous integration features
- Updated bump rake task to use Bump gem
- Removed release rake task, moved it to the CI workflow
- Fixed a bug with the sanitization, which was trying to modify the string directly (causing problems with frozen strings). Changed to return a new, sanitized string.
- Update Rubocop configuration to actually run against files, and make necessary corrections to comply with Rubocop styling.
- Remove deprecated harvesters and associated tests, helpers, etc.
- Updated a few other dependencies that weren't at the newest versions.
- Updated Ruby to 3.2.2, updated gem dependencies to more recent versions.
- Updated the call for identifiers for the json harvester to use the proper "metadataPrefix" parameter, and request the dif identifiers instead of iso.
- Added a CLI method to "ping" the Solr and Source servers for a given data center.
- Added a CLI method "errcode" to get information about the various error codes that may be returned during harvest
- Updated the CLI harvest to return more useful error codes on failure.
- Bug fix: some requires weren't included that needed to be.
- Update Ruby to 2.6.5, update gem dependencies to more recent version.
- Updates to correspond with an update to Solr 8.5.2
- Patch release to include updated CHANGELOG.
- Update dataset-catalog-services URL to only fetch current (not retired) metadata records.
- Add a few more gem release notes to README.
Note: v4.1.0 was prematurely released and, in theory, yanked. However, on the second try at publishing 4.1.0, Rubygems complained about the attempt to republish a gem. The version was therefore bumped again to 4.2.0 as the path of least resistance to a successful publish. v4.1.0 should not be used.
- Update CHANGELOG and release instructions.
- Fix README typo.
Changes:
- Update spatial field formatting to work with Solr 8.1.1.
Changes:
- Update Ruby, Nokogiri, RestClient, Rubocop, and Webmock versions to address security warnings.
- Update syntax as necessary for new versions of Rubocop and RestClient.
Changes
- Constrain ADC and ECHO feeds to only fetch records in the arctic.
Note
- v3.9.1 and v3.10.0 were mistakenly released after version 3.9.0 was tagged. All three versions are identical, although v3.9.0 was never released to rubygems.org.
Bugfix
- Fix deleting old records after harvest for ADE auto suggest.
Bugfix
- Add dependency on ffi-geos to fix issue where
RGeo::Geos.factoryreturnednilon Ubuntu 14 when parsing the BCO-DMO feed.
Bugfix
- Update NOAA WDS Paleo feed URL to use https.
Bugfix
- Fix BCO-DMO harvester to only fail when there are issues with individual
records if
--die-on-failureis given.
Changes
- Change ECHO harvester to harvest 100 records at a time, rather than 1000 to avoid timeout/hanging issues with the large requests.
- Change "CISL"/ACADIS Gateway harvester to "NSF Arctic Data Center"; aoncadis.org redirects to another site, and the data center's name was changed. The feed format was also changed; the harvester was updated to consume the new feed.
Bugfixes
- Update NODC feed URL to use https.
- Update RDA feed URL to use https.
- Update handling of geometries to match new format provided by BCO-DMO feed.
- Update NMI feed URL; the feed was relocated.
- Harvesting tDAR starts from record 0 instead of record 1.
- tDAR harvester no longer attempts to obtain another page of records after all the records have been harvested; where other feeds return an empty response that our harvester handles without issue, tDAR throws an error if the "startRecord" parameter is higher than their last record.
- Exit with a non-0 status when a problem with the whole feed is encountered,
even if
--die-on-failureis not passed. That flag should only cause failures when there are issues with individual records; we don't want harvesting to stop due to a metadata issue with a small number of records. - Include BCO-DMO URL in the harvester output the same way all the other URLs are displayed.
- RuboCop fixes.
New Features
- Add sponsored programs to NSIDC harvesting.
- Add support for ingesting Data Access Links from NSIDC JSON
Bugfixes
- Fix dependency issue with gem "listen".
- Fix bad configuration for OAI feed URLs.
Bugfixes
- Add temporal duration facet for GTN-P data center.
Changes
- Update long name for GTN-P data center.
New Features
- Add harvester for GTN-P.
See v3.4.0.
Bugfix
- Added quote checking for cisl offset parsing check
Bugfix
- Remove strange facet string for temporal duration from NOAA Paleo search results.
New Features
-
Add harvest support for NOAA Paleoclimatology Data Center (NOAA Paleo).
-
Add harvest support for Data Observation Network for Earth (Data ONE). Pivotal 77763710
Bugfixes
- Catch a timeout error earlier in the stack to prevent an infinite loop of retries; this bug caused the PDC harvester to attempt to access the feed 150 times, instead of simply failing after 3 failed attempts. Pivotal 103057378
Changes
- Change NODC harvester's default page size from 100 to 50. The NODC feed is responding with HTTP 500 when requesting records 301-400, but not when requesting 301-350 or 351-400.
New Features
- Add
harvestsupport for Rolling Deck to Repository (R2R) - Add subcommands
-vand--versionto display the installed version of the gem
Changes
- Gem is available via RubyGems
Bugfixes
- Updated deletion constraints such that lucene special characters in dataset names do not cause deletion of that data provider's data to fail.
Features
- Remove gi-cat as a dependency as no harvesters utilize it.
- Harvest the UCAR NCAR - Earth Observing Laboratory (UCAR/NCAR EOL) from EOL's THREDDS endpoint instead of GI-Cat
- Harvest the Norwegian Meteorological Institute feed directly instead of via GI-Cat.
Bugfixes
- Fix broken configuration, where production was attempting to use the Blue, rather than the the production, Solr host for harvesting. (see PCT-410)
Bugfixes
- Fix broken
delete_allcommands.
- Packaged as a gem with a new executable file, providing a new interface to harvest feeds into solr.
- Change the RDA and EOL harvesters to store the data center name as "UCAR NCAR", rather than "UCAR/NCAR". This fixes a bug with deleting the datasets; the query to Solr was failing because the "/" character could not be correctly escaped.
- Upgrade from Ruby version 1.9.3 to 2.2.2
- Compliant with RuboCop v0.32.0
Features
- Add support to harvest RDA directly from their feed, rather than through GI-Cat.
Bugfixes
- Fix missing accented characters in datasets from Polar Data Catalogue
Features
- Added TDAR translator for harvesting into Solr
- Added PDC (Polar Data Catalog) translator for harvesting int Solr
- Revised CISL endpoint to harvest a subset of data. Created translator to harvest directly rather than through GI-Cat
Bugfixes
- Fixed USGS harvesting issue where it was timing out on specific records
- Fixed EOL translator for processing spatial bounds properly
- Validate bounding boxes for documents being added to Solr
Bugfixes
- Set USGS page size from 100 to 10 to reduce Solr load
- Added exception handling for REST POSTs to Solr
Features
- Updated project to use new CI tools and processes