Skip to content

Commit

Permalink
Move datasets to delete first in line
Browse files Browse the repository at this point in the history
We have reports of datasets that get re-harvested with an extra `1` in the URL. We have confirmed these reports.
It seems the harvest is doing the best it can to diagnose if this is a new dataset or not; but still failing in some circumstances.
This probably won't fix the bug; however it will mitigate it. By hopefully running through the datasets removal first, if the spatial harvester is essentially doing a "delete and add" when it should be replacing, then the name of the new dataset won't collide with the one that is marked for deleted but still in the system.
  • Loading branch information
jbrown-xentity authored Oct 22, 2021
1 parent 9354d52 commit 77a8b0f
Showing 1 changed file with 13 additions and 13 deletions.
26 changes: 13 additions & 13 deletions ckanext/spatial/harvesters/waf.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,19 @@ def create_extras(url, date, status):


ids = []
for location in delete:
obj = HarvestObject(job=harvest_job,
extras=create_extras('','', 'delete'),
guid=url_to_ids[location][0],
package_id=url_to_ids[location][1],
)
model.Session.query(HarvestObject).\
filter_by(guid=url_to_ids[location][0]).\
update({'current': False}, False)

obj.save()
ids.append(obj.id)

for location in new:
guid=hashlib.md5(location.encode('utf8','ignore')).hexdigest()
obj = HarvestObject(job=harvest_job,
Expand All @@ -160,19 +173,6 @@ def create_extras(url, date, status):
obj.save()
ids.append(obj.id)

for location in delete:
obj = HarvestObject(job=harvest_job,
extras=create_extras('','', 'delete'),
guid=url_to_ids[location][0],
package_id=url_to_ids[location][1],
)
model.Session.query(HarvestObject).\
filter_by(guid=url_to_ids[location][0]).\
update({'current': False}, False)

obj.save()
ids.append(obj.id)

if len(ids) > 0:
log.debug('{0} objects sent to the next stage: {1} new, {2} change, {3} delete'.format(
len(ids), len(new), len(change), len(delete)))
Expand Down

0 comments on commit 77a8b0f

Please sign in to comment.