Code

Topic Collections + Harvest JSON#596

Code

Merged

Fasand merged 55 commits intomasterWebarchivCZ/Seeder:masterfrom collectionsWebarchivCZ/Seeder:collectionsCopy head branch name to clipboardAug 31, 2021

+2,416-692

Commits

Commits on May 4, 2021

• add TopicCollection.[collection_alias, aggregation_with_same_type], implement get_collection_json

  Fasand

  committed
  fdf4cb3

  Copy full SHA for fdf4cb3

Commits on May 10, 2021

• first shot at creating JSON for Harvest with many TODOs RE #402

  Fasand

  committed
  1e13521

  Copy full SHA for 1e13521

Commits on May 28, 2021

• use DEBUG=True for local docker, don't cache templates if in DEBUG

  Fasand

  committed
  464046a

  Copy full SHA for 464046a

• num sources in a category are computed for PUBLIC sources, not ARCHIVING fixes #569

  Fasand

  committed
  fedea62

  Copy full SHA for fedea62

• Harvest custom sources don't need to have archiving states

  Fasand

  committed
  f93c10a

  Copy full SHA for f93c10a

• add Topic Collection seed URLs link to detail

  Fasand

  committed
  08a6ae2

  Copy full SHA for 08a6ae2

• update www bootstrap to 4.6, fix broken styles

  Fasand

  committed
  bb82c05

  Copy full SHA for bb82c05

• add seed URLs buttons to Topic Collection seeder and web (csv)

  Fasand

  committed
  eeb5c52

  Copy full SHA for eeb5c52

• add collectstatic to prepare.sh

  Fasand

  committed
  2637b3c

  Copy full SHA for 2637b3c

• fix visible-sm Bootstrap classes, remove clearfixes (don't do anything anymore)

  Fasand

  committed
  42ff335

  Copy full SHA for 42ff335

• add active page underlining to Nominate

  Fasand

  committed
  46c8935

  Copy full SHA for 46c8935

• fix Select2 vs Bootstrap form-control styling – selected option overflow

  Fasand

  committed
  651b029

  Copy full SHA for 651b029

• swap current/wayback URLs in WWW TopicCollection detail, swap normal/italic style

  Fasand

  committed
  1d774de

  Copy full SHA for 1d774de

• only show Django toolbar for fasand/petr

  Fasand

  committed
  ee746f1

  Copy full SHA for ee746f1

• topic collections: add wayback_url to non-archived and custom seeds, leave them italic

  Fasand

  committed
  5af8d04

  Copy full SHA for 5af8d04

• add TODO for adding HarvestEditView harvest_type checking

  Fasand

  committed
  e841ad7

  Copy full SHA for e841ad7

• merge harvest-json with master

  Fasand

  committed
  7571c74

  Copy full SHA for 7571c74

• include custom seeds in the OneShot collection

  Fasand

  committed
  1e378ab

  Copy full SHA for 1e378ab

• remove tabs/newlines from custom_seeds when serving JSON, update TODOs, add JsonUrls view for Harvest

  Fasand

  committed
  1586637

  Copy full SHA for 1586637

• add combined aliases and annotations to Harvest JSON

  Fasand

  committed
  18e10ca

  Copy full SHA for 18e10ca

Commits on Aug 3, 2021

• change Harvest.scheduled_on to DateTimeField, use stock DateTimeInput with czech input, datetime-local is causing issues

  Fasand

  committed
  c9819bf

  Copy full SHA for c9819bf

Commits on Aug 5, 2021

• ! add migration to split Internal and External TopicCollections, not yet fully solved

  Fasand

  committed
  d330373

  Copy full SHA for d330373

• refactor Topic Collection views, add new for External, update URLs, add link to sidebar; still not on WWW

  Fasand

  committed
  685eb9f

  Copy full SHA for 685eb9f

• implement ExternalTopicCollection custom_sources and custom_seeds, show from all internal collections on WWW

  Fasand

  committed
  2006b60

  Copy full SHA for 2006b60

• make some TC tests changes but most fail because we're in-between large changes

  Fasand

  committed
  3fae33f

  Copy full SHA for 3fae33f

Commits on Aug 9, 2021

• add Harvest.ReadyToHarvest state before Running, freezes seeds; fix scheduled_on URL reverse

  Fasand

  committed
  2b94f44

  Copy full SHA for 2b94f44

• add migration for ReadyToHarvest status

  Fasand

  committed
  8bc1efd

  Copy full SHA for 8bc1efd

Commits on Aug 17, 2021

• add legacy_dumps README

  Fasand

  committed
  038775a

  Copy full SHA for 038775a

• add comments to Harvest. hash_seeds and construct_collection_json

  Fasand

  committed
  7fcc2ff

  Copy full SHA for 7fcc2ff

• no migration; refactor Harvest, add HarvestConfiguration and new Harvest fields, use them where appropriate in JSON

  Fasand

  committed
  31a8fa5

  Copy full SHA for 31a8fa5

Commits on Aug 18, 2021

• minor changes in harvests.models

  Fasand

  committed
  6556ee8

  Copy full SHA for 6556ee8

• refactor OneShot retrieval: always include custom seeds if they exist, and 0-frequency only if the box is checked

  Fasand

  committed
  8a5818c

  Copy full SHA for 8a5818c

• instead of constantly passing down blacklisted/previously_harvested, save it to self and return if available

  Fasand

  committed
  cb61420

  Copy full SHA for cb61420

• remove TODO

  Fasand

  committed
  cc9432b

  Copy full SHA for cc9432b

• add migration for new Harvest fields

  Fasand

  committed
  17de906

  Copy full SHA for 17de906

• add the harvest_info manage.py command to print an overview table of Harvests in the database

  Fasand

  committed
  518be55

  Copy full SHA for 518be55

Commits on Aug 20, 2021

• add fa_boolean template tag, remove Harvest.get_date_url

  Fasand

  committed
  8b91e5c

  Copy full SHA for 8b91e5c

• add new fields to Harvest Detail, split into several sections

  Fasand

  committed
  07edeb5

  Copy full SHA for 07edeb5

• add new Harvest fields to Edit and Create forms

  Fasand

  committed
  b332522

  Copy full SHA for b332522

• fix datetime-local format issue, auto-translate dataLimit GB <-> bytes

  Fasand

  committed
  24fbbf6

  Copy full SHA for 24fbbf6

Commits on Aug 30, 2021

• add HarvestConfig Admin, incl. dataLimit GB<->bytes calculation

  Fasand

  committed
  1eaf83d

  Copy full SHA for 1eaf83d

• combine HarvestEditForm with CreateForm (init + clean_dataLimit), only keep different Meta

  Fasand

  committed
  942efa4

  Copy full SHA for 942efa4

• import gettext in harvests.admin

  Fasand

  committed
  cf9b18e

  Copy full SHA for cf9b18e

• add HarvestConfiguration serializer and viewset

  Fasand

  committed
  fe72c29

  Copy full SHA for fe72c29

• add HarvestConfiguration loading AJAX, implement in Harvest Add, Edit, and Calendar Add

  Fasand

  committed
  d2305d3

  Copy full SHA for d2305d3

• TopicCollection.get_collection_json now accepts scheduled_on of the Harvest since its scheduled_on is nullable

  Fasand

  committed
  eb2ebf3

  Copy full SHA for eb2ebf3

Commits on Aug 31, 2021

• add Harvest.json_frozen for freezing get_json

  Fasand

  committed
  951898b

  Copy full SHA for 951898b

• freeze JSON along with seeds

  Fasand

  committed
  876ea11

  Copy full SHA for 876ea11

• add TC collection_alias and aggregation to forms+detail, allow un-freezing seeds by setting status = Planned

  Fasand

  committed
  6ea23b3

  Copy full SHA for 6ea23b3

• save Harvest.date_frozen on freeze()

  Fasand

  committed
  3673700

  Copy full SHA for 3673700

• add aggregated attachment_set for ExternalTC

  Fasand

  committed
  01b6aa7

  Copy full SHA for 01b6aa7

• update locale strings; translations for new Harvest + config + TC fields

  Fasand

  committed
  3f04584

  Copy full SHA for 3f04584

• add HarvestConfiguration views+forms for Seeder

  Fasand

  committed
  0d441da

  Copy full SHA for 0d441da

• only allow superusers and Tech users in Seeder HarvestConfig

  Fasand

  committed
  bb9fa7a

  Copy full SHA for bb9fa7a

• add HarvestConfiguration history, translation for config __str__

  Fasand

  committed
  377853f

  Copy full SHA for 377853f