Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Topic Collections + Harvest JSON #596

Merged
merged 55 commits into from
Aug 31, 2021
Merged

Topic Collections + Harvest JSON #596

merged 55 commits into from
Aug 31, 2021

Conversation

Fasand
Copy link
Contributor

@Fasand Fasand commented Aug 9, 2021 •
edited
Loading

Jelikož jsem na TC začal pracovat až po několika změnách na Harvests a základním nastavení pro Harvest JSON, ještě to nechám trochu uležet a PR nakonec bude jak pro Harvest JSON tak pro Topic Collection rozdělení.
Při snaze o rebase masteru jsem vytvořil trochu peklo, proto dávám PR znovu s čistější historií.

  • Bude řešit i Kolekce a json zaznam pro harvesty #402 a některé spojené issues
    • Nová pole pro Harvest, model HarvestConfiguration, skriptík pro vypsání tabulky sklizní
    • Freeze uloží timestamp, zmrazí seeds i json to samostatných polí
    • Pokud se stav nastaví zpět na "Planned", semínka se "odmrazí" – tedy nastaví zpět na None. Tím pádem lze nachystat sklizeň, zmrazit, udělat pár změn, odmrazit a znovu zmrazit aktuální verzi s novým timestampem
    • Skriptík: manage.py harvest_info, případně manage.py harvest_info --plaintext; vypsat do souboru manage.py harvest_info > out.txt
  • Řeší Definice default type HarvestConfiguration #594: nový model, možnost editace pro staff uživatele skrz admina, AJAX načítání
    • Po změně Harvest.harvest_type se automaticky načtou defaults pro daný typ sklizně nebo vyskočí upozornění, že defaults ještě nejsou nastaveny
    • dataLimit se automaticky přepočítává GB <-> bytes; v administraci i Seederu se zadávají GB, interně se ukládá a propisuje do JSONu jako bytes
    • Editovací/detail stránky na Seederu, přístupné pouze superuživatelům a "Tech" uživatelům
  • fix Rozdeleni objektu kolekci #592 : Původní TopicCollection zachována, prezentační data přesunuta do ExternalTopicCollection
  • Částečně řeší Scheduling sklizni s u kolekci s pravidelnou frekvenci #568 : nový stav ReadyToHarvest, žádné změny v samotném schedulingu zatím
  • Seeder views přidány pro ExternalTC, WWW views zobrazují External a tedy semínka/zdroje pro všechny přiřazené InternalTC
  • Velká migrace, která z TopicCollections vytáhne data pro ExternalTC a je revertable

Fasand added 30 commits May 4, 2021 09:32
@Fasand
add TopicCollection.[collection_alias, aggregation_with_same_type], i…
fdf4cb3 
…mplement get_collection_json
@Fasand
first shot at creating JSON for Harvest with many TODOs RE #402
1e13521
@Fasand
use DEBUG=True for local docker, don't cache templates if in DEBUG
464046a
@Fasand
num sources in a category are computed for PUBLIC sources, not ARCHIV…
fedea62 
…ING fixes #569
@Fasand
Harvest custom sources don't need to have archiving states
f93c10a
@Fasand
add Topic Collection seed URLs link to detail
08a6ae2
@Fasand
update www bootstrap to 4.6, fix broken styles
bb82c05
@Fasand
add seed URLs buttons to Topic Collection seeder and web (csv)
eeb5c52
@Fasand
add collectstatic to prepare.sh
2637b3c
@Fasand
fix visible-sm Bootstrap classes, remove clearfixes (don't do anythin…
42ff335 
…g anymore)
@Fasand
add active page underlining to Nominate
46c8935
@Fasand
fix Select2 vs Bootstrap form-control styling – selected option overflow
651b029
@Fasand
swap current/wayback URLs in WWW TopicCollection detail, swap normal/…
1d774de 
…italic style
@Fasand
only show Django toolbar for fasand/petr
ee746f1
@Fasand
topic collections: add wayback_url to non-archived and custom seeds, …
5af8d04 
…leave them italic
@Fasand
add TODO for adding HarvestEditView harvest_type checking
e841ad7
@Fasand
merge harvest-json with master
7571c74
@Fasand
include custom seeds in the OneShot collection
1e378ab
@Fasand
remove tabs/newlines from custom_seeds when serving JSON, update TODO…
1586637 
…s, add JsonUrls view for Harvest
@Fasand
add combined aliases and annotations to Harvest JSON
18e10ca
@Fasand
change Harvest.scheduled_on to DateTimeField, use stock DateTimeInput…
c9819bf 
… with czech input, datetime-local is causing issues
@Fasand
! add migration to split Internal and External TopicCollections, not …
d330373 
…yet fully solved
@Fasand
refactor Topic Collection views, add new for External, update URLs, a…
685eb9f 
…dd link to sidebar; still not on WWW
@Fasand
implement ExternalTopicCollection custom_sources and custom_seeds, sh…
2006b60 
…ow from all internal collections on WWW
@Fasand
make some TC tests changes but most fail because we're in-between lar…
3fae33f 
…ge changes
@Fasand
add Harvest.ReadyToHarvest state before Running, freezes seeds; fix s…
2b94f44 
…cheduled_on URL reverse
@Fasand
add migration for ReadyToHarvest status
8bc1efd
@Fasand
add legacy_dumps README
038775a
@Fasand
add comments to Harvest. hash_seeds and construct_collection_json
7fcc2ff
@Fasand
no migration; refactor Harvest, add HarvestConfiguration and new Harv…
31a8fa5 
…est fields, use them where appropriate in JSON
Fasand added 20 commits August 18, 2021 09:02
@Fasand
minor changes in harvests.models
6556ee8
@Fasand
refactor OneShot retrieval: always include custom seeds if they exist…
8a5818c 
…, and 0-frequency only if the box is checked
@Fasand
instead of constantly passing down blacklisted/previously_harvested, …
cb61420 
…save it to self and return if available
@Fasand
remove TODO
cc9432b
@Fasand
add migration for new Harvest fields
17de906
@Fasand
add the harvest_info manage.py command to print an overview table of …
518be55 
…Harvests in the database
@Fasand
add fa_boolean template tag, remove Harvest.get_date_url
8b91e5c
@Fasand
add new fields to Harvest Detail, split into several sections
07edeb5
@Fasand
add new Harvest fields to Edit and Create forms
b332522
@Fasand
fix datetime-local format issue, auto-translate dataLimit GB <-> bytes
24fbbf6
@Fasand
add HarvestConfig Admin, incl. dataLimit GB<->bytes calculation
1eaf83d
@Fasand
combine HarvestEditForm with CreateForm (init + clean_dataLimit), onl…
942efa4 
…y keep different Meta
@Fasand
import gettext in harvests.admin
cf9b18e
@Fasand
add HarvestConfiguration serializer and viewset
fe72c29
@Fasand
add HarvestConfiguration loading AJAX, implement in Harvest Add, Edit…
d2305d3 
…, and Calendar Add
@Fasand
TopicCollection.get_collection_json now accepts scheduled_on of the H…
eb2ebf3 
…arvest since its scheduled_on is nullable
@Fasand
add Harvest.json_frozen for freezing get_json
951898b
@Fasand
freeze JSON along with seeds
876ea11
@Fasand
add TC collection_alias and aggregation to forms+detail, allow un-fre…
6ea23b3 
…ezing seeds by setting status = Planned
@Fasand
save Harvest.date_frozen on freeze()
3673700
@Fasand Fasand self-assigned this Aug 31, 2021
@Fasand Fasand marked this pull request as ready for review August 31, 2021 10:12
@Fasand Fasand merged commit 151ffdb into master Aug 31, 2021
@Fasand Fasand deleted the collections branch August 31, 2021 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Fasand Fasand

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Rozdeleni objektu kolekci
1 participant
@Fasand