[PROD] Ingest Appeal Docs

[szabozoltan69]

Issue

Recently the ingest_appeal_docs job does not run fine.
Without a header hack (of personal cookie data) the scraping of www.ifrc.org/appeals/ gives:
'reason': 'Forbidden',
'status': 403
(Also pip install brotlipy is needed for the successful decompression of the data.)

The bigger issue is that why is that failure invisible in cronjob items? Normally an erroneous run should be seen there and give a big warning message.

@batpad @thenav56

[szabozoltan69]

We could use such API endpoints instead of scraping:
https://go-api.ifrc.org/api/PublicSiteAppeals?Appealnumber=MDRKZ012&Hidden=false

[tovari]

@arunissun, would you mind to compare staging and prod data of the 'appeal_document' endpoint?

[szabozoltan69]

Some differences can be; prod and staging count:

postgres=> select count(*) from api_appealdocument;
 count 
-------
  8126

postgres=> select count(*) from api_appealdocument;
 count 
-------
  8133

[arunissun]

@tovari I will check the staging and prod data for appeal documents

[tovari]

Thanks @arunissun for the analysis. Looks like the scraper doesn't work well anymore on prod (there are missing docs from 2024).