Can't scrape more than one year's worth of data

[whatuserever]

I'm trying to scrape all available data from my financial accounts. However, when I set a start date for more than one year ago, it gets overwritten with a date that is about one year ago from today.

Looking at the code, this seems to be the root cause:

israeli-bank-scrapers/src/scrapers/base-beinleumi-group.ts

Lines 343 to 345 in 26837de

	const defaultStartMoment = moment().subtract(1, 'years').add(1, 'day');
	const startDate = this.options.startDate || defaultStartMoment.toDate();
	const startMoment = moment.max(defaultStartMoment, moment(startDate));

I noticed that this logic exists in most (all?) scrapers.

Testing Beinleumi and Max without this, i.e. with the following diff, seems to work perfectly fine:

-    const startMoment = moment.max(defaultStartMoment, moment(startDate));
+    const startMoment = moment(startDate);

Is there a reason for this limitation? Can it be removed?

[baruchiro]

I think it should be defined like this only for websites that disallow fetching more than one year before.

[whatuserever]

I figured the history limit for Beinleumi and Max and created PRs. Hopefully others can do the same for the rest :)

[whatuserever]

I see #895 and #896 were approved, but I don't have permissions to merge. @baruchiro could you please merge theme as well? 🙏

[baruchiro]

I'm waiting a while to give others a chance to review.

[whatuserever]

Ah I see. No problem!