Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: allow to reuse a browser by passing a browserContext #884

Merged
merged 15 commits into from
Feb 16, 2025

Conversation

daniel-hauser
Copy link
Contributor

@daniel-hauser daniel-hauser commented Oct 2, 2024

Motivation:

  • Enables efficient scraping of multiple accounts in parallel by allowing to reuse a browser.
  • Enables parallel runs of the same scraper with different users by creating a browser context for each user.

Changes:

  • Modified the page creation logic to allow passing a browserContext instead of a browser.
  • The ScraperOptions interface now has a union of three BrowserOptions interfaces:
    • DefaultBrowserOptions for when the scraper creates the browser.
    • ExternalBrowserOptions for when the caller supplies an external single-use browser.
      • An optional skipCloseBrowser flag was added to allow the caller to manage the browser lifecycle.
    • ExternalBrowserContextOptions for when the caller supplies a BrowserContext and manages the browser lifecycle.

@daniel-hauser daniel-hauser changed the title feat: Allow to reuse a browser by passing a browserContext feat: allow to reuse a browser by passing a browserContext Oct 3, 2024
@daniel-hauser daniel-hauser marked this pull request as ready for review October 5, 2024 20:36

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated no suggestions.

Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/scrapers/base-scraper-with-browser.ts:85

  • [nitpick] The word 'bang' might be confusing. It would be clearer to use 'exclamation mark (!)' instead.
NOTICE - it is discouraged to use bang (!) in general.

src/scrapers/base-isracard-amex.ts:33

  • The ExtendedScraperOptions interface was removed, but there was no mention of its removal in the context provided. Ensure that this interface is not used elsewhere in the codebase.
type CompanyServiceOptions = {

src/scrapers/base-isracard-amex.ts:258

  • [nitpick] The parameter name 'options' is of type 'CompanyServiceOptions'. It would be clearer to rename it to 'companyServiceOptions'.
async function getExtraScrapTransaction(page: Page, options:CompanyServiceOptions, month: Moment, accountIndex: number, transaction: Transaction): Promise<Transaction> {

baruchiro and others added 3 commits February 2, 2025 08:17
…ptions

* **ExternalBrowserOptions**
  - Add example using an externally created browser instance

* **ExternalBrowserContextOptions**
  - Add example using an externally managed browser context
@daniel-hauser
Copy link
Contributor Author

@baruchiro Thanks for approving! Do you have an estimation for the merging of the PR?

@baruchiro baruchiro merged commit fdc55a5 into eshaham:master Feb 16, 2025
5 checks passed
Copy link

🎉 This PR is included in version 5.4.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants