[Bug]: Incognito mode is required (document?)

[adamscybot]

Expected Behavior

As discussed in jef#2698 (comment).

Incognito mode was the thing I was missing. After this, it works fine. Perhaps that should just be hard-coded on as there's no reason not to be using it.

Side point: I'm actively using this now. I will raise PRs if I see/notice anything more. Good job -- you saved me a day!

Current Behavior

See logs. Browser dies, streetmerchant soft restarts.

What operating system are you seeing the problem on?

Windows

Relevant log output

[19:43:41] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Request is already handled! {
  "stack": "Error: Request is already handled!\n    at assert (\\node_modules\\.pnpm\\[email protected]\\node_modules\\puppeteer-core\\lib\\cjs\\puppeteer\\util\\assert.js:18:15)\n    at CdpHTTPRequest.abort (\\node_modules\\.pnpm\\[email protected]\\node_modules\\puppeteer-core\\lib\\cjs\\puppeteer\\api\\HTTPRequest.js:306:32)\n    at proxyRequest0 (\\node_modules\\.pnpm\\[email protected]\\node_modules\\puppeteer-proxy\\dist\\routines\\proxyRequest.js:136:39)\n    at async handleDefaultContextProxy (\\build\\src\\proxy.js:76:9)\n    at async \\build\\src\\util.js:115:21\n    at async CdpHTTPRequest.finalizeInterceptions (\\node_modules\\.pnpm\\[email protected]\\node_modules\\puppeteer-core\\lib\\cjs\\puppeteer\\api\\HTTPRequest.js:151:9)"

Configuration

STORES=awd,box,ebuyer,novatech,nvidia-gb,overclockers,scan
SHOW_ONLY_COUNTRY=UK
SHOW_ONLY_SERIES=5090

Code of Conduct

• I agree to follow this project's Code of Conduct

[andrewmackrodt]

Thanks for creating the issue here:

I was able to reproduce this when having multiple proxies set and INCOGNITO=false, does that match your setup? (edit: explained in original comment). I'm not sure why exactly the request has already been handled so have added a check before calling continue and returning early if that is the case: src/proxy.ts#L100.

Please note, I'm continuously rebasing on-top of upstream and squashing commits in this fork so you may need to git reset --hard if you want to pull any updates.

As for using proxies, I find using INCOGNITO=true yields better results. I do not understand why, but this is true even in a fresh browser instance on the first request. It it also more stable as --proxy-server can be set on the chromium instanc directly rather than relying on puppeteer middleware.

[adamscybot]

Hi yeh, I noticed the rebasing, ive been resetting on origin.

For sure incognito is working. I've accidentally blasted through my bandwidth available on my proxy plan though just now (lol). Are you using any settings to reduce bandwidth (images? not sure how much you can get away with constraining such things).

[andrewmackrodt]

My plan with oxylabs is unlimited but I'm not sure that's the case for my VPS heh, so I've enabled LOW_BANDWIDTH=true, it seems to be ok for these retailers. My current config is:

COUNTRY=uk
DISCORD_WEB_HOOK=<redacted>
IN_STOCK_WAIT_TIME=1200
INCOGNITO=true
LOW_BANDWIDTH=true
PROXY_PROTOCOL=https
PROXY_LIST=<redacted>,<redacted>,.....
PROXY_USERNAME=<redacted>
PROXY_PASSWORD=<redacted>
SCREENSHOT=false
SHOW_ONLY_SERIES=5080,5090
STORES=awd,box,currys,ebuyer,novatech,nvidia-gb,overclockers,scan
USER_AGENT_KEEP_DEFAULT=true
WEB_PORT=8080

I did buy a 5080 already so am doing this for fun at this point, will hopefully upstream the less substantial refactoring changes. Am still on the lookout for a 5080 FE though!

[adamscybot]

Cool. Didn't notice LOW_BANDWIDTH, oops. I will now get an oxylabs sub :P.

Ah you can see I have lost all sense of perspective and gone straight for the 5090.