Timeout errors

[edsu]

This appears to be a site specific issue when attempting to crawl http://stephenratcliffe.blogspot.com with two workers. After a short period of time a series of net:ERR_ABORTED errors appear, which are eventually followed by a series of long running timeout errors:


Once the timeout errors start the screencast window appears blank.

In /crawls/collections/stephenratcliffe/logs/pywb.log I noticed a series of posts being routed through pywb which seemed to generate the aborted connections, and resulted sometimes in a timeout. For example:

127.0.0.1 - - [2022-08-18 15:27:49] "POST /live/resource/postreq?param.recorder.coll=stephenratcliffe&url=https%3A%2F%2Fstephenratcliffe.blogspot.com%2F2015%2F&closest=now&matchType=exact HTTP/1.1" 200 15420 0.178694
127.0.0.1 - - [2022-08-18 15:27:49] "POST /live/resource/postreq?param.recorder.coll=stephenratcliffe&url=https%3A%2F%2Fstephenratcliffe.blogspot.com%2F2015%2F&closest=now&matchType=exact HTTP/1.1" 200 15406 0.182478
Thu Aug 18 15:27:49 2022 - uwsgi_response_write_headers_do(): Broken pipe [core/writer.c line 248] during CONNECT 2.bp.blogspot.com:443 (127.0.0.1)
[pid: 163|app: 0|req: 354/1704] 127.0.0.1 () {24 vars in 416 bytes} [Thu Aug 18 15:27:40 2022] CONNECT 2.bp.blogspot.com:443 => generated 0 bytes in 8853 msecs (HTTP/1.1 200) 0 headers in 0 bytes (2 switches on core 993)

I was using main at 827c153 with the following command:

docker compose run -p 9037:9037 -it crawler crawl --config /crawls/config.yaml

and the following config file placed in the crawls directory:

collection: stephenratcliffe
workers: 2
generateWACZ: true
text: true
screencastPort: 9037
logging: stats,pywb,behaviors,behaviors-debug
seeds:
  - url: https://stephenratcliffe.blogspot.com
    scopeType: host
    exclude:
      - https?://stephenratcliffe.blogpost.com/search.*
      - https?://stephenratcliffe.blogpost.com//search.*
      - https?://stephenratcliffe.blogspot.com/navbar.g.*

[ikreymer]

Thanks for the detailed repro! Hm, I actually have seen this happen on other sites as well, I think the default window context and screencasting is causing this issue in the latest 0.7.0 beta, which uses Chrome/Chromium 101. This issue is not happening with Chrome 91 (0.6.0 release). Of course want to upgrade to latest browser, so will investigate if there's a reasonable fix..

[edsu]

@ikreymer you're right, it is running a lot smoother without screencasting!

I can't quite understand why it is picking up URLs like https://stephenratcliffe.blogspot.com/search?updated-max=2009-05-12 when the config should be excluding it. But that's a separate issue 😵

[ikreymer]

Can you try this branch: https://github.com/webrecorder/browsertrix-crawler/tree/window-context-tweaks, hopefully this fixes it, with current browser and screencasting enabled..

[edsu]

@ikreymer This branch is working a lot better!

[ikreymer]

Should be fixed as of 0.7.0-beta.3 release