Possible problem with geckodriver

[turban1988]

Hi,
When I perform a large scale measurement the crawl “crashes” with the following error message:

2019-02-20 18:49:46,567 - PoolWorker-3[MainThread] - browser_commands - ERROR : BROWSER 1: error visting internal link: Message: Failed to decode response from marionette

All following GET commands fail with the following error:
2019-02-20 18:49:46,583 - PoolWorker-3[MainThread] - browser_commands - ERROR : BROWSER 1: error visting internal link: Message: Tried to run command without establishing a connection

It seems to be related to geckodriver. Did anyone else experience something like this?

[felix4webscience]

Hey,
I did and found the following threads at stackoverflow, which might fit to this topic?
1.) https://stackoverflow.com/questions/49734915/failed-to-decode-response-from-marionette-message-in-python-firefox-headless-s

2.) https://stackoverflow.com/questions/46088442/is-it-firefox-or-geckodriver-which-creates-rust-mozprofile-directory

Were you able to solve that problem?

[turban1988]

I added a random waiting period right after the get_website function call. This seems to fix the problem.

get_website(new_url, sleep, visit_id, webdriver, browser_params, extension_socket)
time.sleep(1 + random.randint(1, 3))

Thanks.

[englehardt]

Related to #255.

[englehardt]

Should be fixed from the changes described in #255 (comment). Feel free to open a new issue if not.