Linux: r.init() not responding - see these checks and solutions at issue #190

[Caside]

I use Ubuntu 20.04 and PyCharm to use RPA.
I followed all steps from https://sikulix-2014.readthedocs.io/en/latest/newslinux.html
When i run scryp:

import rpa as r
r.init()
r.url('https://www.google.com')
r.close()

I got a blank chrome window open and the process hung. I've tried debugging. p.init() goes into an infinite loop and the next commands are not executed. Errors and warnings are not raised.

Maybe there is some better instruction for running this library on Linux?

[prvnsingh]

I am also facing the same the issue ... r.init() hungs up for while (40mins) and returns
[RPA][ERROR] - following happens when starting TagUI...

ps: I did tagui.setup .... it has downloaded all the file and returned me true

[prvnsingh]

After an hour later I interrupted... The code is stuck here...

`Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import tagui as t
t.init()
^CTraceback (most recent call last):
File "", line 1, in
File "/root/anaconda3/lib/python3.7/site-packages/tagui.py", line 550, in init
tagui_out = _tagui_read()
File "/root/anaconda3/lib/python3.7/site-packages/tagui.py", line 118, in _tagui_read
global _process; return _py23_decode(_process.stdout.readline())
KeyboardInterrupt

`

[prvnsingh]

I guess... it because I am using it as a root user on Centos machine.... and chrome has issue in running for root user (need to use --no-sandbox switch) but still not working ..

[kensoh]

Hi RPA for Python users, 🅗🅐🅟🅟🅨 ②⓪②①!

It has been many months since I last worked on this personal side project. Just want to shout out that this project is very much alive, and I hope to look into the 30+ issues raised since July (and bug-fixes if required) soon when I get a moment again.

Below are details if you are interested, why I'm away past few months and why I believe my darkest days are over 🍀🤞🏻🙏🏻

---

Since May last year, I've stopped all work to juggle with my mum's critical illness, taking care of my toddler, and doing housework chores like cleaning and cooking. TBH, my wife and I felt that taking care of a baby and giving her the best nutrition (nursing and home-cooked organic food) and care requires at least 1.5 full-time headcount. There is no life to speak of, other than trying to stay afloat with baby care needs, and keep finding alternative ways to treat my mum, where there're some wins.

Eg importing soy-based tube feeds from Netherlands / Germany / Australia --> after switching away from milk-based tube feeds for 2 weeks she stops requiring dialysis. She started requiring 3x a week dialysis for more than 3 months, after admitting to the hospital. Also, applying herbs like mashed garlic or diced onions to her feet somehow consistently improve her breathing and lung function. Have to keep trying alternative healing methods when her recovery keeps hitting plateau in hospital.

She has been discharged from the hospital a month ago and recovering at home, though very slowly. A longterm medication seems to be one of the contributing causes for her condition. After stopping it, there seems to be improvement to her breathing capacity. She is still bed-bound, but over the course of the year I hope she'll regain strength to move about, and hopefully lungs to recover sufficiently to not require oxygen tank support through a tracheal mask (a hole through her throat).

---

Since November, I've gone back to work at AI Singapore, a government-funded programme to accelerate AI in Singapore. I'm with Product Engineering team, focusing on RPA and TagUI (open-source RPA tool), which RPA for Python is a wrapper for. There are pretty interesting stuffs on TagUI 2021 roadmap. 2020 was the year which my family was going to move to the UK and work, after I receive its tier 1 exceptional tech talent visa. Then everything changed with the virus outbreak, and my mum's illness. The best thing that happened to me in 2020 was being able to go back to work at AI Singapore - I'm grateful.

I've just hired someone to help with housework, cooking and supporting my wife with baby care. I hope this new arrangement will allow me to deliver on my work at AI Singapore, and soon be able to reply countless friends' messages which circumstances leave me with no choice but go missing for months, and then look into the issues raised here!

Eg of the cost of best-in-class nutrition standard researched and set by wife. Baby just felt restless and broke her plate. Even Corelle's break-resistant plate breaks. Why not use plastic plate? Cuz plastic can leech toxins to the food on it. Why not use Silicon plate? Cuz there is no white silicon materials for plate. Why white? Cuz that helps to improve visibility of the food instead of contrasting against some pink colour or some other colours. I would be spending much time clearing up the mess of 70 over pieces of plate fragments, while baby gets nursed to calm down, if not for our newfound helper.

That's why I feel like my darkest days are over and now seeing glimpses of light at the end of the tunnel.

---

My vision for RPA for Python remains - I would like it to be the go-to Python package for RPA.

A best-in-class tool in Python users toolbox which you combine with other cool packages to build amazing stuffs.

[chuckmckinnon]

After an hour later I interrupted... The code is stuck here...

`Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import tagui as t
t.init()
^CTraceback (most recent call last):
File "", line 1, in
File "/root/anaconda3/lib/python3.7/site-packages/tagui.py", line 550, in init
tagui_out = _tagui_read()
File "/root/anaconda3/lib/python3.7/site-packages/tagui.py", line 118, in _tagui_read
global _process; return _py23_decode(_process.stdout.readline())
KeyboardInterrupt

`

I am having exactly these symptoms, and Ctrl-C had me crashing at exactly that line number. It is the _process.stdout.readline() that's hanging.

If in tagui.py's def init() I change shell=True to shell=False, it no longer hangs, but then it errors out telling me that it can't find /path/to/tagui rpa_python headless, which is the entry shell command to invoke the tagui process.

I'm trying to run in a Docker container, Ubuntu 20.04. There may be a way to make that stdout.readline request non-blocking, see https://stackoverflow.com/questions/375427/a-non-blocking-read-on-a-subprocess-pipe-in-python but I'm not enough of a Python developer to know what would have to be modified in tagui.py for that to work.

@kensoh If it helps, my use case is to headlessly login with a username and password to a website once per day, and pull a few numbers from a daily report page for use in a separate dashboard. That dashboard combines data from three sources. Two of those sources have an API but one does not. I am trying to use RPA Python / TagUI to read those numbers from the source that has no API but only a webpage. I'd like the job to run under cron so it runs daily without intervention, and under Docker because I can pass the username and password as environment variables stored on the host system as a read-only file with limited permissions.

The automation runs fine on my local Ubuntu 20.04 machine, even headless, but always hangs in the container. I've also tried Alpine as the base image and things hang in the same place, on r.init() when it's trying to use _process.stdout.readline()

You've built a powerful tool; it's excellent work. I hope this context helps explain why some of us are trying to run inside Docker containers. The isolation and portability they provide is very compelling for certain use cases. Thanks!

[kensoh]

Hi Guys! Can you try the minimal replication script below and share back what happens?

import rpa as r
r.debug(True)
r.init()
r.url('https://www.google.com')
r.close()

debug(True) will enable printing of communication messages between Python process and TagUI process.

The communications have to be blocking so that execution happens synchronously step by step.

Other than for some reason the communications between the 2 processes mess up, another possible point for infinite loop is when TagUI connects to the Chrome process. This is done with the following command. In normal situation, Chrome process is launched with a web socket connection at port 9222 at localhost for TagUI to connect to it. If for some reason this connection is not realised, TagUI will keep looping until Chrome shares that connection.

curl -s localhost:9222/json | grep -A 1 '"url": "about:blank"' | grep webSocketDebuggerUrl | cut -d'"' -f 4

To check for this Chrome connection, you can go to ~/.tagui/src folder and run ./tagui samples/1_yahoo chrome and see if it hangs. Then try with ./tagui samples/1_yahoo headless. If it really hangs, you can change the ~/.tagui/src/tagui file the first line to #!/usr/bin/env bash -x and try again. That will at least give the command that results in the infinite loop so that further troubleshooting can be done.

Note that there is an open issue on headless mode related to a problem introduced in newer Chrome versions. Here's the workaround but I'll put in a proper fix soon - #216

This is a personal project, but for the upstream TagUI project, there is a weekly Zoom call at Thursday 4pm (UTC+8), which you can join to troubleshoot this together, since this seems probable to be an issue with TagUI, if it hangs at connecting to Chrome. Here's the Zoom meeting invitation URL - aisingapore/TagUI#914

[kensoh]

There's also a Telegram group chat for TagUI and RPA for Python users, if you use Telegram - https://t.me/rpa_chat

[kensoh]

Adding on, the fix for newer headless Chrome on Windows is out.

This is fixed in v1.31, available with pip install rpa --upgrade

[chuckmckinnon]

Thanks @kensoh. For me, the minimal replication script hangs in the same place. I will try the curl connection to Chrome and report back.

I may not be able to attend the Zoom call this week but if not, will try to do so next week. Thank you!

[kensoh]

Hi Guys, below is an update to a similar issue #190 for Linux, see if that helps! Closing this issue.

---

Hi Guys, can you try the latest package with pip install rpa --upgrade and let me know if it works now?

• There has been a fix implemented to run on macOS properly, in response to a recent macOS change
• For Ubuntu, still require to install PHP with apt-get update; apt install php;
• For Linux OSes, the default for this package is to use Chrome, so Chrome browser needs to be installed
• For Linux or macOS, Chrome does not allow a user to run it as root, a hack is needed if you really need root
• Using Chromium browser is possible but a simple hack is required to change the package default to use Chromium
• Using Microsoft Edge is also possible but a hack would be needed until upstream TagUI supports Edge option

Below is a working Google Colab notebook with some of the above hacks implemented fyi -
https://colab.research.google.com/drive/13bQO6G_hzE1teX35a3NZ4T5K-ICFFdB5?usp=sharing

Also, this RPA for Python package is based on a forked version of TagUI open-source RPA tool.
Feel free to join Telegram community group chat to post any questions - https://t.me/rpa_chat

[pradishb]

When running ./tagui samples/1_yahoo chrome I got the error below. I installed curl and now r.init() is responding. I noticed that without curl, it can not properly install tagui on the first run. Had to reinstall it. Now, everything works properly.

$ ./tagui samples/1_yahoo chrome
./tagui: line 314: curl: command not found
./tagui: line 314: curl: command not found
./tagui: line 314: curl: command not found
./tagui: line 314: curl: command not found

[kensoh]

Hi @pradishb thanks for your feedback and sharing here!

Yes, curl command comes with Linux by default for most Linux distributions. But for a Linux OS without that, it will require installing it in the way that's done for a particular distribution. Adding for reference, if it is Ubuntu, it's apt install curl.

[b-cil]

@chuckmckinnon did you manage to package your script in docker ? I'm running into the exact same problem and it drives me crazy.

Using ubuntu 18.04, the init() infinite loop (non docker launch) caused by _py23_decode(_process.stdout.readline()) occurs when :

• I move from chrome to chromium with the following r.dump(r.load('/home/pierrem/.tagui/src/tagui').replace('"google-chrome"', '"chromium-browser"').replace('$headless_switch', '--no-sandbox'), '/home/pierrem/.tagui/src/tagui')
• If I launch the pyvirtualdisplay : import pyvirtualdisplay; display = pyvirtualdisplay.Display(); display.start();

The script that works on ubuntu using python3 -u ....py does not work under docker.

[chuckmckinnon]

@chuckmckinnon did you manage to package your script in docker ? I'm running into the exact same problem and it drives me crazy.

I did not. I moved on to a different solution as my intention was just to grab a handful of numbers daily. I'm still intrigued by TagUI and grateful for @kensoh's work. I've just been too busy with other things to come back to this. =)

Good luck!

[b-cil]

Hey,

thank you for answering... To be honest me neither !
I kinda come to the conclusion that TagUi and docker were not compatible due to shell output which I deeply regret...

If someone crack this, I would be definitely interested !

[b-cil]

just out of curiosity, which solution did you choose and is it docker compatible ?

[chuckmckinnon]

We just did a more traditional python script/cron job since what we were screen-scraping for was pretty simple.