[Ask About Hangup Signal]

[riandyrn]

Hello, Gene

I have question regarding hangup signal (syscall.SIGHUP) in Tinode. What are the possible causes for Tinode to receive hangup signal?

So I was trying to run Tinode cluster on EC2 instances behind load balancer. On each instances I was using nohup to run Tinode on the background. When I tried to use it, sometimes only one instances shutdown randomly, sometimes both. When I check the output on nohup.out I notice following consistent output: Signal received: 'hangup', shutting down.

Before I was running Tinode on cluster, I also tried to run Tinode on single instance & sometimes I get the same output. But in cluster setup, it is occurred more often. Especially when I tried to send large payload from another instance.

So what are the possible causes for this issue on Tinode?

Thanks

[or-else]

The signal handler is intended for graceful shutdown. I.e. to tell the server to close connections and do other clean up. The signal is sent by the environment. I.e. it's something external to the server.

Would it be possible that EC2 sends SIGHUP when the process runs out of resources? Maybe out of memory or similar?

[or-else]

See here: https://en.wikipedia.org/wiki/SIGHUP

[or-else]

Is it possible that you launch the server from the shell and then close the shell?

[or-else]

Here is how I launch the server:

sudo sh -c '/opt/tinode/tinode --listen=:443 --tls_enabled=true --config=/etc/tinode.conf --static_data=/opt/tinode/static 2> /var/log/tinode.log &'

Notice that last &. It makes the process start in the background, so when the terminal is closed the process continues to run.

[riandyrn]

Would it be possible that EC2 sends SIGHUP when the process runs out of resources? Maybe out of memory or similar?

Hmm…, yes, I was also expecting the same thing. But I wasn’t sure because I was only running 3 users. So I thought maybe Tinode have somekind of internal mechanism which will send SIGHUP signal when some events or errors occurs which I wasn’t aware of. But from your explanation it confirms that the signal came purely from the environment, not from Tinode itself.

Notice that last &. It makes the process start in the background, so when the terminal is closed the process continues to run.

Yes, I also did similar things using nohup :D

Ok, thank you very much, Gene. At least I got some clue for the issue :D (y)

[riandyrn]

Hello, Gene

I found the cause of SIGHUP signal.

The issue was caused by unexpected behavior (at least for me) of nohup. So apparently nohup doesn’t always ignore SIGHUP signal even though the process already running in background.

Such situation happen when we are already starting the background process then somehow the SSH connection broke (indicated by message Connection to XXX.XXX.XXX.XXX port 22: Broken pipe). The background process would still receive SIGHUP signal.

Thus the correct way to setup background process on nohup, after setting the process as background process, we issue exit command. So the commands would be following:

> nohup $GOPATH/bin/server -config=$GOPATH/src/github.com/tinode/chat/server/cluster-one.conf -static_data=$HOME/tinode/example-react-js/example-react-js/ &
> exit

By using exit we close the ssh session cleanly, thus the nohup background process we started won’t be attached to terminal anymore. But in case of broke connection, the process would be still attached to terminal, so it’s still receive SIGHUP signal.

Here is the same guy which have the same experience like mine:
https://unix.stackexchange.com/questions/261627/nohup-nice-job-aborted-when-disconnected?rq=1

Let me know if you found my understanding is incorrect. I hope this could help anyone facing the same issue.

Thanks.

[or-else]

Makes sense. Good to know. Maybe you can suggest this is the docs, for example here:
https://github.com/tinode/chat/blob/master/INSTALL.md

Thanks!

[mudphilo]

In centOS you can make this script as a service, follow the below steps

1. create a file name tinode and paste the below code, alter the code to match your environme

#!/bin/sh

SCRIPT="/go/bin/server" # or $GOBIN/server
RUNAS=root # user 
NAME="tinode" # app name

PIDFILE="/var/run/$NAME.pid"
LOGFILE="/var/log/$NAME/daemon.log"
CONFIG="$GOPATH/src/github.com/tinode/chat/server/tinode.conf"

start() {
  # check if service already running, exit if service is running
  if [ -f $PIDFILE ] && [ -s $PIDFILE ] && kill -0 $(cat $PIDFILE); then
    echo 'Service already running' >&2
    return 1
  fi

  # reload environment variables, this is not neccessary
  source /root/.bash_profile

  echo 'Starting service?~@?' >&2
  local CMD="$SCRIPT --data=$CONFIG &> \"$LOGFILE\" & echo \$!"
  su -c "$CMD" $RUNAS > "$PIDFILE"  # runs command and get PID
 # Try with this command line instead of above if not workable
 # su -s /bin/sh $RUNAS -c "$CMD" > "$PIDFILE"

  sleep 2
  PID=$(cat $PIDFILE)
    if pgrep -u $RUNAS -f $NAME > /dev/null
    then
      echo "$NAME is now running, the PID is $PID"
    else
      echo ''
      echo "Error! Could not start $NAME!"
    fi
}

stop() {
  if [ ! -f "$PIDFILE" ] || ! kill -0 $(cat "$PIDFILE"); then
    echo 'Service not running' >&2
    return 1
  fi
  echo 'Stopping service?~@?' >&2
  kill -15 $(cat "$PIDFILE") && rm -f "$PIDFILE"
  echo 'Service stopped' >&2
}

uninstall() {
  echo -n "Are you really sure you want to uninstall this service? That cannot be undone. [yes|No] "
  local SURE
  read SURE
  if [ "$SURE" = "yes" ]; then
    stop
    rm -f "$PIDFILE"
    echo "Notice: log file was not removed: $LOGFILE" >&2
    update-rc.d -f $NAME remove
    rm -fv "$0"
  else
    echo "Abort!"
  fi

}

status() {
    printf "%-50s" "Checking $NAME ..."
    if [ -f $PIDFILE ] && [ -s $PIDFILE ]; then
        PID=$(cat $PIDFILE)
            if [ -z "$(ps axf | grep ${PID} | grep -v grep)" ]; then
                printf "%s\n" "The process appears to be dead but pidfile still exists"
            else
                echo "Running, the PID is $PID"
            fi
    else
        printf "%s\n" "Service not running"
    fi
}

update() {
    printf "%-50s" "building $NAME ..."
    stop
    echo "reload environment"
    source ~/.bash_profile
    echo "building with rethinkdb tags"
    go get -tags rethinkdb github.com/tinode/chat/server && go install -tags rethinkdb github.com/tinode/chat/server
    go get -tags rethinkdb github.com/tinode/chat/tinode-db && go install -tags rethinkdb github.com/tinode/chat/tinode-db
    echo "done updating "
    start
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status
    ;;
  uninstall)
    uninstall
    ;;
  restart)
    stop
    start
    ;;
  update)
    update
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|uninstall|update}"
esac

3. Copy this script in /etc/init.d/
4. Make the script executable sudo chmod +x /etc/init.d/tinode
5. Create log file sudo mkdir /var/log/tinode
6. Ready to run it as service like this
   sudo service tinode start
sudo service tinode stop
sudo service tinode restart
sudo service tinode update

This script can be changed a little bit to work on Linux, Ubuntu and MacOS

[bobaikato]

@mudphilo thanks for this script, it's amazing. Can I get your email, please ?

[mudphilo]

Here is my email address [email protected]