Exceptions and undefined variables are not raised within callback handlers

[jpmens]

Exceptions (e.g. missing variables) are not raised from within callback handlers nor is there any indication that the program is failing.

Install latest version of paho.mqtt

python3 -mvenv v3
source v3/bin/activate
pip install pip --upgrade       # => pip-19.0.3
pip install paho-mqtt		# => paho-mqtt-1.4.0

Run the following short program

#!/usr/bin/env python -B

import paho.mqtt.client as paho

def on_connect(mosq, userdata, flags, rc):
    raise(MoreTrouble)
    mqttc.subscribe("test", 0)

def on_message(mosq, userdata, msg):

    bb = this_is_not_defined
    raise(Hell)

mqttc = paho.Client(None, clean_session=True)
mqttc.on_message = on_message
mqttc.on_connect = on_connect

mqttc.connect("localhost", 1883, 60)

while True:
    try:
        mqttc.loop_forever()
    except KeyboardInterrupt:
        mqttc.disconnect()
        exit(0)
    except:
        raise

What happens

The program connects to the MQTT broker, but neither subscribes nor does it show any sign of doing anything on the console.

Expectations

I'd expect the raise within on_connect() to cause an undefined error, and I would expect the raise and the this_is_not_defined variable to also cause such errors.

NameError: name 'MoreTrouble' is not defined

Platforms

I have tested this on

• Mac OS/X 10.12.6 (Python 3.7.2)
• Linux Debian 9 (Python 2.7.13 and Python 3.5.3)

[tarunw07]

Instead of raising the error, v1.4 logs the error (using _easy_log function).
Here is the snapshot from client.py

with self._in_callback_mutex:
    try:
        self.on_connect(self, self._userdata, flags_dict, result)
    except Exception as err:
        self._easy_log(MQTT_LOG_ERR, 'Caught exception in on_connect: %s', err)

def _easy_log(self, level, fmt, *args):
        if self.on_log is not None:
            buf = fmt % args
            try:
                self.on_log(self, self._userdata, level, buf)
            except Exception:
                # Can't _easy_log this, as we'll recurse until we break
                pass # self._logger will pick this up, so we're fine
        if self._logger is not None:
            level_std = LOGGING_LEVEL[level]
            self._logger.log(level_std, fmt, *args)

You can modify your above code to log error messages using on_log callback. For example,

import paho.mqtt.client as paho

def on_connect(mosq, userdata, flags, rc):
    raise(MoreTrouble)
    mqttc.subscribe("test", 0)

def on_message(mosq, userdata, msg):

    bb = this_is_not_defined
    raise(Hell)

def on_log(mqttc, obj, level, string):
    print(string)

mqttc = paho.Client(None, clean_session=True)
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_log = on_log

mqttc.connect("localhost", 1883, 60)

while True:
    try:
        mqttc.loop_forever()
    except KeyboardInterrupt:
        mqttc.disconnect()
        exit(0)
    except:
        raise

[jpmens]

Thank you, @tarunw07, that works, and I wasn't aware of that. Is this "designed to be so"?

I find it would be a great improvement if the documentation were to state that explicitly.

[tarunw07]

I think it is designed to be so. I agree that it should be mentioned in the documentation. I will try to update the documentation if I get any free time.

[kanflo]

Personally I find this behaviour of silently eating exceptions annoying and not a very good practice. A change would be much appreciated ;)

[BazaJayGee66]

When running the above example, the script simply hangs after logging the exception until I pass a KeyboardInterrupt.

How would I gracefully fail the script when raising an exception within callback handlers?

I was attempting to catch bad users/passwords, and raise an exception within on_connect, so that I could handle these failed attempts, however the best I can do is to stop the loop, with the script still hung.

def on_connect(client, userdata, flags, rc):
  if rc == 5:
    client.loop_stop()
    raise MQTTAuthenticationException

[tapionx]

Errors should never pass silently.
Unless explicitly silenced.

https://www.python.org/dev/peps/pep-0020/

This issue is really annoying, exceptions should be raised properly and not be silenced.

[DanielO]

Silently eating exceptions makes debugging and development much harder.
Having it as an option (that defaults to eating) would be reasonable IMHO.

[ukrutt]

Seconding. I'm currently getting an log message, paho.mqtt.client: ERROR: Caught exception in on_message: '<string that I created>'. I did in fact construct this string but I don't know how it raises an exception. A proper crash and traceback would give me this.

[brd]

Argh, I have wasted hours and I am sure many others have too. Please consider changing this.

[trebolcinco]

Wasted hours this week, really miserable. Agree that they should NOT be swallowed, make an option to swallow but default off.

[schneeemensch]

We had issues with this in the past as well. We needed to implement new communication channels to get around this.

[mxmaxime]

Hello, that seems to be fixed right now (thanks a ton!), but how do we handle exceptions? For example, If I have an Exception raised in a callback (function added via message_callback_add for ex), I would like to catch it.

[hardillb]

This was a major change of default behaviour and as such probably should not have happened on point release, it probably should have been saved for 1.6 not a change from 1.5.0 to 1.5.1. While I understand why the change has been made, this is as likely to break existing code as it is to solve peoples frustrations with not knowing what happened.

@mxmaxime It looks like you would have to put a try/expect block as the top level construct in your callback e.g.

def some_callback:
  try:
    ...
  expect err:
    ...

[fleutot]

@hardillb I still have an issue with this: the thread dies with an exception, but I cannot catch it in the caller thread. That's a problem because my application has multiple threads, and I want the whole thing to die in case of error in my callback.

I've solved that in other threaded modules by saving the exception in a self.exc and returning, then throwing that exception upon the thread dying, in the caller thread:

class MyClassError(Exception):
    def __init__(self, message):
        self.message = f"{message}"
        super().__init__(self.message)


class MyClass()::
    # other stuff...
    def rx(self):
        try:
            while True:
                do_your_thing()
        except Exception as e:
            self.exc = MyClassError(f"An unexpected error occurred: {e}")
            return

    def run(self):
        self.rx_thread.start()
        self.rx_thread.join()
        if self.exc is not None:
            raise self.exc

Since run runs in the caller thread, the exception is risen there.

But in the paho case, I don't know where to raise it in order to be caught in the main thread?