Custom Serialization in Python Target

[Depetrol]

This PR enables a new feature for using arbitrary python package names that follows a specific API as the connection serializer in Python target federated execution.

Syntax:

<source_port_reference> -> <destination_port_reference> after <time_value> serializer "<serializer_name>"

This feature is backwards compatible. If <serializer_name> is any of [NATIVE, ROS2, PROTO], it will use these serialization methods instead of finding a python package.

The custom Python package would have to export a Serializer class that has the following API:

class Serializer():
    def serialize(self, obj)->bytes:
        return <bytes_object>
    def deserialize(self, message:bytes):
        return <deserialized_object>

The Serializer class will be initialized at the first send and first receive on both ends of the connection. It is persisted in each networksender and networkreceiver as a state variable. Python objects are serialized with serialize(), sent over the socket and deserialized with deserialize().

This feature allows for user-defined custom serialization for any strategy. For example, the user could define a serialization method that enables fast inter-process communication with shared memory. Potentially ROS2 and PROTO serialization could be implemented with custom python packages too. It also supports different serialization methods for different connections.

[Depetrol]

CI is failing because of #2376

[edwardalee]

This looks like a great start! I've made some suggestions, however.

[cmnrd]

I still cannot say that I like installing something into the user's system as a side effect of the preamble in a test file. I also think that requiring the serializer to be a standalone Python package that is installed separately is not a practical design from a usability perspective. However, I have made those points before and will not continue arguing for them. I cannot give this my approval, but if someone else does, I will not block it.

Dismissing my own review.

[Depetrol]

I agree that installing something into the user's system as a side effect of the preamble in a test file is not a perfect solution, but it doesn't require developers to install it manually in order to run the test, and makes the test self-contained.

However I would argue that installing a standalone Python package is more user friendly compared with using relative imports. There are several reasons:

• There are not many ways to implement a serializer, and most likely the user will use a written custom serializer provided by others(most likely perhaps by us). In this case, installing it as a custom python package is the easiest way for the package to be distributed.
• For a generic serializer, it will likely be used multiple times in multiple locations. It would be a hassle for the users to import it every time that it is used.
• If the serializer has to be imported every time that it is used, it would be almost equivalent to provide a "transparent" serializer that assumes the object being sent over the connection is a bytes object, and send the bytes object as-is. The user could do their serialization before passing the object to the "transparent" serializer. For example, out_port.set(custom_serialize_func(<object>)).
• If the user references the serializer using relative import without installing it in the system environment, this will break because the generated code is in a different location as the .lf code where it is written. The user would also have to specify a files parameter every time the serializer is used to copy the serializer into the generated directory.

[edwardalee]

Maybe you haven't pushed the changes that resolve the issues?

[lhstrh]

@edwardalee sorry I re-requested a review but I hadn't noticed that you just provided feedback already

[edwardalee]

This looks good to me. It depends on this PR