Create flowkit_common module for definitions shared across FlowKit components

[maxalbert]

There is a frequent exchange of messages in the communication between the flowmachine server, flowapi and flowclient. In order to make validation and testing easier and the communication more understandable (esp. during debugging) it is useful for these messages to have a well-defined structure. This message structure needs to be known by the two FlowKit components involved in the message exchange (e.g. flowapi and flowmachine for messages exchanged via zmq). If we want to avoid making one of the modules a dependency of the other it seems sensible to create a (small) module flowkit_common which contains such shared definitions.

For example, the class ZMQReply currently defined in flowmachine/core/server/zmq_helpers.py (as well as in flowapi and flowclient) could live there.

[greenape]

Other obvious thing that would go in this is the JWT generation.

Actually after pondering that a bit more, the test token stuff has a dependency on flowmachine, because it uses the query schemas for producing all access tokens, which would introduce a somewhat hidden interdependence, which is exactly what we want to avoid here.

[greenape]

What I can't quite see the best solution to here is building docker images with a common module tho - can't include stuff from a parallel directory using COPY, obviously want to build using the current commit. So, would we need to wrap up any docker build in a build script?

Basically four options:

1. A common base image (blocked right now because flowauth is on a different python version)
2. A common base image used only in a copy-from instruction (tenable)
3. A build script that copies in the missing files (tenable)
4. Docker images must be built from parent directory (probably a bad idea because it will massively bloat the build context)

[greenape]

Presumably we'd also need to publish the common module on pypi, because Flowmachine is.

[maxalbert]

@greenape Do you feel this is still a valid issue, now that the token generation has been moved into its own module?

The only other thing that is currently shared is the ZMQReply class, whose definition lives in three different components (in flowmachine/flowmachine/core/server/zmq_helpers.py, flowapi/tests/unit/zmq_helpers.py, flowclient/tests/unit/zmq_helpers.py). But I'm happy to sustain this little bit of duplication for the time being because the definition is so small and unlikely to change.

So I'd be happy to close this issue for now and re-open it if we find other pieces of code that should be shared between components.

If in the future we do want a properly shared common module which needs to be included in the docker builds, I think I'd personally vote for your option 3 because it's kinda nice to have build scripts (or equivalent makefile targets) anyway. Partly because it makes it more obvious how to build docker image locally, and partly because it would probably allow us to simplify the CircleCI config, which currently contains a bunch of custom build instructions.

[greenape]

Duped bits of code:

• get_secret_or_env_var
• load_private/public_key (once Public key linkage between FlowAuth and FlowAPI #864) is merged
• generate_token
• ZMQReply

I'm inclined to think that this, in combination with #818 strongly suggests we need some build scripts, and justifies perhaps two common modules.

Worth noting that there is a change coming to Docker that will allow per-Dockerfile .dockerignores, which would allow us to build everything from the top directory and allow us to include common modules.

[greenape]

Closing this - get_secret_or_env_var was extracted to a whole other package independent from FlowKit, and the public key and token stuff is handled by symlinking to the flowkit-jwt-generator module.