Mismatch between local and remote Microscope interface

[Baharis]

I recently started working with Instamatic and found an issue with the code structure that appeared after PR #99 . While I wholeheartedly agree with the overall philosophy behind the change – great props to @viljarjf – it confused me somewhat and led to some rather unexpected traceback and hours of confusion.

The problem

Imagine that you are working on a 2-computer configuration, where Instamatic GUI and camera server are run on Computer A, while microscope server is running on computer B. Following the instructions, you set up your installation, in particular your config files. On computer A, in settings.yaml you point towards microscope/fei.yaml, where you specify interface: fei. With that, on initialization your MicroscopeClient will simulate a Microscope object that has exactly the same interface as FEIMicroscope.

Now on Computer B, you similarly create config files and specify interface:fei. Unfortunately, you quickly realize that the FEIMicroscope class does not specify all methods for the method you want to run. For example, FEIMicroscope.stopStage method is missing. But it's not a big deal, is it? Just quickly patch in stopStage on Computer B's FEIMicroscope, re-run the code and...

    File "...\instamatic\src\instamatic\microscope\client.py", line 86, in __getattr__
        raise AttributeError( 
    AttributeError: `MicroscopeClient` object has no attribute `stopStage`.

What happens is particularly interesting: the server-side FEIMicroscope has the method available but since the MicroscopeClient is created based on a local declaration pulled using get_microscope_class, overloaded MicroscopeClient.__getattr__ does not know about – and thus does not call – the method, even though calling it via MicroscopeClient._eval_dct WOULD succeed.

The application

The problem discussed above looks take in the vacuum; indeed, it seems that the solution requires one to simply pull the changes done to the FEIMicroscope from Computer B to Computer A and voila – problem solved. This is true in a typical scenario where both computers are similar and the amount and frequency of changes is not abnormal. However, if Computer B is particularly different from Computer A and B's FEIMicroscope utilizes libraries or other objects that are not available on A (think significantly different operating systems or proprietary software), sometimes it might be unfeasible or straight-up impossible to sync all the changes.

In my particular case, I am trying to set up Instamatic on Windows XP using Instamatic Tecnai Server. Here, none of Computer A's current Microscope classes match the interface of B's TecnaiMicroscope – see an excerpt of the full table below:

The biggest consequence of this issue is that even though methods are properly implemented AND handled, they are not exposed by the MicroscopeClient, leading to some very confusing errors.

The solution

I believe the current design of Instamatic is outstanding: a Server class can be written completely independently of the main program and, as long as it offers a matching interface, it can handle communication just fine, as proved by the Tecnai Server. Unfortunately, with the current design of MicroscopeClient updated in #99 , if I want to use this feature I need to do one of the following:

1. Add, register, and frequently update a completely new microscope interface on my local branch — requires constant re-basing and merging — unacceptable for long-term development;
2. Add, register, and frequently update a completely new microscope interface on main — requires a new microscope to be registered in the central repository for every new server — annoying to maintain but otherwise doable;
3. Change the mechanism MicroscopeClient interface is initialized, e.g. based on Microscope().__dir__ list rather than Microscope.__dict__ — requires no other (i.e. regular) updates to main — according to my study would mess up wrapping in instamatic/microscope/client.py, line 90, but otherwise should work dynamically.

The question

I think a patch suggested in point 3 shouldn't be particularly difficult and it should facilitate using Instamatic across machines. I will be happy to implement it but I might appreciate some help testing it on different setups. I want to ask specifically @viljarjf if you have other concerns regarding this approach. Have tried doing things this way before and failed due to a reason I did not notice? Or have you missed this issue because your Computer A and B have the same configuration?

Links to the files in question:

• FEIMicroscope class
• MicroscopeClient class
• TecnaiMicroscope class

[Baharis]

Here is one more issue with the current mechanism: it fills MicroscopeClient based on given Microscope's __dict__ which is not preserved upon subclassing, meaning if we ever want to subclass individual interfaces (e.g. split FEIMicroscope into TecnaiMicroscope and TitanMicroscope), it will break entirely.

[Baharis]

While investigating the issue and looking for solution I realized that a similar code in camera_client.py has been written already some time ago by @stefsmeets , so I guess my question stands open to anyone who has experience with this issue.

[viljarjf]

Hi @Baharis, I like your idea of retrieving the available methods from an initialized object rater than the class itself.
The approach for the camera class seems like a good solution to me, where the server has a method to send its available methods, which can be used to populate the table of available methods on the client.
As the connection to the server is established before the table is filled, this should work out of the box for the microscope as well.

A function returning the available methods fits nicely as a part of the server class, which is the case for the camera. External implementations like the one in instamatic Tecnai server would not benefit from the implementation being in the base class, as they would not (necessarily) inherit from that.

Maybe there should be some core instamatic package, which contains the abstract microscope- and camera-interface and the MicroscopeController, and things like the GUI and experiments can be seperate packages.
Then, projects like the tecnai server can have the core as a dependency, and therefore ensure that the implementation exposes the necessary functions.
Having a meta-package with all the components (core, GUI ect.) which keeps the name instamatic should then allow users to install the program as normal.
This does, however, introduce the overhead of dependencies for projects like the tecnai server, rather than the independence you praise.

I don't have much experience of programming this kind of server communication. It feels a little strange to override __getattr__ at runtime in this way, but I can't think of any simpler or better ways myself.

[stefsmeets]

The reason for trying to grab it from the class directly is especially to not have to initialize an object on a PC. This can be a limitation if you are running the client on a PC that is not connected to the microscope / camera. So this needs to be kept in mind.

Btw, this whole server / client thing is a complete hack on my side, and I think using some rpc solution would be better these days, for example: https://docs.python.org/3/library/xmlrpc.html or whatever library is popular these days.

[Baharis]

Thank you so much for your thoughts! For the time being in PR #108 I removed the explicit check via __getattr__. This allows both server and client to fall back to the already built-in exception handling mechanism which I confirmed to work great. If having a functional list of methods and attributes is still desired due to other reasons, I will look for some solutions. I'd also prefer not to add new dependencies or potentially-in-inheritable methods... Now I need to prioritize the issues with FEI's "float μm" vs Jeol's "int nm" and type conversion that prohibit me from running anything other than simplest stage control though.

[stefsmeets]

Closed by #108