Adding a DeepProfiler plugin

[callum-jpg]

While meeting with @fefossa we had some thoughts on the GUI inputs to the DeepProfiler plugin and how the run method will work. In essence, the run method will make subprocess calls to DeepProfiler for each image set. We can later build this out to be more DCP friendly, allowing a user to specify an entire image set list (like a load_data_csv or something).

GUI inputs

1. User defines objects from which centres will be used
2. Experiment Name
   1. This will be in the outputs folder under this name
3. Define an image for each channel
   1. Can duplicate (ie RNA being used as ER)
4. Define save directory
   1. Inside this folder will be where all of the setup folders are
5. Define the git repo directory
   1. This is where the deepprofiler command will be run by subprocess
6. Locations
   1. Single cell
   2. Whole image
7. Define 3 levels of metadata
   1. Plate
   2. Well (Can be fixed)
   3. Site
8. Path to model

Run method
6. Run deepprofiler setup to create the output files in the save directory
7. Copy model to checkpoint dir in deepprofiler setup folders
8. For the input objects, get the object centers in XY

1. measurements.get_current_measurement("my_objects", "Location_Center_X")
2. Construct the locations csv file from the centres and save in the deepprofiler/inputs/locations
   1. The savename will be Metadata_Plate/Well_Site_ObjcetName.csv
3. Create CSV that contains image paths (image set list)
   1. This will contain one row since one image set
4. Edit the deepprofiler config
   1. Tell it the name of the:
      1. checkpoint (ie. the model)
      2. batch size
      3. feature_layer
   2. Add the image dimensions (after checking all are equal)
      1. This is required information for crop generation
5. Save the config file into the input/config folder (created deepprofiler setup)
6. subprocess.checkcall(run deep profiler)

[bethac07]

Overall, very nice work!

Is there a way to hook into the underlying DeepProfiler code so you can pass locations, etc directly to functions in order to avoid needing to write the CSV, config, etc out to disk so that you can call with subprocess? That might take more time to set up up front but work more smoothly. If not, we can certainly follow this plan!

[fefossa]

I've been trying to run DeepProfiler on a jupyter notebook to see if we can figure that out!

1. I'm importing 2 functions:

from deepprofiler.dataset.image_dataset import read_dataset
from deepprofiler.learning.profiling import profile

2. Then I imported the config file and basically copied their code to add more information to the config (paths, metadata, experiment_name, which we can use to get the inputs from the user and edit the config).

3. Now, I run dset = read_dataset(config, mode='profile'), and runs fine (the same output you would have running on the command line).

4. Finally, I run profile(config, dset) starts but it fails (error below).

I'll figure out why that happens, because running in the same environment using the command line, the profiling works fine. I'll make this jupyter into a .py file and run in the command line since I'm not sure if jupyter notebooks handle this well!

C:\Users\fossa\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:374: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  warnings.warn(
C:\Users\fossa\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\keras\utils\generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  warnings.warn('Custom mask layers require a config and must override '
2023-05-04 09:43:10,238 - WARNING - From c:\users\fossa\documents\deepprofiler\deepprofiler\learning\profiling.py:36: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

2023-05-04 09:43:10,239 - WARNING - OMP_NUM_THREADS is no longer used by the default Keras config. To configure the number of threads, use tf.config.threading APIs.
Output exceeds the size limit. Open the full output data in a text editor---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
g:\My Drive\Analysis\DeepProfiler\plugin\functions.ipynb Cell 15 in ()
----> 1 profile(params, dset)

File c:\users\fossa\documents\deepprofiler\deepprofiler\learning\profiling.py:107, in profile(config, dset)
    105 def profile(config, dset):
    106     profile = Profile(config, dset)
--> 107     profile.configure()
    108     dset.scan(profile.extract_features, frame="all", check=profile.check)
    109     print("Profiling: done")

File c:\users\fossa\documents\deepprofiler\deepprofiler\learning\profiling.py:36, in Profile.configure(self)
     34 def configure(self):        
     35     # Main session configuration
---> 36     self.profile_crop_generator.start(tf.compat.v1.keras.backend.get_session())
     38     # Create feature extractor
     39     if self.config["profile"]["checkpoint"] != "None":

File c:\users\fossa\documents\deepprofiler\deepprofiler\imaging\cropping.py:302, in SingleImageCropGenerator.start(self, session)
    300 with tf.compat.v1.variable_scope("train_inputs"):
    301     self.config["train"]["model"]["params"]["batch_size"] = self.config["train"]["validation"]["batch_size"]
--> 302     self.build_input_graph()

File c:\users\fossa\documents\deepprofiler\deepprofiler\imaging\cropping.py:112, in CropGenerator.build_input_graph(self, export_masks)
...
-> 3268   raise RuntimeError("tf.placeholder() is not compatible with "
   3269                      "eager execution.")
   3271 return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)

RuntimeError: tf.placeholder() is not compatible with eager execution.

[bethac07]

We can later build this out to be more DCP friendly, allowing a user to specify an entire image set list (like a load_data_csv or something).

Can you explain a bit more the vision of how this would work, given that CellProfiler will still be grouping at the image level? There is an initial implementation of Distributed-DeepProfiler already, it just needs a few hours of debugging to get it "production-ready".

[bethac07]

Here's the code of the worker file for DDP, in case it gives you stuff you can easily pull in the plugin https://github.com/DistributedScience/Distributed-DeepProfiler/blob/main/worker/dp-worker.py

[callum-jpg]

The vision behind this is not particularly fleshed out, I added this comment more of a "we have thought about it and maybe it could be implemented in future, somehow". We felt that initially creating a RunDeepProfiler module that works in a typical CellProfiler fashion first makes sense.

It's cool to see that there is already a distributed DeepProfiler, though. I think it would be nice to have this scalable up to the point where the RunDeepProfiler module could be added in any typical CellPainting experiment, along with the measurement modules, so embeddings are always made (though maybe this is less desirable as new models are trained, but maybe thats when DistDP would be handy).

Though, coming back to the initial concept, I think my comment on providing an image set list doesn't make sense. Maybe since a particular DCP instance already has an image set grouped per site given to it, the RunDeepProfiler module as proposed here would work. All that said, I'm sure there's nuance that I'm missing

[bethac07]

Edit the deepprofiler config
Tell it the name of the:
checkpoint (ie. the model)
batch size
feature_layer
Add the image dimensions (after checking all are equal)
This is required information for crop generation
Save the config file into the input/config folder (created deepprofiler setup)

You might think about making the user just give you a nicely configured file (that you just move/copy if needed), rather than generating on the fly; it's likely not a huge difference either way but just something to consider.

[fefossa]

So here's the first version of RunDeepProfiler. If you could test on a Mac I'd appreciate it; seems like it only works in test mode on Windows (similar to runilastik issue), so hopefully it works on analysis mode on Mac. @callum-jpg @ErinWeisbart @bethac07

The folder with images, pipeline, config and model files: download here

If you want just to download rundeepprofiler.py and paste into your already cloned repo here it's: https://github.com/fefossa/CellProfiler-plugins/blob/run-deepprofiler/active_plugins/rundeepprofiler.py

Instructions DeepProfiler

1 - Create an environment

conda create --name cp-dp python=3.8

2 - Clone CellProfiler-plugins and DeepProfiler

git clone https://github.com/CellProfiler/CellProfiler-plugins.git

git clone https://github.com/broadinstitute/DeepProfiler.git

3 - Activate env

conda activate cp-dp

4 - Install DeepProfiler

cd DeepProfiler /
pip install -e . 

5 - Install CellProfiler and DeepProfiler dependencies

Run next to install cellprofiler:

cd CellProfiler-plugins/
pip install -e . 

Run next dependencies:

pip install numpy==1.23.0
pip install inflect==6.0.0

6 - Run CellProfiler

Call cellprofiler and see if you can find RunDeeProfiler by adding a new module:

cellprofiler

7 - Select the cellprofiler-plugins folder in Preferences > Save

8 - Use the folder with images and files (here) as an example to run the pipeline test_deepprofiler.cppipe
Don't forget to select the config, model, and DeepProfiler directories to your local paths.

[jamila-griffith]

Hi Fefossa,

I have tried to install Deeprofiler as a cellprofiler plugin. I have installed CellProfiler 4.2.6 as shown above but I come across a lot of package incompatibilities. Just wondering if you have an environment yml file that I can use to install DeepProfiler as a plugin for cellprofiler. I guess more details are needed in this.

[fefossa]

Hi @jamila-griffith
Are you on Windows or a Mac?
At this time, I was using CellProfiler 4.2.5 - I will test it with 4.2.6 to see if I find something!

[jamila-griffith]

Hi Fefossa, I am trying to install that in a linux cluster as I have lots of images to process.

[jamila-griffith]

Hi fefossa, just wondering if you had the chance to try this out and have any update?

Thanks in advance

[jamila-griffith]

Hi fefossa,

Its been a while but I am sure you can give some insights. I am struggling to install deep profiler as a plugin of cellprofiler. I wonder if you can share the settings and other details for the version you have used. Does that work with specific version of cellprofiler, I am using 4.2.6

[bethac07]

Working in my hands after the edits I put into the PR review, with a definition of "working" that is "gets as far as getting to the actual execution, then barfs because M1 mac doesn't support an old enough version of tensorflow for this to work, and in ~10 minutes of trying to get the tensorflow and keras versions right I gave up". I think we'll likely need/want a Docker - I asked in Slack if one exists, if the answer is no, let's plan to eventually make one, for now this should work at least for Linux/Windows/non-M1 mac.

It's worth noting that the plugin didn't warn me that anything had gone wrong, it was only that I noticed I didn't have output files and went looking for them, so I then tried running the subprocess command on its own and then was told about the TF errors. We want to probably capture the results. Will add to my review.

CompletedProcess(args=['python', '/Users/bcimini/Desktop/test/CPDP/DeepProfiler/deepprofiler', '--root=/Users/bcimini/Desktop/test/CPDP', '--config', 'profiling.json', '--metadata', 'index_143075.csv', '--exp', 'FeaturesDeepProfiler', 'profile'], returncode=1, stdout='Using TensorFlow backend.\nTraceback (most recent call last):\n  File "/opt/homebrew/Caskroom/miniconda/base/envs/cp-dp/lib/python3.8/runpy.py", line 194, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n  File "/opt/homebrew/Caskroom/miniconda/base/envs/cp-dp/lib/python3.8/runpy.py", line 87, in _run_code\n    exec(code, run_globals)\n  File "/Users/bcimini/Desktop/test/CPDP/DeepProfiler/deepprofiler/__main__.py", line 16, in <module>\n    import deepprofiler.dataset.sampling\n  File "/Users/bcimini/Desktop/test/CPDP/DeepProfiler/deepprofiler/dataset/sampling.py", line 12, in <module>\n    import deepprofiler.imaging.cropping\n  File "/Users/bcimini/Desktop/test/CPDP/DeepProfiler/deepprofiler/imaging/cropping.py", line 10, in <module>\n    import deepprofiler.imaging.augmentations\n  File "/Users/bcimini/Desktop/test/CPDP/DeepProfiler/deepprofiler/imaging/augmentations.py", line 68, in <module>\n    class AugmentationLayer(tf.compat.v1.keras.layers.Layer):\n  File "/opt/homebrew/Caskroom/miniconda/base/envs/cp-dp/lib/python3.8/site-packages/tensorflow/python/util/lazy_loader.py", line 58, in __getattr__\n    module = self._load()\n  File "/opt/homebrew/Caskroom/miniconda/base/envs/cp-dp/lib/python3.8/site-packages/tensorflow/python/util/lazy_loader.py", line 41, in _load\n    module = importlib.import_module(self.__name__)\n  File "/opt/homebrew/Caskroom/miniconda/base/envs/cp-dp/lib/python3.8/importlib/__init__.py", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\nModuleNotFoundError: No module named \'keras.api\'\n')

[fefossa]

Edited based on your changes and it still works fine! Changed the PR as ready for review.

[bethac07]

As discussed in person, as a naive user I wouldn't expect that the features would be in their own files - I would expect they would show up in CellProfiler measurements. We could think about in a later version of this plugin introducing a switch that allows the users to either leave the files as is, OR have CellProfiler pull in the npz files and put them as "measurements". Since there are so many, that approach also has its issues for as long as we expect our backends will have 2K column limits due to SQLite, but again, that's why we give options.

[fefossa]

A few updates - so I've added the option to show the DeepProfiler features on CellProfiler measurements. Hopefully, that should work now!

Also, I had to change the code inside DeepProfiler so it would check if the user is giving a filename or pathname+filename and just add the root path if there's only a filename. This is necessary because otherwise, we would have to create copies of the images as well.

[jamila-griffith]

Hi, I am trying to install this on linux cluster. Dr Jamila Iqbal| Research Fellow Griffith Institute for Drug Discovery Griffith University Nathan Campus, Building N75, Don Young Rd, NATHAN, Queensland, AUS, 4111 Email: ***@***.***>.***@***.***

…

________________________________ From: Fernanda Fossa ***@***.***> Sent: Tuesday, 19 December 2023 9:20 PM To: CellProfiler/CellProfiler-plugins ***@***.***> Cc: Jamila Iqbal ***@***.***>; Mention ***@***.***> Subject: Re: [CellProfiler/CellProfiler-plugins] Adding a DeepProfiler plugin (Discussion #183) Hi @jamila-griffith<https://github.com/jamila-griffith> Are you on Windows or a Mac? At this time, I was using CellProfiler 4.2.5 - I will test it with 4.2.6 to see if I find something! — Reply to this email directly, view it on GitHub<#183 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWG5JZHQTSWRIUQM24XPA4LYKF2A7AVCNFSM6AAAAAAXUSZINSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQOJWGM3TE>. You are receiving this because you were mentioned.Message ID: ***@***.***>