Fix quality issues

[tibuch]

I might have found two issues related to the quality drop described in #30.

• It looks like the training data was sampled from a much smaller region than the validation data.
• I added the rotation augmentation back.

[thorstenwagner]

I've some doubts regarding this change. See my comment :-)

[thorstenwagner]

I don't think that this is correct.

In cryoCARE_extract_train the validation fraction is already defined as:

validation_fraction=(1.0 - config['split'])

See: https://github.com/juglab/cryoCARE_pip/blob/master/cryocare/scripts/cryoCARE_extract_train_data.py#L28

Given a split of 0.9: With your change 90% of the data would be validation data?

[tibuch]

Here

cryoCARE_pip/cryocare/internals/CryoCAREDataModule.py

Line 239 in 847d9bf

extraction_shape_train[tilt_axis_index] = [0, val_cut_off]

it takes then train_data to go from 0 up to the val_cut_off. Maybe I am confusing it :/

[thorstenwagner]

You are totally right :-)

[thorstenwagner]

Nice catch! Please merge it.

[tibuch]

I didn't manage to run it yet. Secretly hoping that @rdrighetto finds the time 😇

[rdrighetto]

OK, I tested it and now I get this error:

2022-10-05 15:28:10.372969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
  0%|          | 0/500 [00:04<?, ?it/s]
Traceback (most recent call last):
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/bin/cryoCARE_extract_train_data.py", line 45, in <module>
    main()
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/bin/cryoCARE_extract_train_data.py", line 27, in main
    dm.setup(config['odd'], config['even'], n_samples_per_tomo=config['num_slices'],
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/lib/python3.8/site-packages/cryocare/internals/CryoCAREDataModule.py", line 194, in setup
    self.train_dataset = CryoCARE_Dataset(tomo_paths_odd=tomo_paths_odd,
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/lib/python3.8/site-packages/cryocare/internals/CryoCAREDataModule.py", line 46, in __init__
    self.compute_mean_std(n_samples=n_normalization_samples)
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/lib/python3.8/site-packages/cryocare/internals/CryoCAREDataModule.py", line 91, in compute_mean_std
    x, _ = self.__getitem__(i)
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/lib/python3.8/site-packages/cryocare/internals/CryoCAREDataModule.py", line 161, in __getitem__
    return self.augment(np.array(even_subvolume)[..., np.newaxis], np.array(odd_subvolume)[..., np.newaxis])
  File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryo-care/cryoCARE_pip/cryocare_11/lib/python3.8/site-packages/cryocare/internals/CryoCAREDataModule.py", line 137, in augment
    x[i] = np.rot90(x[i], k=rot_k[i], axes=self.rot_axes)
ValueError: could not broadcast input array from shape (1,72,72) into shape (72,72,1)

I tried to fix it by myself without success ☹️. What I did notice is the following:

1. The problem occurs because x and y have shape (72, 72, 72, 1) when augment() is called
2. Therefore, when k=rot_k[i] is 1 or 3 in np.rot90 (i.e. a rotation of 90 or 270 degrees) the resulting array will be of shape (1,72,72) which it will try to put in an array whose original shape is (72,72,1) (i.e. x[i])

As I said, I tried to fix this by getting rid of this 4th dimension within augment() and placing it back when returning from that function, but then the code would break somewhere else, so I decided it's better to stop and call the experts 😅

Thanks again!

[tibuch]

Thanks! I pushed a hot-fix.

I promise, if it does not work this time I will setup everything on my end and stop coding in the github IDE!

[rdrighetto]

Thanks @tibuch! With some additional tweaks to the code cryoCARE_extract_train_data.py now runs apparently flawless (see #34).
I'm doing a full run right now to check the quality of results, will report back when done.

[tibuch]

Okay, as promised I checked it out and took it for a functional test-run on my side. Now it should work.

@rdrighetto could you restart with the current version? I would be interested to see if the results look now better.

Thanks!