Skip to content

Commit

Permalink
Merge branch 'master' into feature/spellchecker
Browse files Browse the repository at this point in the history
  • Loading branch information
@sarthakpati
sarthakpati authored Nov 21, 2024
2 parents 5b1c4e2 + 966f84f commit 60eda6c
Show file tree
Hide file tree
Showing 52 changed files with 1,116 additions and 93 deletions.
2 changes: 1 addition & 1 deletion .devcontainer/onCreateCommand.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ pip install wheel
pip install openvino-dev==2023.0.1 # [OPTIONAL] to generate optimized models for inference
pip install mlcube_docker # [OPTIONAL] to deploy GaNDLF models as MLCube-compliant Docker containers
pip install medmnist==2.1.0
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cpu
pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cpu
2 changes: 1 addition & 1 deletion .devcontainer/postCreateCommand.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# if runnning on a GPU machine, install the GPU version of pytorch
if command -v nvidia-smi &> /dev/null
then
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cu121
fi

pip install -e .
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dependencies/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,5 +100,5 @@ runs:
python -m pip install --upgrade pip==24.0
python -m pip install wheel
python -m pip install openvino-dev==2023.0.1 mlcube_docker
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cpu
pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cpu
pip install -e .
4 changes: 2 additions & 2 deletions Dockerfile-CPU
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3-pip libjpeg8-dev zlib1g-dev python3-dev libpython3.9-dev libffi-dev libgl1
RUN python3.9 -m pip install --upgrade pip==24.0
# EXPLICITLY install cpu versions of torch/torchvision (not all versions have +cpu modes on PyPI...)
RUN python3.9 -m pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cpu
RUN python3.9 -m pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cpu
RUN python3.9 -m pip install openvino-dev==2023.0.1 opencv-python-headless mlcube_docker

# Do some dependency installation separately here to make layer caching more efficient
Expand All @@ -32,7 +32,7 @@ CMD run
# See https://github.com/hexops/dockerfile as a best practices guide.
#RUN addgroup --gid 10001 --system nonroot \
# && adduser --uid 10000 --system --ingroup nonroot --home /home/nonroot nonroot
#
#
#USER nonroot

# Prepare the container for possible model embedding later.
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile-CUDA11.8
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3-pip libjpeg8-dev zlib1g-dev python3-dev libpython3.9-dev libffi-dev libgl1
RUN python3.9 -m pip install --upgrade pip==24.0
RUN python3.9 -m pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu118
RUN python3.9 -m pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cu118
RUN python3.9 -m pip install openvino-dev==2023.0.1 opencv-python-headless mlcube_docker

# Do some dependency installation separately here to make layer caching more efficient
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile-CUDA12.1
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3-pip libjpeg8-dev zlib1g-dev python3-dev libpython3.9-dev libffi-dev libgl1
RUN python3.9 -m pip install --upgrade pip==24.0
RUN python3.9 -m pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
RUN python3.9 -m pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cu121
RUN python3.9 -m pip install openvino-dev==2023.0.1 opencv-python-headless mlcube_docker

# Do some dependency installation separately here to make layer caching more efficient
Expand Down
6 changes: 3 additions & 3 deletions Dockerfile-ROCm
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch
FROM rocm/pytorch:rocm6.1_ubuntu20.04_py3.9_pytorch
LABEL github="https://github.com/mlcommons/GaNDLF"
LABEL docs="https://mlcommons.github.io/GaNDLF/"
LABEL version=1.0
Expand All @@ -10,7 +10,7 @@ RUN apt-get update && apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y python3.9 python3-pip libjpeg8-dev zlib1g-dev python3-dev libpython3.9-dev libffi-dev libgl1
RUN python3.9 -m pip install --upgrade pip==24.0
RUN python3.9 -m pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/rocm6.0
RUN python3.9 -m pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/rocm6.0
RUN python3.9 -m pip install --upgrade pip && python3.9 -m pip install openvino-dev==2023.0.1 opencv-python-headless mlcube_docker
RUN apt-get update && apt-get install -y libgl1

Expand All @@ -36,7 +36,7 @@ CMD run
# See https://github.com/hexops/dockerfile as a best practices guide.
#RUN addgroup --gid 10001 --system nonroot \
# && adduser --uid 10000 --system --ingroup nonroot --home /home/nonroot nonroot
#
#
#USER nonroot

# Prepare the container for possible model embedding later.
Expand Down
14 changes: 8 additions & 6 deletions GANDLF/compute/inference_loop.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from skimage.io import imsave
from tqdm import tqdm
from torch.cuda.amp import autocast
import tiffslide as openslide
import openslide
from GANDLF.data import get_testing_loader
from GANDLF.utils import (
best_model_path_end,
Expand Down Expand Up @@ -344,11 +344,13 @@ def inference_loop(
)
cv2.imwrite(file_to_write, heatmaps[key])

os_image_array = os_image.read_region(
(0, 0),
parameters["slide_level"],
(level_width, level_height),
as_array=True,
# this is needed because openslide returns an RGBA image
os_image_array = np.asarray(
os_image.read_region(
(0, 0),
parameters["slide_level"],
(level_width, level_height),
).convert("RGB")
)
blended_image = cv2.addWeighted(
os_image_array,
Expand Down
17 changes: 10 additions & 7 deletions GANDLF/data/inference_dataloader_histopath.py
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import os
from typing import Optional
import numpy as np
import tiffslide
import openslide
from GANDLF.data.patch_miner.opm.utils import get_patch_size_in_microns, tissue_mask
from skimage.transform import resize
from torch.utils.data.dataset import Dataset
Expand Down Expand Up @@ -51,7 +51,7 @@ def __init__(
self._stride_size = get_patch_size_in_microns(wsi_path, self._stride_size)
self._selected_level = selected_level
self._mask_level = mask_level
self._os_image = tiffslide.open_slide(os.path.join(self._wsi_path))
self._os_image = openslide.open_slide(os.path.join(self._wsi_path))
self._points = []
self._basic_preprocessing()

Expand All @@ -61,11 +61,13 @@ def _basic_preprocessing(self):
try:
mask_xdim, mask_ydim = self._os_image.level_dimensions[self._mask_level]
mask = get_tissue_mask(
self._os_image.read_region(
(0, 0), self._mask_level, (mask_xdim, mask_ydim), as_array=True
# this is needed because openslide returns an RGBA image
np.asarray(
self._os_image.read_region(
(0, 0), self._mask_level, (mask_xdim, mask_ydim)
).convert("RGB")
)
)

if self._selected_level != self._mask_level:
mask = resize(mask, (height, width))
mask = (mask > 0).astype(np.ubyte)
Expand Down Expand Up @@ -134,9 +136,10 @@ def __getitem__(self, idx):
(x_loc, y_loc),
self._selected_level,
(self._patch_size[0], self._patch_size[1]),
as_array=True,
)
# as_array=True, openslide-python returns an RGBA PIL image
).convert("RGB")

patch = np.asarray(patch) # convert the image to ndarray
# this is to ensure that channels come at the beginning
patch = patch.transpose([2, 0, 1])
# this is to ensure that we always have a z-stack before applying any torchio transforms
Expand Down
4 changes: 3 additions & 1 deletion GANDLF/data/patch_miner/opm/patch.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ def read_patch(self):
return np.asarray(
self.slide_object.read_region(
(self.coordinates[1], self.coordinates[0]), self.level, self.size
)
).convert(
"RGB"
) # openslide-python returns an RGBA PIL image
)

def copy(self):
Expand Down
6 changes: 3 additions & 3 deletions GANDLF/data/patch_miner/opm/patch_manager.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from tqdm import tqdm
from pathlib import Path
import pandas as pd
import tiffslide
import openslide


class PatchManager:
Expand Down Expand Up @@ -41,7 +41,7 @@ def set_subjectID(self, subjectID):
def set_slide_path(self, filename):
self.img_path = filename
self.img_path = convert_to_tiff(self.img_path, self.output_dir, "img")
self.slide_object = tiffslide.open_slide(self.img_path)
self.slide_object = openslide.open_slide(self.img_path)
self.slide_dims = self.slide_object.dimensions

def set_label_map(self, path):
Expand All @@ -50,7 +50,7 @@ def set_label_map(self, path):
@param path: path to label map.
"""
self.label_map = convert_to_tiff(path, self.output_dir, "mask")
self.label_map_object = tiffslide.open_slide(self.label_map)
self.label_map_object = openslide.open_slide(self.label_map)

assert all(
x == y for x, y in zip(self.label_map_object.dimensions, self.slide_dims)
Expand Down
14 changes: 7 additions & 7 deletions GANDLF/data/patch_miner/opm/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

# import matplotlib.pyplot as plt
import yaml
import tiffslide
import openslide

# RGB Masking (pen) constants
RGB_RED_CHANNEL = 0
Expand Down Expand Up @@ -428,7 +428,7 @@ def generate_initial_mask(slide_path: str, scale: int) -> Tuple[np.ndarray, tupl
Tuple[np.ndarray, tuple]: The valid mask and the real scale.
"""
# Open slide and get properties
slide = tiffslide.open_slide(slide_path)
slide = openslide.open_slide(slide_path)
slide_dims = slide.dimensions

# Call thumbnail for efficiency, calculate scale relative to whole slide
Expand Down Expand Up @@ -505,26 +505,26 @@ def get_patch_size_in_microns(
"Using mpp to calculate patch size for dimension {}".format(i)
)
# only enter if "m" is present in patch size
input_slide = tiffslide.open_slide(input_slide_path)
input_slide = openslide.open_slide(input_slide_path)
metadata = input_slide.properties
if i == 0:
for _property in [
tiffslide.PROPERTY_NAME_MPP_X,
openslide.PROPERTY_NAME_MPP_X,
"tiff.XResolution",
"XResolution",
]:
if _property in metadata:
magnification = metadata[_property]
magnification = float(metadata[_property])
magnification_prev = magnification
break
elif i == 1:
for _property in [
tiffslide.PROPERTY_NAME_MPP_Y,
openslide.PROPERTY_NAME_MPP_Y,
"tiff.YResolution",
"YResolution",
]:
if _property in metadata:
magnification = metadata[_property]
magnification = float(metadata[_property])
break
if magnification == -1:
# if y-axis data is missing, use x-axis data
Expand Down
2 changes: 0 additions & 2 deletions GANDLF/entrypoints/run.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@
import argparse
import ast

# import traceback
from typing import Optional

from deprecated import deprecated
import click

Expand Down
1 change: 0 additions & 1 deletion GANDLF/losses/hybrid.py
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import torch

from .segmentation import MCD_loss, FocalLoss
from .regression import CCE_Generic, CE, CE_Logits

Expand Down
21 changes: 21 additions & 0 deletions GANDLF/losses/hybrid_new.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
from .regression_new import BinaryCrossEntropyLoss, BinaryCrossEntropyWithLogitsLoss
from .segmentation_new import MulticlassDiceLoss, MulticlassFocalLoss
from .loss_interface import AbstractHybridLoss


class DiceCrossEntropyLoss(AbstractHybridLoss):
def _initialize_all_loss_calculators(self):
return [MulticlassDiceLoss(self.params), BinaryCrossEntropyLoss(self.params)]


class DiceCrossEntropyLossLogits(AbstractHybridLoss):
def _initialize_all_loss_calculators(self):
return [
MulticlassDiceLoss(self.params),
BinaryCrossEntropyWithLogitsLoss(self.params),
]


class DiceFocalLoss(AbstractHybridLoss):
def _initialize_all_loss_calculators(self):
return [MulticlassDiceLoss(self.params), MulticlassFocalLoss(self.params)]
Loading

1 comment on commit 60eda6c

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@check-spelling-bot Report

🔴 Please review

See the 📜action log or 📝 job summary for details.

Unrecognized words (14) 
addtional
adopy
autograd
configration
crcrpar
cudagraph
funcions
integartion
kwonly
lrs
realted
Shohei
thirdparty
torchscript
These words are not needed and should be removed tiffslide

Some files were automatically ignored 🙈

These sample patterns would exclude them:

/classification_medmnist_notebook/medmnist/dataset/[^/]+$
[^/]\.png$
^\Q.gitattributes\E$
^\Q.spelling/.spelling/expect.txt\E$
^\Q__init__.py\E$
^\QGANDLF/data/patch_miner/__init__.py\E$
^\QGANDLF/data/patch_miner/opm/__init__.py\E$
^\QGANDLF/grad_clipping/__init__.py\E$
^\QGANDLF/models/seg_modules/__init__.py\E$
^\QGANDLF/privacy/__init__.py\E$
^\Qtesting/__init__.py\E$
^\Qtutorials/classification_medmnist_notebook/tutorial.ipynb\E$
^samples/images_opm/

You should consider excluding directory paths (e.g. (?:^|/)vendor/), filenames (e.g. (?:^|/)yarn\.lock$), or file extensions (e.g. \.gz$)

You should consider adding them to:

.github/actions/spelling/excludes.txt

File matching is via Perl regular expressions.

To check these files, more of their words need to be in the dictionary than not. You can use patterns.txt to exclude portions, add items to the dictionary (e.g. by adding them to allow.txt), or fix typos.

To accept these unrecognized words as correct, update file exclusions, and remove the previously acknowledged and now absent words, you could run the following commands

... in a clone of the [email protected]:mlcommons/GaNDLF.git repository
on the feature/spellchecker branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/mlcommons/GaNDLF/actions/runs/11956224531/attempts/1'
Available 📚 dictionaries could cover words not in the 📘 dictionary
Dictionary Entries Covers Uniquely
cspell:java/src/java-terms.txt 920 2 2

Consider adding them (in .github/workflows/spellchecker.yml) in jobs:/spelling: for uses: check-spelling/check-spelling@main in its with:

      with:
        extra_dictionaries: |
          cspell:java/src/java-terms.txt

To stop checking additional dictionaries, add (in .github/workflows/spellchecker.yml) for uses: check-spelling/check-spelling@main in its with:

check_extra_dictionaries: ''
Errors (4)

See the 📜action log or 📝 job summary for details.

❌ Errors Count
⚠️ binary-file 19
❌ check-file-path 4
⚠️ large-file 2
⚠️ noisy-file 2

See ❌ Event descriptions for more information.

Please sign in to comment.