File permission error when results are saved

[GenevieveBuckley]

When I run full inference with empanada-napari on a Windows machine, I'm running into this error writing the data out. It's a bit frustrating, because it only ever occurs near the very end of things.

PermissionError: [Errno 13] Permission denied: 'D:\\2022Genevieve\\empanada\\20220721-oocyte-mito-predictions\\oocyte_OocyteMitoModel-20220719_napari.zarr\\1_pred/3.16.30'

At first I thought it was because I'd run out of space on the hard drive, so I cleared out a lot more space. But that wasn't it, because I'm still seeing the same behaviour now.

One thing that jumps out at me is that there is a forward slash in the filepath, eg:...zarr\\1_pred/3.16.30. I'd suggest that might not be playing well with Windows, except for the fact that all the other zarr chunks presumably follow the same filename format, and some/most of them seem to write to disk just fine.

Error message (click to expand)

100%|███████████████████████████████████████████████████████████████████████▉| 3489/3494 [01:58<00:00, 77.55it/s]I100%|████████████████████████████████████████████████████████████████████████| 3494/3494 [01:58<00:00, 29.41it/s]
Class 1, axis yz, has 1508 instances
Creating consensus segmentation for class 1...
Total 1 objects 968
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\empanada\zarr_utils.py", line 26, in fill_zarr_mp
    seg2d = array[index].reshape(-1)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\core.py", line 720, in __getitem__
    result = self.get_basic_selection(pure_selection, fields=fields)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\core.py", line 846, in get_basic_selection
    return self._get_basic_selection_nd(selection=selection, out=out,
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\core.py", line 889, in _get_basic_selection_nd
    return self._get_selection(indexer=indexer, out=out, fields=fields)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\core.py", line 1179, in _get_selection
    self._chunk_getitem(chunk_coords, chunk_selection, out, out_selection,
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\core.py", line 1871, in _chunk_getitem
    cdata = self.chunk_store[ckey]
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\storage.py", line 893, in __getitem__
    return self._fromfile(filepath)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\zarr\storage.py", line 867, in _fromfile
    with open(fn, 'rb') as f:
PermissionError: [Errno 13] Permission denied: 'D:\\2022Genevieve\\empanada\\20220721-oocyte-mito-predictions\\oocyte_OocyteMitoModel-20220719_napari.zarr\\1_pred/3.16.30'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\superqt\utils\_qthreading.py", line 616, in reraise
    raise e
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\superqt\utils\_qthreading.py", line 176, in run
    result = self.work()
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\superqt\utils\_qthreading.py", line 443, in work
    output = self._gen.send(input)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\empanada_napari\inference.py", line 158, in tracker_consensus
    fill_volume(consensus_vol, consensus_tracker.instances)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\empanada\inference\patterns.py", line 224, in fill_volume
    zarr_fill_instances(volume, instances, processes)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\site-packages\empanada\zarr_utils.py", line 109, in zarr_fill_instances
    pool.map(fill_zarr_mp, arg_iter)
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\multiprocessing\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\CryoEM\.conda\envs\napari-empanada\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
PermissionError: [Errno 13] Permission denied: 'D:\\2022Genevieve\\empanada\\20220721-oocyte-mito-predictions\\oocyte_OocyteMitoModel-20220719_napari.zarr\\1_pred/3.16.30'

If you suspect this is an IPython 8.1.1 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Additional traceback details (click to expand)

In [8]: %tb
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\interactiveshell.py:1979, in InteractiveShell.showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
   1975 try:
   1976     # Exception classes can customise their traceback - we
   1977     # use this in IPython.parallel for exceptions occurring
   1978     # in the engines. This should return a list of strings.
-> 1979     stb = value._render_traceback_()
   1980 except Exception:

AttributeError: 'PermissionError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

AssertionError                            Traceback (most recent call last)
Input In [8], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('tb', '')

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\interactiveshell.py:2285, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2283     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2284 with self.builtin_trap:
-> 2285     result = fn(*args, **kwargs)
   2286 return result

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\magics\execution.py:505, in ExecutionMagics.tb(self, s)
    503         interactive_tb.set_mode(mode=original_mode)
    504 else:
--> 505     self.shell.showtraceback()

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\interactiveshell.py:1981, in InteractiveShell.showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
   1979     stb = value._render_traceback_()
   1980 except Exception:
-> 1981     stb = self.InteractiveTB.structured_traceback(etype,
   1982                         value, tb, tb_offset=tb_offset)
   1984 self._showtraceback(etype, value, stb)
   1985 if self.call_pdb:
   1986     # drop into debugger

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\ultratb.py:1105, in AutoFormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
   1103 else:
   1104     self.tb = tb
-> 1105 return FormattedTB.structured_traceback(
   1106     self, etype, value, tb, tb_offset, number_of_lines_of_context)

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\ultratb.py:999, in FormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
    996 mode = self.mode
    997 if mode in self.verbose_modes:
    998     # Verbose modes need a full traceback
--> 999     return VerboseTB.structured_traceback(
   1000         self, etype, value, tb, tb_offset, number_of_lines_of_context
   1001     )
   1002 elif mode == 'Minimal':
   1003     return ListTB.get_exception_only(self, etype, value)

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\ultratb.py:871, in VerboseTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
    869 chained_exc_ids = set()
    870 while evalue:
--> 871     formatted_exceptions += self.format_exception_as_a_whole(etype, evalue, etb, lines_of_context,
    872                                                              chained_exceptions_tb_offset)
    873     exception = self.get_parts_of_chained_exception(evalue)
    875     if exception and not id(exception[1]) in chained_exc_ids:

File ~\.conda\envs\napari-empanada\lib\site-packages\IPython\core\ultratb.py:775, in VerboseTB.format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)
    762 def format_exception_as_a_whole(
    763     self,
    764     etype: type,
   (...)
    768     tb_offset: Optional[int],
    769 ):
    770     """Formats the header, traceback and exception message for a single exception.
    771
    772     This may be called multiple times by Python 3 exception chaining
    773     (PEP 3134).
    774     """
--> 775     assert etb is not None
    776     # some locals
    777     orig_etype = etype

AssertionError:

[conradry]

I wasn't able to reproduce this problem on my Windows workstation. Unfortunately, it's also an unhelpfully vague error message.

From a little research, Permission Denied errors can sometimes happen because of installed antivirus software. For example, see:

• PermissionError when writing to larger zarr array zarr-developers/zarr-python#597
• https://gitter.im/zarr-developers/community?at=5db2d6c014d55a3785732666

You'll probably have better luck raising this issue with the Zarr folks at https://github.com/zarr-developers/zarr-python/issues. Sorry I can't be of more help.

[GenevieveBuckley]

It might just be a flaky computer. I had similar permission errors when trying to save files earlier this week (using a completely unrelated piece of software).

Frustrating, but it's looking more like it was just a coincidence that it happened at the same stage of empanada data processing.