diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst new file mode 100644 index 00000000..30bf489d --- /dev/null +++ b/docs/howto_guides/replacing_asdfs.rst @@ -0,0 +1,81 @@ +.. _dkist:howto-guide:replacing-asdfs: + +Replacing Previously Downloaded ASDF Files +========================================== + +The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. +When this happens, you may see a warning message when opening ASDF files locally after updating your Python tools installation. +To remove the warning you will need to update your local ASDF files as described below. + + +Replacing Sample Data +--------------------- + +The sample datasets included with the Python tools will occasionally be updated with a new release of the package. +When this happens you may see the following warning (or something similar) when you try to load sample data: + +.. code-block:: python + + >>> from dkist.data.sample import VBI_AJQWW + >>> from dkist import load_dataset + >>> ds = load_dataset(VBI_AJQWW) + + asdf.exceptions.AsdfPackageVersionWarning: File 'file:///home/runner/.local/share/dkist/VBI_AJQWW/VBI_L1_20231016T184519_AJQWW_metadata.asdf' was created with extension URI 'asdf://astropy.org/core/extensions/core-1.5.0' (from package asdf-astropy==0.5.0), which is not currently installed + + +To update the local files, you can use the utility function in `dkist.data.sample` to re-download them and force it to replace them using the `overwrite` keyword arg. + + +.. code-block:: python + + from dkist.data.sample import download_all_sample_data + + download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA + + +In this specific instance you will also want to update `asdf-astropy` to v0.7.1 or later, but replacing the ASDF files will also solve any other problems caused by outdated sample data. + +Replacing Other Data +-------------------- + +There are two cases in which metadata files for other datasets might have to be replaced. +First, the metadata itself or the internal structure of the files may have been updated but the name kept the same. +Second, the naming convention for the ASDF files might occasionally change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. + +In the case where the Data Center's copy of the metadata file has been updated but not renamed, the normal download process will not automatically download the updated file: + +.. code-block:: python + + from sunpy.net import Fido, attrs as a + import dkist.net + + # Search for some dataset we already have locally + res = Fido.search(a.dkist.Dataset('AGLKO')) # doctest: +REMOTE_DATA + + # Download the asdf file + f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}") # doctest: +REMOTE_DATA + + +To force Fido to download the new file, we can use the overwrite keyword argument again: + +.. code-block:: python + + f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +REMOTE_DATA + + +In the second case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: + +.. code-block:: python + + >>> from dkist import load_dataset + >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') + + WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] + + +When this happens the newer ASDF file is loaded so the old one can safely be ignored. +However, to remove the warning the old file can simply be deleted or moved elsewhere. + +Note that this behaviour is new in dkist v1.10.0. +In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. +Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible.