From 7e7fa7487ec179ef0973b2e70ebd57bbbee3ea32 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:23:29 +0000 Subject: [PATCH] Better address the problem --- docs/howto_guides/replacing_asdfs.rst | 79 ++++++++++++--------------- 1 file changed, 35 insertions(+), 44 deletions(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 04b89191..79c8ebe1 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -4,78 +4,69 @@ Replacing Previously Downloaded ASDF Files ========================================== The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. -When this happens, you may see a warning message when opening ASDF files locally after updating your Python tools installation. -To remove the warning you will need to update your local ASDF files as described below. +This can lead to two possible problems. +First, when you download ASDF metadata files from the Data Center those files might be expecting a newer version of the Python tools. +Second, ASDF files you have downloaded previously might become outdated and need to be re-downloaded. -Replacing Sample Data ---------------------- +Re-downloading ASDF files +------------------------- -The sample datasets included with the Python tools will occasionally be updated with a new release of the package. -When this happens you may see the following warning (or something similar) when you try to load sample data: +You should periodically re-download your local ASDF files to keep up to date with changes to the dataset metadata. +To do so, you can use the `overwrite` keyword argument when downloading an ASDF file. +For example, to force a refresh of your local copy of the sample data: .. code-block:: python - >>> from dkist.data.sample import VBI_AJQWW # doctest: +REMOTE_DATA - >>> from dkist import load_dataset - >>> ds = load_dataset(VBI_AJQWW) # doctest: +REMOTE_DATA + from dkist.data.sample import download_all_sample_data - asdf.exceptions.AsdfPackageVersionWarning: File 'file:///home/runner/.local/share/dkist/VBI_AJQWW/VBI_L1_20231016T184519_AJQWW_metadata.asdf' was created with extension URI 'asdf://astropy.org/core/extensions/core-1.5.0' (from package asdf-astropy==0.5.0), which is not currently installed + download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA -To update the local files, you can use the utility function in `dkist.data.sample` to re-download them and force it to replace them using the `overwrite` keyword arg. +`Fido.fetch` also takes the `overwrite` argument. +Occasionally, the naming convention for the ASDF files might also change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. +In this case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: .. code-block:: python - from dkist.data.sample import download_all_sample_data - - download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA - - -In this specific instance you will also want to update `asdf-astropy` to v0.7.1 or later, but replacing the ASDF files will also solve any other problems caused by outdated sample data. - -Replacing Other Data --------------------- + >>> from dkist import load_dataset + >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') # doctest: +SKIP -There are two cases in which metadata files for other datasets might have to be replaced. -First, the metadata itself or the internal structure of the files may have been updated but the name kept the same. -Second, the naming convention for the ASDF files might occasionally change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. + WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] -In the case where the Data Center's copy of the metadata file has been updated but not renamed, the normal download process will not automatically download the updated file: -.. code-block:: python +When this happens the newer ASDF file is loaded so the old one can safely be ignored. +However, to remove the warning the old file can simply be deleted or moved elsewhere. - from sunpy.net import Fido, attrs as a - import dkist.net +Note that this behaviour is new in dkist v1.10.0. +In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. +Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. - # Search for some dataset we already have locally - res = Fido.search(a.dkist.Dataset('AGLKO')) # doctest: +REMOTE_DATA +Understanding version warnings when loading ASDFs +------------------------------------------------- - # Download the asdf file - f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}") # doctest: +REMOTE_DATA +When you load a recently-downloaded ASDF file you may see a warning something like this: +:: -To force Fido to download the new file, we can use the overwrite keyword argument again: + AsdfPackageVersionWarning: File '' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed. -.. code-block:: python - f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +REMOTE_DATA +Of course the extension and package it complains about will vary. +This warning means that an extension needed to properly parse the ASDF file is missing or outdated. +To correct this, you should update your Python tools installation with +.. code-block:: bash -In the second case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: + pip install --upgrade dkist -.. code-block:: python - >>> from dkist import load_dataset - >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') # doctest: +REMOTE_DATA +if you installed using `pip` or - WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] +.. code-block:: bash + conda update dkist -When this happens the newer ASDF file is loaded so the old one can safely be ignored. -However, to remove the warning the old file can simply be deleted or moved elsewhere. -Note that this behaviour is new in dkist v1.10.0. -In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. -Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. +if you used `conda`.