From 57a140781acfe70417103a0a37d114ad1df6441f Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 11:31:08 +0000 Subject: [PATCH 01/26] How-to guide on replacing asdfs --- docs/howto_guides/replacing_asdfs.rst | 81 +++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 docs/howto_guides/replacing_asdfs.rst diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst new file mode 100644 index 00000000..30bf489d --- /dev/null +++ b/docs/howto_guides/replacing_asdfs.rst @@ -0,0 +1,81 @@ +.. _dkist:howto-guide:replacing-asdfs: + +Replacing Previously Downloaded ASDF Files +========================================== + +The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. +When this happens, you may see a warning message when opening ASDF files locally after updating your Python tools installation. +To remove the warning you will need to update your local ASDF files as described below. + + +Replacing Sample Data +--------------------- + +The sample datasets included with the Python tools will occasionally be updated with a new release of the package. +When this happens you may see the following warning (or something similar) when you try to load sample data: + +.. code-block:: python + + >>> from dkist.data.sample import VBI_AJQWW + >>> from dkist import load_dataset + >>> ds = load_dataset(VBI_AJQWW) + + asdf.exceptions.AsdfPackageVersionWarning: File 'file:///home/runner/.local/share/dkist/VBI_AJQWW/VBI_L1_20231016T184519_AJQWW_metadata.asdf' was created with extension URI 'asdf://astropy.org/core/extensions/core-1.5.0' (from package asdf-astropy==0.5.0), which is not currently installed + + +To update the local files, you can use the utility function in `dkist.data.sample` to re-download them and force it to replace them using the `overwrite` keyword arg. + + +.. code-block:: python + + from dkist.data.sample import download_all_sample_data + + download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA + + +In this specific instance you will also want to update `asdf-astropy` to v0.7.1 or later, but replacing the ASDF files will also solve any other problems caused by outdated sample data. + +Replacing Other Data +-------------------- + +There are two cases in which metadata files for other datasets might have to be replaced. +First, the metadata itself or the internal structure of the files may have been updated but the name kept the same. +Second, the naming convention for the ASDF files might occasionally change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. + +In the case where the Data Center's copy of the metadata file has been updated but not renamed, the normal download process will not automatically download the updated file: + +.. code-block:: python + + from sunpy.net import Fido, attrs as a + import dkist.net + + # Search for some dataset we already have locally + res = Fido.search(a.dkist.Dataset('AGLKO')) # doctest: +REMOTE_DATA + + # Download the asdf file + f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}") # doctest: +REMOTE_DATA + + +To force Fido to download the new file, we can use the overwrite keyword argument again: + +.. code-block:: python + + f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +REMOTE_DATA + + +In the second case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: + +.. code-block:: python + + >>> from dkist import load_dataset + >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') + + WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] + + +When this happens the newer ASDF file is loaded so the old one can safely be ignored. +However, to remove the warning the old file can simply be deleted or moved elsewhere. + +Note that this behaviour is new in dkist v1.10.0. +In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. +Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. From 6918cb3afede318c2aab39253150d9dd905c7164 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 11:35:21 +0000 Subject: [PATCH 02/26] Changelog --- changelog/531.docs.rst | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog/531.docs.rst diff --git a/changelog/531.docs.rst b/changelog/531.docs.rst new file mode 100644 index 00000000..3650b0cc --- /dev/null +++ b/changelog/531.docs.rst @@ -0,0 +1 @@ +Add a how-to guide explaining how to replace outdated dataset metadata files. From e5491c709296b14a61e40467f3a918d7e0c468ab Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 12:01:37 +0000 Subject: [PATCH 03/26] Bad changelog --- changelog/{531.docs.rst => 531.doc.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changelog/{531.docs.rst => 531.doc.rst} (100%) diff --git a/changelog/531.docs.rst b/changelog/531.doc.rst similarity index 100% rename from changelog/531.docs.rst rename to changelog/531.doc.rst From e0765966a062188ef60763623f8b69a9432350b4 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 12:09:49 +0000 Subject: [PATCH 04/26] Add another remote data tag --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 30bf489d..b1764f8e 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -16,7 +16,7 @@ When this happens you may see the following warning (or something similar) when .. code-block:: python - >>> from dkist.data.sample import VBI_AJQWW + >>> from dkist.data.sample import VBI_AJQWW # doctest: +REMOTE_DATA >>> from dkist import load_dataset >>> ds = load_dataset(VBI_AJQWW) From 5f04fee42458e78450722f93576040e59df6d03f Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 14:23:43 +0000 Subject: [PATCH 05/26] Add another remote data tag --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index b1764f8e..4e275b28 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -68,7 +68,7 @@ In the second case, if the metadata file has been renamed then you may see a war .. code-block:: python >>> from dkist import load_dataset - >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') + >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') # doctest: +REMOTE_DATA WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] From 5e97c79b03cea89ffbff6b1ca59ae21a9948c6a2 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 14:25:10 +0000 Subject: [PATCH 06/26] Even more remote data --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 4e275b28..04b89191 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -18,7 +18,7 @@ When this happens you may see the following warning (or something similar) when >>> from dkist.data.sample import VBI_AJQWW # doctest: +REMOTE_DATA >>> from dkist import load_dataset - >>> ds = load_dataset(VBI_AJQWW) + >>> ds = load_dataset(VBI_AJQWW) # doctest: +REMOTE_DATA asdf.exceptions.AsdfPackageVersionWarning: File 'file:///home/runner/.local/share/dkist/VBI_AJQWW/VBI_L1_20231016T184519_AJQWW_metadata.asdf' was created with extension URI 'asdf://astropy.org/core/extensions/core-1.5.0' (from package asdf-astropy==0.5.0), which is not currently installed From ee1084dac3bd3c3273f033a5734f5592040db6d3 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 14:33:59 +0000 Subject: [PATCH 07/26] Need to include the guide in the doc tree --- docs/howto_guides/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/howto_guides/index.rst b/docs/howto_guides/index.rst index 4e288539..b2e2aa11 100644 --- a/docs/howto_guides/index.rst +++ b/docs/howto_guides/index.rst @@ -11,3 +11,4 @@ If you're starting fresh you might want to check out the :ref:`dkist:tutorial:in :maxdepth: 1 sample_data + replacing_asdfs From 7e7fa7487ec179ef0973b2e70ebd57bbbee3ea32 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:23:29 +0000 Subject: [PATCH 08/26] Better address the problem --- docs/howto_guides/replacing_asdfs.rst | 79 ++++++++++++--------------- 1 file changed, 35 insertions(+), 44 deletions(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 04b89191..79c8ebe1 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -4,78 +4,69 @@ Replacing Previously Downloaded ASDF Files ========================================== The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. -When this happens, you may see a warning message when opening ASDF files locally after updating your Python tools installation. -To remove the warning you will need to update your local ASDF files as described below. +This can lead to two possible problems. +First, when you download ASDF metadata files from the Data Center those files might be expecting a newer version of the Python tools. +Second, ASDF files you have downloaded previously might become outdated and need to be re-downloaded. -Replacing Sample Data ---------------------- +Re-downloading ASDF files +------------------------- -The sample datasets included with the Python tools will occasionally be updated with a new release of the package. -When this happens you may see the following warning (or something similar) when you try to load sample data: +You should periodically re-download your local ASDF files to keep up to date with changes to the dataset metadata. +To do so, you can use the `overwrite` keyword argument when downloading an ASDF file. +For example, to force a refresh of your local copy of the sample data: .. code-block:: python - >>> from dkist.data.sample import VBI_AJQWW # doctest: +REMOTE_DATA - >>> from dkist import load_dataset - >>> ds = load_dataset(VBI_AJQWW) # doctest: +REMOTE_DATA + from dkist.data.sample import download_all_sample_data - asdf.exceptions.AsdfPackageVersionWarning: File 'file:///home/runner/.local/share/dkist/VBI_AJQWW/VBI_L1_20231016T184519_AJQWW_metadata.asdf' was created with extension URI 'asdf://astropy.org/core/extensions/core-1.5.0' (from package asdf-astropy==0.5.0), which is not currently installed + download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA -To update the local files, you can use the utility function in `dkist.data.sample` to re-download them and force it to replace them using the `overwrite` keyword arg. +`Fido.fetch` also takes the `overwrite` argument. +Occasionally, the naming convention for the ASDF files might also change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. +In this case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: .. code-block:: python - from dkist.data.sample import download_all_sample_data - - download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA - - -In this specific instance you will also want to update `asdf-astropy` to v0.7.1 or later, but replacing the ASDF files will also solve any other problems caused by outdated sample data. - -Replacing Other Data --------------------- + >>> from dkist import load_dataset + >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') # doctest: +SKIP -There are two cases in which metadata files for other datasets might have to be replaced. -First, the metadata itself or the internal structure of the files may have been updated but the name kept the same. -Second, the naming convention for the ASDF files might occasionally change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. + WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] -In the case where the Data Center's copy of the metadata file has been updated but not renamed, the normal download process will not automatically download the updated file: -.. code-block:: python +When this happens the newer ASDF file is loaded so the old one can safely be ignored. +However, to remove the warning the old file can simply be deleted or moved elsewhere. - from sunpy.net import Fido, attrs as a - import dkist.net +Note that this behaviour is new in dkist v1.10.0. +In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. +Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. - # Search for some dataset we already have locally - res = Fido.search(a.dkist.Dataset('AGLKO')) # doctest: +REMOTE_DATA +Understanding version warnings when loading ASDFs +------------------------------------------------- - # Download the asdf file - f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}") # doctest: +REMOTE_DATA +When you load a recently-downloaded ASDF file you may see a warning something like this: +:: -To force Fido to download the new file, we can use the overwrite keyword argument again: + AsdfPackageVersionWarning: File '' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed. -.. code-block:: python - f = Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +REMOTE_DATA +Of course the extension and package it complains about will vary. +This warning means that an extension needed to properly parse the ASDF file is missing or outdated. +To correct this, you should update your Python tools installation with +.. code-block:: bash -In the second case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: + pip install --upgrade dkist -.. code-block:: python - >>> from dkist import load_dataset - >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') # doctest: +REMOTE_DATA +if you installed using `pip` or - WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader] +.. code-block:: bash + conda update dkist -When this happens the newer ASDF file is loaded so the old one can safely be ignored. -However, to remove the warning the old file can simply be deleted or moved elsewhere. -Note that this behaviour is new in dkist v1.10.0. -In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. -Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. +if you used `conda`. From d0c4f8da638b32cbc066235c2d05d5fbaf1c6ee3 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:40:34 +0000 Subject: [PATCH 09/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 79c8ebe1..609cc6e9 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -1,6 +1,6 @@ .. _dkist:howto-guide:replacing-asdfs: -Replacing Previously Downloaded ASDF Files +Update Previously Downloaded ASDF Files ========================================== The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. From 320322f81fd08be3e6df9dfdbd72fa5c13f99169 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:40:43 +0000 Subject: [PATCH 10/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 609cc6e9..644f4d35 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -53,7 +53,7 @@ When you load a recently-downloaded ASDF file you may see a warning something li AsdfPackageVersionWarning: File '' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed. -Of course the extension and package it complains about will vary. +The extension name and package it references will vary. This warning means that an extension needed to properly parse the ASDF file is missing or outdated. To correct this, you should update your Python tools installation with From 6f54cb26be3cbc443af75c4671146c4dc63b3c66 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:40:58 +0000 Subject: [PATCH 11/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 644f4d35..3822f090 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -54,7 +54,7 @@ When you load a recently-downloaded ASDF file you may see a warning something li The extension name and package it references will vary. -This warning means that an extension needed to properly parse the ASDF file is missing or outdated. +This warning means that the `dkist` package or one of it's dependencies is out of date. To correct this, you should update your Python tools installation with .. code-block:: bash From d28defa48ada9697a9178b21a0b9990883030e6c Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:41:11 +0000 Subject: [PATCH 12/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 3822f090..daffcaef 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -55,7 +55,7 @@ When you load a recently-downloaded ASDF file you may see a warning something li The extension name and package it references will vary. This warning means that the `dkist` package or one of it's dependencies is out of date. -To correct this, you should update your Python tools installation with +To correct this, you should update your Python tools installation (and it's dependencies) with .. code-block:: bash From 3e93e006b2bbf2a38c6aed7acc355689ee3dec42 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 15:49:11 +0000 Subject: [PATCH 13/26] Move warnings discussion to its own file and rephrase duplicate ASDF warning --- docs/howto_guides/asdf_warnings.rst | 29 ++++++++++++++ docs/howto_guides/index.rst | 1 + docs/howto_guides/replacing_asdfs.rst | 54 ++++++++------------------- 3 files changed, 46 insertions(+), 38 deletions(-) create mode 100644 docs/howto_guides/asdf_warnings.rst diff --git a/docs/howto_guides/asdf_warnings.rst b/docs/howto_guides/asdf_warnings.rst new file mode 100644 index 00000000..899f409a --- /dev/null +++ b/docs/howto_guides/asdf_warnings.rst @@ -0,0 +1,29 @@ +.. _dkist:howto-guide:asdf-warnings + +Understanding version warnings when loading ASDFs +================================================= + +When you load a recently-downloaded ASDF file you may see a warning something like this: + +:: + + AsdfPackageVersionWarning: File '' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed. + + +Of course the extension and package it complains about will vary. +This warning means that an extension needed to properly parse the ASDF file is missing or outdated. +To correct this, you should update your Python tools installation with + +.. code-block:: bash + + pip install --upgrade dkist + + +if you installed using `pip` or + +.. code-block:: bash + + conda update dkist + + +if you used `conda`. diff --git a/docs/howto_guides/index.rst b/docs/howto_guides/index.rst index b2e2aa11..1d6ace5f 100644 --- a/docs/howto_guides/index.rst +++ b/docs/howto_guides/index.rst @@ -12,3 +12,4 @@ If you're starting fresh you might want to check out the :ref:`dkist:tutorial:in sample_data replacing_asdfs + asdf_warnings diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 79c8ebe1..89231123 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -1,7 +1,7 @@ .. _dkist:howto-guide:replacing-asdfs: -Replacing Previously Downloaded ASDF Files -========================================== +Re-downloading ASDF files +========================= The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. This can lead to two possible problems. @@ -9,12 +9,20 @@ First, when you download ASDF metadata files from the Data Center those files mi Second, ASDF files you have downloaded previously might become outdated and need to be re-downloaded. -Re-downloading ASDF files -------------------------- - You should periodically re-download your local ASDF files to keep up to date with changes to the dataset metadata. To do so, you can use the `overwrite` keyword argument when downloading an ASDF file. -For example, to force a refresh of your local copy of the sample data: + +For example: + +.. code-block:: python + + Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP + + +where `res` is the result of a `Fido` search for some dataset you already have saved locally. + +You might also need to force a refresh of the sample data which is included with the Python tools. +Again, we can use the `overwrite` keyword for this. .. code-block:: python @@ -23,10 +31,8 @@ For example, to force a refresh of your local copy of the sample data: download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA -`Fido.fetch` also takes the `overwrite` argument. - -Occasionally, the naming convention for the ASDF files might also change, meaning that the usual checks to stop you downloading a dataset you already have locally will fail and you will end up with two (likely identical) metadata files. -In this case, if the metadata file has been renamed then you may see a warning like this when you load the dataset: +Occasionally, the naming convention for the ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two (likely identical) metadata files. +If you re-download a dataset that you already had locally before the rename, you may see a warning like this when you load the dataset: .. code-block:: python @@ -42,31 +48,3 @@ However, to remove the warning the old file can simply be deleted or moved elsew Note that this behaviour is new in dkist v1.10.0. In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. - -Understanding version warnings when loading ASDFs -------------------------------------------------- - -When you load a recently-downloaded ASDF file you may see a warning something like this: - -:: - - AsdfPackageVersionWarning: File '' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed. - - -Of course the extension and package it complains about will vary. -This warning means that an extension needed to properly parse the ASDF file is missing or outdated. -To correct this, you should update your Python tools installation with - -.. code-block:: bash - - pip install --upgrade dkist - - -if you installed using `pip` or - -.. code-block:: bash - - conda update dkist - - -if you used `conda`. From 76c983655ea703955d452a690cb545d1ca896e6d Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:00:17 +0000 Subject: [PATCH 14/26] @Cadair's suggestions --- docs/howto_guides/asdf_warnings.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/howto_guides/asdf_warnings.rst b/docs/howto_guides/asdf_warnings.rst index 3dc8bdcb..a85131f3 100644 --- a/docs/howto_guides/asdf_warnings.rst +++ b/docs/howto_guides/asdf_warnings.rst @@ -10,9 +10,9 @@ When you load a recently-downloaded ASDF file you may see a warning something li AsdfPackageVersionWarning: File '' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed. -Of course the extension and package it complains about will vary. -This warning means that an extension needed to properly parse the ASDF file is missing or outdated. -To correct this, you should update your Python tools installation with +The extension and package it references will vary. +This warning means that the `dkist` package or one of its dependencies is out of date. +To correct this, you should update your Python tools installation (and its dependencies) with .. code-block:: bash From 8d46bb9fb3d9710e789805d059417905ef3337fe Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:02:15 +0000 Subject: [PATCH 15/26] Update docs/howto_guides/asdf_warnings.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/asdf_warnings.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/asdf_warnings.rst b/docs/howto_guides/asdf_warnings.rst index a85131f3..ba3d9b54 100644 --- a/docs/howto_guides/asdf_warnings.rst +++ b/docs/howto_guides/asdf_warnings.rst @@ -1,7 +1,7 @@ .. _dkist:howto-guide:asdf-warnings Understand version warnings when loading ASDFs -================================================= +============================================== When you load a recently-downloaded ASDF file you may see a warning something like this: From c5bb1673d92cb4dba62b7828fad0f77f1ea5f0a5 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:02:41 +0000 Subject: [PATCH 16/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 81a455eb..62554e85 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -16,7 +16,7 @@ For example: .. code-block:: python - Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP + Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP where `res` is the result of a `Fido` search for some dataset you already have saved locally. From 5b03110213c4534587b232e7efa520c03a29e1a7 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:02:54 +0000 Subject: [PATCH 17/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 62554e85..9986a0e99 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -28,7 +28,7 @@ Again, we can use the `overwrite` keyword for this. from dkist.data.sample import download_all_sample_data - download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA + download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA Occasionally, the naming convention for the ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two (likely identical) metadata files. From 9dd9fb9b0ce2fe7f9888e4a411386e61bb0c8669 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:03:09 +0000 Subject: [PATCH 18/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 9986a0e99..fddb0b07 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -19,7 +19,7 @@ For example: Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP -where `res` is the result of a `Fido` search for some dataset you already have saved locally. +where ``res`` is the result of a ``Fido`` search for some dataset you already have saved locally. You might also need to force a refresh of the sample data which is included with the Python tools. Again, we can use the `overwrite` keyword for this. From 7e84dee2223924ad6c37a808cbf2fba4c4897188 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:03:25 +0000 Subject: [PATCH 19/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index fddb0b07..8f402ae1 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -22,7 +22,7 @@ For example: where ``res`` is the result of a ``Fido`` search for some dataset you already have saved locally. You might also need to force a refresh of the sample data which is included with the Python tools. -Again, we can use the `overwrite` keyword for this. +Again, we can use the ``overwrite`` keyword for this. .. code-block:: python From 5cccefaca837bd5a71f5806faf5e7d3a2f6819f4 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:04:00 +0000 Subject: [PATCH 20/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 8f402ae1..c3a7e695 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -32,7 +32,7 @@ Again, we can use the ``overwrite`` keyword for this. Occasionally, the naming convention for the ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two (likely identical) metadata files. -If you re-download a dataset that you already had locally before the rename, you may see a warning like this when you load the dataset: +If you try and load a dataset where a file with an old name is present you will see a warning similar to this: .. code-block:: python From f8c94925b95723584e5c652c5f2ffb6c054b7b6f Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:04:20 +0000 Subject: [PATCH 21/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index c3a7e695..4f4cbfef 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -31,7 +31,7 @@ Again, we can use the ``overwrite`` keyword for this. download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA -Occasionally, the naming convention for the ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two (likely identical) metadata files. +In the past, the naming convention for the metadata ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two separate metadata files. If you try and load a dataset where a file with an old name is present you will see a warning similar to this: .. code-block:: python From 1015e4d6bf1de78978da4dd3cf91c64e1a7dbd2b Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:04:34 +0000 Subject: [PATCH 22/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 4f4cbfef..d7ab103a 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -45,6 +45,6 @@ If you try and load a dataset where a file with an old name is present you will When this happens the newer ASDF file is loaded so the old one can safely be ignored. However, to remove the warning the old file can simply be deleted or moved elsewhere. -Note that this behaviour is new in dkist v1.10.0. +Note that this warning was added in dkist version 1.10.0. In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. From 7c4967c8cb338667495bf8d57f6a185c0345a018 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:05:25 +0000 Subject: [PATCH 23/26] Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford --- docs/howto_guides/replacing_asdfs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index d7ab103a..71cace49 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -47,4 +47,4 @@ However, to remove the warning the old file can simply be deleted or moved elsew Note that this warning was added in dkist version 1.10.0. In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems. -Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.1 or later if possible. +Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.0 or later. From 2582b1b0b523eca17b84998f5d77def0d7cb73f3 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:14:17 +0000 Subject: [PATCH 24/26] Re- more words --- docs/howto_guides/replacing_asdfs.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index 81a455eb..7dab5d0c 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -4,10 +4,7 @@ Update Previously Downloaded ASDF Files ======================================= The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata. -This can lead to two possible problems. -First, when you download ASDF metadata files from the Data Center those files might be expecting a newer version of the Python tools. -Second, ASDF files you have downloaded previously might become outdated and need to be re-downloaded. - +This means that ASDF files you have downloaded previously might become outdated and need to be re-downloaded. You should periodically re-download your local ASDF files to keep up to date with changes to the dataset metadata. To do so, you can use the `overwrite` keyword argument when downloading an ASDF file. From 47baeead75f5c9ac16fc02a35efb2321305f7120 Mon Sep 17 00:00:00 2001 From: Drew Leonard Date: Tue, 18 Feb 2025 16:22:41 +0000 Subject: [PATCH 25/26] Colons are important apparently --- docs/howto_guides/asdf_warnings.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto_guides/asdf_warnings.rst b/docs/howto_guides/asdf_warnings.rst index ba3d9b54..6b94c495 100644 --- a/docs/howto_guides/asdf_warnings.rst +++ b/docs/howto_guides/asdf_warnings.rst @@ -1,4 +1,4 @@ -.. _dkist:howto-guide:asdf-warnings +.. _dkist:howto-guide:asdf-warnings: Understand version warnings when loading ASDFs ============================================== From 3120940850ffc49262c896d76f487c195be0f99b Mon Sep 17 00:00:00 2001 From: Stuart Mumford Date: Tue, 18 Feb 2025 16:41:30 +0000 Subject: [PATCH 26/26] Apply suggestions from code review --- docs/howto_guides/replacing_asdfs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst index d8446862..350e9dbf 100644 --- a/docs/howto_guides/replacing_asdfs.rst +++ b/docs/howto_guides/replacing_asdfs.rst @@ -13,7 +13,7 @@ For example: .. code-block:: python - Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP + Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP where ``res`` is the result of a ``Fido`` search for some dataset you already have saved locally. @@ -25,7 +25,7 @@ Again, we can use the ``overwrite`` keyword for this. from dkist.data.sample import download_all_sample_data - download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA + download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA In the past, the naming convention for the metadata ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two separate metadata files.