How-to guide on replacing ASDFs (#531)

* How-to guide on replacing asdfs * Changelog * Bad changelog * Add another remote data tag * Add another remote data tag * Even more remote data * Need to include the guide in the doc tree * Better address the problem * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Move warnings discussion to its own file and rephrase duplicate ASDF warning * @Cadair's suggestions * Update docs/howto_guides/asdf_warnings.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Update docs/howto_guides/replacing_asdfs.rst Co-authored-by: Stuart Mumford <[email protected]> * Re- more words * Colons are important apparently * Apply suggestions from code review --------- Co-authored-by: Stuart Mumford <[email protected]>
DKISTDC · Feb 18, 2025 · 003dc21 · 003dc21
1 parent b963047
commit 003dc21
Show file tree

Hide file tree

Showing 4 changed files with 79 additions and 0 deletions.
diff --git a/changelog/531.doc.rst b/changelog/531.doc.rst
@@ -0,0 +1 @@
+Add a how-to guide explaining how to replace outdated dataset metadata files.
diff --git a/docs/howto_guides/asdf_warnings.rst b/docs/howto_guides/asdf_warnings.rst
@@ -0,0 +1,29 @@
+.. _dkist:howto-guide:asdf-warnings:
+
+Understand version warnings when loading ASDFs
+==============================================
+
+When you load a recently-downloaded ASDF file you may see a warning something like this:
+
+::
+
+    AsdfPackageVersionWarning: File '<file you tried to load>' was created with extension URI 'asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.2.0' (from package gwcs==0.24.0), but older package (gwcs==0.22.0) is installed.
+
+
+The extension and package it references will vary.
+This warning means that the `dkist` package or one of its dependencies is out of date.
+To correct this, you should update your Python tools installation (and its dependencies) with
+
+.. code-block:: bash
+
+    pip install --upgrade dkist
+
+
+if you installed using `pip` or
+
+.. code-block:: bash
+
+    conda update dkist
+
+
+if you used `conda`.
diff --git a/docs/howto_guides/index.rst b/docs/howto_guides/index.rst
@@ -11,3 +11,5 @@ If you're starting fresh you might want to check out the :ref:`dkist:tutorial:in
    :maxdepth: 1
 
    sample_data
+   replacing_asdfs
+   asdf_warnings
diff --git a/docs/howto_guides/replacing_asdfs.rst b/docs/howto_guides/replacing_asdfs.rst
@@ -0,0 +1,47 @@
+.. _dkist:howto-guide:replacing-asdfs:
+
+Update Previously Downloaded ASDF Files
+=======================================
+
+The DKIST Data Center will occasionally update the ASDF files for all the datasets it stores, due to changes or corrections to the metadata.
+This means that ASDF files you have downloaded previously might become outdated and need to be re-downloaded.
+
+You should periodically re-download your local ASDF files to keep up to date with changes to the dataset metadata.
+To do so, you can use the `overwrite` keyword argument when downloading an ASDF file.
+
+For example:
+
+.. code-block:: python
+
+    Fido.fetch(res, path="~/sunpy/data/{instrument}/{dataset_id}", overwrite=True) # doctest: +SKIP
+
+
+where ``res`` is the result of a ``Fido`` search for some dataset you already have saved locally.
+
+You might also need to force a refresh of the sample data which is included with the Python tools.
+Again, we can use the ``overwrite`` keyword for this.
+
+.. code-block:: python
+
+    from dkist.data.sample import download_all_sample_data
+
+    download_all_sample_data(overwrite=True) # doctest: +REMOTE_DATA
+
+
+In the past, the naming convention for the metadata ASDF files has also changed, meaning that the usual checks to stop you downloading a dataset you already have locally may fail and you will end up with two separate metadata files.
+If you try and load a dataset where a file with an old name is present you will see a warning similar to this:
+
+.. code-block:: python
+
+    >>> from dkist import load_dataset
+    >>> ds = load_dataset('~/sunpy/data/VISP/AGLKO/') # doctest: +SKIP
+
+    WARNING: DKISTUserWarning: ASDF files with old names (VISP_L1_20221024T212807_AGLKO_user_tools.asdf) were found in this directory and ignored. You may want to delete these files. [dkist.dataset.loader]
+
+
+When this happens the newer ASDF file is loaded so the old one can safely be ignored.
+However, to remove the warning the old file can simply be deleted or moved elsewhere.
+
+Note that this warning was added in dkist version 1.10.0.
+In older versions the loader will return a list containing the corresponding dataset for each ASDF file present, which is likely to cause problems.
+Deleting the old file will still solve the issue, although you should also update your Python tools installation to v1.10.0 or later.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Add a how-to guide explaining how to replace outdated dataset metadata files.