response to Hera ticket

source/FAQ/index.rst

@@ -402,8 +402,271 @@ How do I find out what my project quota is?
 
 Refer to the allocation pages.
 
-How to transfer small files to/from an RDHPCS system?
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. _create-port-tunnel:
+
+How do I set up an ssh port tunnel?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can only establish an ssh tunnel from your initial bastion session. If you
+try to establish a tunnel and see the messages like this:
+
+
+  .. code-block:: shell
+
+    -------------------
+    bind [127.0.0.1]:57037: Address already in use
+    channel_setup_fwd_listener_tcpip: cannot listen to port: 57037
+    Could not request local forwarding.
+    -------------------
+
+You will know that you already have an open session, and cannot
+open a tunnel on this bastion.
+
+To establish a new tunnel, do one of the following:
+
+  * Close any existing sessions on this bastion, **or**,
+  * Open a new session using a bastion where you have no existing sessions.
+
+In the steps below, replace First.Last with your own HPC username, and
+XXXXX with the unique Local Port Number assigned to you when you log
+in to your specified HPC system (Hera/Jet). Use the word "localhost"
+where indicated. It is not a variable, don't substitute anything else.
+Before you perform the first step, close all current sessions on the
+HPC where system you are trying to connect. Once the first session has
+been opened with port forwarding, any further connections (login via
+ssh, copy via scp) will work as expected. You are running these
+commands on your local machine, not within the HPC system terminal.
+
+As long as this ssh window remains open, you will be able to use this
+forwarded port for data transfers. After the first session has been
+opened with the port forwarding, any further connections (login via
+ssh, copy via scp) will work as expected.
+
+
+
+**1. Find your local port number**
+
+To find your unique local port number, log onto your specified HPC
+system (Hera/Jet). Make a note of this number - once you've recorded
+it, close all sessions. Note that this number, which is a fixed
+value for you, will be different on Jet and Hera.
+
+.. image:: /images/linux_xfer1.png
+   :scale: 75%
+
+.. note::
+    Open two terminal windows for this process
+
+**Local Client Window #1**
+
+Enter the appropriate command for your environment. Remember to replace XXXXX
+with the local port number identified in Step 1 or as needed.
+
+For Windows Power Shell, enter:
+
+.. code-block:: shell
+
+     ssh -m [email protected] -L1234e:localhost:12345 [email protected]
+
+
+For Mac or Linux, enter:
+
+.. code-block:: shell
+
+     ssh -L12345:localhost:12345 [email protected]
+
+If you will be running X11 applications with x2go or normal terminals,
+remember to add the -X parameter as follows:
+
+.. code-block:: shell
+
+    ssh -X -L12345:localhost:12345 [email protected]
+
+Note that objects emphasized in this figure should be unique to your
+configuration:
+
+.. image:: /images/linux_xfer2.png
+   :scale: 75%
+
+Verify that the tunnel is working by doing the following in another local
+window from your local machine:
+
+.. code-block:: shell
+
+   ssh -p <port> First.Last@localhost
+
+
+Note that <port> is your local port number used above, First.Last is
+your user ID on the RDHPCS systems and localhost is typed as-is.
+
+You should be prompted for your password; enter your PIN + RSA token
+and you should be able to login. Once you are able to log in, you can
+log out of that session as that was only for testing the tunnel.
+
+**2. Use SCP to Complete the Transfer**
+
+**Local Client Window #2**
+
+Once the session is open, you can use this forwarded port
+for data transfers, as long as this ssh window is kept open. After the
+first session has been opened with the port forwarding, any
+further connections (login via ssh, copy via scp) will work as
+expected.
+
+
+Remember that this is the second terminal session opened on your local
+machine. Once a tunnel has been set up as in Step 1, you
+can use a client such as WinSCP to do the tranfers using that tunnel.
+Please keep in mind that tunnel will exist only as long as the session opened
+in Step 1 is kept alive.
+
+
+.. code-block:: shell
+
+  Hostname: localhost
+  Port: your-assigned-port-used-in-Step1-above
+  File protocol: SFTP
+
+
+
+
+To transfer a file **to** HPC Systems
+
+
+For Windows Power Shell, enter:
+
+.. code-block:: shell
+
+  scp -P XXXXX /local/path/to/file First.Last@localhost:/path/to/file/on/HPCSystems
+
+For Mac or Linux, enter:
+
+.. code-block:: shell
+
+  rsync <put rsync options here> -e 'ssh -l First.Last -p XXXXX' /local/path/to/files First.Last@localhost:/path/to/files/on/HPCSystems
+
+.. note::
+
+   Your username is case sensitive when used in the scp command. Username should be in the form of First.Last.
+
+To transfer a file **from** HPC Systems:
+
+For Windows Power Shell, enter:
+
+.. code-block:: shell
+
+    scp -P XXXXX First.Last@localhost:/path/to/file/on/HPCSystems /local/path/to/file
+
+For Mac or Linux, enter:
+
+.. code-block:: shell
+
+    rsync <put rsync options here> -e 'ssh -l First.Last -p XXXXX' First.Last@localhost:/path/to/files/on/HPCSystems /local/path/to/files
+
+
+In either case, you will be asked for a password. Enter the password
+from your RSA token (not your passphrase). Your response should be
+your PIN+Token code.
+
+
+
+SSH Port Tunnel For PuTTy Windows Systems
+-----------------------------------------
+
+PuTTY is an SSH client, used to configure and initiate connection.
+Navigate to a separate tab to install `PuTTY
+<http://www.putty.org/>`_. If you cannot install software on your
+machine, contact your local systems administrator.
+
+**Configuration**
+
+Enter host information to configure an SSH Terminal Session. The
+example below defines a session to Jet via the Boulder Bastion:
+
+.. image:: /images/putty1.png
+   :scale: 75%
+
+1. Enter Username
+In the left pane under Connection, select "Data" and enter your NOAA
+user name as it appears in your NOAA email address. (Ex: Robin.Lee
+if your NOAA email is [email protected]). User name is case
+sensitive - First.Last. If you do not create a username, you will have
+to enter your user name each time your open a session.
+
+.. image:: /images/putty2.png
+   :scale: 75%
+
+Complete the configuration:
+
+* Select "Session" from the top of the left pane.
+* Select "Save" (between Load and Delete).
+
+**Open a First System Session**
+
+Open the session to make sure it's working, and to record your Local
+Port number to complete the Port Tunneling setup.
+
+* Select the configured session from the "Saved Sessions" list. Select
+  Load, then Open.
+* Enter your unique RSA Passcode.
+
+The RSA passcode is your RSA token PIN followed by 8 digits displayed
+on your RSA token. The digits must be on display when you press enter,
+or access will be denied. When you open a new SSH session, wait for
+the RSA token code to refresh before you enter it.
+
+* Find and record your Local Host number.
+
+.. image:: /images/linux_xfer1.png
+    :scale: 75%
+
+* Click **Exit**, or close the Putty window to end the session.
+
+**Port Tunnel Setup**
+
+To enable data transfers, you will need to set up a Port Tunnel.
+
+* Open Putty.
+* Select the session from the Saved Sessions list, then Load.
+* In the left pane under Connection>SSH select Tunnels.
+* Check Local ports accept connections from other hosts.
+* In the Source Port field, enter your Local Port number
+* In the Destination Port field, enter "localhost:<local port
+  number>", where your local port number matches what was entered in
+  the Source port.
+* Select Local and Auto Radio Buttons.
+* Click the Add Button.
+
+.. image:: /images/putty3.png
+
+To save the configuration change:
+
+* In the left pane, select Session.
+* Select Save.
+
+Select **Open** to Login and verify that the updated session works correctly.
+
+Create a new Port Tunnel for each SSH system you intend to use. Each
+one will have a unique Local Port number.
+
+To add extra saved sessions (ex: for another Bastion) for the same
+system (you already have the Local Port number):
+
+* Load your current saved session
+* Enter the new host name for the other Bastion
+* Give the new session a new name (ex: Jet - Princeton)
+* Select Save. The new session will appear in the list of saved sessions.
+* Select Open to Login and verify the new session works correctly.
+
+
+SSH Port Tunnel For Tectia Windows Systems
+------------------------------------------
+
+See the :ref:`tectia` pages for complete information.
+
+
+How do I transfer small files to/from an RDHPCS system?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The Port Tunnelling approach is useful for transferring small amount
 data to/from RDHPCS systems from your local machine.

source/data/globus_example.rst

@@ -262,8 +262,6 @@ noaa-rrfs-pds): arn:aws:s3:::noaa-rrfs-pds Now you have the info you need.
 Once you are able to see the listing of files you can use the "File Manager" to
 move the files between the desired endpoints.
 
-That should do it!
-
 .. note::
 
   Module globus-cli needs to be loaded before any globus commands are used.
@@ -366,6 +364,8 @@ The mechanism for transferring data between your laptop/workstation
 `Globus Connect Personal`_ for information about setting up your
 laptop/workstation as a Globus Personal Endpoint.
 
+.. _institutional-data-portal:
+
 GFDL Institutional Data Portal
 ==============================
 

source/data/globus_online_data_transfer.rst

@@ -260,8 +260,8 @@ Endpoint.
       files (less than 1MB each one), it would be better to tar the smallfiles.
       Otherwise, if the files are larger, Globus will handle them.
 
-Data Sharing with External Collaborators
-========================================
+Sharing Data Externally
+=======================
 
 .. Note::
 
@@ -272,6 +272,15 @@ accounts on the RDHPCS system. You can share data files with external
 collaborators, both inbound and outbound, using the Untrusted DTNs (UDTNs). The
 process is described in this section.
 
+For data that is short-lived, and not broadly shared with external users use
+RDHPCS end-points. For data that is expected to be available for three 3 months
+or more, use the :ref:`institutional-data-portal` end-point.
+
+.. note::
+
+  Refer to the `GFDL FAIR use and GFDL Data DOI policy
+  <https://www.gfdl.noaa.gov/fair-use-policy/>`_ for external data sharing.
+
 .. Note::
 
   * This data sharing feature is only available only on *untrusted*