MyVariant.info deployment notes

New Data release (or ES upgrade)

• old ES cluster: <myvariant_current>
• new ES cluster: <myvariant_new>

1. Take snapshot of new ES index from dev ES.

2. Start a new data node, change to the new cluster name, and restore snapshot from S3.

3. Enable master and http on this new data node.

4. Point a dev server to use new ES index, and run nose tests to make sure everything passes.

5. Do these steps on one web node (prefer not spot instance):

   1. take it out of LB
   2. stop ES, upgrade ES if needed
   3. change ES cluster name to new cluster name
   4. restart ES to let this web node to join the new ES cluster
   5. run nose tests on this web node, make sure everything passes
   6. add this web node back to LB

6. Take out of the rest of web nodes from LB, and make above changes on one of web nodes, and add it back to LB

7. Create a new ami from a web node, and create new web node spot requests.

8. Once all new spot instances fulfilled, terminate all old SP web nodes and SP requests.

9. Do these steps on all master nodes:

   1. stop ES, upgrade ES if needed
   2. change ES cluster name to new cluster name
   3. restart ES to let them join the new ES cluster

10. Make new data node SP requests:

    1. Take one SP data node, stop ES.
    2. update ES if needed.
    3. change ES cluster name to the new cluster name.
    4. clean up the /data/es_logs and /data/es_data folders.
    5. Make a new AMI, and create new data node spot requests.

11. When all new SP data nodes fulfilled, terminate all the new data nodes and SP requests.

12. As the fallback from this upgrade, we can stop the old data node and keep it for at least a week or so. In case of new cluster failure, we can enable master and http on this old data node as the fallback.