[core] add method to kill aws instance to simulate chaos

[hongchaodeng]

Why are these changes needed?

This is separated from #45364.
Have tested it for chaos_test/test_chaos_basic.py.

How to enable this feature is to be discussed? We can enable it for all chaos tests, maybe? Or we can add a flag to migrate tests one by one?

I will test it on more existing tests and discuss with other teams to decide later.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rynewang]

why do we need paramiko? Maybe I'm wrong but can we just use subprocess with command = "ssh ..."?

If we are to add it, I think we need to add it also somewhere for example python/requirements/test-requirements.txt.

Apart from ssh, you can also curl the IDMSv2 HTTP ports. Here is some code (not tested)

import boto3
import requests

def get_instance_metadata(token, path):
    url = f"http://169.254.169.254/latest/meta-data/{path}"
    headers = {'X-aws-ec2-metadata-token': token}
    response = requests.get(url, headers=headers)
    return response.text

def main():
    # Get the metadata token
    token = requests.put(
        'http://169.254.169.254/latest/api/token',
        headers={'X-aws-ec2-metadata-token-ttl-seconds': '21600'}
    ).text

    # Get instance ID and region
    instance_id = get_instance_metadata(token, 'instance-id')
    region = get_instance_metadata(token, 'placement/region')

    # Create EC2 client
    ec2 = boto3.client('ec2', region_name=region)

    # Terminate the instance
    response = ec2.terminate_instances(InstanceIds=[instance_id])

    # Print the response
    print(response)

if __name__ == "__main__":
    main()

[hongchaodeng]

Sure. I will write it in pure ssh.

We can't write python code because it's not wrapped in ssh command.

[rynewang]

Maybe we can detect if we are on anyscale and else, skip the test so the test does not fail on local desktop.

[hongchaodeng]

That's something to add to the test. This PR doesn't change that, but only add the utility method.

That's also why I separate this from #45364. I can't make the call and need to talk to other decision makers to follow up on config change.

[jjyao]

Suggested change

	def _kill_node(self, ip):
	def _terminate_ec2_instance(self, ip):

[rynewang]

just for the sake of completeness, can we detect if the aws cli return 0 or not, and raise exceptions if it's not returning 0?

[hongchaodeng]

Done.
It will raise CalledProcessError if failed.