Feature: getting started script #475

jafreck · 2018-04-03T23:24:36Z

…tk into feature/getting_started_scripts

paselem · 2018-04-04T22:06:02Z

README.md

@@ -16,28 +16,32 @@ This toolkit is built on top of Azure Batch but does not require any Azure Batch
 - Ability to run _spark submit_ directly from your local machine's CLI

 ## Setup
+
 1. Clone the repo


Are we comfortable enough to just pip install? Technically it should be the same thing, right??

technically, yes. But I elected not to make that change here until our release on pip is "official". I can do it now if you'd prefer.

it just seems so much easier now... even if the version shows up as '0.6.1b' for example...

paselem · 2018-04-04T22:14:10Z

account_setup.py

+                             "Do you want to use this virtual network? (y/n): ".format(virtual_network_name)
+        deny_error = AccountSetupError("Virtual network already exists, not recreating.")
+        unrecognized_input_error = AccountSetupError("Input not recognized.")
+        prompt_for_confirmation(confirmation_prompt, deny_error, unrecognized_input_error)


Never used this before, but does this exit the method?

it will raise an error if either the user declines, or if in 3 tries the input is not recognized. Otherwise this returns none and just continues.

paselem · 2018-04-04T22:16:29Z

account_setup.py

+        filter="roleName eq '{}'".format(role_name)
+    ))
+    contributor_role = roles[0]
+    for i in range(10):


What are you doing here? Is this trying to assign the roles on a retry loop?

For some reason, the GraphRbacManagementClient's service_principal.create() method does not return an AzureOperationPoller to wait on (ala most of the other clients) nor does it block until the object is created server side. It just returns an object which may or may not have been created server side yet.

So I need to wait to give permissions here until the service principal is actually created.

paselem · 2018-04-04T22:17:40Z

account_setup.py

+    subscription_client = SubscriptionClient(creds)
+    tenant_ids = [tenant.id for tenant in subscription_client.tenants.list()]
+    if len(tenant_ids) != 1:
+        raise AccountSetupError("More than one tenant configured on your subscription.")


Is there nothing the user can do here? I'm not even sure what this error means myself...

This means that you're subscription has more than one tenant configured (don't know myself why that is possible/desired). I'm not sure what should be done here, or if we should bother. This seems like an extremely advanced case.

paselem · 2018-04-04T22:18:31Z

account_setup.py

+        "resource_group": prompt_with_default("Resource Group Name", DefaultSettings.resource_group),
+        "storage_account": prompt_with_default("Storage Account Name", DefaultSettings.storage_account),
+        "batch_account": prompt_with_default("Batch Account Name", DefaultSettings.batch_account),
+        # "virtual_network_name": prompt_with_default("Virtual Network Name", DefaultSettings.virtual_network_name),


un-comment?

maybe this could be a 'leave blank for no vnet'

I left this commented since we don't (currently) have a way to output the subnet_id since it is in cluster.yaml and everything else is in secrets.yaml.

paselem · 2018-04-04T22:20:46Z

account_setup.py

+        parameters={
+            'location': kwargs.get("region", DefaultSettings.region),
+        }
+    )


Will this fail w/ a good error msg is the user provides bad inputs? E.g. region == westuss

Right now this will just throw whatever error the ResourceManagementClient throws. I think we should probably catch this error, print a link to docs on azure regions, do a 3 attempt retry loop.

This is the current error:

Traceback (most recent call last): File "account_setup.py", line 336, in <module> resource_group_id = create_resource_group(creds, subscription_id, **kwargs) File "account_setup.py", line 79, in create_resource_group 'location': kwargs.get("region", DefaultSettings.region), File "/home/jake/.local/lib/python3.5/site-packages/azure/mgmt/resource/resources/v2017_05_10/operations/resource_groups_operations. py", line 147, in create_or_update raise exp msrestazure.azure_exceptions.CloudError: Azure Error: LocationNotAvailableForResourceGroup Message: The provided location 'westuss' is not available for resource group. List of available regions is 'centralus,eastasia,southea stasia,eastus,eastus2,westus,westus2,northcentralus,southcentralus,westcentralus,northeurope,westeurope,japaneast,japanwest,brazilsout h,australiasoutheast,australiaeast,westindia,southindia,centralindia,canadacentral,canadaeast,uksouth,ukwest,koreacentral,koreasouth,f rancecentral,eastus2euap,centraluseuap'.

paselem · 2018-04-04T22:23:09Z

docs/01-getting-started-script.md

+
+The script will create and configure the following resources:
+- Resource group
+- Storage account


Should we add a vnet?

Virtual Network (optional)

Possibly, but I'm not sure how we'd do output. We could potentially just leave this out until we have "environments"

paselem

I still feel like we should use pip install and cut down on that step. It's probably not required any more. At that point though, might be worth moving these docs somewhere for users to want the 'latest and greatest' version.

paselem · 2018-04-10T16:57:54Z

account_setup.py

+        "application_credential_name": prompt_with_default("Active Directory Application Credential Name", DefaultSettings.resource_group),
+        "service_principal": prompt_with_default("Service Principal Name", DefaultSettings.service_principal)
+    }
+    print("Creating the Azure resources.")


I just went through this again and parts of this can take a while, especially getting the service principal. I would recommend adding more print statements in this piece of code make it feel like progress is being made.

paselem · 2018-04-10T17:01:50Z

account_setup.py

+    )
+    application_credential = uuid.uuid4()
+    try:
+        display_name = kwargs.get("application_name", DefaultSettings.application_name)


Looks like there may be a bug:

Here is what it looks like in portal:

paselem · 2018-04-10T17:06:33Z

account_setup.py

+        }
+    )
+
+    print("\n# Copy the following into your .aztk/secrets.yaml file\n", secrets)


This is still printing with a space. I believe this will work

print("\n# Copy the following into your .aztk/secrets.yaml file\n{}".format(secrets))

paselem and others added 17 commits March 23, 2018 14:56

initial changes for getting started scripts

ca499b2

add temp error handling

62c1518

rename file - fix typo

4754ca3

add debug strings

3f05898

add handling for existing user

9bac57f

WIP: wait for subprocess to complete to get exit code

9ab8810

WIP: handle existing user and refactor code

a7076bf

WIP: add missing return statements

d93e73e

WIP: fix typo

8c80ba0

start sdk refactor

e6b7d53

mostly working create

23ff3e2

working happy create path

2576393

handle errors for vnet, aad application

010b8c1

make account setup interactive

e33e0c8

add prompt

746e7d0

add docs

51997e6

rename account_setup_refac to account_setup

d34b05a

jafreck added this to the v0.7.0 milestone Apr 3, 2018

jafreck added the in progress label Apr 3, 2018

jafreck added 3 commits April 3, 2018 16:24

Merge branch 'master' into feature/getting_started_scripts

742dc6b

add some logging

d4c261a

Merge branch 'feature/getting_started_scripts' of github.com:azure/az…

c7cf1dc

…tk into feature/getting_started_scripts

jafreck added the work in progress label Apr 4, 2018

pip install msrest, azure-cli-core, import issues

1049419

jafreck removed the work in progress label Apr 4, 2018

jafreck added 4 commits April 4, 2018 11:59

remove in script pip, add shell wrapper program

b59848b

ellipsis to period

96dd518

update branch name for account_setup.sh

29b4111

docstring

35c5ee9

paselem suggested changes Apr 4, 2018

View reviewed changes

jafreck added 3 commits April 4, 2018 17:39

retry resource group creation

3851dd4

fix typo, update retry

56f1919

explicitly set output location

3616f0a

paselem approved these changes Apr 5, 2018

View reviewed changes

jafreck added 6 commits April 5, 2018 10:51

wget overwrite flag, docs update

a6cbd56

add prompt for multi tenants

84a4bc2

fix bug with batch account creation

4ebc4f3

Merge branch 'master' into feature/getting_started_scripts

75b039e

Merge branch 'master' into feature/getting_started_scripts

60bdc18

Merge branch 'master' into feature/getting_started_scripts

30fb1ae

paselem reviewed Apr 10, 2018

View reviewed changes

jafreck added 3 commits April 10, 2018 16:17

add spinner, print statements, fix formatting bug

76e95f5

Merge branch 'master' into feature/getting_started_scripts

d0c7a41

fix param bug

27e45d8

paselem approved these changes Apr 11, 2018

View reviewed changes

jafreck merged commit 7ef721f into master Apr 11, 2018

jafreck removed the in progress label Apr 11, 2018

timotheeguerin deleted the feature/getting_started_scripts branch April 18, 2018 22:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: getting started script #475

Feature: getting started script #475

jafreck commented Apr 3, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

paselem Apr 5, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

paselem Apr 4, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

jafreck Apr 4, 2018

paselem Apr 4, 2018

jafreck Apr 4, 2018

paselem left a comment

paselem Apr 10, 2018

paselem Apr 10, 2018

paselem Apr 10, 2018

Feature: getting started script #475

Feature: getting started script #475

Conversation

jafreck commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paselem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment