feat(ingest/snowflake): handle failures gracefully and raise permissi…

…on failures (datahub-project#6748)

metadata-ingestion/docs/sources/snowflake/snowflake_pre.md

@@ -16,21 +16,19 @@ grant usage on DATABASE "<your-database>" to role datahub_role;
 grant usage on all schemas in database "<your-database>" to role datahub_role;
 grant usage on future schemas in database "<your-database>" to role datahub_role;
 
-// If you are NOT using Snowflake Profiling feature: Grant references privileges to your tables and views 
+// If you are NOT using Snowflake Profiling or Classification feature: Grant references privileges to your tables and views 
 grant references on all tables in database "<your-database>" to role datahub_role;
 grant references on future tables in database "<your-database>" to role datahub_role;
 grant references on all external tables in database "<your-database>" to role datahub_role;
 grant references on future external tables in database "<your-database>" to role datahub_role;
 grant references on all views in database "<your-database>" to role datahub_role;
 grant references on future views in database "<your-database>" to role datahub_role;
 
-// If you ARE using Snowflake Profiling feature: Grant select privileges to your tables and views 
+// If you ARE using Snowflake Profiling or Classification feature: Grant select privileges to your tables
 grant select on all tables in database "<your-database>" to role datahub_role;
 grant select on future tables in database "<your-database>" to role datahub_role;
 grant select on all external tables in database "<your-database>" to role datahub_role;
 grant select on future external tables in database "<your-database>" to role datahub_role;
-grant select on all views in database "<your-database>" to role datahub_role;
-grant select on future views in database "<your-database>" to role datahub_role;
 
 // Create a new DataHub user and assign the DataHub role to it 
 create user datahub_user display_name = 'DataHub' password='' default_role = datahub_role default_warehouse = '<your-warehouse>';
@@ -40,17 +38,26 @@ grant role datahub_role to user datahub_user;
 ```
 
 The details of each granted privilege can be viewed in [snowflake docs](https://docs.snowflake.com/en/user-guide/security-access-control-privileges.html). A summarization of each privilege, and why it is required for this connector:
+
 - `operate` is required on warehouse to execute queries
 - `usage` is required for us to run queries using the warehouse
 - `usage` on `database` and `schema` are required because without it tables and views inside them are not accessible. If an admin does the required grants on `table` but misses the grants on `schema` or the `database` in which the table/view exists then we will not be able to get metadata for the table/view.
 - If metadata is required only on some schemas then you can grant the usage privilieges only on a particular schema like
+
 ```sql
 grant usage on schema "<your-database>"."<your-schema>" to role datahub_role;
 ```
 
 This represents the bare minimum privileges required to extract databases, schemas, views, tables from Snowflake.
 
 If you plan to enable extraction of table lineage, via the `include_table_lineage` config flag or extraction of usage statistics, via the `include_usage_stats` config, you'll also need to grant access to the [Account Usage](https://docs.snowflake.com/en/sql-reference/account-usage.html) system tables, using which the DataHub source extracts information. This can be done by granting access to the `snowflake` database.
+
 ```sql
 grant imported privileges on database snowflake to role datahub_role;
-```
+```
+
+### Caveats
+
+- Some of the features are only available in the Snowflake Enterprise Edition. This doc has notes mentioning where this applies.
+- The underlying Snowflake views that we use to get metadata have a [latency of 45 minutes to 3 hours](https://docs.snowflake.com/en/sql-reference/account-usage.html#differences-between-account-usage-and-information-schema). So we would not be able to get very recent metadata in some cases like queries you ran within that time period etc.
+- If there is any [incident going on for Snowflake](https://status.snowflake.com/) we will not be able to get the metadata until that incident is resolved.

metadata-ingestion/src/datahub/ingestion/source/snowflake/constants.py

@@ -0,0 +1,48 @@
+from enum import Enum
+
+
+class SnowflakeCloudProvider(str, Enum):
+    AWS = "aws"
+    GCP = "gcp"
+    AZURE = "azure"
+
+
+SNOWFLAKE_DEFAULT_CLOUD = SnowflakeCloudProvider.AWS
+
+
+class SnowflakeEdition(str, Enum):
+    STANDARD = "Standard"
+
+    # We use this to represent Enterprise Edition or higher
+    ENTERPRISE = "Enterprise or above"
+
+
+# See https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#region-ids
+# Includes only exceptions to format <provider>_<cloud region with hyphen replaced by _>
+SNOWFLAKE_REGION_CLOUD_REGION_MAPPING = {
+    "aws_us_east_1_gov": (SnowflakeCloudProvider.AWS, "us-east-1"),
+    "azure_uksouth": (SnowflakeCloudProvider.AZURE, "uk-south"),
+    "azure_centralindia": (SnowflakeCloudProvider.AZURE, "central-india.azure"),
+}
+
+# https://docs.snowflake.com/en/sql-reference/snowflake-db.html
+SNOWFLAKE_DATABASE = "SNOWFLAKE"
+
+
+# We will always compare with lowercase
+# Complete list for objectDomain - https://docs.snowflake.com/en/sql-reference/account-usage/access_history.html
+class SnowflakeObjectDomain(str, Enum):
+    TABLE = "table"
+    EXTERNAL_TABLE = "external table"
+    VIEW = "view"
+    MATERIALIZED_VIEW = "materialized view"
+
+
+GENERIC_PERMISSION_ERROR_KEY = "permission-error"
+LINEAGE_PERMISSION_ERROR = "lineage-permission-error"
+
+
+# Snowflake connection arguments
+# https://docs.snowflake.com/en/user-guide/python-connector-api.html#connect
+CLIENT_PREFETCH_THREADS = "client_prefetch_threads"
+CLIENT_SESSION_KEEP_ALIVE = "client_session_keep_alive"