Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary stack trace. #10

Merged
merged 3 commits into from
Nov 9, 2021
Merged

Remove unnecessary stack trace. #10

merged 3 commits into from
Nov 9, 2021

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented Nov 9, 2021

Description

Removes unnecessary stack trace.

For a model:

select *
from {{ ref('my_first_dbt_model') }}
where and id = 1

The error message when run dbt run will be:

% dbt run --models models/example/my_second_dbt_model.sql

...

Completed with 1 error and 0 warnings:

Runtime Error in model my_second_dbt_model (models/example/my_second_dbt_model.sql)
  Query execution failed.
  Error message: org.apache.spark.sql.catalyst.parser.ParseException:
  mismatched input 'id' expecting {<EOF>, ';'}(line 9, pos 10)

  == SQL ==
  /* {"app": "dbt", "dbt_version": "1.0.0b2", "profile_name": "databricks_sql_connector", "target_name": "dev", "node_id": "model.my_new_project.my_second_dbt_model"} */
  create or replace view dbt_demo_20211102.my_second_dbt_model

    as
      -- Use the `ref` function to select from other models

  select *
  from dbt_demo_20211102.my_first_dbt_model
  where and id = 1
  ----------^^^

Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Without the fix:

...

Completed with 1 error and 0 warnings:

Runtime Error in model my_second_dbt_model (models/example/my_second_dbt_model.sql)
  Query execution failed.
  State: ERROR_STATE; Error code: 0; SQLSTATE: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.catalyst.parser.ParseException:
  mismatched input 'id' expecting {<EOF>, ';'}(line 9, pos 10)

  == SQL ==
  /* {"app": "dbt", "dbt_version": "1.0.0b2", "profile_name": "databricks_sql_connector", "target_name": "dev", "node_id": "model.my_new_project.my_second_dbt_model"} */
  create or replace view dbt_demo_20211102.my_second_dbt_model

    as
      -- Use the `ref` function to select from other models

  select *
  from dbt_demo_20211102.my_first_dbt_model
  where and id = 1
  ----------^^^

  	at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:43)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:355)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:195)
  	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  	at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.withLocalProperties(ThriftLocalProperties.scala:114)
  	at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.withLocalProperties$(ThriftLocalProperties.scala:48)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:47)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:195)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:190)
  	at java.security.AccessController.doPrivileged(Native Method)
  	at javax.security.auth.Subject.doAs(Subject.java:422)
  	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:206)
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)
  Caused by: org.apache.spark.sql.catalyst.parser.ParseException:
  mismatched input 'id' expecting {<EOF>, ';'}(line 9, pos 10)

  == SQL ==
  /* {"app": "dbt", "dbt_version": "1.0.0b2", "profile_name": "databricks_sql_connector", "target_name": "dev", "node_id": "model.my_new_project.my_second_dbt_model"} */
  create or replace view dbt_demo_20211102.my_second_dbt_model

    as
      -- Use the `ref` function to select from other models

  select *
  from dbt_demo_20211102.my_first_dbt_model
  where and id = 1
  ----------^^^

  	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:275)
  	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:136)
  	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:66)
  	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:87)
  	at com.databricks.sql.parser.DatabricksSqlParser.$anonfun$parsePlan$1(DatabricksSqlParser.scala:67)
  	at com.databricks.sql.parser.DatabricksSqlParser.parse(DatabricksSqlParser.scala:87)
  	at com.databricks.sql.parser.DatabricksSqlParser.parsePlan(DatabricksSqlParser.scala:64)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$compileQuery$3(SparkExecuteStatementOperation.scala:271)
  	at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
  	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:134)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$compileQuery$2(SparkExecuteStatementOperation.scala:271)
  	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:854)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$compileQuery$1(SparkExecuteStatementOperation.scala:268)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.getOrCreateDF(SparkExecuteStatementOperation.scala:257)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.compileQuery(SparkExecuteStatementOperation.scala:268)
  	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:318)
  	... 16 more

  Error message: None


Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-spark next" section.

@ueshin
Copy link
Collaborator Author

ueshin commented Nov 9, 2021

The unit tests fail even in the upstream https://github.com/dbt-labs/dbt-spark/runs/4142451929?check_suite_focus=true.
I guess some changes in dbt-core cause the test failures. Let's see.

I tried the unit tests in my local and they all pass.

@ueshin
Copy link
Collaborator Author

ueshin commented Nov 9, 2021

I'd merge this now.
The unit tests will be fixed in the next PR to sync with upstream dbt-labs/dbt-spark@6e605cb.

@ueshin ueshin merged commit 02beca7 into databricks:main Nov 9, 2021
@ueshin ueshin deleted the stack_trace branch November 9, 2021 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant