Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(snowflake): convert parameters to qmark style #350

Merged
merged 2 commits into from
Mar 17, 2021

Conversation

hachichaud
Copy link
Contributor

@hachichaud hachichaud commented Mar 17, 2021

Change Summary

The parameters value passed from the backend can be incompatible
with pyformat mapping (dict or array values).
I use convert to qmark method that is already used on other connectors
Another way would be to flatten parameters, but it could still leave
some incompatible values (arrays of dict).

Checklist

  • Unit tests for the changes exist
  • Tests pass on CI and coverage remains at 100%
  • Documentation reflects the changes where applicable

the parameters value passed from the backend can be incompatible
with pyformat mapping (dict or array values).
I use this method that is already used on other conenctors
Another way would be to flatten parameters, but it could still leave
some incompatible values (arrays of dict).
@hachichaud hachichaud self-assigned this Mar 17, 2021
@hachichaud hachichaud force-pushed the fix/parameters-snowflake branch from 7c6704f to fac0e98 Compare March 17, 2021 08:46
@raphaelvignes
Copy link
Contributor

Thanks for this fix 👍

Copy link
Member

@davinov davinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 👍
A very small suggestion regarding delimiters used in examples:

@@ -36,7 +36,7 @@
domain='test_domain',
database='test_database',
warehouse='test_warehouse',
query='test_query with %(foo)s and {{ pokemon }}',
query='test_query with %(foo)s and %(pokemon)s',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

%(...) is an old python syntax, I think we'd prefer to push Jinja' one in our connectors configs
Today, we're already replacing these automatically:

# Replace param templating with jinja templating:

So my suggestion would be to use jinja {{ }} in your tests and documentation as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

les syntaxes compatibles avec execute de snowflake sont

  • pyformat (%()s) => bug car on passe un dictionnaire parameters qui peut comporter des valeurs invalides.
  • qmark => cest dans ce format qu'on convertit notre query pour bypasser le probleme du dictionnaire.

Par contre on ne peut pas utiliser jinja car ce n'est pas safe sur des requetes SQL
cf le code que tu pointes, qui est extrait de la methode no_sql_apply_parameters_to_query :

def nosql_apply_parameters_to_query(query, parameters, handle_errors=False):
    """
    WARNING : DO NOT USE THIS WITH VARIANTS OF SQL
    Instead use your client library parameter substitution method.
    https://www.owasp.org/index.php/Query_Parameterization_Cheat_Sheet
    """

on veut spécifiquement éviter le templating de cette manière.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davinov you already had an issue on that actually! #303
^^

@hachichaud
Copy link
Contributor Author

@hachichaud hachichaud dismissed davinov’s stale review March 17, 2021 12:13

Cannot use jinja for sql query parameters

@hachichaud hachichaud added the bug Something isn't working label Mar 17, 2021
Copy link
Member

@davinov davinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a slack discussion about which style of param to use, let's open an issue/think more about it, but this PR is not the place for that.

@hachichaud hachichaud merged commit 6f09c48 into master Mar 17, 2021
@hachichaud hachichaud deleted the fix/parameters-snowflake branch March 17, 2021 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants