Google spreadsheets #5915

betodealmeida · 2018-09-17T05:22:42Z

This PR allow querying Google spreadsheets via SQL Alchemy, using a new module called gsheetsdb.

It works, but requires pushing some fixes upstream to moz-sql-parser.

SQL Lab also works:

betodealmeida · 2018-09-17T07:47:40Z

Loading the tables in SQL Lab:

betodealmeida · 2018-09-17T07:52:32Z

superset/assets/src/SqlLab/components/SqlEditorLeftBar.jsx

-      this.setState({ tableName });
-      this.props.actions.queryEditorSetSchema(this.props.queryEditor, schemaName);
-      this.fetchTables(this.props.queryEditor.dbId, schemaName);
-    }


The current code allows switching schema by typing new_schema.some_table in the table selector. The code breaks when the table name has a period in it, since it does naive splitting. I opted for removing it here, since it seems a bit unintuitive as well.

hughhhh · 2018-09-19T16:45:54Z

superset/assets/src/SqlLab/components/ExploreResultsButton.jsx

@@ -159,11 +159,12 @@ class ExploreResultsButton extends React.PureComponent {
      </div>);
  }
  render() {
+    const allowsSubquery = this.props.database && this.props.database.allows_subquery;


nit: deconstruct props on the line before then you can just use allows_subquery in disabled param

const { database: { allows_subquery } } = this.props

https://github.com/lyft/marketdx/blob/master/src/components/Diagnosis/index.jsx#L55

hughhhh · 2018-09-19T16:47:45Z

superset/views/core.py

@@ -2223,10 +2223,11 @@ def sqllab_viz(self):
        }))

    @has_access
-    @expose('/table/<database_id>/<table_name>/<schema>/')
+    @expose('/table/<database_id>/<path:table_name>/<schema>/')
    @log_this
    def table(self, database_id, table_name, schema):


What is the path doing for us here? is it a reference the param in the URI?

hughhhh · 2018-09-19T16:49:45Z

superset/db_engine_specs.py

+    }
+
+    @classmethod
+    def select_star(cls, my_db, table_name, engine, schema=None, limit=100,


I would put the limit var as a class var at the top as DEFAULT_LIMIT = 100

betodealmeida · 2018-10-30T19:11:49Z

Time series now work:

I also tested against a spreadsheet behind authentication, and it works.

…heets

codecov-io · 2018-12-07T19:31:28Z

Codecov Report

Merging #5915 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5915      +/-   ##
==========================================
+ Coverage   73.31%   73.33%   +0.01%     
==========================================
  Files          67       67              
  Lines        9594     9600       +6     
==========================================
+ Hits         7034     7040       +6     
  Misses       2560     2560

Impacted Files	Coverage Δ
superset/db_engine_specs.py	`55.42% <100%> (+0.25%)`	⬆️
superset/views/core.py	`75% <100%> (+0.03%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc8acf2...6be792d. Read the comment docs.

mistercrunch · 2018-12-09T05:55:53Z

LGTM

* Google spreadsheets * Fetch table metadata in SQL Lab * Show full URL for spreadsheet * Fix version * Remove sqllab changes (cherry picked from commit f366bbe)

* Google spreadsheets * Fetch table metadata in SQL Lab * Show full URL for spreadsheet * Fix version * Remove sqllab changes (cherry picked from commit f366bbe) (cherry picked from commit 8900695)

neildunlop · 2019-10-08T14:24:41Z

Can anyone point to an example of setting up the datasource? I can setup the connection and test it fine, but once I save it raises an error.

hemantaggarwal · 2019-12-25T09:50:13Z

Can anyone point to an example of setting up the datasource? I can setup the connection and test it fine, but once I save it raises an error.

@neildunlop : Are you able to setup organization google sheet as datasource?
@betodealmeida : I am able to link public sheets but how to do it for organization specific sheets in superset?

cachafla · 2020-01-14T23:05:43Z

@betodealmeida ping. I'd like some help setting this up as well. I tried setting up a datasource with a URL such as:

gsheets://docs.google.com/spreadsheets/d/<spreadsheet-id>/edit#gid=0

But I get the following error:

2020-01-14 22:51:53,769:ERROR:root:'from'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/superset/views/core.py", line 1705, in testconn
    conn.scalar(select([1]))
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 920, in scalar
    return self.execute(object_, *multiparams, **params).scalar()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1468, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/db.py", line 36, in g
    return f(self, *args, **kwargs)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/db.py", line 137, in execute
    query, headers, self.credentials)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/query.py", line 96, in execute
    from_ = extract_url(query)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/url.py", line 51, in extract_url
    return parse_sql(sql)['from']
KeyError: 'from'

I haven't found any documentation that shows how to setup GSheets a data source, even though this PR shows it's possible.

hemantaggarwal · 2020-01-15T05:00:08Z

@betodealmeida ping. I'd like some help setting this up as well. I tried setting up a datasource with a URL such as:

gsheets://docs.google.com/spreadsheets/d/<spreadsheet-id>/edit#gid=0

But I get the following error:

2020-01-14 22:51:53,769:ERROR:root:'from'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/superset/views/core.py", line 1705, in testconn
    conn.scalar(select([1]))
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 920, in scalar
    return self.execute(object_, *multiparams, **params).scalar()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1468, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/db.py", line 36, in g
    return f(self, *args, **kwargs)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/db.py", line 137, in execute
    query, headers, self.credentials)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/query.py", line 96, in execute
    from_ = extract_url(query)
  File "/home/superset/.local/lib/python3.6/site-packages/gsheetsdb/url.py", line 51, in extract_url
    return parse_sql(sql)['from']
KeyError: 'from'

I haven't found any documentation that shows how to setup GSheets a data source, even though this PR shows it's possible.

In data source put only "gsheets://" instead of full link. If you test it may give error but just save the data source and query the google sheet in SQL editor using full sheet link. But sheet should be public.

ispulkit · 2020-06-29T13:35:04Z

Can you explain a bit about the workaround for private and org-based gSheets?

betodealmeida added 2 commits September 16, 2018 22:20

Google spreadsheets

eb1446d

Fetch table metadata in SQL Lab

8e821f4

Show full URL for spreadsheet

84ae884

betodealmeida commented Sep 17, 2018

View reviewed changes

hughhhh reviewed Sep 19, 2018

View reviewed changes

betodealmeida mentioned this pull request Sep 21, 2018

Add support for granularity functions betodealmeida/gsheets-db-api#2

Closed

Make time series work

9afa85f

betodealmeida changed the title ~~[WIP] Google spreadsheets~~ Google spreadsheets Oct 30, 2018

kristw added the enhancement:request Enhancement request submitted by anyone from the community label Dec 3, 2018

betodealmeida added 3 commits December 6, 2018 14:02

Merge branch 'master' of github.com:apache/incubator-superset into gs…

f85b4ba

…heets

Fix version

9cece7e

Remove sqllab changes

6be792d

betodealmeida merged commit f366bbe into apache:master Dec 10, 2018

betodealmeida mentioned this pull request Dec 10, 2018

Update requests version #6510

Merged

betodealmeida mentioned this pull request Dec 14, 2018

Increase size of column name in table ab_view_meu #6532

Merged

francishemingway mentioned this pull request Jun 10, 2019

Spreadsheet, or Google Sheets, data source #7565

Closed

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.34.0 labels Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google spreadsheets #5915

Google spreadsheets #5915

betodealmeida commented Sep 17, 2018 •

edited

Loading

betodealmeida commented Sep 17, 2018

betodealmeida Sep 17, 2018

hughhhh Sep 19, 2018

hughhhh Sep 19, 2018

hughhhh Sep 19, 2018

betodealmeida commented Oct 30, 2018

codecov-io commented Dec 7, 2018 •

edited

Loading

mistercrunch commented Dec 9, 2018

neildunlop commented Oct 8, 2019

hemantaggarwal commented Dec 25, 2019 •

edited

Loading

cachafla commented Jan 14, 2020

hemantaggarwal commented Jan 15, 2020

ispulkit commented Jun 29, 2020

Google spreadsheets #5915

Google spreadsheets #5915

Conversation

betodealmeida commented Sep 17, 2018 • edited Loading

betodealmeida commented Sep 17, 2018

betodealmeida Sep 17, 2018

Choose a reason for hiding this comment

hughhhh Sep 19, 2018

Choose a reason for hiding this comment

hughhhh Sep 19, 2018

Choose a reason for hiding this comment

hughhhh Sep 19, 2018

Choose a reason for hiding this comment

betodealmeida commented Oct 30, 2018

codecov-io commented Dec 7, 2018 • edited Loading

Codecov Report

mistercrunch commented Dec 9, 2018

neildunlop commented Oct 8, 2019

hemantaggarwal commented Dec 25, 2019 • edited Loading

cachafla commented Jan 14, 2020

hemantaggarwal commented Jan 15, 2020

ispulkit commented Jun 29, 2020

betodealmeida commented Sep 17, 2018 •

edited

Loading

codecov-io commented Dec 7, 2018 •

edited

Loading

hemantaggarwal commented Dec 25, 2019 •

edited

Loading