Merge pull request #51 from quantumblacklabs/release/0.14.1

Release 0.14.1
kedro-org · May 31, 2019 · 5faa415 · 5faa415
2 parents 95ba3de + a95617f
commit 5faa415
Show file tree

Hide file tree

Showing 57 changed files with 1,087 additions and 232 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -37,6 +37,12 @@ jobs:
           name: Run all end to end tests
           command: |
             make e2e-tests
+       - run:
+           name: Check docs build on Python 3.6+
+           command: |
+             if python -c "import sys; sys.exit(sys.version_info[:2] == (3, 5))" ; then
+                 make build-docs
+             fi
        - run:
           name: Check for legal compliance
           command: |

diff --git a/.isort.cfg b/.isort.cfg
@@ -7,4 +7,5 @@ force_grid_wrap=0
 use_parentheses=True
 line_length=88
 skip_glob=**/template/**
-known_first_party=kedro
+known_first_party=features,kedro,tests
+default_section=THIRDPARTY
diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -0,0 +1,30 @@
+# .readthedocs.yml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  builder: html
+  configuration: docs/conf.py
+
+# Build documentation with MkDocs
+# mkdocs:
+# configuration: mkdocs.yml
+
+# Optionally build your docs in additional formats such as PDF and ePub
+#formats: all
+
+# Optionally set the version of Python and requirements required to build your docs
+python:
+  version: 3.6
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs
+    - requirements: test_requirements.txt
+
+
diff --git a/README.md b/README.md
@@ -1,13 +1,13 @@
-![Kedro Logo Banner](https://github.com/quantumblacklabs/kedro/blob/master/img/kedro_banner.jpg)
+![Kedro Logo Banner](https://raw.githubusercontent.com/quantumblacklabs/kedro/master/img/kedro_banner.jpg)
 
 `develop` | `master`  
 ----------|---------  
-[![CircleCI](https://circleci.com/gh/quantumblacklabs/kedro/tree/develop.svg?style=svg)](https://circleci.com/gh/quantumblacklabs/kedro/tree/develop) | [![CircleCI](https://circleci.com/gh/quantumblacklabs/kedro/tree/master.svg?style=svg)](https://circleci.com/gh/quantumblacklabs/kedro/tree/master)
+[![CircleCI](https://circleci.com/gh/quantumblacklabs/kedro/tree/develop.svg?style=shield)](https://circleci.com/gh/quantumblacklabs/kedro/tree/develop) | [![CircleCI](https://circleci.com/gh/quantumblacklabs/kedro/tree/master.svg?style=shield)](https://circleci.com/gh/quantumblacklabs/kedro/tree/master)
 
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![Python Version](https://img.shields.io/pypi/pyversions/kedro.svg)](https://pypi.org/project/kedro/)
+[![Python Version](https://img.shields.io/badge/python-3.5%20%7C%203.6%20%7C%203.7-blue.svg)](https://pypi.org/project/kedro/)
 [![PyPI version](https://badge.fury.io/py/kedro.svg)](https://pypi.org/project/kedro/)
-[![Documentation](https://img.shields.io/badge/docs-passing-brightgreen.svg)](https://kedro.readthedocs.io/)
+[![Documentation](https://readthedocs.org/projects/kedro/badge/?version=latest)](https://kedro.readthedocs.io/)
 [![Code Style: Black](https://img.shields.io/badge/code%20style-black-black.svg)](https://github.com/ambv/black)
 
 # What is Kedro?
@@ -62,7 +62,7 @@ For more detailed installation instructions, including how to setup Python virtu
   - Kedro-Docker, a tool for packing and shipping Kedro projects within containers
 - Kedro can be deployed locally, on-premise and cloud (AWS, Azure and GCP) servers, or clusters (EMR, Azure HDinsight, GCP and Databricks)
 
-![Kedro-Viz Pipeline Visualisation](https://github.com/quantumblacklabs/kedro/blob/master/img/pipeline_visualisation.png)
+![Kedro-Viz Pipeline Visualisation](https://raw.githubusercontent.com/quantumblacklabs/kedro/master/img/pipeline_visualisation.png)
 *Random pipeline visualisation using Kedro-Viz (coming soon)*
 
 ## How do I use Kedro?

diff --git a/RELEASE.md b/RELEASE.md
@@ -1,12 +1,25 @@
+# Release 0.14.1
+
+## Major features and improvements
+* New I/O module `HDFS3DataSet`.
+
+## Bug fixes and other changes
+* Improved API docs.
+* Template `run.py` will throw a warning instead of error if `credentials.yml`
+  not present.
+
+## Breaking changes to the API
+None
+
 # Release 0.14.0:
 
 The initial release of Kedro.
 
 ## Thanks to our main contributors
 
-[Nikolaos Tsaousis](https://github.com/tsanikgr), [Ivan Danov](https://github.com/idanov), [Dmitrii Deriabin](https://github.com/DmitryDeryabin), [Gordon Wrigley](https://github.com/tolomea), [Yetunde Dada](https://github.com/yetudada), [Nasef Khan](https://github.com/nakhan98), [Kiyohito Kunii](https://github.com/921kiyo), [Nikolaos Kaltsas](https://github.com/nikos-kal), [Meisam Emamjome](https://github.com/misamae), [Peteris Erins](https://github.com/Pet3ris), [Lorena Balan](https://github.com/lorenabalan)
+[Nikolaos Tsaousis](https://github.com/tsanikgr), [Ivan Danov](https://github.com/idanov), [Dmitrii Deriabin](https://github.com/DmitryDeryabin), [Gordon Wrigley](https://github.com/tolomea), [Yetunde Dada](https://github.com/yetudada), [Nasef Khan](https://github.com/nakhan98), [Kiyohito Kunii](https://github.com/921kiyo), [Nikolaos Kaltsas](https://github.com/nikos-kal), [Meisam Emamjome](https://github.com/misamae), [Peteris Erins](https://github.com/Pet3ris), [Lorena Balan](https://github.com/lorenabalan), [Richard Westenra](https://github.com/richardwestenra)
 
-## Thanks for supporting contributions 
+## Thanks for supporting contributions
 
 Jo Stichbury, Aris Valtazanos, Fabian Peters, Guilherme Braccialli, Joel Schwarzmann, Miguel Beltre, Mohammed ElNabawy, Deepyaman Datta, Shubham Agrawal, Oleg Andreyev, Mayur Chougule, William Ashford, Ed Cannon, Nikhilesh Nukala, Sean Bailey, Vikram Tegginamath, Thomas Huijskens, Musa Bilal
 

diff --git a/..._api_docs/_templates/autosummary/base.rst → docs/_templates/autosummary/base.rst b/..._api_docs/_templates/autosummary/base.rst → docs/_templates/autosummary/base.rst
diff --git a/docs/_templates/autosummary/class.rst b/docs/_templates/autosummary/class.rst
@@ -0,0 +1,33 @@
+{{ fullname | escape | underline }}
+
+.. currentmodule:: {{ module }}
+
+.. autoclass:: {{ objname }}
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :inherited-members:
+
+   {% block attributes %}
+   {% if attributes %}
+   .. rubric:: Attributes
+
+   .. autosummary::
+      {% for item in all_attributes %}
+      {%- if not item.startswith('_') %}
+      {{ name }}.{{ item }}
+      {%- endif -%}
+      {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
+   {% block methods %}
+   {% if methods %}
+   .. rubric:: Methods
+
+   .. autosummary::
+      {% for item in methods %}
+      {{ name }}.{{ item }}
+      {%- endfor %}
+    {% endif %}
+   {% endblock %}
diff --git a/docs/_templates/autosummary/module.rst b/docs/_templates/autosummary/module.rst
@@ -0,0 +1,58 @@
+{{ fullname | escape | underline }}
+
+.. rubric:: Description
+
+.. automodule:: {{ fullname }}
+
+   {% block public_modules %}
+   {% if public_modules %}
+   .. rubric:: Modules
+
+   .. autosummary::
+      :toctree:
+      :template: autosummary/module.rst
+   {% for item in public_modules %}
+      {{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
+   {% block functions %}
+   {% if functions %}
+   .. rubric:: Functions
+
+   .. autosummary::
+      :toctree:
+   {% for item in functions %}
+      {%- if not item.startswith('_') %}
+      {{ item }}
+      {% endif %}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
+   {% block classes %}
+   {% if classes %}
+   .. rubric:: Classes
+
+   .. autosummary::
+      :toctree:
+      :template: autosummary/class.rst
+   {% for item in classes %}
+      {{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
+
+   {% block exceptions %}
+   {% if exceptions %}
+   .. rubric:: Exceptions
+
+   .. autosummary::
+      :toctree:
+      :template: autosummary/class.rst
+   {% for item in exceptions %}
+      {{ item }}
+   {%- endfor %}
+   {% endif %}
+   {% endblock %}
diff --git a/docs/build-docs.sh b/docs/build-docs.sh
@@ -38,9 +38,8 @@ rm -rf docs/tmp-build-artifacts
 cp -a docs/source docs/tmp-build-artifacts
 mv docs/tmp-build-artifacts/05_api_docs/* docs/tmp-build-artifacts/
 rm -rf docs/tmp-build-artifacts/05_api_docs/
-sphinx-apidoc --module-first -o docs/tmp-build-artifacts kedro
 rm -rf docs/build
 mkdir -p docs/build/html/_static
 cp -r docs/source/css docs/build/html/_static
-sphinx-build docs/tmp-build-artifacts docs/build/html -c docs/ -E -a
+sphinx-build -c docs/ -W -E -a -j auto docs/tmp-build-artifacts docs/build/html
 rm -rf docs/tmp-build-artifacts
diff --git a/docs/conf.py b/docs/conf.py
@@ -14,15 +14,19 @@
 # serve to show the default.
 
 import importlib
+import os
 import re
+import shutil
 import sys
+from distutils.dir_util import copy_tree
 from inspect import getmembers, isclass, isfunction
+from pathlib import Path
 from typing import List, Tuple
 
 from click import secho, style
+from recommonmark.transform import AutoStructify
 
 from kedro import __version__ as release
-from recommonmark.transform import AutoStructify
 
 # -- Project information -----------------------------------------------------
 
@@ -33,8 +37,12 @@
 # The short X.Y version.
 version = re.match(r"^([0-9]+\.[0-9]+).*", release).group(1)
 
+
 # -- General configuration ---------------------------------------------------
 
+on_rtd = os.getenv("READTHEDOCS") == "True"  # running on ReadTheDocs
+
+
 # If your documentation needs a minimal Sphinx version, state it here.
 #
 # needs_sphinx = '1.0'
@@ -83,7 +91,10 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path .
-exclude_patterns = ["**cli*", "**.ipynb_checkpoints", "_templates", "modules.rst"]
+exclude_patterns = ["**.ipynb_checkpoints", "_templates", "modules.rst"]
+if on_rtd:
+    exclude_patterns.append("source")
+
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = "sphinx"
@@ -94,6 +105,8 @@
 # a list of builtin themes.
 #
 html_theme = "sphinx_rtd_theme"
+here = Path(__file__).parent.absolute()
+html_logo = str(here / "kedro_logo.svg")
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -371,7 +384,22 @@ def skip(app, what, name, obj, skip, options):
     return skip
 
 
+def _prepare_cwd_for_rtd(app, config):
+    """Get current working directory to the state expected
+    by the ReadTheDocs builder. Shortly, it does the same as
+    ./build-docs.sh script except not running `sphinx-build` step."""
+    copy_tree(str(here / "source"), str(here))
+    copy_tree(str(here / "05_api_docs"), str(here))
+    shutil.rmtree(str(here / "05_api_docs"))
+    shutil.rmtree(str(here / "_build"), ignore_errors=True)
+    _css = here / "_build" / "html" / "_static" / "css"
+    copy_tree(str(here / "css"), str(_css))
+    shutil.rmtree(str(here / "css"))
+
+
 def setup(app):
+    if on_rtd:
+        app.connect("config-inited", _prepare_cwd_for_rtd)
     app.connect("autodoc-process-docstring", autodoc_process_docstring)
     app.connect("autodoc-skip-member", skip)
     app.add_stylesheet("css/qb1-sphinx-rtd.css")

diff --git a/docs/kedro_logo.svg b/docs/kedro_logo.svg
diff --git a/docs/source/02_getting_started/03_new_project.md b/docs/source/02_getting_started/03_new_project.md
@@ -55,5 +55,10 @@ If you want to work with a Kedro project that has already been created, you may
 To make sure you have the required dependencies, run (in your virtual environment):
 
 ```bash
-pip install -r src/requirements.txt
+pip install -U -r src/requirements.txt
 ```
+Or run the following if you have Kedro installed in your virtual environment:
+```bash
+kedro install
+```
+
diff --git a/docs/source/04_user_guide/01_setting_up_vscode.md b/docs/source/04_user_guide/01_setting_up_vscode.md
@@ -1,6 +1,6 @@
 # Setting up Visual Studio Code
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 Start by opening a new project directory in VS Code and installing the Python plugin under **Tools and languages**:
 

diff --git a/docs/source/04_user_guide/02_setting_up_pycharm.md b/docs/source/04_user_guide/02_setting_up_pycharm.md
@@ -1,6 +1,6 @@
 # Setting up PyCharm
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 This section will present a quick guide on how to configure [PyCharm](https://www.jetbrains.com/pycharm/) as a development environment for working on Kedro projects.
 

diff --git a/docs/source/04_user_guide/03_configuration.md b/docs/source/04_user_guide/03_configuration.md
@@ -1,6 +1,6 @@
 # Configuration
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 This section contains detailed information about configuration. You may also want to consult the relevant API documentation on [kedro.config](/kedro.config.rst).
 

diff --git a/docs/source/04_user_guide/04_data_catalog.md b/docs/source/04_user_guide/04_data_catalog.md
@@ -1,6 +1,6 @@
 # The Data Catalog
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 This section introduces `catalog.yml`, the project-shareable Data Catalog. The file is located in `conf/base` and is a registry of all data sources available for use by a project; it manages loading and saving of data. 
 

diff --git a/docs/source/04_user_guide/05_nodes_and_pipelines.md b/docs/source/04_user_guide/05_nodes_and_pipelines.md
@@ -1,6 +1,6 @@
 # Nodes and pipelines
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 In this section we introduce pipelines and nodes.
 
 Relevant API documentation:

diff --git a/docs/source/04_user_guide/06_logging.md b/docs/source/04_user_guide/06_logging.md
@@ -1,5 +1,6 @@
 # Logging
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 Kedro uses, and facilitates the use of Python’s `logging` library, by providing a default logging configuration. This can be found in `conf/base/logging.yml` in every project generated using Kedro’s CLI `kedro new` command.
 

diff --git a/docs/source/04_user_guide/07_advanced_io.md b/docs/source/04_user_guide/07_advanced_io.md
@@ -1,6 +1,6 @@
 # Advanced IO
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 In this tutorial, you will learn about advanced uses of the [Kedro IO](/kedro.io.rst) module and understand the underlying implementation.
 
@@ -176,6 +176,7 @@ Currently the following datasets support versioning:
 - `CSVLocalDataSet`
 - `CSVS3DataSet`
 - `HDFLocalDataSet`
+- `HDFS3DataSet`
 - `JSONLocalDataSet`
 - `ParquetLocalDataSet`
 - `PickleLocalDataSet`

diff --git a/docs/source/04_user_guide/08_pyspark.md b/docs/source/04_user_guide/08_pyspark.md
@@ -1,10 +1,10 @@
 # Working with PySpark
 
-> *Note:* This documentation is based on `Kedro 0.14.0`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
 
 In this tutorial we explain how to work with `PySpark` in a Kedro pipeline.
 
-Relevant API documentation: [SparkDataSet](/kedro.contrib.io.pyspark)
+Relevant API documentation: [SparkDataSet](/kedro.contrib.io.pyspark.SparkDataSet), [SparkJDBCDataSet](/kedro.contrib.io.pyspark.SparkJDBCDataSet)
 
 ## Initialising a `SparkSession`
 
@@ -58,7 +58,7 @@ Since `SparkSession` is a [singleton](https://python-3-patterns-idioms-test.read
 
 Having created a `SparkSession`, you can load your data using `PySpark`'s [DataFrameReader](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader). 
 
-To do so, please use the provided [SparkDataSet](/kedro.contrib.io.pyspark):
+To do so, please use the provided [SparkDataSet](/kedro.contrib.io.pyspark.SparkDataSet):
 
 ### Code API
 

diff --git a/docs/source/04_user_guide/09_developing_plugins.md b/docs/source/04_user_guide/09_developing_plugins.md
@@ -1,5 +1,7 @@
 # Developing Kedro plugins
 
+> *Note:* This documentation is based on `Kedro 0.14.1`, if you spot anything that is incorrect then please create an [issue](https://github.com/quantumblacklabs/kedro/issues) or pull request.
+
 ## Overview
 
 Kedro uses various entry points in the [`pkg_resources` entry_point system](https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins) to provide plugin functionality.