initialize the repository

.flake8

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[flake8]
+max-line-length = 88
+ignore = E203, E266, E501, W503
+exclude =
+  # Exclude generated code.
+  .dev/**
+  .test/**
+  **/proto/**
+  **/gapic/**
+  **/services/**
+  **/types/**
+  **/.terraform/**
+  *_pb2.py
+
+  # Standard linting exemptions.
+  **/.nox/**
+  __pycache__,
+  .git,
+  *.pyc,
+  conf.py

.github/CONTRIBUTING.md

@@ -0,0 +1,28 @@
+# How to Contribute
+
+We'd love to accept your patches and contributions to this project. There are
+just a few small guidelines you need to follow.
+
+## Contributor License Agreement
+
+Contributions to this project must be accompanied by a Contributor License
+Agreement. You (or your employer) retain the copyright to your contribution;
+this simply gives us permission to use and redistribute your contributions as
+part of the project. Head over to <https://cla.developers.google.com/> to see
+your current agreements on file or to sign a new one.
+
+You generally only need to submit a CLA once, so if you've already submitted one
+(even if it was for a different project), you probably don't need to do it
+again.
+
+## Code reviews
+
+All submissions, including submissions by project members, require review. We
+use GitHub pull requests for this purpose. Consult
+[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
+information on using pull requests.
+
+## Community Guidelines
+
+This project follows [Google's Open Source Community
+Guidelines](https://opensource.google/conduct/).

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,16 @@
+
+## Description
+
+Tell us about what's going on or what your suggestions are.
+
+Some helpful details or guidelines:
+
+- Include the steps to reproduce an error or issue you're experiencing, along with your environment details
+- Share code snippets, logs, diagrams, or other content that can help us further understand your situation
+- Avoid sharing sensitive, private information
+- Label your issue to indicate which component(s) this request is related to.
+
+
+## Checklist
+- [ ] I created this issue in accordance with the [Code of Conduct](https://github.com/GoogleCloudPlatform/public-datasets-pipelines/blob/main/CODE_OF_CONDUCT.md).
+- [ ] This issue is appropriately labeled.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,14 @@
+## Description
+
+Based on #<ISSUE-NUMBER>
+
+Note: It's recommended to open an issue first for context and discussion.
+
+## Checklist
+- [ ] Tests pass.
+- [ ] Linters pass.
+- [ ] Please merge this PR for me once it is approved.
+- [ ] If this PR adds/edits/deletes a feature, I have updated the [`README`](https://github.com/GoogleCloudPlatform/public-datasets-pipelines/blob/main/README.md) accordingly.
+- [ ] If this PR adds a new dataset or pipeline, it was reviewed and approved by the Google Cloud Public Datasets team beforehand.
+- [ ] If this PR adds a new dataset or pipeline, I put all my code inside  `datasets/<YOUR-DATASET>` and nothing outside of that directory.
+- [ ] If this PR adds a new dataset or pipeline, I updated the [`CONTRIBUTORS` file](https://github.com/GoogleCloudPlatform/public-datasets-pipelines/blob/main/CONTRIBUTORS) with the contributors or maintainers of that new dataset.

.github/workflows/python-lint.yaml

@@ -0,0 +1,19 @@
+name: Lint Python
+on: [pull_request]
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/setup-python@v2
+      - name: Install dependencies
+        run:
+          python -m pip install flake8 isort
+      - uses: psf/[email protected]
+        with:
+          args: ". --check"
+      - name: Run Flake8 Linter
+        run: flake8
+      - name: Run isort
+        run: isort --check-only --profile black --diff .

.github/workflows/terraform-lint.yaml

@@ -0,0 +1,13 @@
+name: Lint Terraform
+on: [pull_request]
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: hashicorp/setup-terraform@v1
+        with:
+          terraform_version: 0.12.25
+      - name: Terraform fmt
+        run: terraform fmt -check -recursive
+        continue-on-error: true

.github/workflows/unit-tests.yaml

@@ -0,0 +1,25 @@
+name: Run unit tests
+on: [pull_request]
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8]
+    steps:
+      - uses: actions/checkout@v2
+      - uses: hashicorp/setup-terraform@v1
+        with:
+          terraform_version: 0.12.25
+      - name: Setup Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install pipenv
+        run: pip install pipenv
+      - name: Install dependencies
+        run: pipenv install --ignore-pipfile --dev
+      - name: Initialize Airflow
+        run: pipenv run airflow db init
+      - name: Run tests
+        run: pipenv run python -m pytest -v

.github/workflows/yaml-lint.yaml

@@ -0,0 +1,13 @@
+name: Lint YAML
+on: [pull_request]
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Setup Python
+        uses: actions/setup-python@v2
+      - name: Install yamllint
+        run: python -m pip install yamllint
+      - name: Lint
+        run: yamllint .

.gitignore

@@ -0,0 +1,27 @@
+# python caches
+__pycache__
+
+# venv
+lib
+lib64
+share
+bin
+pyvenv.cfg
+
+# terraform caches
+.terraform*
+terraform.tfstate
+.terragrunt-cache
+
+# airflow cache
+tmp
+
+# env or editor config
+.python-version
+.vscode
+
+# generated files and folders
+.dev
+.test
+.staging
+.prod

.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+# Copyright 2021 Google LLC
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+#     https://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+repos:
+  - repo: https://github.com/gruntwork-io/pre-commit
+    rev: v0.1.12
+    hooks:
+      - id: terraform-fmt
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.4.0
+    hooks:
+      - id: check-yaml
+  - repo: https://github.com/psf/black
+    rev: 20.8b1
+    hooks:
+      - id: black
+        name: black
+        language_version: python3
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: '3.8.4'
+    hooks:
+      - id: flake8
+  - repo: https://github.com/pycqa/isort
+    rev: '5.6.4'
+    hooks:
+      - id: isort
+        args: ["--profile", "black", "--filter-files"]

.yamllint.yaml

@@ -0,0 +1,23 @@
+# Copyright 2021 Google LLC
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+#     https://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+extends: default
+
+rules:
+  document-start: disable
+  line-length: disable
+  truthy: disable
+
+ignore: |
+  **/.terraform

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to Adler Santos ([email protected]). All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq