Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix dbt deps issues (#778 #994 #895) #1110

Merged
merged 10 commits into from
Nov 14, 2018
Merged

Fix dbt deps issues (#778 #994 #895) #1110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7757c85
add some helpful exceptions about command running
Nov 5, 2018
59b6f78
Raise an exception on rc!=0 in run_cmd, raise more specific exception…
Nov 5, 2018
ec46606
Tests, fixup ci
Nov 5, 2018
e866caa
add support for cross-drive moves
Nov 5, 2018
531d7c6
use environment variables or per-run temp directories to assign the t…
Nov 5, 2018
412b165
fix issue where we incorrectly logged stack traces
Nov 6, 2018
3773843
in deps, when "git" is not in the path, link users to help docs
Nov 13, 2018
3515422
Merge branch 'dev/grace-kelly' into fix/dbt-deps-issues
Nov 13, 2018
d35e549
Handle cross-drive windows permissions issues by undoing git's readon…
Nov 14, 2018
8840996
user feedback: log download directory in dbt deps
Nov 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 13 additions & 43 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,113 +1,83 @@
version: 2
jobs:
unit:
docker: &py36_postgres
- image: python:3.6
docker: &test_and_postgres
- image: fishtownjacob/test-container
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this built by Dockerfile in this repo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I use this gross and lame shell script to do it:

#!/bin/bash
set -xeuo pipefail
commit_message="$1"
shift
if [[ -z ${commit_message} ]]; then
    echo 'no commit message'
    exit 1
fi
docker build --tag 'test-container:latest' .
container_id="$(docker run -it test-container:latest hostname | tr -d '[[:space:]]')"
docker commit -a 'Jacob Beck <[email protected]>' -m "${commit_message}" ${container_id} test-container
docker tag test-container:latest fishtownjacob/test-container:latest
docker push fishtownjacob/test-container:latest

- image: postgres
name: database
environment: &pgenv
POSTGRES_USER: "root"
POSTGRES_PASSWORD: "password"
POSTGRES_DB: "dbt"
steps:
- checkout
- run: apt-get update && apt-get install -y python-dev python3-dev postgresql
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run: &setupdb
name: Setup postgres
command: bash test/setup_db.sh
environment:
PGHOST: 127.0.0.1
PGHOST: database
PGUSER: root
PGPASSWORD: password
PGDATABASE: postgres
- run: tox -e pep8,unit-py27,unit-py36
integration-postgres-py36:
docker: *py36_postgres
docker: *test_and_postgres
steps:
- checkout
- run: apt-get update && apt-get install -y python3-dev postgresql
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run: *setupdb
- run:
name: Run tests
command: tox -e integration-postgres-py36
integration-snowflake-py36:
docker: &py36
- image: python:3.6
docker: &test_only
- image: fishtownjacob/test-container
steps:
- checkout
- run: apt-get update && apt-get install -y python3-dev
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run:
name: Run tests
command: tox -e integration-snowflake-py36
no_output_timeout: 1h
integration-redshift-py36:
docker: *py36
docker: *test_only
steps:
- checkout
- run: apt-get update && apt-get install -y python3-dev postgresql
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run:
name: Run tests
command: tox -e integration-redshift-py36
integration-bigquery-py36:
docker: *py36
docker: *test_only
steps:
- checkout
- run: apt-get update && apt-get install -y python3-dev
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run:
name: Run tests
command: tox -e integration-bigquery-py36
integration-postgres-py27:
docker: &py27_postgres
- image: python:2.7
- image: postgres
environment: *pgenv
docker: *test_and_postgres
steps:
- checkout
- run: apt-get update && apt-get install -y python-dev postgresql
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run: *setupdb
- run:
name: Run tests
command: tox -e integration-postgres-py27
integration-snowflake-py27:
docker: &py27
- image: python:2.7
docker: *test_only
steps:
- checkout
- run: apt-get update && apt-get install -y python-dev
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run:
name: Run tests
command: tox -e integration-snowflake-py27
no_output_timeout: 1h
integration-redshift-py27:
docker: *py27
docker: *test_only
steps:
- checkout
- run: apt-get update && apt-get install -y python-dev postgresql
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run:
name: Run tests
command: tox -e integration-redshift-py27
integration-bigquery-py27:
docker: *py27
docker: *test_only
steps:
- checkout
- run: apt-get update && apt-get install -y python-dev
- run: echo 127.0.0.1 database | tee -a /etc/hosts
- run: pip install virtualenvwrapper tox
- run:
name: Run tests
command: tox -e integration-bigquery-py27
Expand Down
1 change: 1 addition & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*
10 changes: 6 additions & 4 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
FROM python:3.6

RUN apt-get update

RUN apt-get install -y python-pip netcat
RUN apt-get install -y python-dev python3-dev
# do these on one line so changes trigger apt-get update
RUN apt-get update && \
apt-get install -y python-pip netcat python-dev python3-dev postgresql

RUN pip install pip --upgrade
RUN pip install virtualenv
RUN pip install virtualenvwrapper
RUN pip install tox

RUN useradd -mU dbt_test_user
USER dbt_test_user

WORKDIR /usr/src/app
RUN cd /usr/src/app
33 changes: 21 additions & 12 deletions dbt/clients/git.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,7 @@ def list_tags(cwd):
return tags


def checkout(cwd, repo, branch=None):
if branch is None:
branch = 'master'

def _checkout(cwd, repo, branch):
logger.debug(' Checking out branch {}.'.format(branch))

run_cmd(cwd, ['git', 'remote', 'set-branches', 'origin', branch])
Expand All @@ -44,12 +41,17 @@ def checkout(cwd, repo, branch=None):
spec = 'origin/{}'.format(branch)

out, err = run_cmd(cwd, ['git', 'reset', '--hard', spec])
stderr = err.decode('utf-8').strip()
return out, err

if stderr.startswith('fatal:'):
dbt.exceptions.bad_package_spec(repo, branch, stderr)
else:
return out, err

def checkout(cwd, repo, branch=None):
if branch is None:
branch = 'master'
try:
return _checkout(cwd, repo, branch)
except dbt.exceptions.CommandResultError as exc:
stderr = exc.stderr.decode('utf-8').strip()
dbt.exceptions.bad_package_spec(repo, branch, stderr)


def get_current_sha(cwd):
Expand All @@ -64,9 +66,16 @@ def remove_remote(cwd):

def clone_and_checkout(repo, cwd, dirname=None, remove_git_dir=False,
branch=None):
_, err = clone(repo, cwd, dirname=dirname, remove_git_dir=remove_git_dir)
exists = re.match("fatal: destination path '(.+)' already exists",
err.decode('utf-8'))
exists = None
try:
_, err = clone(repo, cwd, dirname=dirname,
remove_git_dir=remove_git_dir)
except dbt.exceptions.CommandResultError as exc:
err = exc.stderr.decode('utf-8')
exists = re.match("fatal: destination path '(.+)' already exists", err)
if not exists: # something else is wrong, raise it
raise

directory = None
start_sha = None
if exists:
Expand Down
Loading