Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document --backup import behaviour #1788

Closed
Closed
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
8981b40
Document --backup import behaviour
charlesbaynham Sep 20, 2020
ded9bef
Restyled by prettier
restyled-commits Sep 20, 2020
e97d2da
Update content/docs/command-reference/import.md
jorgeorpinel Sep 20, 2020
5eb6ef7
Update content/docs/command-reference/import.md
jorgeorpinel Sep 20, 2020
f6ef819
Update content/docs/command-reference/import.md
jorgeorpinel Sep 20, 2020
a707366
Update content/docs/command-reference/import.md
jorgeorpinel Sep 20, 2020
a7aa282
Update content/docs/command-reference/import.md
jorgeorpinel Sep 20, 2020
15db349
Update content/docs/command-reference/import.md
jorgeorpinel Sep 20, 2020
476eb79
Apply suggestions from code review
charlesbaynham Sep 20, 2020
a7f4fd0
Move backup mode section and demote to h3
charlesbaynham Sep 20, 2020
6423f0f
Link back to backup mode section from dvc files page
charlesbaynham Sep 20, 2020
72e86bd
Links
charlesbaynham Sep 20, 2020
11e0286
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
880ef7a
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
a3c4201
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
d7549ef
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
dc7ea4b
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
ba32153
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
356f9dc
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
ab459bb
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
bd9fe56
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
60c69c5
Update import.md
charlesbaynham Sep 21, 2020
6ab19b0
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
1772f54
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
788b1c5
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
8b809c0
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
bec8136
Update content/docs/command-reference/import.md
jorgeorpinel Sep 21, 2020
b11a3ec
Update content/docs/user-guide/dvc-files-and-directories.md
jorgeorpinel Sep 21, 2020
732020a
Update content/docs/user-guide/dvc-files-and-directories.md
jorgeorpinel Sep 21, 2020
576e2bf
Update content/docs/command-reference/import.md
jorgeorpinel Sep 22, 2020
9e95f2c
Update content/docs/command-reference/import.md
jorgeorpinel Sep 22, 2020
df08768
Update content/docs/command-reference/import.md
jorgeorpinel Sep 22, 2020
747aee8
Update 'backup' to 'store' and add error link
charlesbaynham Oct 1, 2020
6da47c7
Allow merge with master
charlesbaynham Oct 1, 2020
1b52cdf
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
3e257cb
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
e8c036a
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
a6b9810
Update content/docs/user-guide/dvc-files-and-directories.md
jorgeorpinel Oct 1, 2020
7607697
Update content/docs/user-guide/troubleshooting.md
jorgeorpinel Oct 1, 2020
744db79
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
a02c827
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
73dd979
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
e98f488
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
4bef59b
Update content/docs/command-reference/import.md
jorgeorpinel Oct 1, 2020
c8d447c
Update content/docs/user-guide/dvc-files-and-directories.md
jorgeorpinel Oct 2, 2020
322ffa0
Update content/docs/user-guide/troubleshooting.md
jorgeorpinel Oct 2, 2020
4b48347
Update content/docs/user-guide/troubleshooting.md
jorgeorpinel Oct 2, 2020
1d29252
Update content/docs/user-guide/troubleshooting.md
jorgeorpinel Oct 2, 2020
1a39418
Update content/docs/user-guide/troubleshooting.md
jorgeorpinel Oct 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion content/docs/command-reference/import.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ the import.
## Synopsis

```usage
usage: dvc import [-h] [-q | -v] [-o <path>] [--rev <commit>] url path
usage: dvc import [-h] [-q | -v]
[-o <path>] [--file <filename>] [--rev <commit>] [--store]
url path

positional arguments:
url Location of DVC or Git repository to download from
Expand Down Expand Up @@ -74,6 +76,25 @@ Note that import stages are considered always
they won't be updated. Use `dvc update` to update the downloaded data artifact
from the source repo.

### Enabling remote storage

To avoid duplicating large files, imported data is not pushed to
[remote storage](/doc/command-reference/remote) by default (see `dvc push`).
This means that `dvc pull`, `dvc fetch`, and `dvc repro` retrieve imports from
their source (found in the `deps` field of the `.dvc` file). So if the original
source of an import is deleted, then the files are not backed up anywhere, and
data may be lost.

As an alternative, you can use the `--store` option so the import is pushed to a
DVC remote when `dvc push` is run. This also means that `dvc pull/fetch/repro`
no longer access import sources, going to the remote instead, like with regular
project data tracked with `dvc add`. You can still use `dvc update` to get the
latest import data from its source.

Remote storage is enabled with a `store: true` key/value pair in the `outs`
section of the import `.dvc` file. To switch to regular import behavior, change
its value to `false` manually, or re-import the data without the `--store` option.

## Options

- `-o <path>`, `--out <path>` - specify a path to the desired location in the
Expand All @@ -91,6 +112,10 @@ from the source repo.
> [Importing and updating fixed revisions](#example-importing-and-updating-fixed-revisions)
> example below).

- `-s`, `--store` - [enables remote storage](#enabling-remote-storage) for the
import, so it can be pushed and pulled like regular project data (e.g. in
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
case the original source is lost).

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down
4 changes: 4 additions & 0 deletions content/docs/user-guide/dvc-files-and-directories.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,10 @@ An _output entry_ (`outs`) consists of these fields:
_checksum_ for HDFS.
- `cache`: Whether or not this file or directory is <abbr>cached</abbr> (`true`
by default, if not present). See the `--no-commit` option of `dvc add`.
- `store`: Whether or not this file or directory is pushed and pulled to/from
[remote storage](/doc/command-reference/remote) (`true` by default for all
outputs except for imports, where it is `false` by default). See
[import storage](/doc/command-reference/import#enabling-remote-storage).
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

A _dependency entry_ (`deps`) consists of these fields:

Expand Down
13 changes: 13 additions & 0 deletions content/docs/user-guide/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,16 @@ Unable to detect supported link types, as the
[cache directory](/doc/command-reference/config#cache) doesn't exist. It is
usually created automatically by DVC commands that need it, but you can create
it manually (e.g. `mkdir .dvc/cache`) to enable this check.

## Checkout failed with `store: false` files {#checkout-failed-no-store}
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

Unable to checkout this output since it is not present in the cache and has been
marked to not be stored on remotes.

If `store: false` is added to the `outs` section of a `.dvc` file, that output
will not be stored on remotes when `dvc push` is used, and dvc will not attempt
to retreive it when `dvc pull` or `dvc fetch` is called. This error means that
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
the file is not already in the local cache, so dvc cannot find it. You must
provide the file locally since it is not tracked on dvc-managed remote storage.

See [import storage](/doc/command-reference/import#enabling-remote-storage).