Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge master #782

Merged
merged 65 commits into from
Mar 23, 2016
Merged
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
452e274
Handle eventally consistent blob lists in storage ITs
mziccard Mar 2, 2016
622dcc9
Replace values().iterator() with iterateAll() in blob list ITs
mziccard Mar 3, 2016
50750d4
Merge pull request #704 from mziccard/fix-storage-its
mziccard Mar 3, 2016
8dd8786
Remove redundant throws from services method signatures
mziccard Mar 4, 2016
7eb957d
Merge pull request #712 from mziccard/remove-throws
aozarov Mar 4, 2016
c486452
Add NoAuthCredentials
Mar 6, 2016
8c353ec
Add javadoc
Mar 7, 2016
0c4af89
change NoAuthCredentials javadoc wording
Mar 7, 2016
d3224b3
Merge pull request #719 from ajkannan/add-no-auth
ajkannan Mar 7, 2016
cded234
Fix writes with 0 length
aozarov Mar 7, 2016
36ebabd
Fix test and issue #723
aozarov Mar 8, 2016
e70387d
Merge pull request #725 from aozarov/master
mziccard Mar 8, 2016
d930b4b
Remove BlobListOption.recursive option and fix delimiter handling
mziccard Mar 8, 2016
0f4a7a0
Release version 0.1.5
mziccard Mar 8, 2016
2611514
Merge pull request #729 from mziccard/master
mziccard Mar 8, 2016
1f1f4b0
Updating version in README files. [ci skip]
Mar 8, 2016
aa6c173
Update version to 0.1.6-SNAPSHOT
mziccard Mar 8, 2016
7c42116
Merge pull request #730 from mziccard/master
aozarov Mar 8, 2016
de9b0d6
Refactor Storage's delimiter support
mziccard Mar 9, 2016
5dd3c30
Merge pull request #726 from mziccard/fix-list-prefix
mziccard Mar 9, 2016
407c436
Create service-specific spi packages
mziccard Mar 10, 2016
fd7d84d
Reorder and add imports
mziccard Mar 10, 2016
0d38c67
Merge pull request #742 from mziccard/rename-spi
ajkannan Mar 11, 2016
a595f6a
Use groups to separate packages in javadoc's index
mziccard Mar 10, 2016
90d0917
Add group for testing packages
mziccard Mar 10, 2016
24a712a
Rename Main packages to API packages, reorder groups, add service's SPIs
mziccard Mar 10, 2016
ac1ba66
Add more detailed javadoc to Blob and Storage signUrl
mziccard Mar 11, 2016
1425513
Merge pull request #738 from mziccard/javadoc-group-packages
ajkannan Mar 11, 2016
8a3b4f2
Remove final from Blob
jean-philippe-martin Mar 11, 2016
ddd02aa
Merge pull request #744 from jean-philippe-martin/blob_notfinal
aozarov Mar 11, 2016
ddb1ade
Rephrase signUrl javadoc for better clarity
mziccard Mar 13, 2016
d97c188
Rename maxResults to pageSize
mziccard Mar 14, 2016
86e23d5
Fix flaky RemoteGcsHelperTest.testForceDeleteTimeout
mziccard Mar 14, 2016
9e5c173
Merge pull request #745 from mziccard/rename-max-results
mderka Mar 14, 2016
646f3c9
Merge pull request #747 from mziccard/master
aozarov Mar 14, 2016
4e0248f
Add better javadoc for signUrl examples
mziccard Mar 15, 2016
3b083a6
Merge pull request #743 from mziccard/signurl-javadoc
mziccard Mar 15, 2016
c2c6628
Add get, replace, and test for IAM
Mar 1, 2016
7d48ec7
Merge pull request #718 from ajkannan/get-set-test-iam
ajkannan Mar 15, 2016
8ae8a1b
Create base class for serialization tests
mziccard Mar 16, 2016
967c5c5
Add tests for restorable classes. Fix serialization issues
mziccard Mar 17, 2016
034432a
Remove serialization test for ApplicationDefaultAuthCredentials
mziccard Mar 17, 2016
ce59793
Add restorableObjects method to BaseSerializationTest
mziccard Mar 17, 2016
0a113cd
Remove default contentType from Storage's compose
mziccard Mar 18, 2016
8840b33
Remove default contentType from Bucket's create
mziccard Mar 18, 2016
5c4e288
Make NoAuthCredentials constructor private
mziccard Mar 18, 2016
19e0c6e
Add IamPolicy to SerializationTest
mziccard Mar 18, 2016
83ddb08
Add equals and hashCode to exceptions. Add exceptions to serializatio…
mziccard Mar 18, 2016
353d2db
Merge pull request #754 from mziccard/serialization-tests
aozarov Mar 18, 2016
22636e2
Remove checks for contentType in CopyRequest
mziccard Mar 18, 2016
3e5db7a
Refactor CopyRequest: remove targetId, add boolean overrideInfo
mziccard Mar 20, 2016
1855ae1
Add better javadoc to Storage.copy and CopyWriter
mziccard Mar 20, 2016
5dd0292
Rephrase RewriteChannel and Storage.copy javadoc. Add final to CopyWr…
mziccard Mar 20, 2016
fa64fa5
Merge pull request #762 from mziccard/remove-compose-content-type
mziccard Mar 20, 2016
3e9e1a0
IAM docs and functional methods for policies
Mar 16, 2016
c05e738
Add binding entries as necessary in IamPolicy.Builder
Mar 21, 2016
7bbe67f
Take care of minor changes
Mar 21, 2016
5fbb41c
Merge pull request #760 from ajkannan/iam-docs-and-cleanup
ajkannan Mar 21, 2016
3a97ae1
Make role and permission strings to allow for service-specific values
Mar 22, 2016
52bf6c1
remove varargs for testPermissions and other minor fixes
Mar 22, 2016
a78f0a7
Merge pull request #772 from ajkannan/support-more-iam-roles
ajkannan Mar 22, 2016
b7c6da4
Rename startPageToken to pageToken in Storage and BigQuery
mziccard Mar 22, 2016
689bbb4
Merge pull request #774 from mziccard/rename-next-page-token
ajkannan Mar 22, 2016
d974887
Merge remote-tracking branch 'upstream/master' into dns-alpha
mderka Mar 23, 2016
e946b76
Updated the version in pom.
mderka Mar 23, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove checks for contentType in CopyRequest
- update CopyRequest to hold both targetId and targetInfo
- avoid sending storage object in StorageRpc.rewrite when only targetId is set
- refactor CopyWriter and state
- add integration tests
  • Loading branch information
mziccard committed Mar 19, 2016

Verified

This commit was signed with the committer’s verified signature.
pradyunsg Pradyun Gedam
commit 22636e276005b809e7bef8cc869a38cba786b162
Original file line number Diff line number Diff line change
@@ -65,11 +65,11 @@ public class CopyWriter implements Restorable<CopyWriter> {
*
* @throws StorageException upon failure
*/
public BlobInfo result() {
public Blob result() {
while (!isDone()) {
copyChunk();
}
return BlobInfo.fromPb(rewriteResponse.result);
return Blob.fromPb(serviceOptions.service(), rewriteResponse.result);
}

/**
@@ -120,8 +120,12 @@ public RestorableState<CopyWriter> capture() {
serviceOptions,
BlobId.fromPb(rewriteResponse.rewriteRequest.source),
rewriteResponse.rewriteRequest.sourceOptions,
BlobInfo.fromPb(rewriteResponse.rewriteRequest.target),
BlobId.of(rewriteResponse.rewriteRequest.targetBucket,
rewriteResponse.rewriteRequest.targetName),
rewriteResponse.rewriteRequest.targetOptions)
.targetInfo(rewriteResponse.rewriteRequest.targetObject != null
? BlobInfo.fromPb(rewriteResponse.rewriteRequest.targetObject) : null)
.result(rewriteResponse.result != null ? BlobInfo.fromPb(rewriteResponse.result) : null)
.blobSize(blobSize())
.isDone(isDone())
.megabytesCopiedPerChunk(rewriteResponse.rewriteRequest.megabytesRewrittenPerCall)
@@ -132,12 +136,13 @@ public RestorableState<CopyWriter> capture() {

static class StateImpl implements RestorableState<CopyWriter>, Serializable {

private static final long serialVersionUID = 8279287678903181701L;
private static final long serialVersionUID = 1693964441435822700L;

private final StorageOptions serviceOptions;
private final BlobId source;
private final Map<StorageRpc.Option, ?> sourceOptions;
private final BlobInfo target;
private final BlobId targetId;
private final BlobInfo targetInfo;
private final Map<StorageRpc.Option, ?> targetOptions;
private final BlobInfo result;
private final long blobSize;
@@ -150,7 +155,8 @@ static class StateImpl implements RestorableState<CopyWriter>, Serializable {
this.serviceOptions = builder.serviceOptions;
this.source = builder.source;
this.sourceOptions = builder.sourceOptions;
this.target = builder.target;
this.targetId = builder.targetId;
this.targetInfo = builder.targetInfo;
this.targetOptions = builder.targetOptions;
this.result = builder.result;
this.blobSize = builder.blobSize;
@@ -165,8 +171,9 @@ static class Builder {
private final StorageOptions serviceOptions;
private final BlobId source;
private final Map<StorageRpc.Option, ?> sourceOptions;
private final BlobInfo target;
private final BlobId targetId;
private final Map<StorageRpc.Option, ?> targetOptions;
private BlobInfo targetInfo;
private BlobInfo result;
private long blobSize;
private boolean isDone;
@@ -176,14 +183,19 @@ static class Builder {

private Builder(StorageOptions options, BlobId source,
Map<StorageRpc.Option, ?> sourceOptions,
BlobInfo target, Map<StorageRpc.Option, ?> targetOptions) {
BlobId targetId, Map<StorageRpc.Option, ?> targetOptions) {
this.serviceOptions = options;
this.source = source;
this.sourceOptions = sourceOptions;
this.target = target;
this.targetId = targetId;
this.targetOptions = targetOptions;
}

Builder targetInfo(BlobInfo targetInfo) {
this.targetInfo = targetInfo;
return this;
}

Builder result(BlobInfo result) {
this.result = result;
return this;
@@ -220,15 +232,16 @@ RestorableState<CopyWriter> build() {
}

static Builder builder(StorageOptions options, BlobId source,
Map<StorageRpc.Option, ?> sourceOptions, BlobInfo target,
Map<StorageRpc.Option, ?> sourceOptions, BlobId targetId,
Map<StorageRpc.Option, ?> targetOptions) {
return new Builder(options, source, sourceOptions, target, targetOptions);
return new Builder(options, source, sourceOptions, targetId, targetOptions);
}

@Override
public CopyWriter restore() {
RewriteRequest rewriteRequest = new RewriteRequest(
source.toPb(), sourceOptions, target.toPb(), targetOptions, megabytesCopiedPerChunk);
RewriteRequest rewriteRequest = new RewriteRequest(source.toPb(), sourceOptions,
targetId.bucket(), targetId.name(), targetInfo != null ? targetInfo.toPb() : null,
targetOptions, megabytesCopiedPerChunk);
RewriteResponse rewriteResponse = new RewriteResponse(rewriteRequest,
result != null ? result.toPb() : null, blobSize, isDone, rewriteToken,
totalBytesCopied);
@@ -237,8 +250,9 @@ public CopyWriter restore() {

@Override
public int hashCode() {
return Objects.hash(serviceOptions, source, sourceOptions, target, targetOptions, result,
blobSize, isDone, megabytesCopiedPerChunk, rewriteToken, totalBytesCopied);
return Objects.hash(serviceOptions, source, sourceOptions, targetId, targetInfo,
targetOptions, result, blobSize, isDone, megabytesCopiedPerChunk, rewriteToken,
totalBytesCopied);
}

@Override
@@ -253,7 +267,8 @@ public boolean equals(Object obj) {
return Objects.equals(this.serviceOptions, other.serviceOptions)
&& Objects.equals(this.source, other.source)
&& Objects.equals(this.sourceOptions, other.sourceOptions)
&& Objects.equals(this.target, other.target)
&& Objects.equals(this.targetId, other.targetId)
&& Objects.equals(this.targetInfo, other.targetInfo)
&& Objects.equals(this.targetOptions, other.targetOptions)
&& Objects.equals(this.result, other.result)
&& Objects.equals(this.rewriteToken, other.rewriteToken)
@@ -267,10 +282,14 @@ public boolean equals(Object obj) {
public String toString() {
return MoreObjects.toStringHelper(this)
.add("source", source)
.add("target", target)
.add("isDone", isDone)
.add("totalBytesRewritten", totalBytesCopied)
.add("targetId", targetId)
.add("targetInfo", targetInfo)
.add("result", result)
.add("blobSize", blobSize)
.add("isDone", isDone)
.add("rewriteToken", rewriteToken)
.add("totalBytesCopied", totalBytesCopied)
.add("megabytesCopiedPerChunk", megabytesCopiedPerChunk)
.toString();
}
}
Original file line number Diff line number Diff line change
@@ -962,7 +962,8 @@ class CopyRequest implements Serializable {

private final BlobId source;
private final List<BlobSourceOption> sourceOptions;
private final BlobInfo target;
private final BlobId targetId;
private final BlobInfo targetInfo;
private final List<BlobTargetOption> targetOptions;
private final Long megabytesCopiedPerChunk;

@@ -971,7 +972,8 @@ public static class Builder {
private final Set<BlobSourceOption> sourceOptions = new LinkedHashSet<>();
private final Set<BlobTargetOption> targetOptions = new LinkedHashSet<>();
private BlobId source;
private BlobInfo target;
private BlobId targetId;
private BlobInfo targetInfo;
private Long megabytesCopiedPerChunk;

/**
@@ -1019,39 +1021,37 @@ public Builder sourceOptions(Iterable<BlobSourceOption> options) {
*
* @return the builder
*/
public Builder target(BlobId target) {
this.target = BlobInfo.builder(target).build();
public Builder target(BlobId targetId) {
this.targetId = targetId;
return this;
}

/**
* Sets the copy target and target options. {@code target} parameter is used to override
* source blob information (e.g. {@code contentType}, {@code contentLanguage}). {@code
* target.contentType} is a required field.
* source blob information (e.g. {@code contentType}, {@code contentLanguage}). Target blob
* information is set exactly to {@code target}, no information is inherited from the source
* blob. If not set, target blob information is inherited from the source blob.
*
* @return the builder
* @throws IllegalArgumentException if {@code target.contentType} is {@code null}
*/
public Builder target(BlobInfo target, BlobTargetOption... options)
throws IllegalArgumentException {
checkContentType(target);
this.target = target;
public Builder target(BlobInfo targetInfo, BlobTargetOption... options) {
this.targetId = targetInfo.blobId();
this.targetInfo = targetInfo;
Collections.addAll(targetOptions, options);
return this;
}

/**
* Sets the copy target and target options. {@code target} parameter is used to override
* source blob information (e.g. {@code contentType}, {@code contentLanguage}). {@code
* target.contentType} is a required field.
* source blob information (e.g. {@code contentType}, {@code contentLanguage}). Target blob
* information is set exactly to {@code target}, no information is inherited from the source
* blob. If not set, target blob information is inherited from the source blob.
*
* @return the builder
* @throws IllegalArgumentException if {@code target.contentType} is {@code null}
*/
public Builder target(BlobInfo target, Iterable<BlobTargetOption> options)
throws IllegalArgumentException {
checkContentType(target);
this.target = target;
public Builder target(BlobInfo targetInfo, Iterable<BlobTargetOption> options) {
this.targetId = targetInfo.blobId();
this.targetInfo = targetInfo;
Iterables.addAll(targetOptions, options);
return this;
}
@@ -1072,16 +1072,15 @@ public Builder megabytesCopiedPerChunk(Long megabytesCopiedPerChunk) {
* Creates a {@code CopyRequest} object.
*/
public CopyRequest build() {
checkNotNull(source);
checkNotNull(target);
return new CopyRequest(this);
}
}

private CopyRequest(Builder builder) {
source = checkNotNull(builder.source);
sourceOptions = ImmutableList.copyOf(builder.sourceOptions);
target = checkNotNull(builder.target);
targetId = checkNotNull(builder.targetId);
targetInfo = builder.targetInfo;
targetOptions = ImmutableList.copyOf(builder.targetOptions);
megabytesCopiedPerChunk = builder.megabytesCopiedPerChunk;
}
@@ -1101,10 +1100,20 @@ public List<BlobSourceOption> sourceOptions() {
}

/**
* Returns the {@link BlobInfo} for the target blob.
* Returns the {@link BlobId} for the target blob.
*/
public BlobInfo target() {
return target;
public BlobId targetId() {
return targetId;
}

/**
* Returns the {@link BlobInfo} for the target blob. If set, this value is used to replace
* source blob information (e.g. {@code contentType}, {@code contentLanguage}). Target blob
* information is set exactly to this value, no information is inherited from the source blob.
* If not set, target blob information is inherited from the source blob.
*/
public BlobInfo targetInfo() {
return targetInfo;
}

/**
@@ -1125,34 +1134,27 @@ public Long megabytesCopiedPerChunk() {

/**
* Creates a copy request. {@code target} parameter is used to override source blob information
* (e.g. {@code contentType}, {@code contentLanguage}). {@code target.contentType} is a required
* field.
* (e.g. {@code contentType}, {@code contentLanguage}).
*
* @param sourceBucket name of the bucket containing the source blob
* @param sourceBlob name of the source blob
* @param target a {@code BlobInfo} object for the target blob
* @return a copy request
* @throws IllegalArgumentException if {@code target.contentType} is {@code null}
*/
public static CopyRequest of(String sourceBucket, String sourceBlob, BlobInfo target)
throws IllegalArgumentException {
checkContentType(target);
public static CopyRequest of(String sourceBucket, String sourceBlob, BlobInfo target) {
return builder().source(sourceBucket, sourceBlob).target(target).build();
}

/**
* Creates a copy request. {@code target} parameter is used to override source blob information
* (e.g. {@code contentType}, {@code contentLanguage}). {@code target.contentType} is a required
* field.
* Creates a copy request. {@code target} parameter is used to replace source blob information
* (e.g. {@code contentType}, {@code contentLanguage}). Target blob information is set exactly
* to {@code target}, no information is inherited from the source blob.
*
* @param sourceBlobId a {@code BlobId} object for the source blob
* @param target a {@code BlobInfo} object for the target blob
* @return a copy request
* @throws IllegalArgumentException if {@code target.contentType} is {@code null}
*/
public static CopyRequest of(BlobId sourceBlobId, BlobInfo target)
throws IllegalArgumentException {
checkContentType(target);
public static CopyRequest of(BlobId sourceBlobId, BlobInfo target) {
return builder().source(sourceBlobId).target(target).build();
}

@@ -1214,10 +1216,6 @@ public static CopyRequest of(BlobId sourceBlobId, BlobId targetBlobId) {
public static Builder builder() {
return new Builder();
}

private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentException {
checkArgument(blobInfo.contentType() != null, "Blob content type can not be null");
}
}

/**
Original file line number Diff line number Diff line change
@@ -413,15 +413,20 @@ public CopyWriter copy(final CopyRequest copyRequest) {
final StorageObject source = copyRequest.source().toPb();
final Map<StorageRpc.Option, ?> sourceOptions =
optionMap(copyRequest.source().generation(), null, copyRequest.sourceOptions(), true);
final StorageObject target = copyRequest.target().toPb();
final Map<StorageRpc.Option, ?> targetOptions = optionMap(copyRequest.target().generation(),
copyRequest.target().metageneration(), copyRequest.targetOptions());
final BlobId targetId = copyRequest.targetId();
final StorageObject targetObject =
copyRequest.targetInfo() != null ? copyRequest.targetInfo().toPb() : null;
final Map<StorageRpc.Option, ?> targetOptions = optionMap(
copyRequest.targetInfo() != null ? copyRequest.targetInfo().generation() : null,
copyRequest.targetInfo() != null ? copyRequest.targetInfo().metageneration() : null,
copyRequest.targetOptions());
try {
RewriteResponse rewriteResponse = runWithRetries(new Callable<RewriteResponse>() {
@Override
public RewriteResponse call() {
return storageRpc.openRewrite(new StorageRpc.RewriteRequest(source, sourceOptions, target,
targetOptions, copyRequest.megabytesCopiedPerChunk()));
return storageRpc.openRewrite(new StorageRpc.RewriteRequest(source, sourceOptions,
targetId.bucket(), targetId.name(), targetObject, targetOptions,
copyRequest.megabytesCopiedPerChunk()));
}
}, options().retryParams(), EXCEPTION_HANDLER);
return new CopyWriter(options(), rewriteResponse);
Original file line number Diff line number Diff line change
@@ -580,8 +580,8 @@ private RewriteResponse rewrite(RewriteRequest req, String token) {
Long maxBytesRewrittenPerCall = req.megabytesRewrittenPerCall != null
? req.megabytesRewrittenPerCall * MEGABYTE : null;
com.google.api.services.storage.model.RewriteResponse rewriteResponse = storage.objects()
.rewrite(req.source.getBucket(), req.source.getName(), req.target.getBucket(),
req.target.getName(), req.target.getContentType() != null ? req.target : null)
.rewrite(req.source.getBucket(), req.source.getName(), req.targetBucket,
req.targetName, req.targetObject)
.setSourceGeneration(req.source.getGeneration())
.setRewriteToken(token)
.setMaxBytesRewrittenPerCall(maxBytesRewrittenPerCall)
Original file line number Diff line number Diff line change
@@ -138,16 +138,20 @@ class RewriteRequest {

public final StorageObject source;
public final Map<StorageRpc.Option, ?> sourceOptions;
public final StorageObject target;
public final String targetBucket;
public final String targetName;
public final StorageObject targetObject;
public final Map<StorageRpc.Option, ?> targetOptions;
public final Long megabytesRewrittenPerCall;

public RewriteRequest(StorageObject source, Map<StorageRpc.Option, ?> sourceOptions,
StorageObject target, Map<StorageRpc.Option, ?> targetOptions,
Long megabytesRewrittenPerCall) {
String targetBucket, String targetName, StorageObject targetObject,
Map<StorageRpc.Option, ?> targetOptions, Long megabytesRewrittenPerCall) {
this.source = source;
this.sourceOptions = sourceOptions;
this.target = target;
this.targetBucket = targetBucket;
this.targetName = targetName;
this.targetObject = targetObject;
this.targetOptions = targetOptions;
this.megabytesRewrittenPerCall = megabytesRewrittenPerCall;
}
@@ -163,14 +167,17 @@ public boolean equals(Object obj) {
final RewriteRequest other = (RewriteRequest) obj;
return Objects.equals(this.source, other.source)
&& Objects.equals(this.sourceOptions, other.sourceOptions)
&& Objects.equals(this.target, other.target)
&& Objects.equals(this.targetBucket, other.targetBucket)
&& Objects.equals(this.targetName, other.targetName)
&& Objects.equals(this.targetObject, other.targetObject)
&& Objects.equals(this.targetOptions, other.targetOptions)
&& Objects.equals(this.megabytesRewrittenPerCall, other.megabytesRewrittenPerCall);
}

@Override
public int hashCode() {
return Objects.hash(source, sourceOptions, target, targetOptions, megabytesRewrittenPerCall);
return Objects.hash(source, sourceOptions, targetBucket, targetName, targetObject,
targetOptions, megabytesRewrittenPerCall);
}
}

Loading