initial metadata store implementation #165

reset · 2016-01-21T23:14:18Z

This is the first pass at adding a metadata storage to the Depot server

Repo server stores metadata in lmdb, an embedded in-memory database
Added PackageIdent struct which encapsulates functionality which was sprinkled throughout modules in the codebase for working with package identifiers. This struct is passed as an argument to many functions which previously took either four &strs or two &strs and two Option<&str>s
Added DataStore struct which encapsulates the lmdb environment and all databases
Added PkgDatabase which is populated from reading the metadata files of package archives
Added PkgIndex which is a special database that allows sorted duplicate data to form an index. This database can be queried to find the latest version of a package using a cursor.
Added ViewDatabase which contains the names of repositories and the packages which are a part of them. This allows us to segregate particular packages to only being available in certain environments while not duplicating data on disk for the actual package archive.
Added data_store and data_object modules which contain an abstraction on top of the FFI layer of lmdb
Add new command line option depot-repair
Add new command line option repo-create
Add new command line option repo-list
Uploading a package which already exists on the depot will no longer result in a forced upload
Moved package archive specific functions to package::archive module from package

Known Issues

Starting multiple transactions from different threads is not currently working. Additional child transactions can be started, but not root transactions. This behaviour should be possible within LMDB.
Cursor functions don't return useable values for keys, but the data of the current item is good to read and available

chef-delivery · 2016-01-21T23:20:14Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-22T12:50:16Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-22T23:36:25Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-23T09:00:52Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-23T09:38:07Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-23T11:13:46Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-23T11:25:23Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-24T07:25:17Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-27T13:46:42Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-01-27T23:12:51Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

fnichol · 2016-01-30T19:46:13Z

src/bldr/package/mod.rs

+}
+
+impl<'a> From<&'a str> for PackageIdent {
+    fn from(value: &'a str) -> PackageIdent {


This is awesome!

chef-delivery · 2016-02-25T02:01:38Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

adamhjk · 2016-02-25T05:06:38Z

src/bldr/package/mod.rs

+        PackageIdent {
+            derivation: deriv.into(),
+            name: name.into(),
+            version: version.map(|v| v.into()),


fnichol · 2016-02-25T15:51:22Z

When I try to run bldr depot, it looks like the data directory isn't created if it does not exist:

root@398e53586e8b:/src# bldr depot
bldr(MN): Starting Bldr Depot at /opt/bldr/srvc/bldr/data
bldr(CR): Repo listening on 0.0.0.0:9632
bldr(DS)[src/bldr/repo/data_store.rs:451:24]: Undefined(2, "No such file or directory")

root@398e53586e8b:/src# ls /opt/bldr/srvc/bldr/data
ls: cannot access /opt/bldr/srvc/bldr/data: No such file or directory

root@398e53586e8b:/src# mkdir -p /opt/bldr/srvc/bldr/data

root@398e53586e8b:/src# bldr depot
bldr(MN): Starting Bldr Depot at /opt/bldr/srvc/bldr/data
bldr(CR): Repo listening on 0.0.0.0:9632

Can we ensure that the data directory exists before starting the depot?

reset · 2016-02-25T19:40:14Z

@fnichol absolutely, good catch!

chef-delivery · 2016-02-25T20:25:09Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-02-25T21:05:41Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-02-26T06:11:51Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

chef-delivery · 2016-02-26T07:10:03Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

reset · 2016-02-26T08:00:43Z

Ok, this PR contains the fixes that @adamhjk and I spoke about on video chat last night. I'm going to merge this one in so we're ready for demos tomorrow!

chef-delivery · 2016-02-26T08:07:58Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

reset · 2016-02-26T08:09:09Z

@delivery approve

chef-delivery · 2016-02-26T08:09:11Z

Failed to unknown change: afd21805-27c3-4cf2-b3d6-25462315f5d6
Delivery does not understand this command. @reset

reset · 2016-02-26T08:10:12Z

@delivery approve

Merged change afd21805-27c3-4cf2-b3d6-25462315f5d6 From review branch datastore into master Signed-off-by: reset <[email protected]>

chef-delivery · 2016-02-26T08:23:52Z

This PR has passed 'Acceptance' and is ready to be delivered!
Use: '@delivery deliver' when validated in acceptance.

chef-delivery added the Delivery Change Created label Jan 21, 2016

reset force-pushed the datastore branch 2 times, most recently from baa1ef3 to 6ab47b7 Compare January 22, 2016 12:47

reset force-pushed the datastore branch from 6ab47b7 to 6488226 Compare January 22, 2016 23:33

reset force-pushed the datastore branch from 6488226 to 5ecf969 Compare January 23, 2016 08:57

reset force-pushed the datastore branch from 5ecf969 to 9d17820 Compare January 23, 2016 09:35

reset force-pushed the datastore branch from 9d17820 to 8b01a1a Compare January 23, 2016 11:10

reset force-pushed the datastore branch from 8b01a1a to 1f01c4a Compare January 23, 2016 11:22

reset force-pushed the datastore branch from 1f01c4a to e96834e Compare January 24, 2016 07:22

reset force-pushed the datastore branch 3 times, most recently from 4dad098 to b607447 Compare January 27, 2016 13:43

reset force-pushed the datastore branch from b607447 to 0fbcc61 Compare January 27, 2016 23:08

reset force-pushed the datastore branch 6 times, most recently from baedafa to bb11e49 Compare January 29, 2016 06:49

fnichol reviewed Jan 30, 2016
View reviewed changes

src/bldr/package/mod.rs

}

impl<'a> From<&'a str> for PackageIdent {

fn from(value: &'a str) -> PackageIdent {

Copy link

Collaborator

fnichol Jan 30, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome!

reset force-pushed the datastore branch from fdeee0d to bc63a0c Compare February 25, 2016 01:37

adamhjk reviewed Feb 25, 2016
View reviewed changes

reset force-pushed the datastore branch from bc63a0c to 871e377 Compare February 25, 2016 20:15

reset force-pushed the datastore branch from 871e377 to c2ac286 Compare February 25, 2016 20:49

reset force-pushed the datastore branch 2 times, most recently from 9101246 to e2b73f4 Compare February 26, 2016 06:01

reset force-pushed the datastore branch from e2b73f4 to e2277b8 Compare February 26, 2016 07:02

initial metadata store implementation

1953511

reset force-pushed the datastore branch from e2277b8 to 1953511 Compare February 26, 2016 07:59

chef-delivery added a commit that referenced this pull request Feb 26, 2016

Merge pull request #165 from chef/datastore

8554671

Merged change afd21805-27c3-4cf2-b3d6-25462315f5d6 From review branch datastore into master Signed-off-by: reset <[email protected]>

chef-delivery merged commit 8554671 into master Feb 26, 2016

chef-delivery deleted the datastore branch February 26, 2016 08:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial metadata store implementation #165

initial metadata store implementation #165

reset commented Jan 21, 2016

chef-delivery commented Jan 21, 2016

chef-delivery commented Jan 22, 2016

chef-delivery commented Jan 22, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 24, 2016

chef-delivery commented Jan 27, 2016

chef-delivery commented Jan 27, 2016

fnichol Jan 30, 2016

chef-delivery commented Feb 25, 2016

adamhjk Feb 25, 2016

fnichol commented Feb 25, 2016

reset commented Feb 25, 2016

chef-delivery commented Feb 25, 2016

chef-delivery commented Feb 25, 2016

chef-delivery commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

reset commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

reset commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

reset commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

initial metadata store implementation #165

initial metadata store implementation #165

Conversation

reset commented Jan 21, 2016

Known Issues

chef-delivery commented Jan 21, 2016

chef-delivery commented Jan 22, 2016

chef-delivery commented Jan 22, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 23, 2016

chef-delivery commented Jan 24, 2016

chef-delivery commented Jan 27, 2016

chef-delivery commented Jan 27, 2016

fnichol Jan 30, 2016

Choose a reason for hiding this comment

chef-delivery commented Feb 25, 2016

adamhjk Feb 25, 2016

Choose a reason for hiding this comment

fnichol commented Feb 25, 2016

reset commented Feb 25, 2016

chef-delivery commented Feb 25, 2016

chef-delivery commented Feb 25, 2016

chef-delivery commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

reset commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

reset commented Feb 26, 2016

chef-delivery commented Feb 26, 2016

reset commented Feb 26, 2016

chef-delivery commented Feb 26, 2016