CollectionFS cfs:tempstore and cfs:worker to fix concurrency issues with GridFS uploads

packages/tempstore/.npm/package/.gitignore

@@ -0,0 +1 @@
+node_modules

packages/tempstore/.npm/package/README

@@ -0,0 +1,7 @@
+This directory and the files immediately inside it are automatically generated
+when you change this package's NPM dependencies. Commit the files in this
+directory (npm-shrinkwrap.json, .gitignore, and this README) to source control
+so that others run the same versions of sub-dependencies.
+
+You should NOT check in the node_modules directory that Meteor automatically
+creates; if you are using git, the .gitignore file tells git to ignore it.

packages/tempstore/.travis.yml

@@ -0,0 +1,5 @@
+language: node_js
+node_js:
+  - "0.10"
+before_install:
+  - "curl -L http://git.io/s0Zu-w | /bin/sh"

packages/tempstore/.versions

@@ -0,0 +1,29 @@
+[email protected]
+[email protected]
+[email protected]
+cfs:[email protected]
+cfs:[email protected]
+cfs:[email protected]
+cfs:[email protected]
+cfs:[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+raix:[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+[email protected]

packages/tempstore/CHANGELOG.md

@@ -0,0 +1,169 @@
+# Changelog
+
+## vCurrent
+## [v0.1.2] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.1.2)
+#### 17/12/14 by Morten Henriksen
+## [v0.1.1] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.1.1)
+#### 17/12/14 by Morten Henriksen
+- mbr update, remove versions.json
+
+- Bump to version 0.1.1
+
+## [v0.1.0] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.1.0)
+#### 17/12/14 by Morten Henriksen
+- mbr update versions and fix warnings
+
+- fix 0.9.1 package scope
+
+- don't rely on package names; fix for 0.9.1
+
+- 0.9.1 support
+
+## [v0.0.29] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.29)
+#### 28/08/14 by Morten Henriksen
+- Meteor Package System Update
+
+## [v0.0.28] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.28)
+#### 27/08/14 by Eric Dobbertin
+- change package name to lowercase
+
+## [v0.0.27] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.27)
+#### 17/06/14 by Eric Dobbertin
+- add `FS.TempStore.removeAll` method
+
+## [v0.0.26] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.26)
+#### 30/04/14 by Eric Dobbertin
+## [v0.0.25] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.25)
+#### 30/04/14 by Eric Dobbertin
+- use third-party combined-stream node pkg as attempt to resolve pesky streaming issues
+
+## [v0.0.24] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.24)
+#### 29/04/14 by Eric Dobbertin
+- generate api docs
+
+- fileKey methods now expect an FS.File always, so we give them one
+
+- small FS.File API change
+
+## [v0.0.23] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.23)
+#### 12/04/14 by Eric Dobbertin
+- test for packages since we're assigning default error functions for stores
+
+## [v0.0.22] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.22)
+#### 12/04/14 by Eric Dobbertin
+- avoid errors if file already removed from temp store
+
+## [v0.0.21] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.21)
+#### 12/04/14 by Eric Dobbertin
+## [v0.0.20] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.20)
+#### 08/04/14 by Eric Dobbertin
+- cleanup stored/uploaded events and further improve chunk tracking
+
+## [v0.0.19] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.19)
+#### 08/04/14 by Eric Dobbertin
+- use internal tracking collection
+
+- Have TempStore set the size
+
+- Add the SA on stored result
+
+- allow unset chunkSum
+
+## [v0.0.18] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.18)
+#### 06/04/14 by Eric Dobbertin
+- delete chunkCount and chunkSize props from fileObj after upload is complete
+
+## [v0.0.17] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.17)
+#### 06/04/14 by Eric Dobbertin
+- We now wait to mount storage until it's needed (first upload begins); this ensures that we are able to accurately check for the cfs-worker package, which loads after this one. It also makes the code a bit cleaner.
+
+## [v0.0.16] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.16)
+#### 04/04/14 by Morten Henriksen
+- Temporary workaround: We currently we generate a mongoId if gridFS is used for TempStore
+
+- Note: At the moment tempStore will only use gridfs if no filesystem is installed
+
+## [v0.0.15] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.15)
+#### 02/04/14 by Morten Henriksen
+- Use the stored event and object instead (result object is not used at the moment - but we could store an id at some point)
+
+## [v0.0.14] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.14)
+#### 31/03/14 by Eric Dobbertin
+- use latest releases
+
+- use latest releases
+
+## [v0.0.13] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.13)
+#### 31/03/14 by Morten Henriksen
+- Try to use latest when using weak deps
+
+## [v0.0.12] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.12)
+#### 30/03/14 by Morten Henriksen
+## [v0.0.11] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.11)
+#### 30/03/14 by Morten Henriksen
+- Set noon callback - we just want the file gone
+
+## [v0.0.10] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.10)
+#### 29/03/14 by Morten Henriksen
+- add filesystem and gridfs as weak deps
+
+## [v0.0.9] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.9)
+#### 29/03/14 by Morten Henriksen
+- Add check to see if FS.TempStore.Storage is set
+
+## [v0.0.8] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.8)
+#### 29/03/14 by Morten Henriksen
+- Converting TempStore to use SA api
+
+## [v0.0.7] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.7)
+#### 25/03/14 by Morten Henriksen
+- use `new Date`
+
+- Have TempStore emit relevant events
+
+## [v0.0.6] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.6)
+#### 23/03/14 by Morten Henriksen
+- Rollback to specific git dependency
+
+- use collectionFS travis version force update
+
+## [v0.0.5] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.5)
+#### 22/03/14 by Morten Henriksen
+- try to fix travis test by using general package references
+
+## [v0.0.4] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.4)
+#### 21/03/14 by Morten Henriksen
+- fix chunk files not actually being deleted
+
+## [v0.0.3] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.3)
+#### 18/03/14 by Morten Henriksen
+- * TempStore is now an EventEmitter   * progress event   * uploaded   * (start) should perhaps be created   * remove * Added FS.TempStore.listParts - will return lookup object listing the parts already uploaded
+
+- Allow chunk to be undefined an thereby have the createWriteStream follow normal streaming api
+
+- Allow undefined in chunkPath
+
+- added comments
+
+- bug hunting
+
+- Add streaming WIP
+
+- rename temp store collection to 'cfs.tempstore'
+
+- fix ensureForFile
+
+- track tempstore chunks in our own collection rather than in the file object
+
+- change to accept buffer; less converting
+
+- prevent bytesUploaded from getting bigger than size
+
+## [v0.0.2] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.2)
+#### 15/02/14 by Morten Henriksen
+- fix typo
+
+## [v0.0.1] (https://github.com/CollectionFS/Meteor-cfs-tempstore/tree/v0.0.1)
+#### 13/02/14 by Morten Henriksen
+- init commit
+

packages/tempstore/LICENSE.md

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 [@raix](https://github.com/raix) and [@aldeed](https://github.com/aldeed), aka Morten N.O. Nørgaard Henriksen, [email protected]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

packages/tempstore/README.md

@@ -0,0 +1,24 @@
+cfs:tempstore
+=========================
+
+This is a Meteor package used by
+[CollectionFS](https://github.com/CollectionFS/Meteor-CollectionFS). It provides
+an API for quickly storing chunks of file data in temporary files. If also supports deleting those chunks, and combining them into one
+binary object and attaching it to an FS.File instance.
+
+You don't need to manually add this package to your app, but you could replace
+this package with your own if you want to handle temporary storage in another
+way.
+
+> `FS.TempStore` uses the `cfs:storage-adapter` compatible Storage Adapters, both `FS.Store.FileSystem` and `FS.Store.GridFS` will be defaulted. *for more information read the [internal.api.md](internal.api.md)*
+
+##Documentation
+[API Documentation](api.md)
+
+##Contribute
+Here's the [complete API documentation](internal.api.md), including private methods.
+
+Update docs, `npm install docmeteor`
+```bash
+$ docmeteor
+```

packages/tempstore/api.md

@@ -0,0 +1,112 @@
+## cfs-tempstore Public API ##
+
+CollectionFS, temporary storage
+
+_API documentation automatically generated by [docmeteor](https://github.com/raix/docmeteor)._
+
+##Temporary Storage
+
+Temporary storage is used for chunked uploads until all chunks are received
+and all copies have been made or given up. In some cases, the original file
+is stored only in temporary storage (for example, if all copies do some
+manipulation in beforeSave). This is why we use the temporary file as the
+basis for each saved copy, and then remove it after all copies are saved.
+
+Every chunk is saved as an individual temporary file. This is safer than
+attempting to write multiple incoming chunks to different positions in a
+single temporary file, which can lead to write conflicts.
+
+Using temp files also allows us to easily resume uploads, even if the server
+restarts, and to keep the working memory clear.
+The FS.TempStore emits events that others are able to listen to
+-
+
+### <a name="FS.TempStore"></a>*fs*.TempStore {object}&nbsp;&nbsp;<sub><i>Server</i></sub> ###
+
+*This property __TempStore__ is defined in `FS`*
+it's an event emitter*
+
+> ```FS.TempStore = new EventEmitter();``` [tempStore.js:28](tempStore.js#L28)
+
+
+
+-
+We will not mount a storage adapter until needed. This allows us to check for the
+existance of FS.FileWorker, which is loaded after this package because it
+depends on this package.
+
+-
+XXX: TODO
+FS.TempStore.on('stored', function(fileObj, chunkCount, result) {
+This should work if we pass on result from the SA on stored event...
+fileObj.update({ $set: { chunkSum: 1, chunkCount: chunkCount, size: result.size } });
+});
+Stream implementation
+-
+
+### <a name="FS.TempStore.removeFile"></a>*fsTempstore*.removeFile(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
+
+*This method __removeFile__ is defined in `FS.TempStore`*
+
+__Arguments__
+
+* __fileObj__ *{[FS.File](#FS.File)}*  
+
+This function removes the file from tempstorage - it cares not if file is
+already removed or not found, goal is reached anyway.
+
+> ```FS.TempStore.removeFile = function(fileObj) { ...``` [tempStore.js:169](tempStore.js#L169)
+
+
+-
+
+### <a name="FS.TempStore.createWriteStream"></a>*fsTempstore*.createWriteStream(fileObj, [options])&nbsp;&nbsp;<sub><i>Server</i></sub> ###
+
+*This method __createWriteStream__ is defined in `FS.TempStore`*
+
+__Arguments__
+
+* __fileObj__ *{[FS.File](#FS.File)}*  
+
+ File to store in temporary storage
+
+* __options__ *{[Number ](#Number )|[ String](# String)}*  (Optional)
+
+__Returns__  *{Stream}*
+Writeable stream
+
+
+`options` of different types mean differnt things:
+`undefined` We store the file in one part
+(Normal server-side api usage)*
+`Number` the number is the part number total
+(multipart uploads will use this api)*
+`String` the string is the name of the `store` that wants to store file data
+(stores that want to sync their data to the rest of the files stores will use this)*
+
+> Note: fileObj must be mounted on a `FS.Collection`, it makes no sense to store otherwise
+
+> ```FS.TempStore.createWriteStream = function(fileObj, options) { ...``` [tempStore.js:217](tempStore.js#L217)
+
+
+-
+
+### <a name="FS.TempStore.createReadStream"></a>*fsTempstore*.createReadStream(fileObj)&nbsp;&nbsp;<sub><i>Server</i></sub> ###
+
+*This method __createReadStream__ is defined in `FS.TempStore`*
+
+__Arguments__
+
+* __fileObj__ *{[FS.File](#FS.File)}*  
+
+ The file to read
+
+
+__Returns__  *{Stream}*
+Returns readable stream
+
+
+
+> ```FS.TempStore.createReadStream = function(fileObj) { ...``` [tempStore.js:313](tempStore.js#L313)
+
+