MUPX: files lost on re-deploy and restart

[LazerJesus]

i have set the storagePath relative to user like suggested here
Images = new Meteor.Files({ collectionName: 'Images', storagePath: '~/tmp/meteor/imgs' });
but when i restart meteor in development or redeploy to the project with mupx, all imgs are gone.

[dr-dimitru]

Make sure you aren't overwriting folder where files is stored on deploy.
I recommend to use rsync with -azuiP flags.

And currently I won't recommend to use relative path to user. I'm emphatically recommend to always use absolute path. Or at least left default value for storagePath to see if it's working properly.

[LazerJesus]

then i would have to rsync to my local machine everytime I push code to the server, and when I forget once, the data would be overwritten?
or do i missunderstand?

[dr-dimitru]

Can't clearly answer as I don't now your server's structure and how it's configured, and I've never worked with MUP/MUPX

• You don't need to sync your server with local machine (unless you want to keep all uploaded files locally)
• If storagePath is out of your Meteor/NodeJS directory uploaded files shouldn't be overwritten
• Have you tried to use package's default storagePath?
• Have you tried to use absolute value for storagePath?

[LazerJesus]

i have tried storagePath: '/tmp/UploadedFiles', no storagePath and
storagePath: '~/tmp/meteor/imgs'.

I am running an Ubuntu 14.04.4 x64 droplet on digitalOcean

[dr-dimitru]

• Do you able to obtain uploaded files in those directories?
• Do you upload compiled app (NodeJS) or as "Meteor App"?

[LazerJesus]

• what do you mean by this?
• mupx writes this: Deploy app 'anna_app' (linux) to the console when they start to deploy. so i assume its "Meteor App". but any images that i have locally do not get uploaded

[dr-dimitru]

1. Can you go to /tmp/UploadedFiles on you server and ls -la to see uploaded files
2. Yes, MUPX won't deploy anything what is not related to application codebase - this is expected behaviour

Deploy you application, and then upload files

[LazerJesus]

• the tmp folder at /opt/anna_app/tmp contains nothing but bundle.tar.gz. and /tmp is empty.

[dr-dimitru]

Do you have any error/warning at server's console?
Meteor-Files should warn you if storagePath is not writable/accessible

[LazerJesus]

i can upload. see the portfolio page
the image you see is uploaded via the package.
the current storagePath is '/tmp/UploadedFiles'
i have no warnings.

[dr-dimitru]

I guess your server just wiping out /tmp folder frequently, as it's basically temporary folder

[LazerJesus]

but the files are still online.

[dr-dimitru]

Sorry, but I did't get you.

1. What is not working right now?

[LazerJesus]

when i do mupx deploy again, all the files are gone.
so everytime I make a change to the code, i have to reupload all the projects.

[dr-dimitru]

probably mupx builds app inside /tmp/ and probably wipes out tmp folder after build, try to use /data/UploadedFiles

[LazerJesus]

with storagePath: '/data/UploadedFiles' the files still go missing after deploy.

and another thing happened a couple of times now - when uploading an image, onUploaded never gets called. the upload just stops at a 100%.
most of the times onUploaded gets called and everything works. but when this happens. i have to reset the whole project.

[dr-dimitru]

1. Could you please post your server's logs, as this error clearly says what path must be a string
2. Could you please post more of your code, where this package is used?

[LazerJesus]

    'change #file-input': function (e, t) {
        var project = this._id;
        var advance = true;
        var r = 0;
        var files = e.currentTarget.files;

        function insert(file, rank) {
            log('insert');
            t.currentFile.set(Images.insert({
                file: file,
                meta: {rank: rank, project: project, big: false},
                onUploaded: function (error, fileObj) {
                    log('onUploaded');
                    if (error) {
                        console.log(error);
                        return false;
                    } else {
                        Projects.update({_id: project}, {$push: {imgs: fileObj._id}}, function (err, s) {
                            if (err) {
                                console.log('Projects.update Error: ' + err);
                                // return false;
                            } else {
                                console.log('Projects updated: ' + s);
                                t.currentFile.set(false);
                                advance = true;
                            }
                        });
                    }
                }
            }));
        }

        function loop() {
            if (advance == false) {
                setTimeout(function () {
                    loop()
                }, 500);
            } else {
                advance = false;
                insert(files[r], r);
                r++;
                if (r < files.length) {
                    loop()
                }
            }
        }

        loop();
    }

i had to set it up this way to allow for multiple imgs to be uploaded with one action. when I had just passed the images in without this loop(), advance and timeout architecture, it called Projects.update just once and added only one the last image to the project.

what logs do you need?

[dr-dimitru]

Logs from your server's console

[LazerJesus]

the screenshot that i shared, those where all.

[dr-dimitru]

To insert multiple images use _.each:

'change #file-input': function (e, t) {
   _.each(e.currentTarget.files, function (file) {
    Images.insert({/* ... */});
  });
}

[dr-dimitru]

BTW which version of Meteor and Package are you using?

[LazerJesus]

ostrio:[email protected]
meteor1.3.1

when I use _.each, onUploaded only gets called once for the last image. and thus only the last image gets added to the projects.imgs array

[dr-dimitru]

1. Unfortunately I can't answer why directory gets whipped out after you have deployed app via MUPX. As this question is not related to Meteor-Files package directly, please address this question to Meteor-UP team
2. About multiple file upload - I'll provide an expressive example soon
3. About server-side error - The path must be a string means storagePath isn't set or not a string, to debug this:
   • Post here your server-side code related to this package
   • Run package in debug mode (pass debug: true to Constructor) and post here server's logs

[LazerJesus]

1. ok i'll ask them and when we find out what it is, i will tell you.

Images = new Meteor.Files({
    collectionName: 'Images',
    storagePath: '/data/UploadedFiles',
    debug: true
});

uploading the code right now. logs follow

[LazerJesus]

ok, so i looked at the logs and saw that this issue always occurred after i had deleted a file from the Images collection.
I rewrote the .remove function to be a serverside method. now the issue stopped occurring. one question about that. Images.remove vs Images.collection.remove - whats the difference?

about the "images go missing after using mupx deploy - there are others with the same issue.
but i need a quick solution. how do you deploy your applications? i'll just use your way…

[dr-dimitru]

1. MeteorFiles.remove({}) removes record from MongoDB and unlink file from FS (removes file physically)
2. MeteorFiles.collection.remove({}) just removes record from MongoDB
3. You can allow .remove() from client via allowClientCode flag passed into Constructor
4. Our deploy system is very simple, but server's structure is complicated
   • The nodejs apps is powered by Phusion Passenger + Nginx and behind proxy gate (for load balancing an durability)

Deploy scheme:

On your local machine (assuming you app called: myapp)

# in meteor's app folder
meteor build ../ --architecture os.linux.x86_64 
rsync -auhiP -e 'ssh -p 22' ../myapp.tar.gz ostrio@SERVER_IP:/home/user/

On server:

# go to directory with transferred archive
cd /home/user

# Unpack archive
tar -xvf myapp.tar.gz
rm myapp.tar.gz

# Block below will allow to serve static files via nginx
# we set server's root in nginx config to /var/www/myapp/public
mkdir -p ./bundle/public
cp ./bundle/programs/web.browser/*.css ./bundle/public/
cp ./bundle/programs/web.browser/*.js ./bundle/public/
cp -R ./bundle/programs/web.browser/app/* ./bundle/public/

# change to your user
chown -R user:usergroup ./bundle

cd ./bundle/programs/server
# install npm dependencies as your user
su -s /bin/bash -c 'npm install --production' user
# install extra dependencies if needed
# su -s /bin/bash -c 'npm install [email protected] [email protected] --production --save' user
cd ../../../

# Remove old compiled JSs and CSSs
rm -Rf /var/www/myapp/programs/web.browser/*
rm /var/www/myapp/public/*.js /var/www/myapp/public/*.css

# Update project's files
rsync -azuhiP ./bundle/ /var/www/myapp

# Fix permissions
chown -R user:usergroup /var/www/myapp
chmod -R 744 /var/www/myapp
chmod 755 /var/www/myapp
chmod -R 755 /var/www/myapp/public

passenger-config restart-app /var/www/myapp
# Show status
passenger-status -v
# Show logs
tail -n 100 -f /var/log/nginx/error.log

• Do not forget to change SERVER_IP, user:usergroup, myapp and path to project on server (we used /var/www) to your values
• You can omit/change part with passenger, for example to god, forever, etc.

[LazerJesus]

ok so i just spent the last three days to get your package to work in my configuration and i cant get it done.

i teach this stuff to myself and this is too much.

i will need to find a different package to host images. sry

[dr-dimitru]

this thread clearly explains why your FS is changing on every deploy

[Aur0r]

This is clearly a mup/mupx deploy issue since there is no volume mounted that persists your data in the docker container, so this has nothing to do with this package. I am copying a backup of all the files uploaded by ostrio:files to the newly deployed instance after deployment and that solves the issue for now.
BUT for CI reasons if you directly want to deploy from Jenkins/Travis to your server then I can see the issue. For this, you would have to write your own script at the moment, and also the copying of the files takes time.
I think currently you would have to fork the meteor-up project and add the volume to the container creation routine and then if it is good make a pull request to have this functionality. Because currently I can't see a ticket for it (https://github.com/kadirahq/meteor-up/issues) in the active repository of meteor-up on github (https://github.com/kadirahq/meteor-up).

[dr-dimitru]

@FinnFrotscher could we close this issue?

[LazerJesus]

yes. i am now using S3.

[dr-dimitru]

@FinnFrotscher with this package of CFS?

[LazerJesus]

@dr-dimitru no, directly via their API