Run failed to create sandbox

[donkirkby]

Because of a problem with the purge task, the disk filled up. That meant that new runs didn't have room to create a sandbox.

The problem is that the runs were left in the "New" state, when they should have been in the "Failed" state. If a run doesn't successfully submit a slurm job, it should go in the "Failed" state.

2019-03-18 13:41:42[ERROR]django.request.handle_uncaught_exception(): Internal Server Error: /api/containerruns/
Traceback (most recent call last):
  File "/path/to/venv/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/path/to/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/path/to/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/path/to/venv/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/path/to/venv/lib/python2.7/site-packages/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/path/to/venv/lib/python2.7/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/path/to/venv/lib/python2.7/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/path/to/venv/lib/python2.7/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/path/to/venv/lib/python2.7/site-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/path/to/venv/lib/python2.7/site-packages/django/utils/decorators.py", line 185, in inner
    return func(*args, **kwargs)
  File "/path/to/venv/lib/python2.7/site-packages/django/db/transaction.py", line 267, in __exit__
    connection.set_autocommit(True)
  File "/path/to/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 416, in set_autocommit
    self.run_and_clear_commit_hooks()
  File "/path/to/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 651, in run_and_clear_commit_hooks
    func()
  File "/path/to/Kive/kive/container/models.py", line 1018, in schedule
    self.create_sandbox()
  File "/path/to/Kive/kive/container/models.py", line 1007, in create_sandbox
    full_sandbox_path = mkdtemp(prefix=prefix, dir=sandbox_root)
  File "/usr/lib64/python2.7/tempfile.py", line 329, in mkdtemp
    _os.mkdir(file, 0700)
OSError: [Errno 28] No space left on device: '/path/to/data/kive/ContainerRuns/userdon_run59644_9uHw_f'

[donkirkby]

Easier steps to reproduce:

1. Make the ContainerRuns folder read-only: chmod ug-w ContainerRuns/
2. Try to start a run. It will display the error, "Runs have not been started. Please correct the errors and try again."
3. Go to the Container Runs page.

Expected behaviour: either there is no new run, or the new run is in the failed state.

Actual behaviour: the new run is in the new state, and it will never start.