Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup job fails with "Directory not found" error #8245

Closed
obi1kenobi opened this issue Apr 30, 2018 · 10 comments
Closed

Backup job fails with "Directory not found" error #8245

obi1kenobi opened this issue Apr 30, 2018 · 10 comments
Assignees
Milestone

Comments

@obi1kenobi
Copy link
Contributor

obi1kenobi commented Apr 30, 2018

OrientDB Version: 2.2.32

Java Version: Open JDK 8

OS: Linux

OrientDB is configured to do periodic backups of a given database.

Expected behavior

  • The backup job should succeed, barring issues like low disk space etc.
  • If the backup job fails for any reason, no backup file should be generated.

Actual behavior

  • The backup job fails with the following error:
java.lang.IllegalArgumentException: Directory /path/to/database/<database_name>.45.wal not found
  at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
  at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
  at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
  at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
  at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
  at com.orientechnologies.orient.server.handler.OAutomaticBackup.fullBackupDatabase(OAutomaticBackup.java:353)
  at com.orientechnologies.orient.server.handler.OAutomaticBackup$1.run(OAutomaticBackup.java:183)
  at java.util.TimerThread.mainLoop(Timer.java:555)
  at java.util.TimerThread.run(Timer.java:505)

Steps to reproduce

Unfortunately, I am not quite sure how to reproduce the issue. However, given that the error now appears regularly, perhaps it is related to the size of the database?

@obi1kenobi
Copy link
Contributor Author

There is evidence that the issues started when we upgraded from 2.2.30 to 2.2.32. Hopefully that helps narrow down the problem.

@andrii0lomakin
Copy link
Member

andrii0lomakin commented Apr 30, 2018 via email

@obi1kenobi
Copy link
Contributor Author

Hi @Laa, thanks for the speedy reply.

It doesn't seem like 2.2.34 is listed on the website: https://orientdb.com/releases/

Is there a different location where I should download it from?

@andrii0lomakin
Copy link
Member

andrii0lomakin commented Apr 30, 2018 via email

@obi1kenobi
Copy link
Contributor Author

Thanks, I'll give it a shot.

@obi1kenobi
Copy link
Contributor Author

Hey @Laa, I upgraded to 2.2.34 and now there's a different problem.

While the generated backup file is valid, and can be loaded onto a new server just fine, it is missing a large amount of data. A diff between the real database and the database restored from backup shows that the backup only contains 70-75% of the data of the original data. The missing data is data that was originally created at least a few days prior to the backup, and was not modified during the backup process.

Any advice would be much appreciated -- thanks in advance!

@andrii0lomakin
Copy link
Member

@obi1kenobi I do not get it completely. Could you provide a detailed example?
How do you make a backup. Backup is just copies and zip files which are flushed to file system. So I doubt that is ODB issue.

@obi1kenobi
Copy link
Contributor Author

Using the OrientDB automatic backup plugin:

{
  "enabled": true,
  "mode": "FULL_BACKUP",
  "exportOptions": "",
  "delay": "<time_here>",
  "firstTime": "<time_here_too>",
  "targetDirectory": "<path_here>",
  "targetFileName": "<filename_here>",
  "compressionLevel": 9,
  "bufferSize": 1048576
}

We don't do any additional processing of the backup file. Attempting to load the generated backup file produces a database that OrientDB loads just fine, but appears to be incomplete as I described.

I'll keep an eye on the situation and if it happens again, I'll try to get you log files that you can look at.

In the meantime, I think there's at least one bug (at least in 2.2.32, not sure if it was fixed in 2.2.34 since I can't find a changelog) that you should be able to easily reproduce. If the backup process fails, I would expect no backup file to be generated, but 2.2.32 would still output a "backup.zip" file with whatever fraction of the backup managed to complete. This behavior hides the error and makes monitoring more difficult -- it means that the monitoring system has to read OrientDB log output and parse it for error messages, rather than just looking for the backup file to be generated and complaining if it doesn't find it.

Thanks for looking into this!

@andrii0lomakin andrii0lomakin self-assigned this May 3, 2018
@andrii0lomakin andrii0lomakin added this to the 2.2.x (next hotfix) milestone May 3, 2018
andrii0lomakin added a commit that referenced this issue May 4, 2018
andrii0lomakin added a commit that referenced this issue May 4, 2018
@andrii0lomakin
Copy link
Member

@obi1kenobi issue with removing of the backup file in case of error is fixed.

@obi1kenobi
Copy link
Contributor Author

Thank you -- looking forward to the next release :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants