Dockerized Runbot - Missing Image #111
Comments
|
I've experienced this on my own Runbot before too. I can fairly easily replicate the issue by pushing to the same branch a bunch of times. What I think is happening is that our sandbox between builds is being broken - a previous build is stopping or deleting an image/container when it is done, but a later build is still using it. |
|
What version of docker you have installed in your own Runbot? Maybe is a symbolic link issue (or maybe in your case a |
|
Docker version 1.12.3, build 6b644ec Docker-gc 0.1.1 (just in case) |
|
The projects with the same issue Are there |
|
Like in the code? Nah no shell or file manipulations. I'm pretty sure it's not the code, because I have had instances in which the PR build breaks but the branch build does not. Merging the code or creating/pushing a new branch will typically yield a 🍏 too. Interesting thing is - once it happens for a branch, it'll just keep happening. That branch is basically toast for the remainder of Runbot eternity. More examples can be seen on my runbot, vertical-medical repo Build For referencing the example, this is the diff causing issues. |
|
The case for build 39 is a little different. Getting addons dependencies
�[91mfatal: Remote branch origin/feature/10.0/product_service_duration not found in upstream origin
�[0m�[91mTraceback (most recent call last):
File "/root/maintainer-quality-tools/travis/clone_oca_dependencies", line 123, in <module>
�[0m�[91m run(deps_checkout_dir, build_dir)
File "/root/maintainer-quality-tools/travis/clone_oca_dependencies", line 98, in run
url, branch)
File "/root/maintainer-quality-tools/travis/clone_oca_dependencies", line 66, in git_checkout
subprocess.check_call(command)
File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
�[0m�[91msubprocess.CalledProcessError�[0m�[91m: Command '['git', 'clone', '-q', 'https://github.com/laslabs/product-attribute.git', '-b', 'origin/feature/10.0/product_service_duration', '/root/dependencies/product-attribute']' returned non-zero exit status 128�[0m�[91m
�[0mThe command '/bin/sh -c /install' returned a non-zero code: 1Running the same command: |
|
...I mean, the issue was this line You could fix it: +product-attribute https://github.com/laslabs/product-attribute.git origin/feature/10.0/product_service_duration
-product-attribute https://github.com/laslabs/product-attribute.git feature/10.0/product_service_durationRunning locally: Do you have another example without clone issue? |
|
Ahh well crap so my example was bad. That also makes sense why it's also the only one I've seen fail on both branches, so I should have known it was effed. Here's another, same Runbot just use the |
|
Interesting now that I'm looking, that is a clone issue too - Github timeout (which also makes sense why I've seen it only with parallel branches). And the one reported in the OCA/server-tools#611 (comment) is actually an issue with the environment. So really it seems in all instances, the build is just failing before the tests. I'm curious as to the other reasons it has been failing like this now. |
|
The real fail is creating symbolic link That line is in The command should be: But the path received was |
|
Yeah what the heck? I remember this one from the other day too now: |
|
I can confirm that this issue is because we have 2 server with different environments.
@gurneyalex Could you help us to get the differences between runbot2 (fine!) and runbot1 (failing), please?
|
|
Or wait this looks like as a flaky error It's not a runbot1 vs runbot2 error because I have the same issue with runbot2 too in a few cases: |
|
I saw differences between a test_10_job.log fine and other one failing:
|
|
@lasley You have installed a newest, then we can confirm or reject this theory if you have a similar case of this issue in your runbot. |
|
@moylop260 I did not find this specific issue in my log grep's. The step you noted is the RVM install though, and I haven't updated my prod runbot to our new changes yet. Shall I do that and monitor? |
|
If it can help, looking at OCA's runbot server, I think the first error in the logs is: |
|
I saw that the real issue is a command not found from step 21 Could you run in your runbot server the following command |
|
@sylvain-garancher That is not the real error because we have that output from green case and red one. (I just updated my table with that case) |
|
@gurneyalex FYI our http://runbot.vauxoo.com has working fine with the following version: ...
Server:
Version: 1.12.2
API version: 1.24
Go version: go1.6.3
Git commit: bb80604
Built: Tue Oct 11 18:29:41 2016
OS/Arch: linux/amd64For this case where a old version of docker is installed you will need apply a manual change base on docker-release-1.12.2 documentation.
For new version of docker they are using You will need the following configuration: echo """
[Service]
TasksMax=infinity
""" > /etc/systemd/system/docker.service.d/docker.confMaybe we (@ruiztulio and me) could help you to configure the servers together. Tell us if is a good idea for you |
|
we are currently running travis2docker-3.0.9-py2.7.egg and Docker version 1.6.2, build 7c8fca2 |
|
I'll handle the upgrade this week end. |
|
@moylop260 The output of this is quite long... This one is on runbot1: This one is on runbot2: |
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug
|
Reviewing the files from
Thanks in advance. |
|
FYI I saw that the lines inserted come from this project:
Forget it @max3903 help me to disable it, thanks |
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug - Use a copy of ids list instead of original pointer
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug - Use a copy of ids list instead of original pointer
- Helps to fix OCA#111 - Use a unique name of image - Use a unique path of t2d scripts - Remove use of global sys.args (Using directly the t2d class) - Add _log db and logger in order to self-debug - Use a copy of ids list instead of original pointer
|
For record, this issue was a bug in travis2docker with non-ideal cases. PreviousWe have a class with a class MyClass(object):
my_var = None
def my_method(self, error=None):
for local_variable in range(10, 11):
self.my_var = local_variable
assert not error, error
self.my_var = NoneIdeal case works fine! 🍏 obj1 = MyClass()
obj1.my_method("")
print obj1.my_var # None: Good!But if there is a error the variable is dirty 🔴 obj2 = MyClass()
obj2.my_method(error="no ideal case")
print obj2.my_var # It's not None has the value 10 :(We have a non-ideal case in OCA's runbot (e.g. OCA/OCB) and the flaky error was injected. Traceback (most recent call last):
File "travisfile2dockerfile", line 11, in <module>
sys.exit(main())
File "travis2docker/cli.py", line 184, in main
raise InvalidRepoBranchError(msg)
travis2docker.exceptions.InvalidRepoBranchError: The repo or the branch is incorrect value, because It can not got the .travis.yml content from [email protected]:OCA/OCB.git 10.0.
Please, verify access repository,
verify exists url and revision,
verify exists .travis.ymlFixesThen we need the followings fixes: class MyClass(object):
- my_var = None
+ # Fix mutable variable https://github.com/Vauxoo/travis2docker/commit/7128cc2014
+ def __init__(self):
+ self.my_var = None
def my_method(self, error=None):
+ # Fix reset to start and end: https://github.com/Vauxoo/travis2docker/commit/2b3afcce1b5565c422
+ self.my_var = None
for local_variable in range(10, 11):
self.my_var = local_variable
assert not error, error
self.my_var = NoneAdd a .travis.yml to OCA/OCB#10.0 or disable travis2docker check from runbot. (Disabled by @max3903) Sorry for this issue and thanks for help. |
|
Seriously how the hell did you figure this out @moylop260?! I just wanted to drop my wtf, let me know if I can help any 😄 |
|
Offtopic: |
|
Yeah I just ran into that a few days ago. It would be nice if we could get a report of the skipping build branches. Maybe an export from Runbot filtered by skipped, grouped by branch? |
|
Maybe our friend @max3903 could help us with that report |
|
Do we have any sort of mass-commit scripts/framework or anything? I know I've seen mass commits before, which is why I ask. |
|
They are done for the occasion. You can take a look to https://github.com/OCA/maintainer-tools/blob/master/tools/migrate_branch.py for example that some changes made on live on GitHub. |
|
hello all, I'm back from a long and mostly offline vacation which was really needed. @moylop260 congratulations for figuring this out and fixing it! 😻 @lasley I usually do these with bash scripts and the help of the clone_everything script in maintainer tools (1. clone everything and then make a loop on all repos and all branches, sed stuff and commit + push). |
|
@gurneyalex Could you help us? |
|
@moylop260 3.0.13 installed |
|
Thanks |
Randomly Runbot is failing with the following message in the all log (from Runbot build & OCA/server-tools#611 (comment)):
Build looks fine:
The text was updated successfully, but these errors were encountered: