Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky auth failure on cluster setup #310

Open
ShaneHarvey opened this issue Dec 8, 2023 · 2 comments
Open

Flaky auth failure on cluster setup #310

ShaneHarvey opened this issue Dec 8, 2023 · 2 comments

Comments

@ShaneHarvey
Copy link
Collaborator

Flaky auth failure on cluster setup:

https://spruce.mongodb.com/task/mongo_python_driver_tests_python_version_rhel8_test_ssl__platform~rhel8_auth_ssl~auth_ssl_python_version~3.7_coverage~coverage_test_4.2_replica_set_8422edf3abced2b9492123bf72ac91ab3586a69d_23_12_07_21_46_46/logs?execution=0

 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,152 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37373, host=localhost:27019
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,152 [DEBUG] mongo_orchestration.servers:342 - pid=37373, hostname=localhost:27019
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,727 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37379, host=localhost:27018
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,727 [DEBUG] mongo_orchestration.servers:342 - pid=37379, hostname=localhost:27018
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,869 [DEBUG] mongo_orchestration.process:234 - process '/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/mongodb/bin/mongod' has started: pid=37383, host=localhost:27017
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,870 [DEBUG] mongo_orchestration.servers:342 - pid=37383, hostname=localhost:27017
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,881 [ERROR] mongo_orchestration.apps:68 - <function rs_create at 0x7fef4a9a3920>
 [2023/12/07 14:07:00.236] Traceback (most recent call last):
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/__init__.py", line 66, in wrap
 [2023/12/07 14:07:00.236]     return f(*arg, **kwd)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/replica_sets.py", line 80, in rs_create
 [2023/12/07 14:07:00.236]     result = _rs_create(data)
 [2023/12/07 14:07:00.236]              ^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/apps/replica_sets.py", line 37, in _rs_create
 [2023/12/07 14:07:00.236]     rs_id = ReplicaSets().create(params)
 [2023/12/07 14:07:00.236]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 648, in create
 [2023/12/07 14:07:00.236]     repl = ReplicaSet(rs_params)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 120, in __init__
 [2023/12/07 14:07:00.236]     self.restart_with_auth()
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 159, in restart_with_auth
 [2023/12/07 14:07:00.236]     self.restart(config_callback=add_auth)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/replica_sets.py", line 617, in restart
 [2023/12/07 14:07:00.236]     f.result()
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
 [2023/12/07 14:07:00.236]     return self.__get_result()
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
 [2023/12/07 14:07:00.236]     raise self._exception
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
 [2023/12/07 14:07:00.236]     result = self.fn(*self.args, **self.kwargs)
 [2023/12/07 14:07:00.236]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 449, in restart
 [2023/12/07 14:07:00.236]     return self.start(timeout)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 351, in start
 [2023/12/07 14:07:00.236]     self.run_command('isMaster')
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 272, in run_command
 [2023/12/07 14:07:00.236]     result = getattr(self.connection.admin, mode)(command, name, **d)
 [2023/12/07 14:07:00.236]                      ^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/servers.py", line 215, in connection
 [2023/12/07 14:07:00.236]     connected(c)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/mongo_orchestration/common.py", line 130, in connected
 [2023/12/07 14:07:00.236]     client.admin.command('isMaster')
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/_csot.py", line 107, in csot_wrapper
 [2023/12/07 14:07:00.236]     return func(self, *args, **kwargs)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/database.py", line 890, in command
 [2023/12/07 14:07:00.236]     with self.__client._conn_for_reads(read_preference, session) as (
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1330, in _conn_from_server
 [2023/12/07 14:07:00.236]     with self._checkout(server, session) as conn:
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/mongo_client.py", line 1252, in _checkout
 [2023/12/07 14:07:00.236]     with server.checkout(handler=err_handler) as conn:
 [2023/12/07 14:07:00.236]   File "/opt/python/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
 [2023/12/07 14:07:00.236]     return next(self.gen)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1637, in checkout
 [2023/12/07 14:07:00.236]     conn = self._get_conn(handler=handler)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1756, in _get_conn
 [2023/12/07 14:07:00.236]     conn = self.connect(handler=handler)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1607, in connect
 [2023/12/07 14:07:00.236]     conn.authenticate()
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 1078, in authenticate
 [2023/12/07 14:07:00.236]     auth.authenticate(creds, self, reauthenticate=reauthenticate)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 625, in authenticate
 [2023/12/07 14:07:00.236]     auth_func(credentials, conn)
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 530, in _authenticate_default
 [2023/12/07 14:07:00.236]     return _authenticate_scram(credentials, conn, "SCRAM-SHA-1")
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/auth.py", line 257, in _authenticate_scram
 [2023/12/07 14:07:00.236]     res = conn.command(source, cmd)
 [2023/12/07 14:07:00.236]           ^^^^^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/helpers.py", line 322, in inner
 [2023/12/07 14:07:00.236]     return func(*args, **kwargs)
 [2023/12/07 14:07:00.236]            ^^^^^^^^^^^^^^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/pool.py", line 968, in command
 [2023/12/07 14:07:00.236]     return command(
 [2023/12/07 14:07:00.236]            ^^^^^^^^
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/network.py", line 192, in command
 [2023/12/07 14:07:00.236]     helpers._check_command_response(
 [2023/12/07 14:07:00.236]   File "/data/mci/86dc7aaa7c1c28db899d1df66c00949c/drivers-tools/.evergreen/orchestration/venv/lib/python3.11/site-packages/pymongo/helpers.py", line 230, in _check_command_response
 [2023/12/07 14:07:00.236]     raise OperationFailure(errmsg, code, response, max_wire_version)
 [2023/12/07 14:07:00.236] pymongo.errors.OperationFailure: Authentication failed., full error: {'ok': 0.0, 'errmsg': 'Authentication failed.', 'code': 18, 'codeName': 'AuthenticationFailed'}
 [2023/12/07 14:07:00.236] 2023-12-07 22:06:59,886 [DEBUG] mongo_orchestration.apps:55 - send_result(500)
 [2023/12/07 14:07:00.240] curl: (22) The requested URL returned error: 500 Internal Server Error

Perhaps the user was rolled back in the restart process? We should double check that we are using write concern majority when adding users.
@ShaneHarvey
Copy link
Collaborator Author

ShaneHarvey commented Dec 8, 2023 •
edited
Loading

We already create the user with the correct write concern:

        # Explicitly set write concern to number of data-bearing members.
        # If we add a user later, we need to guarantee that every node
        # has the user before we authenticate ('majority' is insufficient).
        self._write_concern = len(
            [m for m in members
             if not m.get('rsParams', {}).get('arbiterOnly')]
        )

@ShaneHarvey
Copy link
Collaborator Author

This error can happen when an existing cluster is running on port X and we try to start a new cluster on the same port.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ShaneHarvey