pynamodb startswith works with moto but not under pytest #1996

dedominicisfa · 2018-12-14T11:54:32Z

environment:
windows 10 64 bit python 3.6

see zip for code to reproduce issue.
do a virtualenv and install requirements. model, test_model and test_dynamodb contain the same code, except that the 2nd and 3rd are wrapped in order for pytest to work. The 3rd, doesn't use moto, but a dynamodb local listening on localhost 8000.
If you execute
python model.py,
everything works correctly.
if, in the test dir, you execute
pytest
the 'query' call using moto mock which uses the startswith, fails with
============================= test session starts =============================
platform win32 -- Python 3.6.4, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
rootdir: C:\Users\FIM514, inifile: pytest.ini
collected 1 item

test_model.py F [100%]

================================== FAILURES ===================================
______________________________ MyTest.test_main _______________________________

self = <test.test_model.MyTest testMethod=test_main>

def test_main(self):
    model.create_table(wait=True)
    model(f1='1', f2='2').save()
    print(next(model.query('1', model.f2=='2')))

  print(next(model.query('1', model.f2.startswith('2'))))

test_model.py:24:

..\envs\temp\lib\site-packages\pynamodb\pagination.py:173: in next
self._get_next_page()
..\envs\temp\lib\site-packages\pynamodb\pagination.py:158: in _get_next_page
page = next(self.page_iter)
..\envs\temp\lib\site-packages\pynamodb\pagination.py:103: in next
page = self._operation(*self._args, **self._kwargs)
..\envs\temp\lib\site-packages\pynamodb\connection\table.py:257: in query
select=select)
..\envs\temp\lib\site-packages\pynamodb\connection\base.py:1409: in query
return self.dispatch(QUERY, operation_kwargs)
..\envs\temp\lib\site-packages\pynamodb\connection\base.py:313: in dispatch
data = self._make_api_call(operation_name, operation_kwargs)
..\envs\temp\lib\site-packages\pynamodb\connection\base.py:362: in _make_api_call
proxies=proxies,
..\envs\temp\lib\site-packages\requests\sessions.py:646: in send
r = adapter.send(request, **kwargs)
..\envs\temp\lib\site-packages\responses.py:622: in unbound_on_send
return self._on_request(adapter, request, *a, **kwargs)
..\envs\temp\lib\site-packages\responses.py:600: in _on_request
response = adapter.build_response(request, match.get_response(request))
..\envs\temp\lib\site-packages\moto\core\models.py:151: in get_response
result = self.callback(request)
..\envs\temp\lib\site-packages\moto\core\utils.py:173: in call
result = self.callback(request, request.url, request.headers)
..\envs\temp\lib\site-packages\moto\core\responses.py:117: in dispatch
return cls()._dispatch(*args, **kwargs)
..\envs\temp\lib\site-packages\moto\core\responses.py:200: in _dispatch
return self.call_action()
..\envs\temp\lib\site-packages\moto\core\utils.py:264: in _wrapper
response = f(*args, **kwargs)
..\envs\temp\lib\site-packages\moto\dynamodb2\responses.py:64: in call_action
response = getattr(self, endpoint)()

self = <moto.dynamodb2.responses.DynamoHandler object at 0x000002DC6554C6D8>

def query(self):
    name = self.body['TableName']
    # {u'KeyConditionExpression': u'#n0 = :v0', u'ExpressionAttributeValues': {u':v0': {u'S': u'johndoe'}}, u'ExpressionAttributeNames': {u'#n0': u'username'}}
    key_condition_expression = self.body.get('KeyConditionExpression')
    projection_expression = self.body.get('ProjectionExpression')
    expression_attribute_names = self.body.get('ExpressionAttributeNames', {})
    filter_expression = self.body.get('FilterExpression')
    expression_attribute_values = self.body.get('ExpressionAttributeValues', {})

    if projection_expression and expression_attribute_names:
        expressions = [x.strip() for x in projection_expression.split(',')]
        for expression in expressions:
            if expression in expression_attribute_names:
                projection_expression = projection_expression.replace(expression, expression_attribute_names[expression])

    filter_kwargs = {}

    if key_condition_expression:
        value_alias_map = self.body.get('ExpressionAttributeValues', {})

        table = self.dynamodb_backend.get_table(name)

        # If table does not exist
        if table is None:
            return self.error('com.amazonaws.dynamodb.v20120810#ResourceNotFoundException',
                              'Requested resource not found')

        index_name = self.body.get('IndexName')
        if index_name:
            all_indexes = (table.global_indexes or []) + \
                (table.indexes or [])
            indexes_by_name = dict((i['IndexName'], i)
                                   for i in all_indexes)
            if index_name not in indexes_by_name:
                raise ValueError('Invalid index: %s for table: %s. Available indexes are: %s' % (
                    index_name, name, ', '.join(indexes_by_name.keys())
                ))

            index = indexes_by_name[index_name]['KeySchema']
        else:
            index = table.schema

        reverse_attribute_lookup = dict((v, k) for k, v in
                                        six.iteritems(self.body.get('ExpressionAttributeNames', {})))

        if " AND " in key_condition_expression:
            expressions = key_condition_expression.split(" AND ", 1)

            index_hash_key = [key for key in index if key['KeyType'] == 'HASH'][0]
            hash_key_var = reverse_attribute_lookup.get(index_hash_key['AttributeName'],
                                                        index_hash_key['AttributeName'])
            hash_key_regex = r'(^|[\s(]){0}\b'.format(hash_key_var)
            i, hash_key_expression = next((i, e) for i, e in enumerate(expressions)
                                          if re.search(hash_key_regex, e))
            hash_key_expression = hash_key_expression.strip('()')
            expressions.pop(i)

            # TODO implement more than one range expression and OR operators
            range_key_expression = expressions[0].strip('()')
            range_key_expression_components = range_key_expression.split()
            range_comparison = range_key_expression_components[1]

            if 'AND' in range_key_expression:
                range_comparison = 'BETWEEN'
                range_values = [
                    value_alias_map[range_key_expression_components[2]],
                    value_alias_map[range_key_expression_components[4]],
                ]
            elif 'begins_with' in range_key_expression:
                range_comparison = 'BEGINS_WITH'
                range_values = [

                  value_alias_map[range_key_expression_components[1]],

E KeyError: '(#1,'

..\envs\temp\lib\site-packages\moto\dynamodb2\responses.py:382: KeyError
---------------------------- Captured stdout call -----------------------------
table<1, 2>

the query call in test_dynamodb, once I start dynamodb-local so I don't use moto mock, works correctly.
I expect the 2nd case to be working correctly too.
test.zip

The text was updated successfully, but these errors were encountered:

spulec · 2019-07-09T02:31:20Z

Can you please add the code needed to reproduce this to the issue text? Thanks

elzilrac · 2019-07-16T20:36:00Z

Also running into this issue. Here's a small example (using pynamodb for table declaration)

import unittest
from pynamodb.attributes import UnicodeAttribute
from pynamodb.attributes import NumberAttribute
from pynamodb.models import Model
from moto import mock_dynamodb2


class TestTable(Model):

    class Meta:
        table_name = 'TestTable'
        read_capacity_units = 25
        write_capacity_units = 25

    user_id = NumberAttribute(hash_key=True)
    table_id = UnicodeAttribute(range_key=True)


@mock_dynamodb2
class TestTestTable(unittest.TestCase):

    def setUp(self):
        TestTable.create_table(wait=True)

    def tearDown(self):
        TestTable.delete_table()

    def test_beginswith(self):
        try:
            return TestTable.query(
                1234,
                range_key_condition=(TestTable.table_id.startswith('something')),
                scan_index_forward=False,
                limit=1,
            ).next()
        except StopIteration:
            return None

Python 3.7.3
boto==2.49.0
boto3==1.9.133
botocore==1.12.133
moto==1.3.7
pynamodb==4.0.0b1

spulec · 2019-07-20T15:56:09Z

I think that might be the same as #416

Can you try activating the mock in your setUp with a manual mock.

elzilrac · 2019-07-22T22:52:41Z

Like this? (I get the keyerror with the startswith, but doing a plain query works fine)

import unittest
from pynamodb.attributes import UnicodeAttribute
from pynamodb.attributes import NumberAttribute
from pynamodb.models import Model
from moto import mock_dynamodb2


class TestTable(Model):

    class Meta:
        table_name = 'TestTable'
        read_capacity_units = 25
        write_capacity_units = 25

    user_id = NumberAttribute(hash_key=True)
    table_id = UnicodeAttribute(range_key=True)


class TestTestTable(unittest.TestCase):

    def test_beginswith(self):  # Keyerror
        mock = mock_dynamodb2()
        mock.start()
        TestTable.create_table(wait=True)

        try:
            return TestTable.query(
                1234,
                range_key_condition=(TestTable.table_id.startswith('something')),
                scan_index_forward=False,
                limit=1,
            ).next()
        except StopIteration:
            return None

        TestTable.delete_table()
        mock.stop()

    def test_query(self):  # Works fine
        mock = mock_dynamodb2()
        mock.start()
        TestTable.create_table(wait=True)

        try:
            return TestTable.query(
                1234,
            ).next()
        except StopIteration:
            return None

        TestTable.delete_table()
        mock.stop()

spulec · 2019-07-24T03:50:32Z

Can you print the traceback for the KeyError? It sounds like it might be a different issue, but at least the mock is being activated.

elzilrac · 2019-07-25T21:55:21Z

self = <test.test.TestTestTable testMethod=test_beginswith>

    def test_beginswith(self):  # Keyerror
        mock = mock_dynamodb2()
        mock.start()
        TestTable.create_table(wait=True)
    
        try:
            return TestTable.query(
                1234,
                range_key_condition=(TestTable.table_id.startswith('something')),
                scan_index_forward=False,
>               limit=1,
            ).next()

test/test.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv/lib/python3.7/site-packages/pynamodb/pagination.py:187: in next
    return self.__next__()
venv/lib/python3.7/site-packages/pynamodb/pagination.py:173: in __next__
    self._get_next_page()
venv/lib/python3.7/site-packages/pynamodb/pagination.py:158: in _get_next_page
    page = next(self.page_iter)
venv/lib/python3.7/site-packages/pynamodb/pagination.py:103: in __next__
    page = self._operation(*self._args, **self._kwargs)
venv/lib/python3.7/site-packages/pynamodb/connection/table.py:261: in query
    select=select)
venv/lib/python3.7/site-packages/pynamodb/connection/base.py:1424: in query
    return self.dispatch(QUERY, operation_kwargs)
venv/lib/python3.7/site-packages/pynamodb/connection/base.py:316: in dispatch
    data = self._make_api_call(operation_name, operation_kwargs)
venv/lib/python3.7/site-packages/pynamodb/connection/base.py:369: in _make_api_call
    event_responses = self.client._endpoint._event_emitter.emit(event_name, request=prepared_request)
venv/lib/python3.7/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
venv/lib/python3.7/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
venv/lib/python3.7/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
venv/lib/python3.7/site-packages/moto/core/models.py:292: in __call__
    status, headers, body = response_callback(request, request.url, request.headers)
venv/lib/python3.7/site-packages/moto/core/responses.py:117: in dispatch
    return cls()._dispatch(*args, **kwargs)
venv/lib/python3.7/site-packages/moto/core/responses.py:200: in _dispatch
    return self.call_action()
venv/lib/python3.7/site-packages/moto/core/utils.py:264: in _wrapper
    response = f(*args, **kwargs)
venv/lib/python3.7/site-packages/moto/dynamodb2/responses.py:64: in call_action
    response = getattr(self, endpoint)()

(sorry, I've not been getting notifications of this thread, so my responses have been a bit delayed)

spulec · 2019-07-26T02:40:11Z

Does the traceback go deeper than that? That particular line in responses.py seems unusual to throw a KeyError.

elzilrac · 2019-07-26T22:09:22Z

Nope, sorry.

venv/lib/python3.7/site-packages/moto/dynamodb2/responses.py:64: in call_action
    response = getattr(self, endpoint)()

is the last thing returned.

spulec · 2019-07-28T22:18:39Z

Can you put a pdb inside of dynamodb2/responses.py:query and step through it until the exception happens? I'm trying to isolate where this is coming from and I'm struggling to reproduce it.

elzilrac · 2019-07-29T18:47:47Z

test/test.py > python3.7/site-packages/moto/dynamodb2/responses.py(315)query()
    314         # {u'KeyConditionExpression': u'#n0 = :v0', u'ExpressionAttributeValues': {u':v0': {u'S': u'johndoe'}}, u'ExpressionAttributeNames': {u'#n0': u'username'}}
--> 315         key_condition_expression = self.body.get('KeyConditionExpression')
    316         projection_expression = self.body.get('ProjectionExpression')

ipdb> self.body                                                                                                   
{'TableName': 'TestTable', 'KeyConditionExpression': '(#0 = :0 AND begins_with (#1, :1))', 'Limit': 1, 'ScanIndexForward': False, 'ExpressionAttributeNames': {'#0': 'user_id', '#1': 'table_id'}, 'ExpressionAttributeValues': {':0': {'N': '1234'}, ':1': {'S': 'something'}}, 'ReturnConsumedCapacity': 'TOTAL'}

...

ipdb> key_condition_expression                                                                                    
'(#0 = :0 AND begins_with (#1, :1))'
ipdb> projection_expression                                                                                       
ipdb> expression_attribute_names                                                                                  
{'#0': 'user_id', '#1': 'table_id'}
ipdb> filter_expression                                                                                           
ipdb> expression_attribute_values                                                                                 
{':0': {'N': '1234'}, ':1': {'S': 'something'}}
ipdb>

...

> python3.7/site-packages/moto/dynamodb2/responses.py(381)query()
    380                 elif 'begins_with' in range_key_expression:
--> 381                     range_comparison = 'BEGINS_WITH'
    382                     range_values = [

ipdb> n                                                                                                           
> python3.7/site-packages/moto/dynamodb2/responses.py(383)query()
    382                     range_values = [
--> 383                         value_alias_map[range_key_expression_components[1]],
    384                     ]

ipdb> n                                                                                                           
KeyError: '(#1,'
> python3.7/site-packages/moto/dynamodb2/responses.py(383)query()
    382                     range_values = [
--> 383                         value_alias_map[range_key_expression_components[1]],
    384                     ]

ipdb> range_key_expression_components                                                                             
['begins_with', '(#1,', ':1']
ipdb> value_alias_map                                                                                             
{':0': {'N': '1234'}, ':1': {'S': 'something'}}

Looks like it was expecting the ':1' to be the 1st index and not '(#1'

Doing some more digging (eg comparing to the tests in test_dynamodb2/test_dynamodb_table_with_range_key.py), and in my test case, there ends up being a space after "begins_with".... whereas

# test_dynamodb_table_with_range_key.py
ipdb> range_key_expression                                                     
'begins_with(#n1, :v1'

vs

# test.py
begins_with (#1, :1))'

getmoto#1996

spulec · 2019-07-30T03:24:23Z

Ah, got it. So it sounds like we are being too strict in what we are parsing. I'm going to set this as a feature to make the parsing be looser.

lradeck · 2019-09-09T17:39:08Z

I have this problem too, any news here?

A query fails if it has a space between `begins_with` and `(`, for example: ```begins_with (getmoto#1, :1)``` Fix getmoto#1996

klawson4311 · 2021-02-25T16:12:05Z

Same issue, same lines, except that in my case, I'm querying a GSI.
pynamodb = "~=4.3.2"
pytest = "==6.2.1"
moto = "==1.3.10"

Model.gsi.query(
                hash_key=hash_key,
                range_key_condition=Model.key.startswith(search_term),
                filter_condition=((Model.status == "A"),
                limit=1,
            )

bblommers · 2021-02-25T16:37:01Z

Hi @klawson, might be worth upgrading moto to the latest version.
If the problem still persists with 2.x, please open a new issue with a reproducible test case.

klawson4311 · 2021-02-25T17:06:28Z

affirmative. Updating to 2.x resolved this issue. Thank you!

spulec added the debugging Working with user to figure out if there is an issue label Jul 20, 2019

elzilrac added a commit to elzilrac/moto that referenced this issue Jul 29, 2019

Change range_key_expression_components for begins_with

fc02ceb

getmoto#1996

elzilrac mentioned this issue Jul 29, 2019

Change range_key_expression_components for begins_with #2341

Closed

spulec added enhancement and removed debugging Working with user to figure out if there is an issue labels Jul 30, 2019

shmygol added a commit to shmygol/moto that referenced this issue Nov 21, 2019

Improve parsing DynamoDB begins_with expression

6ea6001

A query fails if it has a space between `begins_with` and `(`, for example: ```begins_with (getmoto#1, :1)``` Fix getmoto#1996

shmygol mentioned this issue Nov 21, 2019

Improve parsing DynamoDB begins_with expression #2583

Merged

mikegrima closed this as completed in #2583 Dec 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pynamodb startswith works with moto but not under pytest #1996

pynamodb startswith works with moto but not under pytest #1996

dedominicisfa commented Dec 14, 2018

spulec commented Jul 9, 2019

elzilrac commented Jul 16, 2019

spulec commented Jul 20, 2019

elzilrac commented Jul 22, 2019

spulec commented Jul 24, 2019

elzilrac commented Jul 25, 2019

spulec commented Jul 26, 2019

elzilrac commented Jul 26, 2019

spulec commented Jul 28, 2019

elzilrac commented Jul 29, 2019 •

edited

Loading

spulec commented Jul 30, 2019

lradeck commented Sep 9, 2019

klawson4311 commented Feb 25, 2021

bblommers commented Feb 25, 2021

klawson4311 commented Feb 25, 2021

pynamodb startswith works with moto but not under pytest #1996

pynamodb startswith works with moto but not under pytest #1996

Comments

dedominicisfa commented Dec 14, 2018

spulec commented Jul 9, 2019

elzilrac commented Jul 16, 2019

spulec commented Jul 20, 2019

elzilrac commented Jul 22, 2019

spulec commented Jul 24, 2019

elzilrac commented Jul 25, 2019

spulec commented Jul 26, 2019

elzilrac commented Jul 26, 2019

spulec commented Jul 28, 2019

elzilrac commented Jul 29, 2019 • edited Loading

spulec commented Jul 30, 2019

lradeck commented Sep 9, 2019

klawson4311 commented Feb 25, 2021

bblommers commented Feb 25, 2021

klawson4311 commented Feb 25, 2021

elzilrac commented Jul 29, 2019 •

edited

Loading