Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-39337: Add a test case for normalizing of codec names #19069

Merged
merged 17 commits into from
Oct 8, 2020
Merged

bpo-39337: Add a test case for normalizing of codec names #19069

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e9ecac0
add a test case to against nomralizeing() of codecs
shihai1991 Mar 19, 2020
cf46290
remove redundant operation
shihai1991 Mar 19, 2020
59911e1
not clear all codec_search_paths
shihai1991 Mar 20, 2020
aa3c4d0
remove codecs_unregister()
shihai1991 Mar 21, 2020
aa4bf51
remove useless head file
shihai1991 Mar 21, 2020
8ce5956
remove useless import
shihai1991 Mar 21, 2020
41a1672
retrigger checks
shihai1991 Apr 2, 2020
00b300c
Merge remote-tracking branch 'origin/master' into bpo_1635741_1
shihai1991 Sep 8, 2020
f48e5d9
add codecs_unregister() in testinternalcapi
shihai1991 Sep 10, 2020
3795420
update code
shihai1991 Sep 10, 2020
e3b9365
Merge remote-tracking branch 'origin/master' into bpo_1635741_1
shihai1991 Sep 29, 2020
a2887f3
Using codecs.unregister() in test case
shihai1991 Sep 29, 2020
632a448
apply Tal's comment
shihai1991 Sep 29, 2020
bb6e78c
apply victor's comment
shihai1991 Oct 3, 2020
72e2435
apply tal's comment
shihai1991 Oct 3, 2020
26fcbe4
apply victor's comment
shihai1991 Oct 5, 2020
37b80e0
apply victor's comment
shihai1991 Oct 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions Lib/test/test_codecs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from test import support
from test.support import os_helper
from test.support import warnings_helper
import _testinternalcapi

try:
import _testcapi
Expand Down Expand Up @@ -3403,5 +3404,19 @@ def test_rot13_func(self):
'To be, or not to be, that is the question')


class NormalizedTest(unittest.TestCase):
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
"""Test the normalizestring function via codecs module"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that the codec names are normalized via the normalizestring function is an implementation detail, so the test should be written ignoring that detail, to the extent possible. This includes its documentation, so...

Suggested change
"""Test the normalizestring function via codecs module"""
"""Test codec name normalization"""

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I considered this details before. Why I am still leave this detail?
IMHO, leave enough details can help developers to maintain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're welcome to add a comment after the doc-string. I wouldn't want the doc-string to mention normalizestring, though, unless we change the test to call normalizestring() directly instead of codecs.lookup(), which I don't recommend.

From my experience, however, such comments often become outdated when the implementation changes. And outdated comments causing confusion are usually worse than no comments.

Also, if someone needs to see the details, they'll likely need to go through the code starting at codecs.lookup() anyways. It's relatively straightforward to follow: codecs imports lookup from _codecs, implemented in Modules/_codecsmodule.c. lookup() is a simple wrapper for _PyCodec_Lookup from Python/codecs.c. With the imports and wrappers out of the way, _PyCodec_Lookup calls normalizestring() near the top of the function, immediately after a bit of initialization.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my experience, however, such comments often become outdated when the implementation changes. And outdated comments causing confusion are usually worse than no comments.

Thanks for your share, Tal. It's make sense :)

def test_normalized_encoding(self):
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
def search_function(encoding):
if encoding == "aaa_8":
return (1, 2, 3, 4)
else:
return (None, None, None, None)
codecs.register(search_function)
self.assertEqual((1, 2, 3, 4), codecs.lookup('AAA-8'))
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
self.assertEqual((None, None, None, None), codecs.lookup('BBB-8'))
_testinternalcapi.codecs_unregister(search_function)


if __name__ == "__main__":
unittest.main()
42 changes: 38 additions & 4 deletions Modules/_testinternalcapi.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@
#define PY_SSIZE_T_CLEAN

#include "Python.h"
#include "pycore_bitutils.h" // _Py_bswap32()
#include "pycore_initconfig.h" // _Py_GetConfigsAsDict()
#include "pycore_hashtable.h" // _Py_hashtable_new()
#include "pycore_gc.h" // PyGC_Head
#include "pycore_bitutils.h" // _Py_bswap32()
#include "pycore_initconfig.h" // _Py_GetConfigsAsDict()
#include "pycore_hashtable.h" // _Py_hashtable_new()
#include "pycore_gc.h" // PyGC_Head
#include "internal/pycore_interp.h" // PyInterpreterState_Get()


static PyObject *
Expand All @@ -35,6 +36,38 @@ get_recursion_depth(PyObject *self, PyObject *Py_UNUSED(args))
}


static PyObject*
codecs_unregister(PyObject *self, PyObject *args)
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
{
PyObject *search_function;

if (!PyArg_ParseTuple(args, "O:codecs_unregister",
&search_function)) {
return NULL;
}
if (search_function == NULL) {
return NULL;
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
}

PyInterpreterState *interp = PyInterpreterState_Get();
if (interp->codec_search_path == NULL) {
return NULL;
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
}

Py_ssize_t n = PyList_Size(interp->codec_search_path);
PyObject *list = PyList_New(0);
for (Py_ssize_t i = 0; i < n; i++) {
PyObject *item = PyList_GetItem(interp->codec_search_path, i);
shihai1991 marked this conversation as resolved.
Show resolved Hide resolved
if (item != search_function) {
PyList_Append(list, item);
}
}
Py_SETREF(interp->codec_search_path, list);

Py_RETURN_NONE;
taleinat marked this conversation as resolved.
Show resolved Hide resolved
}


static PyObject*
test_bswap(PyObject *self, PyObject *Py_UNUSED(args))
{
Expand Down Expand Up @@ -234,6 +267,7 @@ test_hashtable(PyObject *self, PyObject *Py_UNUSED(args))
static PyMethodDef TestMethods[] = {
{"get_configs", get_configs, METH_NOARGS},
{"get_recursion_depth", get_recursion_depth, METH_NOARGS},
{"codecs_unregister", codecs_unregister, METH_VARARGS},
{"test_bswap", test_bswap, METH_NOARGS},
{"test_popcount", test_popcount, METH_NOARGS},
{"test_bit_length", test_bit_length, METH_NOARGS},
Expand Down