Merge branch 'main' into sqlite-rowcount/pythongh-79579-alt

Doc/c-api/call.rst

@@ -144,8 +144,6 @@ Vectorcall Support API
    However, the function ``PyVectorcall_NARGS`` should be used to allow
    for future extensions.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.8
 
 .. c:function:: vectorcallfunc PyVectorcall_Function(PyObject *op)
@@ -158,8 +156,6 @@ Vectorcall Support API
    This is mostly useful to check whether or not *op* supports vectorcall,
    which can be done by checking ``PyVectorcall_Function(op) != NULL``.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.8
 
 .. c:function:: PyObject* PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *dict)
@@ -172,8 +168,6 @@ Vectorcall Support API
    It does not check the :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag
    and it does not fall back to ``tp_call``.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.8
 
 
@@ -256,8 +250,6 @@ please see individual documentation for details.
    Return the result of the call on success, or raise an exception and return
    *NULL* on failure.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.9
 
 
@@ -343,8 +335,6 @@ please see individual documentation for details.
    Return the result of the call on success, or raise an exception and return
    *NULL* on failure.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.9
 
 
@@ -357,8 +347,6 @@ please see individual documentation for details.
    Return the result of the call on success, or raise an exception and return
    *NULL* on failure.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.9
 
 
@@ -372,8 +360,6 @@ please see individual documentation for details.
    Return the result of the call on success, or raise an exception and return
    *NULL* on failure.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.9
 
 .. c:function:: PyObject* PyObject_VectorcallDict(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwdict)
@@ -388,8 +374,6 @@ please see individual documentation for details.
    already has a dictionary ready to use for the keyword arguments,
    but not a tuple for the positional arguments.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.9
 
 .. c:function:: PyObject* PyObject_VectorcallMethod(PyObject *name, PyObject *const *args, size_t nargsf, PyObject *kwnames)
@@ -410,8 +394,6 @@ please see individual documentation for details.
    Return the result of the call on success, or raise an exception and return
    *NULL* on failure.
 
-   This function is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.9
 
 

Doc/c-api/init_config.rst

@@ -735,9 +735,8 @@ PyConfig
 
       * ``"utf-8"`` if :c:member:`PyPreConfig.utf8_mode` is non-zero.
       * ``"ascii"`` if Python detects that ``nl_langinfo(CODESET)`` announces
-        the ASCII encoding (or Roman8 encoding on HP-UX), whereas the
-        ``mbstowcs()`` function decodes from a different encoding (usually
-        Latin1).
+        the ASCII encoding, whereas the ``mbstowcs()`` function
+        decodes from a different encoding (usually Latin1).
       * ``"utf-8"`` if ``nl_langinfo(CODESET)`` returns an empty string.
       * Otherwise, use the :term:`locale encoding`:
         ``nl_langinfo(CODESET)`` result.

Doc/c-api/structures.rst

@@ -321,8 +321,6 @@ There are these calling conventions:
    or possibly ``NULL`` if there are no keywords.  The values of the keyword
    arguments are stored in the *args* array, after the positional arguments.
 
-   This is not part of the :ref:`limited API <stable>`.
-
    .. versionadded:: 3.7
 
 

Doc/c-api/type.rst

@@ -193,11 +193,12 @@ The following functions and structs are used to create
 .. c:function:: PyObject* PyType_FromMetaclass(PyTypeObject *metaclass, PyObject *module, PyType_Spec *spec, PyObject *bases)
 
    Create and return a :ref:`heap type <heap-types>` from the *spec*
-   (:const:`Py_TPFLAGS_HEAPTYPE`).
+   (see :const:`Py_TPFLAGS_HEAPTYPE`).
 
    The metaclass *metaclass* is used to construct the resulting type object.
-   When *metaclass* is ``NULL``, the default :c:type:`PyType_Type` is used
-   instead. Note that metaclasses that override
+   When *metaclass* is ``NULL``, the metaclass is derived from *bases*
+   (or *Py_tp_base[s]* slots if *bases* is ``NULL``, see below).
+   Note that metaclasses that override
    :c:member:`~PyTypeObject.tp_new` are not supported.
 
    The *bases* argument can be used to specify base classes; it can either
@@ -215,6 +216,19 @@ The following functions and structs are used to create
 
    This function calls :c:func:`PyType_Ready` on the new type.
 
+   Note that this function does *not* fully match the behavior of
+   calling :py:class:`type() <type>` or using the :keyword:`class` statement.
+   With user-provided base types or metaclasses, prefer
+   :ref:`calling <capi-call>` :py:class:`type` (or the metaclass)
+   over ``PyType_From*`` functions.
+   Specifically:
+
+   * :py:meth:`~object.__new__` is not called on the new class
+     (and it must be set to ``type.__new__``).
+   * :py:meth:`~object.__init__` is not called on the new class.
+   * :py:meth:`~object.__init_subclass__` is not called on any bases.
+   * :py:meth:`~object.__set_name__` is not called on new descriptors.
+
    .. versionadded:: 3.12
 
 .. c:function:: PyObject* PyType_FromModuleAndSpec(PyObject *module, PyType_Spec *spec, PyObject *bases)
@@ -228,17 +242,33 @@ The following functions and structs are used to create
       The function now accepts a single class as the *bases* argument and
       ``NULL`` as the ``tp_doc`` slot.
 
+   .. versionchanged:: 3.12
+
+      The function now finds and uses a metaclass corresponding to the provided
+      base classes.  Previously, only :class:`type` instances were returned.
+
 
 .. c:function:: PyObject* PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
 
    Equivalent to ``PyType_FromMetaclass(NULL, NULL, spec, bases)``.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.12
+
+      The function now finds and uses a metaclass corresponding to the provided
+      base classes.  Previously, only :class:`type` instances were returned.
+
 .. c:function:: PyObject* PyType_FromSpec(PyType_Spec *spec)
 
    Equivalent to ``PyType_FromMetaclass(NULL, NULL, spec, NULL)``.
 
+   .. versionchanged:: 3.12
+
+      The function now finds and uses a metaclass corresponding to the
+      base classes provided in *Py_tp_base[s]* slots.
+      Previously, only :class:`type` instances were returned.
+
 .. c:type:: PyType_Spec
 
    Structure defining a type's behavior.
@@ -266,6 +296,8 @@ The following functions and structs are used to create
       Array of :c:type:`PyType_Slot` structures.
       Terminated by the special slot value ``{0, NULL}``.
 
+      Each slot ID should be specified at most once.
+
 .. c:type:: PyType_Slot
 
    Structure defining optional functionality of a type, containing a slot ID

Doc/c-api/typeobj.rst

@@ -727,12 +727,6 @@ and :c:type:`PyType_Type` effectively act as defaults.)
       When a user sets :attr:`__call__` in Python code, only *tp_call* is updated,
       likely making it inconsistent with the vectorcall function.
 
-   .. note::
-
-      The semantics of the ``tp_vectorcall_offset`` slot are provisional and
-      expected to be finalized in Python 3.9.
-      If you use vectorcall, plan for updating your code for Python 3.9.
-
    .. versionchanged:: 3.8
 
       Before version 3.8, this slot was named ``tp_print``.

Doc/faq/library.rst

@@ -483,8 +483,14 @@ including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and
 How do I copy a file?
 ---------------------
 
-The :mod:`shutil` module contains a :func:`~shutil.copyfile` function.  Note
-that on MacOS 9 it doesn't copy the resource fork and Finder info.
+The :mod:`shutil` module contains a :func:`~shutil.copyfile` function.
+Note that on Windows NTFS volumes, it does not copy
+`alternate data streams
+<https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(ADS)>`_
+nor `resource forks <https://en.wikipedia.org/wiki/Resource_fork>`__
+on macOS HFS+ volumes, though both are now rarely used.
+It also doesn't copy file permissions and metadata, though using
+:func:`shutil.copy2` instead will preserve most (though not all) of it.
 
 
 How do I read (or write) binary data?

Doc/howto/logging-cookbook.rst

@@ -3022,6 +3022,95 @@ refer to the comments in the code snippet for more detailed information.
     if __name__=='__main__':
         main()
 
+Logging to syslog with RFC5424 support
+--------------------------------------
+
+Although :rfc:`5424` dates from 2009, most syslog servers are configured by detault to
+use the older :rfc:`3164`, which hails from 2001. When ``logging`` was added to Python
+in 2003, it supported the earlier (and only existing) protocol at the time. Since
+RFC5424 came out, as there has not been widespread deployment of it in syslog
+servers, the :class:`~logging.handlers.SysLogHandler` functionality has not been
+updated.
+
+RFC 5424 contains some useful features such as support for structured data, and if you
+need to be able to log to a syslog server with support for it, you can do so with a
+subclassed handler which looks something like this::
+
+    import datetime
+    import logging.handlers
+    import re
+    import socket
+    import time
+
+    class SysLogHandler5424(logging.handlers.SysLogHandler):
+
+        tz_offset = re.compile(r'([+-]\d{2})(\d{2})$')
+        escaped = re.compile(r'([\]"\\])')
+
+        def __init__(self, *args, **kwargs):
+            self.msgid = kwargs.pop('msgid', None)
+            self.appname = kwargs.pop('appname', None)
+            super().__init__(*args, **kwargs)
+
+        def format(self, record):
+            version = 1
+            asctime = datetime.datetime.fromtimestamp(record.created).isoformat()
+            m = self.tz_offset.match(time.strftime('%z'))
+            has_offset = False
+            if m and time.timezone:
+                hrs, mins = m.groups()
+                if int(hrs) or int(mins):
+                    has_offset = True
+            if not has_offset:
+                asctime += 'Z'
+            else:
+                asctime += f'{hrs}:{mins}'
+            try:
+                hostname = socket.gethostname()
+            except Exception:
+                hostname = '-'
+            appname = self.appname or '-'
+            procid = record.process
+            msgid = '-'
+            msg = super().format(record)
+            sdata = '-'
+            if hasattr(record, 'structured_data'):
+                sd = record.structured_data
+                # This should be a dict where the keys are SD-ID and the value is a
+                # dict mapping PARAM-NAME to PARAM-VALUE (refer to the RFC for what these
+                # mean)
+                # There's no error checking here - it's purely for illustration, and you
+                # can adapt this code for use in production environments
+                parts = []
+
+                def replacer(m):
+                    g = m.groups()
+                    return '\\' + g[0]
+
+                for sdid, dv in sd.items():
+                    part = f'[{sdid}'
+                    for k, v in dv.items():
+                        s = str(v)
+                        s = self.escaped.sub(replacer, s)
+                        part += f' {k}="{s}"'
+                    part += ']'
+                    parts.append(part)
+                sdata = ''.join(parts)
+            return f'{version} {asctime} {hostname} {appname} {procid} {msgid} {sdata} {msg}'
+
+You'll need to be familiar with RFC 5424 to fully understand the above code, and it
+may be that you have slightly different needs (e.g. for how you pass structural data
+to the log). Nevertheless, the above should be adaptable to your speciric needs. With
+the above handler, you'd pass structured data using something like this::
+
+    sd = {
+        'foo@12345': {'bar': 'baz', 'baz': 'bozz', 'fizz': r'buzz'},
+        'foo@54321': {'rab': 'baz', 'zab': 'bozz', 'zzif': r'buzz'}
+    }
+    extra = {'structured_data': sd}
+    i = 1
+    logger.debug('Message %d', i, extra=extra)
+
 
 .. patterns-to-avoid:
 

Doc/howto/sockets.rst

@@ -252,20 +252,25 @@ Binary Data
 -----------
 
 It is perfectly possible to send binary data over a socket. The major problem is
-that not all machines use the same formats for binary data. For example, a
-Motorola chip will represent a 16 bit integer with the value 1 as the two hex
-bytes 00 01. Intel and DEC, however, are byte-reversed - that same 1 is 01 00.
+that not all machines use the same formats for binary data. For example,
+`network byte order <https://en.wikipedia.org/wiki/Endianness#Networking>`_
+is big-endian, with the most significant byte first,
+so a 16 bit integer with the value ``1`` would be the two hex bytes ``00 01``.
+However, most common processors (x86/AMD64, ARM, RISC-V), are little-endian,
+with the least significant byte first - that same ``1`` would be ``01 00``.
+
 Socket libraries have calls for converting 16 and 32 bit integers - ``ntohl,
 htonl, ntohs, htons`` where "n" means *network* and "h" means *host*, "s" means
 *short* and "l" means *long*. Where network order is host order, these do
 nothing, but where the machine is byte-reversed, these swap the bytes around
 appropriately.
 
-In these days of 32 bit machines, the ascii representation of binary data is
+In these days of 64-bit machines, the ASCII representation of binary data is
 frequently smaller than the binary representation. That's because a surprising
-amount of the time, all those longs have the value 0, or maybe 1. The string "0"
-would be two bytes, while binary is four. Of course, this doesn't fit well with
-fixed-length messages. Decisions, decisions.
+amount of the time, most integers have the value 0, or maybe 1.
+The string ``"0"`` would be two bytes, while a full 64-bit integer would be 8.
+Of course, this doesn't fit well with fixed-length messages.
+Decisions, decisions.
 
 
 Disconnecting

Doc/library/logging.rst

@@ -30,9 +30,17 @@ is that all Python modules can participate in logging, so your application log
 can include your own messages integrated with messages from third-party
 modules.
 
+The simplest example:
+
+.. code-block:: none
+
+    >>> import logging
+    >>> logging.warning('Watch out!')
+    WARNING:root:Watch out!
+
 The module provides a lot of functionality and flexibility.  If you are
-unfamiliar with logging, the best way to get to grips with it is to see the
-tutorials (see the links on the right).
+unfamiliar with logging, the best way to get to grips with it is to view the
+tutorials (**see the links above and on the right**).
 
 The basic classes defined by the module, together with their functions, are
 listed below.

Doc/library/mmap.rst

@@ -102,7 +102,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
 
    To ensure validity of the created memory mapping the file specified
    by the descriptor *fileno* is internally automatically synchronized
-   with physical backing store on macOS and OpenVMS.
+   with the physical backing store on macOS.
 
    This example shows a simple way of using :class:`~mmap.mmap`::
 

Doc/library/os.path.rst

@@ -469,7 +469,7 @@ the :mod:`glob` module.)
       ("c:", "/dir")
 
    If the path contains a UNC path, drive will contain the host name
-   and share, up to but not including the fourth separator::
+   and share::
 
       >>> splitdrive("//host/computer/dir")
       ("//host/computer", "/dir")