mod_wsgi error with base64 image and pkgsrc 2013Q3 #181

bahamat · 2014-03-03T09:48:41Z

If I include mod_wsgi in my Apache config I get the following error:

httpd: Syntax error on line 486 of /opt/local/etc/httpd/httpd.conf: Syntax error on line 6 of /opt/local/etc/httpd/wsgi-vhost.conf: Cannot load lib/httpd/mod_wsgi.so into server: ld.so.1: httpd: fatal: relocation error: file /opt/local/lib/httpd/mod_wsgi.so: symbol unixd_config: referenced symbol not found

Line 6 of that file is:

LoadModule wsgi_module lib/httpd/mod_wsgi.so

The text was updated successfully, but these errors were encountered:

bahamat · 2014-03-03T21:51:18Z

I figured out what I was doing wrong. This happened because apache-2.4.6 and ap22-py27-wsgi were installed.

When using either the 2.4 or 2.2 version of either it works.

FITS (Flexible Image Transport System) is a data format most used in astronomy. PyFITS is a Python module for reading, writing, and manipulating FITS files. The module uses Python's object-oriented features to provide quick, easy, and efficient access to FITS files. The use of Python's array syntax enables immediate access to any FITS extension, header cards, or data items. Changes to 2.4.0 (in py-pyfits): Changelog =========== 3.2 (2013-11-26) ---------------- Highlights ^^^^^^^^^^ - Rewrote CFITSIO-based backend for handling tile compression of FITS files. It now uses a standard CFITSIO instead of heavily modified pieces of CFITSIO as before. PyFITS ships with its own copy of CFITSIO v3.35 which supports the latest version of the Tiled Image Convention (v2.3), but system packagers may choose instead to strip this out in favor of a system-installed version of CFITSIO. Earlier versions may work, but nothing earlier than 3.28 has been tested yet. (#169) - Added support for reading and writing tables using the Q format for columns. The Q format is identical to the P format (variable-length arrays) except that it uses 64-bit integers for the data descriptors, allowing more than 4 GB of variable-length array data in a single table. (#160) - Added initial support for table columns containing pseudo-unsigned integers. This is currently enabled by using the ``uint=True`` option when opening files; any table columns with the correct BZERO value will be interpreted and returned as arrays of unsigned integers. - Some refactoring of the table and ``FITS_rec`` modules in order to better separate the details of the FITS binary and ASCII table data structures from the HDU data structures that encapsulate them. Most of these changes should not be apparent to users (but see API Changes below). API Changes ^^^^^^^^^^^ - Assigning to values in ``ColDefs.names``, ``ColDefs.formats``, ``ColDefs.nulls`` and other attributes of ``ColDefs`` instances that return lists of column properties is no longer supported. Assigning to those lists will no longer update the corresponding columns. Instead, please just modify the ``Column`` instances directly (``Column.name``, ``Column.null``, etc.) - The ``pyfits.new_table`` function is marked "pending deprecation". This does not mean it will be removed outright or that its functionality has changed. It will likely be replaced in the future for a function with similar, if not subtly different functionality. A better, if not slightly more verbose approach is to use ``pyfits.FITS_rec.from_columns`` to create a new ``FITS_rec`` table--this has the same interface as ``pyfits.new_table``. The difference is that it returns a plan ``FITS_rec`` array, and not an HDU instance. This ``FITS_rec`` object can then be used as the data argument in the constructors for ``BinTableHDU`` (for binary tables) or ``TableHDU`` (for ASCII tables). This is analogous to creating an ``ImageHDU`` by passing in an image array. ``pyfits.FITS_rec.from_columns`` is just a simpler way of creating a FITS-compatible recarray from a FITS column specification. - The ``updateHeader``, ``updateHeaderData``, and ``updateCompressedData`` methods of the ``CompDataHDU`` class are pending deprecation and moved to internal methods. The operation of these methods depended too much on internal state to be used safely by users; instead they are invoked automatically in the appropriate places when reading/writing compressed image HDUs. - The ``CompDataHDU.compData`` attribute is pending deprecation in favor of the clearer and more PEP-8 compatible ``CompDataHDU.compressed_data``. - The constructor for ``CompDataHDU`` has been changed to accept new keyword arguments. The new keyword arguments are essentially the same, but are in underscore_separated format rather than camelCase format. The old arguments are still pending deprecation. - The internal attributes of HDU classes ``_hdrLoc``, ``_datLoc``, and ``_datSpan`` have been replaced with ``_header_offset``, ``_data_offset``, and ``_data_size`` respectively. The old attribute names are still pending deprecation. This should only be of interest to advanced users who have created their own HDU subclasses. - The following previously deprecated functions and methods have been removed entirely: ``createCard``, ``createCardFromString``, ``upperKey``, ``ColDefs.data``, ``setExtensionNameCaseSensitive``, ``_File.getfile``, ``_TableBaseHDU.get_coldefs``, ``Header.has_key``, ``Header.ascardlist``. If you run your code with a previous version of PyFITS (>= 3.0, < 3.2) with the ``python -Wd`` argument, warnings for all deprecated interfaces still in use will be displayed. - Interfaces that were pending deprecation are now fully deprecated. These include: ``create_card``, ``create_card_from_string``, ``upper_key``, ``Header.get_history``, and ``Header.get_comment``. - The ``.name`` attribute on HDUs is now directly tied to the HDU's header, so that if ``.header['EXTNAME']`` changes so does ``.name`` and vice-versa. - The ``pyfits.file.PYTHON_MODES`` constant dict was renamed to ``pyfits.file.PYFITS_MODES`` which better reflects its purpose. This is rarely used by client code, however. Support for the old name will be removed by PyFITS 3.4. Other Changes and Additions ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - The new compression code also adds support for the ZQUANTIZ and ZDITHER0 keywords added in more recent versions of this FITS Tile Compression spec. This includes support for lossless compression with GZIP. (#198) By default no dithering is used, but the ``SUBTRACTIVE_DITHER_1`` and ``SUBTRACTIVE_DITHER_2`` methods can be enabled by passing the correct constants to the ``quantize_method`` argument to the ``CompImageHDU`` constuctor. A seed can be manually specified, or automatically generated using either the system clock or checksum-based methods via the ``dither_seed`` argument. See the documentation for ``CompImageHDU`` for more details. (#198) (spacetelescope/PYFITS#32) - Images compressed with the Tile Compression standard can now be larger than 4 GB through support of the Q format. (#159) - All HDUs now have a ``.ver`` ``.level`` attribute that returns the value of the EXTVAL and EXTLEVEL keywords from that HDU's header, if the exist. This was added for consistency with the ``.name`` attribute which returns the EXTNAME value from the header. - Then ``Column`` and ``ColDefs`` classes have new ``.dtype`` attributes which give the Numpy dtype for the column data in the first case, and the full Numpy compound dtype for each table row in the latter case. - There was an issue where new tables created defaulted the values in all string columns to '0.0'. Now string columns are filled with empty strings by default--this seems a less surprising default, but it may cause differences with tables created with older versions of PyFITS. - Improved round-tripping and preservation of manually assigned column attributes (``TNULLn``, ``TSCALn``, etc.) in table HDU headers. (astropy/astropy#996) Bug Fixes ^^^^^^^^^ - Binary tables containing compressed images may, optionally, contain other columns unrelated to the tile compression convention. Although this is an uncommon use case, it is permitted by the standard. (#159) - Reworked some of the file I/O routines to allow simpler, more consistent mapping between OS-level file modes ('rb', 'wb', 'ab', etc.) and the more "PyFITS-specific" modes used by PyFITS like "readonly" and "update". That is, if reading a FITS file from an open file object, it doesn't matter as much what "mode" it was opened in so long as it has the right capabilities (read/write/etc.) Also works around bugs in the Python io module in 2.6+ with regard to file modes. (spacetelescope/PyFITS#33) - Fixed an obscure issue that can occur on systems that don't have flush to memory-mapped files implemented (namely GNU Hurd). (astropy/astropy#968) 3.1.3 (2013-11-26) ------------------ - Disallowed assigning NaN and Inf floating point values as header values, since the FITS standard does not define a way to represent them in. Because this is undefined, the previous behavior did not make sense and produced invalid FITS files. (spacetelescope/PyFITS#11) - Added a workaround for a bug in 64-bit OSX that could cause truncation when writing files greater than 2^32 bytes in size. (spacetelescope/PyFITS#28) - Fixed a long-standing issue where writing binary tables did not correctly write the TFORMn keywords for variable-length array columns (they ommitted the max array length parameter of the format). This was thought fixed in v3.1.2, but it was only fixed there for compressed image HDUs and not for binary tables in general. - Fixed an obscure issue that can occur on systems that don't have flush to memory-mapped files implemented (namely GNU Hurd). (Backported from 3.2) 3.0.12 (2013-11-26) ------------------- - Disallowed assigning NaN and Inf floating point values as header values, since the FITS standard does not define a way to represent them in. Because this is undefined, the previous behavior did not make sense and produced invalid FITS files. (Backported from 3.1.3) - Added a workaround for a bug in 64-bit OSX that could cause truncation when writing files greater than 2^32 bytes in size. (Backported from 3.1.3) - Fixed a long-standing issue where writing binary tables did not correctly write the TFORMn keywords for variable-length array columns (they ommitted the max array length parameter of the format). This was thought fixed in v3.1.2, but it was only fixed there for compressed image HDUs and not for binary tables in general. (Backported from 3.1.3) - Fixed an obscure issue that can occur on systems that don't have flush to memory-mapped files implemented (namely GNU Hurd). (Backported from 3.2) 3.1.2 (2013-04-22) ------------------ - When an error occurs opening a file in fitsdiff the exception message will now at least mention which file had the error. (#168) - Fixed support for opening gzipped FITS files by filename in a writeable mode (PyFITS has supported writing to gzip files for some time now, but only enabled it when GzipFile objects were passed to ``pyfits.open()`` due to some legacy code preventing full gzip support. (#195) - Added a more helpful error message in the case of malformatted FITS files that contain non-float NULL values in an ASCII table but are missing the required TNULLn keywords in the header. (#197) - Fixed an (apparently long-standing) issue where writing compressed images did not correctly write the TFORMn keywords for variable-length array columns (they ommitted the max array length parameter of the format). (#199) - Slightly refactored how tables containing variable-length array columns are handled to add two improvements: Fixes an issue where accessing the data after a call to the `pyfits.getdata` convenience function caused an exception, and allows the VLA data to be read from an existing mmap of the FITS file. (#200) - Fixed a bug that could occur when opening a table containing multi-dimensional columns (i.e. via the TDIMn keyword) and then writing it out to a new file. (#201) - Added use of the console_scripts entry point to install the fitsdiff and fitscheck scripts, which if nothing else provides better Windows support. The generated scripts now override the ones explicitly defined in the scripts/ directory (which were just trivial stubs to begin with). (#202) - Fixed a bug on Python 3 where attempting to open a non-existent file on Python 3 caused a seemingly unrelated traceback. (#203) - Fixed a bug in fitsdiff that reported two header keywords containing NaN as value as different. (#204) - Fixed an issue in the tests that caused some tests to fail if pyfits is installed with read-only permissions. (#208) - Fixed a bug where instantiating a ``BinTableHDU`` from a numpy array containing boolean fields converted all the values to ``False``. (#215) - Fixed an issue where passing an array of integers into the constructor of ``Column()`` when the column type is floats of the same byte width caused the column array to become garbled. (#218) - Fixed inconsistent behavior in creating CONTINUE cards from byte strings versus unicode strings in Python 2--CONTINUE cards can now be created properly from unicode strings (so long as they are convertable to ASCII). (spacetelescope/PyFITS#1) - Fixed a couple cases where creating a new table using TDIMn in some of the columns could caused a crash. (spacetelescope/PyFITS#3) - Fixed a bug in parsing HIERARCH keywords that do not have a space after the first equals sign (before the value). (spacetelescope/PyFITS#5) - Prevented extra leading whitespace on HIERARCH keywords from being treated as part of the keyword. (spacetelescope/PyFITS#6) - Fixed a bug where HIERARCH keywords containing lower-case letters was mistakenly marked as invalid during header validation. (spacetelescope/PyFITS#7) - Fixed an issue that was ancillary to (spacetelescope/PyFITS#7) where the ``Header.index()`` method did not work correctly with HIERARCH keywords containing lower-case letters. 3.0.11 (2013-04-17) ------------------- - Fixed support for opening gzipped FITS files by filename in a writeable mode (PyFITS has supported writing to gzip files for some time now, but only enabled it when GzipFile objects were passed to ``pyfits.open()`` due to some legacy code preventing full gzip support. Backported from 3.1.2. (#195) - Added a more helpful error message in the case of malformatted FITS files that contain non-float NULL values in an ASCII table but are missing the required TNULLn keywords in the header. Backported from 3.1.2. (#197) - Fixed an (apparently long-standing) issue where writing compressed images did not correctly write the TFORMn keywords for variable-length array columns (they ommitted the max array length parameter of the format). Backported from 3.1.2. (#199) - Slightly refactored how tables containing variable-length array columns are handled to add two improvements: Fixes an issue where accessing the data after a call to the `pyfits.getdata` convenience function caused an exception, and allows the VLA data to be read from an existing mmap of the FITS file. Backported from 3.1.2. (#200) - Fixed a bug that could occur when opening a table containing multi-dimensional columns (i.e. via the TDIMn keyword) and then writing it out to a new file. Backported from 3.1.2. (#201) - Fixed a bug on Python 3 where attempting to open a non-existent file on Python 3 caused a seemingly unrelated traceback. Backported from 3.1.2. (#203) - Fixed a bug in fitsdiff that reported two header keywords containing NaN as value as different. Backported from 3.1.2. (#204) - Fixed an issue in the tests that caused some tests to fail if pyfits is installed with read-only permissions. Backported from 3.1.2. (#208) - Fixed a bug where instantiating a ``BinTableHDU`` from a numpy array containing boolean fields converted all the values to ``False``. Backported from 3.1.2. (#215) - Fixed an issue where passing an array of integers into the constructor of ``Column()`` when the column type is floats of the same byte width caused the column array to become garbled. Backported from 3.1.2. (#218) - Fixed a couple cases where creating a new table using TDIMn in some of the columns could caused a crash. Backported from 3.1.2. (spacetelescope/PyFITS#3) 3.1.1 (2013-01-02) ------------------ This is a bug fix release for the 3.1.x series. Bug Fixes ^^^^^^^^^ - Improved handling of scaled images and pseudo-unsigned integer images in compressed image HDUs. They now work more transparently like normal image HDUs with support for the ``do_not_scale_image_data`` and ``uint`` options, as well as ``scale_back`` and ``save_backup``. The ``.scale()`` method works better too. (#88) - Permits non-string values for the EXTNAME keyword when reading in a file, rather than throwing an exception due to the malformatting. Added verification for the format of the EXTNAME keyword when writing. (#96) - Added support for EXTNAME and EXTVER in PRIMARY HDUs. That is, if EXTNAME is specified in the header, it will also be reflected in the ``.name`` attribute and in ``pyfits.info()``. These keywords used to be verboten in PRIMARY HDUs, but the latest version of the FITS standard allows them. (#151) - HCOMPRESS can again be used to compress data cubes (and higher-dimensional arrays) so long as the tile size is effectively 2-dimensional. In fact, PyFITS will automatically use compatible tile sizes even if they're not explicitly specified. (#171) - Added support for the optional ``endcard`` parameter in the ``Header.fromtextfile()`` and ``Header.totextfile()`` methods. Although ``endcard=False`` was a reasonable default assumption, there are still text dumps of FITS headers that include the END card, so this should have been more flexible. (#176) - Fixed a crash when running fitsdiff on two empty (that is, zero row) tables. (#178) - Fixed an issue where opening files containing random groups HDUs in update mode could cause an unnecessary rewrite of the file even if none of the data is modified. (#179) - Fixed a bug that could caused a deadlock in the filesystem on OSX if PyFITS is used with Numpy 1.7 in some cases. (#180) - Fixed a crash when generating diff reports from diffs using the ``ignore_comments`` options. (#181) - Fixed some bugs with WCS Paper IV record-valued keyword cards: - Cards that looked kind of like RVKCs but were not intended to be were over-permissively treated as such--commentary keywords like COMMENT and HISTORY were particularly affected. (#183) - Looking up a card in a header by its standard FITS keyword only should always return the raw value of that card. That way cards containing values that happen to valid RVKCs but were not intended to be will still be treated like normal cards. (#184) - Looking up a RVKC in a header with only part of the field-specifier (for example "DP1.AXIS" instead of "DP1.AXIS.1") was implicitly treated as a wildcard lookup. (#184) - Fixed a crash when diffing two FITS files where at least one contains a compressed image HDU which was not recognized as an image instead of a table. (#187) - Fixed bugs in the backwards compatibility layer for the ``CardList.index`` and ``CardList.count`` methods. (#190) - Improved ``__repr__`` and text file representation of cards with long values that are split into CONTINUE cards. (#193) - Fixed a crash when trying to assign a long (> 72 character) value to blank ('') keywords. This also changed how blank keywords are represented--there are still exactly 8 spaces before any commentary content can begin; this *may* affect the exact display of header cards that assumed there could be fewer spaces in a blank keyword card before the content begins. However, the current approach is more in line with the requirements of the FITS standard. (#194) 3.0.10 (2013-01-02) ------------------- - Improved handling of scaled images and pseudo-unsigned integer images in compressed image HDUs. They now work more transparently like normal image HDUs with support for the ``do_not_scale_image_data`` and ``uint`` options, as well as ``scale_back`` and ``save_backup``. The ``.scale()`` method works better too. Backported from 3.1.1. (#88) - Permits non-string values for the EXTNAME keyword when reading in a file, rather than throwing an exception due to the malformatting. Added verification for the format of the EXTNAME keyword when writing. Backported from 3.1.1. (#96) - Added support for EXTNAME and EXTVER in PRIMARY HDUs. That is, if EXTNAME is specified in the header, it will also be reflected in the ``.name`` attribute and in ``pyfits.info()``. These keywords used to be verbotten in PRIMARY HDUs, but the latest version of the FITS standard allows them. Backported from 3.1.1. (#151) - HCOMPRESS can again be used to compress data cubes (and higher-dimensional arrays) so long as the tile size is effectively 2-dimensional. In fact, PyFITS will not automatically use compatible tile sizes even if they're not explicitly specified. Backported from 3.1.1. (#171) - Fixed a bug when writing out files containing zero-width table columns, where the TFIELDS keyword would be updated incorrectly, leaving the table largely unreadable. Backported from 3.1.0. (#174) - Fixed an issue where opening files containing random groups HDUs in update mode could cause an unnecessary rewrite of the file even if none of the data is modified. Backported from 3.1.1. (#179) - Fixed a bug that could caused a deadlock in the filesystem on OSX if PyFITS is used with Numpy 1.7 in some cases. Backported from 3.1.1. (#180) 3.1 (2012-08-08) ---------------- Highlights ^^^^^^^^^^ - The ``Header`` object has been significantly reworked, and ``CardList`` objects are now deprecated (their functionality folded into the ``Header`` class). See API Changes below for more details. - Memory maps are now used by default to access HDU data. See API Changes below for more details. - Now includes a new version of the ``fitsdiff`` program for comparing two FITS files, and a new FITS comparison API used by ``fitsdiff``. See New Features below. API Changes ^^^^^^^^^^^ - The ``Header`` class has been rewritten, and the ``CardList`` class is deprecated. Most of the basic details of working with FITS headers are unchanged, and will not be noticed by most users. But there are differences in some areas that will be of interest to advanced users, and to application developers. For full details of the changes, see the "Header Interface Transition Guide" section in the PyFITS documentation. See ticket #64 on the PyFITS Trac for futher details and background. Some highlights are listed below: * The Header class now fully implements the Python dict interface, and can be used interchangably with a dict, where the keys are header keywords. * New keywords can be added to the header using normal keyword assignment (previously it was necessary to use ``Header.update`` to add new keywords). For example:: >>> header['NAXIS'] = 2 will update the existing 'FOO' keyword if it already exists, or add a new one if it doesn't exist, just like a dict. * It is possible to assign both a value and a comment at the same time using a tuple:: >>> header['NAXIS'] = (2, 'Number of axes') * To add/update a new card and ensure it's added in a specific location, use ``Header.set()``:: >>> header.set('NAXIS', 2, 'Number of axes', after='BITPIX') This works the same as the old ``Header.update()``. ``Header.update()`` still works in the old way too, but is deprecated. * Although ``Card`` objects still exist, it generally is not necessary to work with them directly. ``Header.ascardlist()``/``Header.ascard`` are deprecated and should not be used. To directly access the ``Card`` objects in a header, use ``Header.cards``. * To access card comments, it is still possible to either go through the card itself, or through ``Header.comments``. For example:: >>> header.cards['NAXIS'].comment Number of axes >>> header.comments['NAXIS'] Number of axes * ``Card`` objects can now be used interchangeably with ``(keyword, value, comment)`` 3-tuples. They still have ``.value`` and ``.comment`` attributes as well. The ``.key`` attribute has been renamed to ``.keyword`` for consistency, though ``.key`` is still supported (but deprecated). - Memory mapping is now used by default to access HDU data. That is, ``pyfits.open()`` uses ``memmap=True`` as the default. This provides better performance in the majority of use cases--there are only some I/O intensive applications where it might not be desirable. Enabling mmap by default also enabled finding and fixing a large number of bugs in PyFITS' handling of memory-mapped data (most of these bug fixes were backported to PyFITS 3.0.5). (#85) * A new ``pyfits.USE_MEMMAP`` global variable was added. Set ``pyfits.USE_MEMMAP = False`` to change the default memmap setting for opening files. This is especially useful for controlling the behavior in applications where pyfits is deeply embedded. * Likewise, a new ``PYFITS_USE_MEMMAP`` environment variable is supported. Set ``PYFITS_USE_MEMMAP = 0`` in your environment to change the default behavior. - The ``size()`` method on HDU objects is now a ``.size`` property--this returns the size in bytes of the data portion of the HDU, and in most cases is equivalent to ``hdu.data.nbytes`` (#83) - ``BinTableHDU.tdump`` and ``BinTableHDU.tcreate`` are deprecated--use ``BinTableHDU.dump`` and ``BinTableHDU.load`` instead. The new methods output the table data in a slightly different format from previous versions, which places quotes around each value. This format is compatible with data dumps from previous versions of PyFITS, but not vice-versa due to a parsing bug in older versions. - Likewise the ``pyfits.tdump`` and ``pyfits.tcreate`` convenience function versions of these methods have been renamed ``pyfits.tabledump`` and ``pyfits.tableload``. The old deprecated, but currently retained for backwards compatibility. (r1125) - A new global variable ``pyfits.EXTENSION_NAME_CASE_SENSITIVE`` was added. This serves as a replacement for ``pyfits.setExtensionNameCaseSensitive`` which is not deprecated and may be removed in a future version. To enable case-sensitivity of extension names (i.e. treat 'sci' as distict from 'SCI') set ``pyfits.EXTENSION_NAME_CASE_SENSITIVE = True``. The default is ``False``. (r1139) - A new global configuration variable ``pyfits.STRIP_HEADER_WHITESPACE`` was added. By default, if a string value in a header contains trailing whitespace, that whitespace is automatically removed when the value is read. Now if you set ``pyfits.STRIP_HEADER_WHITESPACE = False`` all whitespace is preserved. (#146) - The old ``classExtensions`` extension mechanism (which was deprecated in PyFITS 3.0) is removed outright. To our knowledge it was no longer used anywhere. (r1309) - Warning messages from PyFITS issued through the Python warnings API are now output to stderr instead of stdout, as is the default. PyFITS no longer modifies the default behavior of the warnings module with respect to which stream it outputs to. (r1319) - The ``checksum`` argument to ``pyfits.open()`` now accepts a value of 'remove', which causes any existing CHECKSUM/DATASUM keywords to be ignored, and removed when the file is saved. New Features ^^^^^^^^^^^^ - Added support for the proposed "FITS" extension HDU type. See http://listmgr.cv.nrao.edu/pipermail/fitsbits/2002-April/001094.html. FITS HDUs contain an entire FITS file embedded in their data section. `FitsHDU` objects work like other HDU types in PyFITS. Their ``.data`` attribute returns the raw data array. However, they have a special ``.hdulist`` attribute which processes the data as a FITS file and returns it as an in-memory HDUList object. FitsHDU objects also support a ``FitsHDU.fromhdulist()`` classmethod which returns a new `FitsHDU` object that embeds the supplied HDUList. (#80) - Added a new ``.is_image`` attribute on HDU objects, which is True if the HDU data is an 'image' as opposed to a table or something else. Here the meaning of 'image' is fairly loose, and mostly just means a Primary or Image extension HDU, or possibly a compressed image HDU (#71) - Added an ``HDUList.fromstring`` classmethod which can parse a FITS file already in memory and instantiate and ``HDUList`` object from it. This could be useful for integrating PyFITS with other libraries that work on FITS file, such as CFITSIO. It may also be useful in streaming applications. The name is a slight misnomer, in that it actually accepts any Python object that implements the buffer interface, which includes ``bytes``, ``bytearray``, ``memoryview``, ``numpy.ndarray``, etc. (#90) - Added a new ``pyfits.diff`` module which contains facilities for comparing FITS files. One can use the ``pyfits.diff.FITSDiff`` class to compare two FITS files in their entirety. There is also a ``pyfits.diff.HeaderDiff`` class for just comparing two FITS headers, and other similar interfaces. See the PyFITS Documentation for more details on this interface. The ``pyfits.diff`` module powers the new ``fitsdiff`` program installed with PyFITS. After installing PyFITS, run ``fitsdiff --help`` for usage details. - ``pyfits.open()`` now accepts a ``scale_back`` argument. If set to ``True``, this automatically scales the data using the original BZERO and BSCALE parameters the file had when it was first opened, if any, as well as the original BITPIX. For example, if the original BITPIX were 16, this would be equivalent to calling ``hdu.scale('int16', 'old')`` just before calling ``flush()`` or ``close()`` on the file. This option applies to all HDUs in the file. (#120) - ``pyfits.open()`` now accepts a ``save_backup`` argument. If set to ``True``, this automatically saves a backup of the original file before flushing any changes to it (this of course only applies to update and append mode). This may be especially useful when working with scaled image data. (#121) Changes in Behavior ^^^^^^^^^^^^^^^^^^^ - Warnings from PyFITS are not output to stderr by default, instead of stdout as it has been for some time. This is contrary to most users' expectations and makes it more difficult for them to separate output from PyFITS from the desired output for their scripts. (r1319) Bug Fixes ^^^^^^^^^ - Fixed ``pyfits.tcreate()`` (now ``pyfits.tableload()``) to be more robust when encountering blank lines in a column definition file (#14) - Fixed a fairly rare crash that could occur in the handling of CONTINUE cards when using Numpy 1.4 or lower (though 1.4 is the oldest version supported by PyFITS). (r1330) - Fixed ``_BaseHDU.fromstring`` to actually correctly instantiate an HDU object from a string/buffer containing the header and data of that HDU. This allowed for the implementation of ``HDUList.fromstring`` described above. (#90) - Fixed a rare corner case where, in some use cases, (mildly, recoverably) malformatted float values in headers were not properly returned as floats. (#137) - Fixed a corollary to the previous bug where float values with a leading zero before the decimal point had the leading zero unnecessarily removed when saving changes to the file (eg. "0.001" would be written back as ".001" even if no changes were otherwise made to the file). (#137) - When opening a file containing CHECKSUM and/or DATASUM keywords in update mode, the CHECKSUM/DATASUM are updated and preserved even if the file was opened with checksum=False. This change in behavior prevents checksums from being unintentionally removed. (#148) - Fixed a bug where ``ImageHDU.scale(option='old')`` wasn't working at all--it was not restoring the image to its original BSCALE and BZERO values. (#162) - Fixed a bug when writing out files containing zero-width table columns, where the TFIELDS keyword would be updated incorrectly, leaving the table largely unreadable. This fix will be backported to the 3.0.x series in version 3.0.10. (#174) 3.0.9 (2012-08-06) ------------------ This is a bug fix release for the 3.0.x series. Bug Fixes ^^^^^^^^^ - Fixed ``Header.values()``/``Header.itervalues()`` and ``Header.items()``/ ``Header.iteritems()`` to correctly return the different values for duplicate keywords (particularly commentary keywords like HISTORY and COMMENT). This makes the old Header implementation slightly more compatible with the new implementation in PyFITS 3.1. (#127) .. note:: This fix did not change the existing behavior from earlier PyFITS versions where ``Header.keys()`` returns all keywords in the header with duplicates removed. PyFITS 3.1 changes that behavior, so that ``Header.keys()`` includes duplicates. - Fixed a bug where ``ImageHDU.scale(option='old')`` wasn't working at all--it was not restoring the image to its original BSCALE and BZERO values. (#162) - Fixed a bug where opening a file containing compressed image HDUs in 'update' mode and then immediately closing it without making any changes caused the file to be rewritten unncessarily. (#167) - Fixed two memory leaks that could occur when writing compressed image data, or in some cases when opening files containing compressed image HDUs in 'update' mode. (#168) 3.0.8 (2012-06-04) ------------------ Changes in Behavior ^^^^^^^^^^^^^^^^^^^ - Prior to this release, image data sections did not work with scaled data--that is, images with non-trivial BSCALE and/or BZERO values. Previously, in order to read such images in sections, it was necessary to manually apply the BSCALE+BZERO to each section. It's worth noting that sections *did* support pseudo-unsigned ints (flakily). This change just extends that support for general BSCALE+BZERO values. Bug Fixes ^^^^^^^^^ - Fixed a bug that prevented updates to values in boolean table columns from being saved. This turned out to be a symptom of a deeper problem that could prevent other table updates from being saved as well. (#139) - Fixed a corner case in which a keyword comment ending with the string "END" could, in some circumstances, cause headers (and the rest of the file after that point) to be misread. (#142) - Fixed support for scaled image data and psuedo-unsigned ints in image data sections (``hdu.section``). Previously this was not supported at all. At some point support was supposedly added, but it was buggy and incomplete. Now the feature seems to work much better. (#143) - Fixed the documentation to point out that image data sections *do* support non-contiguous slices (and have for a long time). The documentation was never updated to reflect this, and misinformed users that only contiguous slices were supported, leading to some confusion. (#144) - Fixed a bug where creating an ``HDUList`` object containing multiple PRIMARY HDUs caused an infinite recursion when validating the object prior to writing to a file. (#145) - Fixed a rare but serious case where saving an update to a file that previously had a CHECKSUM and/or DATASUM keyword, but removed the checksum in saving, could cause the file to be slightly corrupted and unreadable. (#147) - Fixed problems with reading "non-standard" FITS files with primary headers containing SIMPLE = F. PyFITS has never made many guarantees as to how such files are handled. But it should at least be possible to read their headers, and the data if possible. Saving changes to such a file should not try to prepend an unwanted valid PRIMARY HDU. (#157) - Fixed a bug where opening an image with ``disable_image_compression = True`` caused compression to be disabled for all subsequent ``pyfits.open()`` calls. (r1651) 3.0.7 (2012-04-10) ------------------ Changes in Behavior ^^^^^^^^^^^^^^^^^^^ - Slices of GroupData objects now return new GroupData objects instead of extended multi-row _Group objects. This is analogous to how PyFITS 3.0 fixed FITS_rec slicing, and should have been fixed for GroupData at the same time. The old behavior caused bugs where functions internal to Numpy expected that slicing an ndarray would return a new ndarray. As this is a rare usecase with a rare feature most users are unlikely to be affected by this change. - The previously internal _Group object for representing individual group records in a GroupData object are renamed Group and are now a public interface. However, there's almost no good reason to create Group objects directly, so it shouldn't be considered a "new feature". - An annoyance from PyFITS 3.0.6 was fixed, where the value of the EXTEND keyword was always being set to F if there are not actually any extension HDUs. It was unnecessary to modify this value. Bug Fixes ^^^^^^^^^ - Fixed GroupData objects to return new GroupData objects when sliced instead of _Group record objects. See "Changes in behavior" above for more details. - Fixed slicing of Group objects--previously it was not possible to slice slice them at all. - Made it possible to assign `np.bool_` objects as header values. (#123) - Fixed overly strict handling of the EXTEND keyword; see "Changes in behavior" above. (#124) - Fixed many cases where an HDU's header would be marked as "modified" by PyFITS and rewritten, even when no changes to the header are necessary. (#125) - Fixed a bug where the values of the PTYPEn keywords in a random groups HDU were forced to be all lower-case when saving the file. (#130) - Removed an unnecessary inline import in `ExtensionHDU.__setattr__` that was causing some slowdown when opening files containing a large number of extensions, plus a few other small (but not insignficant) performance improvements thanks to Julian Taylor. (#133) - Fixed a regression where header blocks containing invalid end-of-header padding (i.e. null bytes instead of spaces) couldn't be parsed by PyFITS. Such headers can be parsed again, but a warning is raised, as such headers are not valid FITS. (#136) - Fixed a memory leak where table data in random groups HDUs weren't being garbage collected. (#138) 3.0.6 (2012-02-29) ------------------ Highlights ^^^^^^^^^^ The main reason for this release is to fix an issue that was introduced in PyFITS 3.0.5 where merely opening a file containing scaled data (that is, with non-trivial BSCALE and BZERO keywords) in 'update' mode would cause the data to be automatically rescaled--possibly converting the data from ints to floats--as soon as the file is closed, even if the application did not touch the data. Now PyFITS will only rescale the data in an extension when the data is actually accessed by the application. So opening a file in 'update' mode in order to modify the header or append new extensions will not cause any change to the data in existing extensions. This release also fixes a few Windows-specific bugs found through more extensive Windows testing, and other miscellaneous bugs. Bug Fixes ^^^^^^^^^ - More accurate error messages when opening files containing invalid header cards. (#109) - Fixed a possible reference cycle/memory leak that was caught through more extensive testing on Windows. (#112) - Fixed 'ostream' mode to open the underlying file in 'wb' mode instead of 'w' mode. (#112) - Fixed a Windows-only issue where trying to save updates to a resized FITS file could result in a crash due to there being open mmaps on that file. (#112) - Fixed a crash when trying to create a FITS table (i.e. with new_table()) from a Numpy array containing bool fields. (#113) - Fixed a bug where manually initializing an ``HDUList`` with a list of of HDUs wouldn't set the correct EXTEND keyword value on the primary HDU. (#114) - Fixed a crash that could occur when trying to deepcopy a Header in Python < 2.7. (#115) - Fixed an issue where merely opening a scaled image in 'update' mode would cause the data to be converted to floats when the file is closed. (#119) 3.0.5 (2012-01-30) ------------------ - Fixed a crash that could occur when accessing image sections of files opened with memmap=True. (r1211) - Fixed the inconsistency in the behavior of files opened in 'readonly' mode when memmap=True vs. when memmap=False. In the latter case, although changes to array data were not saved to disk, it was possible to update the array data in memory. On the other hand with memmap=True, 'readonly' mode prevented even in-memory modification to the data. This is what 'copyonwrite' mode was for, but difference in behavior was confusing. Now 'readonly' is equivalent to 'copyonwrite' when using memmap. If the old behavior of denying changes to the array data is necessary, a new 'denywrite' mode may be used, though it is only applicable to files opened with memmap. (r1275) - Fixed an issue where files opened with memmap=True would return image data as a raw numpy.memmap object, which can cause some unexpected behaviors--instead memmap object is viewed as a numpy.ndarray. (r1285) - Fixed an issue in Python 3 where a workaround for a bug in Numpy on Python 3 interacted badly with some other software, namely to vo.table package (and possibly others). (r1320, r1337, and #110) - Fixed buggy behavior in the handling of SIGINTs (i.e. Ctrl-C keyboard interrupts) while flushing changes to a FITS file. PyFITS already prevented SIGINTs from causing an incomplete flush, but did not clean up the signal handlers properly afterwards, or reraise the keyboard interrupt once the flush was complete. (r1321) - Fixed a crash that could occur in Python 3 when opening files with checksum checking enabled. (r1336) - Fixed a small bug that could cause a crash in the `StreamingHDU` interface when using Numpy below version 1.5. - Fixed a crash that could occur when creating a new `CompImageHDU` from an array of big-endian data. (#104) - Fixed a crash when opening a file with extra zero padding at the end. Though FITS files should not have such padding, it's not explictly forbidden by the format either, and PyFITS shouldn't stumble over it. (#106) - Fixed a major slowdown in opening tables containing large columns of string values. (#111) 3.0.4 (2011-11-22) ------------------ - Fixed a crash when writing HCOMPRESS compressed images that could happen on Python 2.5 and 2.6. (r1217) - Fixed a crash when slicing an table in a file opened in 'readonly' mode with memmap=True. (r1230) - Writing changes to a file or writing to a new file verifies the output in 'fix' mode by default instead of 'exception'--that is, PyFITS will automatically fix common FITS format errors rather than raising an exception. (r1243) - Fixed a bug where convenience functions such as getval() and getheader() crashed when specifying just 'PRIMARY' as the extension to use (r1263). - Fixed a bug that prevented passing keyword arguments (beyond the standard data and header arguments) as positional arguments to the constructors of extension HDU classes. - Fixed some tests that were failing on Windows--in this case the tests themselves failed to close some temp files and Windows refused to delete them while there were still open handles on them. (r1295) - Fixed an issue with floating point formatting in header values on Python 2.5 for Windows (and possibly other platforms). The exponent was zero-padded to 3 digits; although the FITS standard makes no specification on this, the formatting is now normalized to always pad the exponent to two digits. (r1295) - Fixed a bug where long commentary cards (such as HISTORY and COMMENT) were broken into multiple CONTINUE cards. However, commentary cards are not expected to be found in CONTINUE cards. Instead these long cards are broken into multiple commentary cards. (#97) - GZIP/ZIP-compressed FITS files can be detected and opened regardless of their filename extension. (#99) - Fixed a serious bug where opening scaled images in 'update' mode and then closing the file without touching the data would cause the file to be corrupted. (#101) 3.0.3 (2011-10-05) ------------------ - Fixed several small bugs involving corner cases in record-valued keyword cards (#70) - In some cases HDU creation failed if the first keyword value in the header was not a string value (#89) - Fixed a crash when trying to compute the HDU checksum when the data array contains an odd number of bytes (#91) - Disabled an unnecessary warning that was displayed on opening compressed HDUs with disable_image_compression = True (#92) - Fixed a typo in code for handling HCOMPRESS compressed images. 3.0.2 (2011-09-23) ------------------ - The ``BinTableHDU.tcreate`` method and by extension the ``pyfits.tcreate`` function don't get tripped up by blank lines anymore (#14) - The presence, value, and position of the EXTEND keyword in Primary HDUs is verified when reading/writing a FITS file (#32) - Improved documentation (in warning messages as well as in the handbook) that PyFITS uses zero-based indexing (as one would expect for C/Python code, but contrary to the PyFITS standard which was written with FORTRAN in mind) (#68) - Fixed a bug where updating a header card comment could cause the value to be lost if it had not already been read from the card image string. - Fixed a related bug where changes made directly to Card object in a header (i.e. assigning directly to card.value or card.comment) would not propagate when flushing changes to the file (#69) [Note: This and the bug above it were originally reported as being fixed in version 3.0.1, but the fix was never included in the release.] - Improved file handling, particularly in Python 3 which had a few small file I/O-related bugs (#76) - Fixed a bug where updating a FITS file would sometimes cause it to lose its original file permissions (#79) - Fixed the handling of TDIMn keywords; 3.0 added support for them, but got the axis order backards (they were treated as though they were row-major) (#82) - Fixed a crash when a FITS file containing scaled data is opened and immediately written to a new file without explicitly viewing the data first (#84) - Fixed a bug where creating a table with columns named either 'names' or 'formats' resulted in an infinite recursion (#86) 3.0.1 (2011-09-12) ------------------ - Fixed a bug where updating a header card comment could cause the value to be lost if it had not already been read from the card image string. - Changed ``_TableBaseHDU.data`` so that if the data contain an empty table a ``FITS_rec`` object with zero rows is returned rather than ``None``. - The ``.key`` attribute of ``RecordValuedKeywordCards`` now returns the full keyword+field-specifier value, instead of just the plain keyword (#46) - Fixed a related bug where changes made directly to Card object in a header (i.e. assigning directly to card.value or card.comment) would not propagate when flushing changes to the file (#69) - Fixed a bug where writing a table with zero rows could fail in some cases (#72) - Miscellanous small bug fixes that were causing some tests to fail, particularly on Python 3 (#74, #75) - Fixed a bug where creating a table column from an array in non-native byte order would not preserve the byte order, thus interpreting the column array using the wrong byte order (#77) 3.0.0 (2011-08-23) -------------------- - Contains major changes, bumping the version to 3.0 - Large amounts of refactoring and reorganization of the code; tried to preserve public API backwards-compatibility with older versions (private API has many changes and is not guaranteed to be backwards-compatible). There are a few small public API changes to be aware of: * The pyfits.rec module has been removed completely. If your version of numpy does not have the numpy.core.records module it is too old to be used with PyFITS. * The ``Header.ascardlist()`` method is deprecated--use the ``.ascard`` attribute instead. * ``Card`` instances have a new ``.cardimage`` attribute that should be used rather than ``.ascardimage()``, which may become deprecated. * The ``Card.fromstring()`` method is now a classmethod. It returns a new ``Card`` instance rather than modifying an existing instance. * The ``req_cards()`` method on HDU instances has changed: The ``pos`` argument is not longer a string. It is either an integer value (meaning the card's position must match that value) or it can be a function that takes the card's position as it's argument, and returns True if the position is valid. Likewise, the ``test`` argument no longer takes a string, but instead a function that validates the card's value and returns True or False. * The ``get_coldefs()`` method of table HDUs is deprecated. Use the ``.columns`` attribute instead. * The ``ColDefs.data`` attribute is deprecated--use ``ColDefs.columns`` instead (though in general you shouldn't mess with it directly--it might become internal at some point). * ``FITS_record`` objects take ``start`` and ``end`` as arguments instead of ``startColumn`` and ``endColumn`` (these are rarely created manually, so it's unlikely that this change will affect anyone). * ``BinTableHDU.tcreate()`` is now a classmethod, and returns a new ``BinTableHDU`` instance. * Use ``ExtensionHDU`` and ``NonstandardExtHDU`` for making new extension HDU classes. They are now public interfaces, wheres previously they were private and prefixed with underscores. * Possibly others--please report if you find any changes that cause difficulties. - Calls to deprecated functions will display a Deprecation warning. However, in Python 2.7 and up Deprecation warnings are ignored by default, so run Python with the `-Wd` option to see if you're using any deprecated functions. If we get close to actually removing any functions, we might make the Deprecation warnings display by default. - Added basic Python 3 support - Added support for multi-dimensional columns in tables as specified by the TDIMn keywords (#47) - Fixed a major memory leak that occurred when creating new tables with the ``new_table()`` function (#49) be padded with zero-bytes) vs ASCII tables (where strings are padded with spaces) (#15) - Fixed a bug in which the case of Random Access Group parameters names was not preserved when writing (#41) - Added support for binary table fields with zero width (#42) - Added support for wider integer types in ASCII tables; although this is non- standard, some GEIS images require it (#45) - Fixed a bug that caused the index_of() method of HDULists to crash when the HDUList object is created from scratch (#48) - Fixed the behavior of string padding in binary tables (where strings should be padded with nulls instead of spaces) - Fixed a rare issue that caused excessive memory usage when computing checksums using a non-standard block size (see r818) - Add support for forced uint data in image sections (#53) - Fixed an issue where variable-length array columns were not extended when creating a new table with more rows than the original (#54) - Fixed tuple and list-based indexing of FITS_rec objects (#55) - Fixed an issue where BZERO and BSCALE keywords were appended to headers in the wrong location (#56) - ``FITS_record`` objects (table rows) have full slicing support, including stepping, etc. (#59) - Fixed a bug where updating multiple files simultaneously (such as when running parallel processes) could lead to a race condition with mktemp() (#61) - Fixed a bug where compressed image headers were not in the order expected by the funpack utility (#62)

@tomykaira

v0.8.2, 2013-11-20 ([changes](simplecov-ruby/simplecov@v0.8.1...v0.8.2)) ===================== ## Bugfixes * Replaced the locking behaviour [via lockfile gem](simplecov-ruby/simplecov#185) with plain Ruby explicit file locking when merging results. This should make simplecov merging to behave well on Windows again. See [#258](simplecov-ruby/simplecov#258) and [#223](simplecov-ruby/simplecov#223) (thanks to @tomykaira) v0.8.1, 2013-11-10 ([changes](simplecov-ruby/simplecov@v0.8.0...v0.8.1)) ===================== ## Bugfixes * Fixed a regression introduced in 0.8.0 - the Forwardable STDLIB module is now required explicitly. See [#256](simplecov-ruby/simplecov#256) (thanks to @kylev) v0.8.0, 2013-11-10 ([changes](simplecov-ruby/simplecov@v0.7.1...v0.8.0)) ===================== **Note: Yanked the same day because of the regression that 0.8.1 fixes, see above** ## TL;DR It's been way too long since the last official release 0.7.1, but this was partly due to it proving itself quite stable in most circumstances. This release brings various further stability improvements to result set merging (especially when working with parallel_tests), the configuration, source file encodings, and command name guessing. The 0.8 line is the last one to cooperate with Ruby < 1.9. Starting with 0.9, SimpleCov will assume to be running in Ruby 1.9+, and will not try to detect or bail silently on older Ruby versions. An appropriate deprecation warning has been added. ## Features * Configuration blocks now have access to variables and methods outside of the block's scope. See [#238](simplecov-ruby/simplecov#238) (thanks to @ms-tg) * You can now have a global `~/.simplecov` configuration file. See [#195](simplecov-ruby/simplecov#195) (thanks to @spagalloco) * simplecov-html now uses the MIT-licensed colorbox plugin. Some adjustments when viewing source files, including retaining the currently open file on refresh have been added. See [simplecov-html #15](simplecov-ruby/simplecov-html#15) (thanks to @chetan) * Adds support for Rails 4 command guessing, removes default group `vendor/plugins`. See [#181](simplecov-ruby/simplecov#181) and [#203](simplecov-ruby/simplecov#203) (thanks to @semanticart and @phallstrom) * You can now load simplecov without the default settings by doing `require 'simplecov/no_defaults'` or setting `ENV['SIMPLECOV_NO_DEFAULTS']`. Check `simplecov/defaults` to see what preconfigurations are getting dropped by using this. See [#209](simplecov-ruby/simplecov#209) (thanks to @ileitch) * The result set merging now uses the `lockfile` gem to avoid race conditions. See [#185](simplecov-ruby/simplecov#185) (thanks to @jshraibman-mdsol). * Automatically detect the usage of parallel_tests and adjust the command name with the test env number accordingly, See [#64](simplecov-ruby/simplecov#64) and [#185](simplecov-ruby/simplecov#185) (thanks to @jshraibman-mdsol). ## Enhancements * Rename adapters to "profiles" given that they are bundles of settings. The old adapter methods are deprecated, but remain available for now. See [#207](simplecov-ruby/simplecov#207) (thanks to @mikerobe) * Tweaks to the automatic test suite naming. In particular, `rspec/features` should now be correctly attributed to RSpec, not Cucumber. See [#212](simplecov-ruby/simplecov#212) (thanks to @ersatzryan and @betelgeuse) * MiniTest should now be identified correctly by the command name guesser. See [#244](simplecov-ruby/simplecov#244) (thanks to @envygeeks) * Makes SimpleCov resilient to inclusion of mathn library. See [#175](simplecov-ruby/simplecov#175) and [#140](simplecov-ruby/simplecov#140) (thanks to @scotje) * Allow coverage_dir to be an absolute path. * See [#190](simplecov-ruby/simplecov#190) (thanks to @jshraibman-mdsol) * The internal cucumber test suite now uses Capybara 2. See [#206](simplecov-ruby/simplecov#206) (thanks to @infertux) * Work-arounds for the Coverage library shipped in JRuby 1.6 to behave in line with MRI. See [#174](simplecov-ruby/simplecov#174) (thanks to @grddev) * Fix warning: instance variable @exit_status not initialized. See [#242](simplecov-ruby/simplecov#242) and [#213](simplecov-ruby/simplecov#213) (thanks to @sferik and @infertux) ## Bugfixes * Correct result calculations for people using :nocov: tags. See [#215](simplecov-ruby/simplecov#215) (thanks to @aokolish) * Average hits per line for groups of files is now computed correctly. See [#192](http://github.com/colszowka/simplecov/pull/192) and [#179](http://github.com/colszowka/simplecov/issues/179) (thanks to @Graysonwright) * Compatability with BINARY internal encoding. See [#194](simplecov-ruby/simplecov#194) and [#127](simplecov-ruby/simplecov#127) (thanks to @justfalter) * Special characters in `SimpleCov.root` are now correctly escaped before being used as a RegExp. See [#204](simplecov-ruby/simplecov#204) and [#237](simplecov-ruby/simplecov#237) (thanks to @rli9)

@tomykaira

v0.8.2, 2013-11-20 ([changes](simplecov-ruby/simplecov@v0.8.1...v0.8.2)) ===================== ## Bugfixes * Replaced the locking behaviour [via lockfile gem](simplecov-ruby/simplecov#185) with plain Ruby explicit file locking when merging results. This should make simplecov merging to behave well on Windows again. See [#258](simplecov-ruby/simplecov#258) and [#223](simplecov-ruby/simplecov#223) (thanks to @tomykaira) v0.8.1, 2013-11-10 ([changes](simplecov-ruby/simplecov@v0.8.0...v0.8.1)) ===================== ## Bugfixes * Fixed a regression introduced in 0.8.0 - the Forwardable STDLIB module is now required explicitly. See [#256](simplecov-ruby/simplecov#256) (thanks to @kylev) v0.8.0, 2013-11-10 ([changes](simplecov-ruby/simplecov@v0.7.1...v0.8.0)) ===================== **Note: Yanked the same day because of the regression that 0.8.1 fixes, see above** ## TL;DR It's been way too long since the last official release 0.7.1, but this was partly due to it proving itself quite stable in most circumstances. This release brings various further stability improvements to result set merging (especially when working with parallel_tests), the configuration, source file encodings, and command name guessing. The 0.8 line is the last one to cooperate with Ruby < 1.9. Starting with 0.9, SimpleCov will assume to be running in Ruby 1.9+, and will not try to detect or bail silently on older Ruby versions. An appropriate deprecation warning has been added. ## Features * Configuration blocks now have access to variables and methods outside of the block's scope. See [#238](simplecov-ruby/simplecov#238) (thanks to @ms-tg) * You can now have a global `~/.simplecov` configuration file. See [#195](simplecov-ruby/simplecov#195) (thanks to @spagalloco) * simplecov-html now uses the MIT-licensed colorbox plugin. Some adjustments when viewing source files, including retaining the currently open file on refresh have been added. See [simplecov-html #15](simplecov-ruby/simplecov-html#15) (thanks to @chetan) * Adds support for Rails 4 command guessing, removes default group `vendor/plugins`. See [#181](simplecov-ruby/simplecov#181) and [#203](simplecov-ruby/simplecov#203) (thanks to @semanticart and @phallstrom) * You can now load simplecov without the default settings by doing `require 'simplecov/no_defaults'` or setting `ENV['SIMPLECOV_NO_DEFAULTS']`. Check `simplecov/defaults` to see what preconfigurations are getting dropped by using this. See [#209](simplecov-ruby/simplecov#209) (thanks to @ileitch) * The result set merging now uses the `lockfile` gem to avoid race conditions. See [#185](simplecov-ruby/simplecov#185) (thanks to @jshraibman-mdsol). * Automatically detect the usage of parallel_tests and adjust the command name with the test env number accordingly, See [#64](simplecov-ruby/simplecov#64) and [#185](simplecov-ruby/simplecov#185) (thanks to @jshraibman-mdsol). ## Enhancements * Rename adapters to "profiles" given that they are bundles of settings. The old adapter methods are deprecated, but remain available for now. See [#207](simplecov-ruby/simplecov#207) (thanks to @mikerobe) * Tweaks to the automatic test suite naming. In particular, `rspec/features` should now be correctly attributed to RSpec, not Cucumber. See [#212](simplecov-ruby/simplecov#212) (thanks to @ersatzryan and @betelgeuse) * MiniTest should now be identified correctly by the command name guesser. See [#244](simplecov-ruby/simplecov#244) (thanks to @envygeeks) * Makes SimpleCov resilient to inclusion of mathn library. See [#175](simplecov-ruby/simplecov#175) and [#140](simplecov-ruby/simplecov#140) (thanks to @scotje) * Allow coverage_dir to be an absolute path. * See [#190](simplecov-ruby/simplecov#190) (thanks to @jshraibman-mdsol) * The internal cucumber test suite now uses Capybara 2. See [#206](simplecov-ruby/simplecov#206) (thanks to @infertux) * Work-arounds for the Coverage library shipped in JRuby 1.6 to behave in line with MRI. See [#174](simplecov-ruby/simplecov#174) (thanks to @grddev) * Fix warning: instance variable @exit_status not initialized. See [#242](simplecov-ruby/simplecov#242) and [#213](simplecov-ruby/simplecov#213) (thanks to @sferik and @infertux) ## Bugfixes * Correct result calculations for people using :nocov: tags. See [#215](simplecov-ruby/simplecov#215) (thanks to @aokolish) * Average hits per line for groups of files is now computed correctly. See [#192](http://github.com/colszowka/simplecov/pull/192) and [#179](http://github.com/colszowka/simplecov/issues/179) (thanks to @Graysonwright) * Compatability with BINARY internal encoding. See [#194](simplecov-ruby/simplecov#194) and [#127](simplecov-ruby/simplecov#127) (thanks to @justfalter) * Special characters in `SimpleCov.root` are now correctly escaped before being used as a RegExp. See [#204](simplecov-ruby/simplecov#204) and [#237](simplecov-ruby/simplecov#237) (thanks to @rli9)

v1.0.10 -- 09 Apr 2014 --------------------- - TLSSChannel: use malloc/realloc/free instead of their legacy Local* variants (fixes #222) - VCard: remove \r from vcard photos. Fixes a recent change Facebook made to their vcard pictures (patch by Fernando Sanchez) - Jingle: fixed replying; distinguish between 'from' and 'initiator'; added Jingle::Session::setInitiator() - Jingle: fixed ::ICEUDP to actually add candidates; added ::Session::initiator(), ::setHandler(), ::sessionAccept( PluginList ); fixed storing of new sessions in ::SessionManager (patches by Erich Keane) - Jingle: removed Jingle::setInitiator() and ::setResponder() (now provided by ctor) - AtomicRefCount: fixed compilation on iOS (patch by Erich Keane) - Jingle::Plugin: added JinglePluginType, pluginType(), and findPlugin() to easily look for and retrieve specific plugins - ConnectionBOSH: fixed return value for recv() (patch by Sudarshan Prasad) - Parser: get rid of bogus isValid() (fixes #180, #224) v1.0.9 -- 15 Oct 2013 --------------------- - changed colon to dash in uid generation to possibly fix #191 - added Channel Binding (needed for SASL SCRAM-SHA-1-PLUS) to SChannel on win32 (untested) - fixed SCRAM-SHA-1-PLUS v1.0.8 -- 15 Oct 2013 --------------------- - ConnectionTLS: make stacked TLS/SSL connections work again with HTTP proxies - added SASL SCRAM-SHA-1/SCRAM-SHA-1-PLUS authentication mechanisms (GnuTLS & OpenSSL only) (#201) - properly seed the RNG - SHA::hex(): finalize() only once v1.0.7.1 -- 11 Oct 2013 --------------------- - fixed/updated the win32 project files v1.0.7 -- 11 Oct 2013 --------------------- - added Jingle (XEP-0166) - added Jingle ICE-UDP Transport (XEP-0176) - added Jingle File Transfer (XEP-0234) - fixed compilation in iOS7 SDK (thanks to Kurt Vermeersch) - fixed bug in stanza handling/counting related to Stream Management (patch by Norbert Riedlin) - added protected ClientBase::stanzasSent() to return sent stanzas (if Stream Management enabled) v1.0.6 -- 04 Sep 2013 --------------------- - ClientBase: removed check for empty message body --> messages of type chat with empty bodies will be passed on to listeners - required for Message Carbons - MessageSession: removed check for empty message body --> messages of type chat with empty bodies will be passed on to listeners - required for Message Carbons - don't send presence after stream resumption - added ClientBase::sendQueue() - documentation updates and fixes v1.0.5 -- 02 Sep 2013 --------------------- - added support for Stanza Forwarding (XEP-0297) - added support for Message Carbons (XEP-0280) v1.0.4 -- 30 Aug 2013 --------------------- - added support for Stream Management (XEP-0198) - Fix some iterator usage for portability related to erase (patch by Daniel Bowen) - Ensure setting a connection does not leave a deleted value in the member variable for a time (patch by Daniel Bowen) - Add operator< and relatives to JID so that it can be in a map (patch by Daniel Bowen) - Sandboxing on Apple doesn't like getprotobyname (patch by Daniel Bowen) - MessageSession::send(): removed default argument of 2nd parameter to remove ambiguity of MS::send( string ) (#206) (source incompatible!) - VCard: renamed setPhoto( string ) to setPhotoUri( string ) (#166) (source and binary incompatible) v1.0.3 -- 22 Jul 2013 --------------------- - Changed license to GPLv3 - removed space from VS project name (--> gloox-1.0) - VCardUpdate: fixed handling of empty hash (#203) - VCardUpdate: added hasPhoto() to inidicate whether there was a photo tag (#203) - compilation fixed when using getaddrinfo (patch by Roy van Dam) - Receipt: recognize id attribute (patch by Dídac Pérez) (#208) - MessageSession: added MS::send( string& ) to properly provide a base for MUCMS::send( string& ) (#206) - really fixed memory leak in prep::idna() - gloox.vcproj: removed not-yet-present tlsgnutlsserver.cpp/.h v1.0.2 -- 05 Jul 2013 --------------------- - SOCKS5Bytestream: Don't wait for incoming data, notify about open stream immediately upon connection (patch by Erik Horemans) - fixed/updated Code::Blocks and VS project files (fixes #197, #198) - fixed memory leak in ClientBase (fixes #204) - fixed memory leak in prep::idna() v1.0.1 -- 29 Jun 2013 --------------------- - Added support for Serverless Messaging (XEP-0174) - TLSOpenSSLServer: compilation fix - don't bail on DNS TCP queries - fixed µs timeout value (now defaults to 1.000.000) - omit port in initial greeting (usually -1 anyway) - fixed SHA1 hashes of 55 byte strings (#164) - fixed CFLAGS and LIBS in pkg-config file (#163) - fixed SOCKS5Bytestream double close notification - tell gcrypt that we're using pthreads (if available) - ClientBase: send IQ error response for unsupported features - ClientBase: fixed potential infinite loop on IQ error - ClientBase: fixed NTLM auth - ClientBase, SEF: mutex-protected SE handling - PubSub: added 'subscribe & configure' - PubSub: added optional subid - SOCKS5BytestreamServer: expose local socket - RosterManager: don't use string-comparison on JIDs; use JID class - NonSASLAuth: fixed resource usage by deprecating ClientBase::m_selectedResource - InBandBytestream: don't call handler in dtor - util: fixed long2string() - fixed a few leaks in GnutTLS client code (#181) - VCard: made getters const (#186) (binary-incompatible change!) - PubsubManager: fixed using wrong Tag (#190) - Search: fixed search() (#193) - DNS: fix socket leak if no network connection is available (#192) - PubsubManager: unconditionally call handleItemPublication() (#194) - configure: Added -lgcrypt (dependency of GNUTLS)

Last pkgsrc was version from 2002. Changes in 2.04: Add new programming language ObjectiveâC. Add new bracket style option "style=google" (-A14). Add new option "indent-preproc-cond" (xw) to indent preprocessor conditional statements (#118). Add new bracket modify options "attach-namespaces", "attach-classes", "attach-inlines", and "attach-extern-c". Add new option "indent-modifiers" (-xG) to indent class access modifiers one-half indent (#130). Add new option "remove-brackets" (-xj) to remove brackets from single line conditional statements. Add new option "remove-comment-prefix" (-xp) to remove the leading '*' from multi-line comments. Add new option "align-method-colon" (-xM) to align ObjectiveâC method colons. Add new option "pad-method-colon=#" (-xP#) to space pad ObjectiveâC method colons. Add new options "pad-method-prefix" (-xQ), and "unpad-method-prefix" (-xR) to pad the ObjectiveâC "-" and "+" method prefix. Add new dll entry point AStyleMainUtf16 for processing C# UTF-16 strings. Add formatting of C++11 raw string literals (#222). Add "style=knf" as an alternative to "style=linux". Remove depreciated "bracket=" options. Improve recognition and formatting of pointers and references (#174 and other changes). Improve the recognition of block-opening brackets. Improve code using a static code analyzer (#195). Change "max-code-length" to include ObjectiveâC methods. Change "indent-elseifs" and "break-blocks" to look ahead only if in command-type brackets (speed improvement). Fix linux bracket styles to break the opening bracket in inline function definitions (#185). Fix indentation of switch block comments (#164). Fix enums to indent with tabs when requested (#92, #121). Fix formatting of rvalue reference without a name in a declaration (#219). Fix "pad-first-paren-out" to not pad if the following parens are empty (#232). Fix end-of-statement reset when comments follow closing bracket. Fix the ASBeautifier active and waiting stacks to delete the ASBeautifier objects before deleting the pointers. Fix ASBeautifier "init" to delete the tempStack vectors before deleting the tempStack. Fix Linux piping problem by changing "cin" input to build a stringstream before formatting. Fix to identify the correct bracket type when 'extern "C"' is part of the enum definition. Fix to clear 'extern "C"' variables when the block is closed. Fix unindented 'extern "C"' to not indent when in a #else preprocessor directive. Fix not always correctly formatting linux type brackets for enum blocks. Fix align-pointer in a range-based for statement (#217). Fix pointer-reference argument alignment to correctly position a following comment (#235). Fix to not attach a bracket to a line ending in a backslash '\' (#186, #214, #220). Fix to recognize templates using multiple lines (#85, #87, #136). Fix formatting of template continuation lines (#85, #87, #136). Fix to allow '^' as an array operator (#233). Fix an "enum" argument being mistaken for an enumeration (#211). Fix to recognize a non-instatement array after a "},{" sequence. Fix "pad-oper" to not pad before a following comma. Fix recognition of an operator when the calculation contains a bitwise "not" '~' (#166). Fix to allow a preprocessor statement within a preprocessor define (#238). Fix preprocessor comparison to check for whole words (#246). Fix "add-brackets" when a line contains more than one paren pairs (#181). Fix to allow Mac old CR line endings in the options file (#129). Refactor to aid debugging and improve design and decomposition: Move ALL preliminary indentation calculations to computePreliminaryIndentation() in ASBeautifier. Move calculation of 'force tab' indents to preLineWS() in ASBeautifier. Combine methods init() and init(ASSourceIterator*) in ASBeautifier. Extract method adjustParsedLineIndentation() in ASBeautifier. Extract method parseCurrentLine() in ASEnhancer. Remove astyle_main.cpp unused functions getFilesUnchanged, getOptionsFileRequired, and setOptionsFileRequired. Older changes included in distfile, or available on http://astyle.sourceforge.net/notes.html

Version 2.0 ----------- (released Nov 9, 2014) - Default lexer encoding is now "guess", i.e. UTF-8 / Locale / Latin1 is tried in that order. - Major update to Swift lexer (pull request #410). - Multiple fixes to lexer guessing in conflicting cases: * recognize HTML5 by doctype * recognize XML by XML declaration * don't recognize C/C++ as SystemVerilog - Simplified regexes and builtin lists. Version 2.0rc1 -------------- (released Oct 16, 2014) - Dropped Python 2.4 and 2.5 compatibility. This is in favor of single-source compatibility between Python 2.6, 2.7 and 3.3+. - New website and documentation based on Sphinx (finally!) - Lexers added: * APL (#969) * Agda and Literate Agda (pull request #203) * Alloy (pull request #355) * AmbientTalk * BlitzBasic (pull request #197) * ChaiScript (pull request #24) * Chapel (pull request #256) * Cirru (pull request #275) * Clay (pull request #184) * ColdFusion CFC (pull request #283) * Cryptol and Literate Cryptol (pull request #344) * Cypher (pull request #257) * Docker config files * EBNF (pull request #193) * Eiffel (pull request #273) * GAP (pull request #311) * Golo (pull request #309) * Handlebars (pull request #186) * Hy (pull request #238) * Idris and Literate Idris (pull request #210) * Igor Pro (pull request #172) * Inform 6/7 (pull request #281) * Intel objdump (pull request #279) * Isabelle (pull request #386) * Jasmin (pull request #349) * JSON-LD (pull request #289) * Kal (pull request #233) * Lean (pull request #399) * LSL (pull request #296) * Limbo (pull request #291) * Liquid (#977) * MQL (pull request #285) * MaskJS (pull request #280) * Mozilla preprocessors * Mathematica (pull request #245) * NesC (pull request #166) * Nit (pull request #375) * Nix (pull request #267) * Pan * Pawn (pull request #211) * Perl 6 (pull request #181) * Pig (pull request #304) * Pike (pull request #237) * QBasic (pull request #182) * Red (pull request #341) * ResourceBundle (#1038) * Rexx (pull request #199) * Rql (pull request #251) * Rsl * SPARQL (pull request #78) * Slim (pull request #366) * Swift (pull request #371) * Swig (pull request #168) * TADS 3 (pull request #407) * Todo.txt todo lists * Twig (pull request #404) - Added a helper to "optimize" regular expressions that match one of many literal words; this can save 20% and more lexing time with lexers that highlight many keywords or builtins. - New styles: "xcode" and "igor", similar to the default highlighting of the respective IDEs. - The command-line "pygmentize" tool now tries a little harder to find the correct encoding for files and the terminal (#979). - Added "inencoding" option for lexers to override "encoding" analogous to "outencoding" (#800). - Added line-by-line "streaming" mode for pygmentize with the "-s" option. (pull request #165) Only fully works for lexers that have no constructs spanning lines! - Added an "envname" option to the LaTeX formatter to select a replacement verbatim environment (pull request #235). - Updated the Makefile lexer to yield a little more useful highlighting. - Lexer aliases passed to ``get_lexer_by_name()`` are now case-insensitive. - File name matching in lexers and formatters will now use a regex cache for speed (pull request #205). - Pygments will now recognize "vim" modelines when guessing the lexer for a file based on content (pull request #118). - Major restructure of the ``pygments.lexers`` module namespace. There are now many more modules with less lexers per module. Old modules are still around and re-export the lexers they previously contained. - The NameHighlightFilter now works with any Name.* token type (#790). - Python 3 lexer: add new exceptions from PEP 3151. - Opa lexer: add new keywords (pull request #170). - Julia lexer: add keywords and underscore-separated number literals (pull request #176). - Lasso lexer: fix method highlighting, update builtins. Fix guessing so that plain XML isn't always taken as Lasso (pull request #163). - Objective C/C++ lexers: allow "@" prefixing any expression (#871). - Ruby lexer: fix lexing of Name::Space tokens (#860) and of symbols in hashes (#873). - Stan lexer: update for version 2.4.0 of the language (pull request #162, PR#255, PR#377). - JavaScript lexer: add the "yield" keyword (pull request #196). - HTTP lexer: support for PATCH method (pull request #190). - Koka lexer: update to newest language spec (pull request #201). - Haxe lexer: rewrite and support for Haxe 3 (pull request #174). - Prolog lexer: add different kinds of numeric literals (#864). - F# lexer: rewrite with newest spec for F# 3.0 (#842), fix a bug with dotted chains (#948). - Kotlin lexer: general update (pull request #271). - Rebol lexer: fix comment detection and analyse_text (pull request #261). - LLVM lexer: update keywords to v3.4 (pull request #258). - PHP lexer: add new keywords and binary literals (pull request #222). - external/markdown-processor.py updated to newest python-markdown (pull request #221). - CSS lexer: some highlighting order fixes (pull request #231). - Ceylon lexer: fix parsing of nested multiline comments (#915). - C family lexers: fix parsing of indented preprocessor directives (#944). - Rust lexer: update to 0.9 language version (pull request #270, PR#388). - Elixir lexer: update to 0.15 language version (pull request #392). - Fix swallowing incomplete tracebacks in Python console lexer (#874).

---------------- Documentation ************* - Extend support to Python 3.4, deprecating Python 3.2. - Issue #198: Mention Zake as a sophisticated kazoo mock testing library. - Issue #181: Add documentation on basic logging setup.

Ok MAINTAINER bsiegert. While doing that, update to current release, 0.29.0. Changes since 0.24.0: Version 0.29.0 -------------- Compatibility notes: - when upgrading to 0.29.0 you need to upgrade client as well as server installations due to the locking and commandline interface changes otherwise you'll get an error msg about a RPC protocol mismatch or a wrong commandline option. if you run a server that needs to support both old and new clients, it is suggested that you have a "borg-0.28.2" and a "borg-0.29.0" command. clients then can choose via e.g. "borg --remote-path=borg-0.29.0 ...". - the default waiting time for a lock changed from infinity to 1 second for a better interactive user experience. if the repo you want to access is currently locked, borg will now terminate after 1s with an error message. if you have scripts that shall wait for the lock for a longer time, use --lock-wait N (with N being the maximum wait time in seconds). Bug fixes: - hash table tuning (better chosen hashtable load factor 0.75 and prime initial size of 1031 gave ~1000x speedup in some scenarios) - avoid creation of an orphan lock for one case, #285 - --keep-tag-files: fix file mode and multiple tag files in one directory, #432 - fixes for "borg upgrade" (attic repo converter), #466 - remove --progress isatty magic (and also --no-progress option) again, #476 - borg init: display proper repo URL - fix format of umask in help pages, #463 New features: - implement --lock-wait, support timeout for UpgradableLock, #210 - implement borg break-lock command, #157 - include system info below traceback, #324 - sane remote logging, remote stderr, #461: - remote log output: intercept it and log it via local logging system, with "Remote: " prefixed to message. log remote tracebacks. - remote stderr: output it to local stderr with "Remote: " prefixed. - add --debug and --info (same as --verbose) to set the log level of the builtin logging configuration (which otherwise defaults to warning), #426 note: there are few messages emitted at DEBUG level currently. - optionally configure logging via env var BORG_LOGGING_CONF - add --filter option for status characters: e.g. to show only the added or modified files (and also errors), use "borg create -v --filter=AME ...". - more progress indicators, #394 - use ISO-8601 date and time format, #375 - "borg check --prefix" to restrict archive checking to that name prefix, #206 Other changes: - hashindex_add C implementation (speed up cache re-sync for new archives) - increase FUSE read_size to 1024 (speed up metadata operations) - check/delete/prune --save-space: free unused segments quickly, #239 - increase rpc protocol version to 2 (see also Compatibility notes), #458 - silence borg by default (via default log level WARNING) - get rid of C compiler warnings, #391 - upgrade OS X FUSE to 3.0.9 on the OS X binary build system - use python 3.5.1 to build binaries - docs: - new mailing list [email protected], #468 - readthedocs: color and logo improvements - load coverage icons over SSL (avoids mixed content) - more precise binary installation steps - update release procedure docs about OS X FUSE - FAQ entry about unexpected 'A' status for unchanged file(s), #403 - add docs about 'E' file status - add "borg upgrade" docs, #464 - add developer docs about output and logging - clarify encryption, add note about client-side encryption - add resources section, with videos, talks, presentations, #149 - Borg moved to Arch Linux [community] - fix wrong installation instructions for archlinux Version 0.28.2 -------------- New features: - borg create --exclude-if-present TAGFILE - exclude directories that have the given file from the backup. You can additionally give --keep-tag-files to preserve just the directory roots and the tag-files (but not backup other directory contents), #395, attic #128, attic #142 Other changes: - do not create docs sources at build time (just have them in the repo), completely remove have_cython() hack, do not use the "mock" library at build time, #384 - avoid hidden import, make it easier for PyInstaller, easier fix for #218 - docs: - add description of item flags / status output, fixes #402 - explain how to regenerate usage and API files (build_api or build_usage) and when to commit usage files directly into git, #384 - minor install docs improvements Version 0.28.1 -------------- Bug fixes: - do not try to build api / usage docs for production install, fixes unexpected "mock" build dependency, #384 Other changes: - avoid using msgpack.packb at import time - fix formatting issue in changes.rst - fix build on readthedocs Version 0.28.0 -------------- Compatibility notes: - changed return codes (exit codes), see docs. in short: old: 0 = ok, 1 = error. now: 0 = ok, 1 = warning, 2 = error New features: - refactor return codes (exit codes), fixes #61 - add --show-rc option enable "terminating with X status, rc N" output, fixes 58, #351 - borg create backups atime and ctime additionally to mtime, fixes #317 - extract: support atime additionally to mtime - FUSE: support ctime and atime additionally to mtime - support borg --version - emit a warning if we have a slow msgpack installed - borg list --prefix=thishostname- REPO, fixes #205 - Debug commands (do not use except if you know what you do: debug-get-obj, debug-put-obj, debug-delete-obj, debug-dump-archive-items. Bug fixes: - setup.py: fix bug related to BORG_LZ4_PREFIX processing - fix "check" for repos that have incomplete chunks, fixes #364 - borg mount: fix unlocking of repository at umount time, fixes #331 - fix reading files without touching their atime, #334 - non-ascii ACL fixes for Linux, FreeBSD and OS X, #277 - fix acl_use_local_uid_gid() and add a test for it, attic #359 - borg upgrade: do not upgrade repositories in place by default, #299 - fix cascading failure with the index conversion code, #269 - borg check: implement 'cmdline' archive metadata value decoding, #311 - fix RobustUnpacker, it missed some metadata keys (new atime and ctime keys were missing, but also bsdflags). add check for unknown metadata keys. - create from stdin: also save atime, ctime (cosmetic) - use default_notty=False for confirmations, fixes #345 - vagrant: fix msgpack installation on centos, fixes #342 - deal with unicode errors for symlinks in same way as for regular files and have a helpful warning message about how to fix wrong locale setup, fixes #382 - add ACL keys the RobustUnpacker must know about Other changes: - improve file size displays, more flexible size formatters - explicitly commit to the units standard, #289 - archiver: add E status (means that an error occured when processing this (single) item - do binary releases via "github releases", closes #214 - create: use -x and --one-file-system (was: --do-not-cross-mountpoints), #296 - a lot of changes related to using "logging" module and screen output, #233 - show progress display if on a tty, output more progress information, #303 - factor out status output so it is consistent, fix surrogates removal, maybe fixes #309 - move away from RawConfigParser to ConfigParser - archive checker: better error logging, give chunk_id and sequence numbers (can be used together with borg debug-dump-archive-items). - do not mention the deprecated passphrase mode - emit a deprecation warning for --compression N (giving a just a number) - misc .coverragerc fixes (and coverage measurement improvements), fixes #319 - refactor confirmation code, reduce code duplication, add tests - prettier error messages, fixes #307, #57 - tests: - add a test to find disk-full issues, #327 - travis: also run tests on Python 3.5 - travis: use tox -r so it rebuilds the tox environments - test the generated pyinstaller-based binary by archiver unit tests, #215 - vagrant: tests: announce whether fakeroot is used or not - vagrant: add vagrant user to fuse group for debianoid systems also - vagrant: llfuse install on darwin needs pkgconfig installed - vagrant: use pyinstaller from develop branch, fixes #336 - benchmarks: test create, extract, list, delete, info, check, help, fixes #146 - benchmarks: test with both the binary and the python code - archiver tests: test with both the binary and the python code, fixes #215 - make basic test more robust - docs: - moved docs to borgbackup.readthedocs.org, #155 - a lot of fixes and improvements, use mobile-friendly RTD standard theme - use zlib,6 compression in some examples, fixes #275 - add missing rename usage to docs, closes #279 - include the help offered by borg help <topic> in the usage docs, fixes #293 - include a list of major changes compared to attic into README, fixes #224 - add OS X install instructions, #197 - more details about the release process, #260 - fix linux glibc requirement (binaries built on debian7 now) - build: move usage and API generation to setup.py - update docs about return codes, #61 - remove api docs (too much breakage on rtd) - borgbackup install + basics presentation (asciinema) - describe the current style guide in documentation - add section about debug commands - warn about not running out of space - add example for rename - improve chunker params docs, fixes #362 - minor development docs update Version 0.27.0 -------------- New features: - "borg upgrade" command - attic -> borg one time converter / migration, #21 - temporary hack to avoid using lots of disk space for chunks.archive.d, #235: To use it: rm -rf chunks.archive.d ; touch chunks.archive.d - respect XDG_CACHE_HOME, attic #181 - add support for arbitrary SSH commands, attic #99 - borg delete --cache-only REPO (only delete cache, not REPO), attic #123 Bug fixes: - use Debian 7 (wheezy) to build pyinstaller borgbackup binaries, fixes slow down observed when running the Centos6-built binary on Ubuntu, #222 - do not crash on empty lock.roster, fixes #232 - fix multiple issues with the cache config version check, #234 - fix segment entry header size check, attic #352 plus other error handling improvements / code deduplication there. - always give segment and offset in repo IntegrityErrors Other changes: - stop producing binary wheels, remove docs about it, #147 - docs: - add warning about prune - generate usage include files only as needed - development docs: add Vagrant section - update / improve / reformat FAQ - hint to single-file pyinstaller binaries from README Version 0.26.1 -------------- This is a minor update, just docs and new pyinstaller binaries. - docs update about python and binary requirements - better docs for --read-special, fix #220 - re-built the binaries, fix #218 and #213 (glibc version issue) - update web site about single-file pyinstaller binaries Note: if you did a python-based installation, there is no need to upgrade. Version 0.26.0 -------------- New features: - Faster cache sync (do all in one pass, remove tar/compression stuff), #163 - BORG_REPO env var to specify the default repo, #168 - read special files as if they were regular files, #79 - implement borg create --dry-run, attic issue #267 - Normalize paths before pattern matching on OS X, #143 - support OpenBSD and NetBSD (except xattrs/ACLs) - support / run tests on Python 3.5 Bug fixes: - borg mount repo: use absolute path, attic #200, attic #137 - chunker: use off_t to get 64bit on 32bit platform, #178 - initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0) - fix reaction to "no" answer at delete repo prompt, #182 - setup.py: detect lz4.h header file location - to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194) - fix for obtaining ``char *`` from temporary Python value (old code causes a compile error on Mint 17.2) - llfuse 0.41 install troubles on some platforms, require < 0.41 (UnicodeDecodeError exception due to non-ascii llfuse setup.py) - cython code: add some int types to get rid of unspecific python add / subtract operations (avoid ``undefined symbol FPE_``... error on some platforms) - fix verbose mode display of stdin backup - extract: warn if a include pattern never matched, fixes #209, implement counters for Include/ExcludePatterns - archive names with slashes are invalid, attic issue #180 - chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined - fixes building on OpenBSD. Other changes: - detect inconsistency / corruption / hash collision, #170 - replace versioneer with setuptools_scm, #106 - docs: - pkg-config is needed for llfuse installation - be more clear about pruning, attic issue #132 - unit tests: - xattr: ignore security.selinux attribute showing up - ext3 seems to need a bit more space for a sparse file - do not test lzma level 9 compression (avoid MemoryError) - work around strange mtime granularity issue on netbsd, fixes #204 - ignore st_rdev if file is not a block/char device, fixes #203 - stay away from the setgid and sticky mode bits - use Vagrant to do easy cross-platform testing (#196), currently: - Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit - Ubuntu 12.04 32bit, Ubuntu 14.04 64bit - Centos 7 64bit - FreeBSD 10.2 64bit - OpenBSD 5.7 64bit - NetBSD 6.1.5 64bit - Darwin (OS X Yosemite) Version 0.25.0 -------------- Compatibility notes: - lz4 compression library (liblz4) is a new requirement (#156) - the new compression code is very compatible: as long as you stay with zlib compression, older borg releases will still be able to read data from a repo/archive made with the new code (note: this is not the case for the default "none" compression, use "zlib,0" if you want a "no compression" mode that can be read by older borg). Also the new code is able to read repos and archives made with older borg versions (for all zlib levels 0..9). Deprecations: - --compression N (with N being a number, as in 0.24) is deprecated. We keep the --compression 0..9 for now to not break scripts, but it is deprecated and will be removed later, so better fix your scripts now: --compression 0 (as in 0.24) is the same as --compression zlib,0 (now). BUT: if you do not want compression, you rather want --compression none (which is the default). --compression 1 (in 0.24) is the same as --compression zlib,1 (now) --compression 9 (in 0.24) is the same as --compression zlib,9 (now) New features: - create --compression none (default, means: do not compress, just pass through data "as is". this is more efficient than zlib level 0 as used in borg 0.24) - create --compression lz4 (super-fast, but not very high compression) - create --compression zlib,N (slower, higher compression, default for N is 6) - create --compression lzma,N (slowest, highest compression, default N is 6) - honor the nodump flag (UF_NODUMP) and do not backup such items - list --short just outputs a simple list of the files/directories in an archive Bug fixes: - fixed --chunker-params parameter order confusion / malfunction, fixes #154 - close fds of segments we delete (during compaction) - close files which fell out the lrucache - fadvise DONTNEED now is only called for the byte range actually read, not for the whole file, fixes #158. - fix issue with negative "all archives" size, fixes #165 - restore_xattrs: ignore if setxattr fails with EACCES, fixes #162 Other changes: - remove fakeroot requirement for tests, tests run faster without fakeroot (test setup does not fail any more without fakeroot, so you can run with or without fakeroot), fixes #151 and #91. - more tests for archiver - recover_segment(): don't assume we have an fd for segment - lrucache refactoring / cleanup, add dispose function, py.test tests - generalize hashindex code for any key length (less hardcoding) - lock roster: catch file not found in remove() method and ignore it - travis CI: use requirements file - improved docs: - replace hack for llfuse with proper solution (install libfuse-dev) - update docs about compression - update development docs about fakeroot - internals: add some words about lock files / locking system - support: mention BountySource and for what it can be used - theme: use a lighter green - add pypi, wheel, dist package based install docs - split install docs into system-specific preparations and generic instructions

pkgsrc changes: - set license - revamp build definitions - revamp blk3.mk and bump API_DEPENDS to 0.13 (recent software depends on new ode-config) - drop all local patches as not needed upstream changes: Changes for 0.13 New features: Stable, implicit gyroscopic forces (patch #185 by Joseph Cooper). Bodies with gyroscopic forces won't gain energy anymore. New joint: transmission joint (patch #182 by Dimitris Papavasiliou). Contacts now have rolling friction (patch #184 by Joseph Cooper). Removed the need for defining dSINGLE/dDOUBLE; this is stored now in the generated ode/precision.h header. New joint: Double Ball (AKA "distance joint"). New joint: Double Hinge. Threaded execution support interface added. Optional built-in threading implementation added. Internal threading implementation is excluded by default and to be used, it must be enabled with configure/premake: --disable-threading-intf (--no-threading-intf for Windows/Premake). This allows disabling threading interface support. Bug fixes Joint feedback forces application fixed in QuickStep implementation Bug #89 by Luc: (dJointAddSliderForce() adds a zero force when the parent body is NULL) Bug #88 by Luc: heightfield data assigned to a wrong field in dGeomHeightfieldSetHeightfieldData Fixed cylinder AABB computation. Removed ALLOCA calls from dHashSpace; it should not depend on stack size limits anymore. Bugs in AMotor joint (patch #181) Bugs in PU joint (patch #186) Fixed issue with friction and non-friction constraints being intermixed during constraints random reordering in QuickStep. Fixed matrix inversion bug (patch #183). Fixed issue with some kinds of joints (Ball, DBall, DHinge, Fixed) might overwrite world ERP value with their custom ERP during getInfo2() call and that inappropriate value would then be passed to subsequent joints in solver instead of world ERP. Fixed issues reported in patches #151 and #22 (collisions with SAPSpace and QuadTreeSpace might not work because geometries list was misused in them). Fixed IsPointInPolygon in convex.cpp (patch #160 by Janis Rucis) Fixed zero comparisons in OPCODE to use relative error instead of absolute epsilon value (found by Bill Sellers) Fixed dhinge's last constraint to properly handle rotations. Fixed plane2d joint: uninitialized variables (reported by Dimitris Papavasiliou) Bug #80 by Georg Martius: better handling of capsule-box with deep penetrations. Fixed zero-mu issues: now either mu or mu2 can be set to zero. Other changes Public headers cleaned to cause less pollution of typedefs and macros. Changes for 0.12 Added python bindings, contributed by Gideon Klompje. Integrated libccd from Daniel Fiser, provides new colliders based on GJK, EPA and MPR. Now cylinders can collide with cylinders! New functions have been added: dWorldUseSharedWorkingMemory() dWorldCleanupWorkingMemory() dWorldSetStepMemoryReservationPolicy() dWorldSetStepMemoryManager() dGeomLowLevelControl(): change/query OPCODE trimesh-sphere contact merging behavior at runtime. dGeomGetRelPointPos() (patch #2991622) dGeomGetPosRelPoint() (patch #2991622) dGeomVectorToWorld() (patch #2991622) dGeomVectorFromWorld() (patch #2991622) dWorldStepFast1 API removed along with dWorld[Get/Set]AutoEnableDepthSF1 dWorldStep() and dWorldQuickStep() have been changed to return boolean success status, and avoid allocation on stack. dInitODE2() changed to automatically call AllocateODEDataForThread(dAllocateFlagBasicData) after library initialization. Made sure neither dSINGLE or dDOUBLE is defined by default; the user should always explicitly specify the precision. Fixed a bug in dGeomCopyOffsetRotation(). Macros changed to static inline functions in odemath.h and related files. Improvement for trimesh-plane collision. odecpp classes changed to be inheritable and easily expandable QuadTreeSpace implementation corrected to avoid object-block relation ambiguity due to numeric errors. Fixed bug affecting disabled joints and dWorldStep. Added extern "C" wrapper to dWorldExportDIF(). Fixed bug #2937076: don't try to build demos if drawstuff is disabled. Applied patch #2931174 to make demos work for recent MacOS. Applied patch #2931177 to fix the demos' framerate on X11. Applied patch #2995450 to generate up to four contacts for box-plane collision test and fix contact depths. Applied patch #3030783 to fix drawstuff dimensions being ignored in OSX GLUT port. Applied patch #3429454: fix compilation on some platforms. Heightfield zone boundaries calculation code fixed to also consider whole next cell after the AABB if the AABB ends exactly at the cell boundary. New demo: demo_tracks. Fixed a box-capsule bug: more reasonable normal for deep penetrations. Many internal fixes to world stepping and collision detection code. Changes for 0.11.1 Made the new trimesh collider the default. Added a "-texturepath" option to drawstuff. Heightfield rotation fixed to avoid NaNs while rotating infinite MIN/MAX heights. Incorrect parameter order fixed on contact merging in Sphere-Trimesh collisions. Fixed bug #2685170: use the C99 __func__ instead of __FUNCTION__ when a C99 implementation is available. Remove unused code in demo_joints.cpp. Fix bug in collision categories in demo_jointPU. Added a possibility to initialize/close ODE multiple times recursively has been added. Also, now a call to dSpaceSetManualCleanup() is required for each space right after creation if ODE has been initialized in thread data manual cleanup mode. Thread local data has been cleaned up from OPCODE and OdeTls. New house of cards demo, which stresses the friction handling stability. Changes for 0.11 This release contains many joint-related bug fixes, and some new features like kinematic bodies, better convex-convex collision detection and some extra functions. Fix bug: Fix problem when attaching no body to a joint. Fix problem with dJointGetUniversalAngle2 when the joint is attached to only a body 2. The sign was inverted. Fix bug reported by Tilman: dxJointPU::getInfo1 was setting twice the limit of limot1 to zero and not limot2. Fixed a bug in dSpaceCollide2: if both geoms are not in spaces they would not have valid AABBs. Swapped geometries returned in contacts for OPCODE Trimesh-Plane collisions as they were returned in unnatural order being different from that in GIMPACT Fixed an inverted sign problem on positions of joints PU and PR. Fix bug: when a slider, piston, pu or pr joint had only one body attached to position 2, dJointAttach(jId, 0, bId). The body was not push in the right direction to move back between the limits. Fix bug in dJointGetPUAxis2: the axis was not multiplied with the the rotation matrix of the good body. Fix bug if there is only one body on the PU joint the axis returned was not the right one. New functions: dJointSetUniversalAxis1Offset and dJointSetUniversalAxis2Offset. New functions: dJointEnable, dJointDisable, dJointIsEnabled (patch #2454764). New functions: dGeomTriMeshSetTriMergeCallback/dGeomTriMeshGetTriMergeCallback. New function: void dJointSetPUAnchorOffset (deprecating dJointSetPUAnchorDelta). New functions: dJointGetPRAngle and dJointGetPRAngleRate. New functions: dBodyGetGyroscopicMode and dBodySetGyroscopicMode (patch #2019242). Applied patch #2538046: Heightfield AABB bounds patch. Applied patch #2381592, which adds support for Kinematic Bodies. Applied a patch to make GIMPACT trimesh-ray collisions to be consistent with those in OPCODE (bug #2214623). Applied a patch to make side1, side2 fields of contact structure be always initialized. Applied patch #2158425: This patch enable GIMPACT to work on 64-bit machine. Applied patch #2080674: Improved dBodySetRotation; now exact rotation matrices are preserved until the next simulation step. Enable the motor on the rotoide part of the PR joint. Added Blender script to create ODE convex geoms under tools. Convex-Convex collision detection code is finally (nearly) stable. Changes for 0.10.1 This is a bugfix release, so you shouldn't expect anything to stop working from 0.10. New function: dJointSetPistonAnchorOffset Add new function dJointSetHingeAxisDelta Fix problem with dJointGetPistonPosition and dJointGetPistonPositionRate when the joint is attached to only a body 2. The sign was inversed. Update the slider joint to have the same behavior as the other joint when there is only a body2 attached to it. Rename the new function dJointSetHingeAxisDelta to dJointSetHingeAxisOffset. This will remove confusion with the old function dJointSetHingeAnchorDelta. Bug fix: Max Correcting Vel behavior is now the same as before 0.10. Changes for 0.10 New functions: dInitODE2(), dAllocateODEDataForThread(), dCleanupODEAllDataForThread(). This adds support for ability to run collision detection from multiple threads for separate spaces. New functions: dCheckConfiguration(), dGetConfiguration(), to test how ODE was compiled. New function: dJointGetNumBodies (patch #1901550). New function: dSpaceGetClass (patch #1901637). New function: dSetColliderOverride(), to register custom collision handling functions. Added damping and MaxAngularVel() functions. Applied patch #1335202: Contact Joint Motion (see demo_motion.cpp). Applied patch #1851394: support for GIMPACT with double precision, dCollide fix. Applied patch #1813079 (moved callback). Added possibility to collide a space of lower sublevel as a geometry against another space of a higher level with dSpaceCollide2. dSpaceSetSublevel/dSpaceGetSublevel are used for sublevel assignment/retrieval. Fixed a bug in dJointXXXGetInfo. The value in limot.limit was not always updated. (Ex: If hi and lo limit were changed). New Joint: Prismatic Universal (patch #1828454). New Joint: Piston, with demo. Disabled building shared library by default with autotools. ODE shouldn't be installed as a system library. Fixed drawstuff build issues on OSX. Removed release and debug flags for configure.in: CPPFLAGS, CFLAGS, CXXFLAGS should be set by the user to their liking, respecting autotools policies. Added 'Sweep and Prune' collision space. Optimizations, many bug fixes, and code cleanup. Changes from 0.8 to 0.9 New Feature: CONTACTS_UNIMPORTANT flag added for dCollide() to allow canceling any contact improvements and return any contacts found as soon as possible New feature: Add erp/cfm parameters to fixed and ball joint. SF PATCH 1478244 Assert on zero normalizations, add dSafeNormalize. Added boolean result for dNormalize3() and dNormalize4() so that normalization success could be checked SF Patch 1808007: fix gimpact compiler warning Apply SF patch 1800579: dGeomTriMeshSetData does not work with GIMPACT Fix: heightfield collision could return less contacts than was possible Fix: the deepest contact might be not selected if there was a lack of contact slots in trimesh-trimesh and trimesh-box cases (function exited immediately and did not check for possible contact coincidences). Improvement: optimizations for GIMPACT versions of trimesh-capsule, trimesh-trimesh, trimesh-plane, trimesh-sphere Improvement: dCollideConvexPlane optimized Fix: cylinder-box collision did not respect requested contact count limit Fix SF issue 1729096 by adding dMassSetTrimeshTotal Apply SF patch 1685107: Enforce 1 contact request Apply SF Patch 1697014: Update JointPR code Catch SF BUG 1621938: gimpact only does stride 12. patch 1769851 by Jon Watte: recalc aabb when new setting new data Added Richard Barrington's model loader for XNA (to be used with ODE.NET) 1586733 collision_trimesh_trimesh.cpp OPCODE fix Added Francisco Leon's alternate trimesh-trimesh collider. 1691402 Heighfield performance patch patch 1696933 Fix bug in dWorldExportDIF when joint in "limbo" Renamed "tests" to "demos" and split out unit test project. Updated premake scripts and Visual Studio files; automake files still need to be updated. Make building demos optional. SF patch 1650563 lib64 fixes, SF patch 1646598 Warn against using double precision with gimpact Many more bug fixes and optimizations. Changes from 0.7 to 0.8 New Rotoide - Prismatic Joint Optimisation when dGYROSCOPIC is disabled (default) dJointGetUniversalAngles to efficiently get both angles at once. Changes from 0.6 to 0.7 New heightfield primitive for terrains. Trimesh-Plane collision. Ray-Cylinder collision. Integration of the plane-2d joint. Improved support for convex hulls. Fix for joint limit handling. Half-space optimisation for axis aligned planes. Changes from 0.5 to 0.6 Added flat-capped cylinders. CCylinder has been renamed as Capsule. Added support for geom offsets. Improved stepping function (dWorldQuickStep) Much improved support for trimesh collision shapes Added support for 64-bit systems Started support for convex hulls New and improved build system Lots of bug fixes, stability, and performance improvements Geom Offset Overview Geom offsets allow geom objects to be offset from a body's position. This is useful for cases where an object has an offset centre of mass, or is made up out of several geoms each with their own local position. Native support for geom offsets means that the geomTransform container is now obselete. The real advantage over geomTransform is that there is a much easier interface for setting, getting, and modifying a geom's offset from its body. The elimination of geomTransform simplifies callbacks which no longer have to special case that geom class type. In terms of performance, the new code has a negligible performance difference and a memory improvement compared to using a geomTransform. Geom Offset and Bodies An offset only makes sense for geoms which are connected to bodies. It is an error to attempt to set an offset for a geom which is not connected to a body. When a geom is disconnected from a body, any offset will be automatically eliminated. Thus, the possible states a geom can be in are: Geom Geom on body Geom on body, with an offset To create an offset, just call on of the geomSetOffset functions. The offset will be automatically created if it does not yet exist. Geom offsets are normally set by specifying the offset in local coordinates. An extra set of functions are provided to allow an offset to be determined by providing world coordinates and subtracting them from the current body position. These second set of functions, geomSetOffsetWorldPosition(),etc, allow the user to essentially say "keep the body where it is, and move its geom to this position in the world". Geom Offset API This is defined in the doxygen documentation. Geom Offset affect on existing API The following functions have been updated such that they transparently work with offsets. void dGeomSetPosition (dGeomID, dReal x, dReal y, dReal z) void dGeomSetRotation (dGeomID, const dMatrix3 R) void dGeomSetQuaternion (dGeomID, const dQuaternion) const dReal * dGeomGetPosition (dGeomID) const dReal * dGeomGetRotation (dGeomID) void dGeomGetQuaternion (dGeomID, dQuaternion result) void dGeomGetAABB (dGeomID, dReal aabb[6]) The functions that get a geom's position will return its final position in world coordinates. If a geom is offset, it is no longer true that dGeomGetPosition()==dBodyGetPosition(). Instead, each will return its own world position, with the geom being offset from the body. The functions that set a geom's position will move the body to the position such that the geom will have the position passed in. So if the geom has a positional offset of (0,0,1), and we call dGeomSetPosition(g,10,10,10), the end result will be the body's position being set to (10,10,9). As such, either dGeomSetPosition or dBodySetPosition can be used, depending on which is more convenient. The other position is automatically updated. Geom Offset and geomTransform GeomTransforms are still supported, and all existing code transparently works. When writing new code, however, it is far easier to work with geoms and offsets. The change eliminates a large amount of special casing needed to do when working with geomTransforms. Setting an offset is trivial, and does not require inserting or removing new GeomTransforms from spaces. There is no need to special case getting an offset geom's world position, or setting an offset geom's world position. It is especially useful in the collide callback. The old way required getting the object as the real geom (but without being able to easily grab its Transform, or even its body!), or as its Transform (forcing special casing code to extract the real geom in that case).

@savraj

NEWS: Version 2.5.3 ------------- - Updated zoneinfo to 2016d - Fixed parser bug where unambiguous datetimes fail to parse when dayfirst is set to true. (gh issue #233, pr #234) - Bug in zoneinfo file on platforms such as Google App Engine which do not do not allow importing of subprocess.check_call was reported and fixed by @savraj (gh issue #239, gh pr #240) - Fixed incorrect version in documentation (gh issue #235, pr #243) Version 2.5.2 ------------- - Updated zoneinfo to 2016c - Fixed parser bug where yearfirst and dayfirst parameters were not being respected when no separator was present. (gh issue #81 and #217, pr #229) Version 2.5.1 ------------- - Updated zoneinfo to 2016b - Changed MANIFEST.in to explicitly include test suite in source distributions, with help from @koobs (gh issue #193, pr #194, #201, #221) - Explicitly set all line-endings to LF, except for the NEWS file, on a per-repository basis (gh pr #218) - Fixed an issue with improper caching behavior in rruleset objects (gh issue #104, pr #207) - Changed to an explicit error when rrulestr strings contain a missing BYDAY (gh issue #162, pr #211) - tzfile now correctly handles files containing leapcnt (although the leapcnt information is not actually used). Contributed by @hjoukl (gh issue #146, pr #147) - Fixed recursive import issue with tz module (gh pr #204) - Added compatibility between tzwin objects and datetime.time objects (gh issue #216, gh pr #219) - Refactored monolithic test suite by module (gh issue #61, pr #200 and #206) - Improved test coverage in the relativedelta module (gh pr #215) - Adjusted documentation to reflect possibly counter-intuitive properties of RFC-5545-compliant rrules, and other documentation improvements in the rrule module (gh issue #105, gh issue #149 - pointer to the solution by @phep, pr #213). Version 2.5.0 ------------- - Updated zoneinfo to 2016a - zoneinfo_metadata file version increased to 2.0 - the updated updatezinfo.py script will work with older zoneinfo_metadata.json files, but new metadata files will not work with older updatezinfo.py versions. Additionally, we have started hosting our own mirror of the Olson databases on a github pages site (https://dateutil.github.io/tzdata/) (gh pr #183) - dateutil zoneinfo tarballs now contain the full zoneinfo_metadata file used to generate them. (gh issue #27, gh pr #85) - relativedelta can now be safely subclassed without derived objects reverting to base relativedelta objects as a result of arithmetic operations. (lp:1010199, gh issue #44, pr #49) - relativedelta 'weeks' parameter can now be set and retrieved as a property of relativedelta instances. (lp: 727525, gh issue #45, pr #49) - relativedelta now explicitly supports fractional relative weeks, days, hours, minutes and seconds. Fractional values in absolute parameters (year, day, etc) are now deprecated. (gh issue #40, pr #190) - relativedelta objects previously did not use microseconds to determine of two relativedelta objects were equal. This oversight has been corrected. Contributed by @elprans (gh pr #113) - rrule now has an xafter() method for retrieving multiple recurrences after a specified date. (gh pr #38) - str(rrule) now returns an RFC2445-compliant rrule string, contributed by @schinckel and @armicron (lp:1406305, gh issue #47, prs #50, #62 and #160) - rrule performance under certain conditions has been significantly improved thanks to a patch contributed by @dekoza, based on an article by Brian Beck (@exogen) (gh pr #136) - The use of both the 'until' and 'count' parameters is now deprecated as inconsistent with RFC2445 (gh pr #62, #185) - Parsing an empty string will now raise a ValueError, rather than returning the datetime passed to the 'default' parameter. (gh issue #78, pr #187) - tzwinlocal objects now have a meaningful repr() and str() implementation (gh issue #148, prs #184 and #186) - Added equality logic for tzwin and tzwinlocal objects. (gh issue #151, pr #180, #184) - Added some flexibility in subclassing timelex, and switched the default behavior over to using string methods rather than comparing against a fixed list. (gh pr #122, #139) - An issue causing tzstr() to crash on Python 2.x was fixed. (lp: 1331576, gh issue #51, pr #55) - An issue with string encoding causing exceptions under certain circumstances when tzname() is called was fixed. (gh issue #60, #74, pr #75) - Parser issue where calling parse() on dates with no day specified when the day of the month in the default datetime (which is "today" if unspecified) is greater than the number of days in the parsed month was fixed (this issue tended to crop up between the 29th and 31st of the month, for obvious reasons) (canonical gh issue #25, pr #30, #191) - Fixed parser issue causing fuzzy_with_tokens to raise an unexpected exception in certain circumstances. Contributed by @MichaelAquilina (gh pr #91) - Fixed parser issue where years > 100 AD were incorrectly parsed. Contributed by @Bachmann1234 (gh pr #130) - Fixed parser issue where commas were not a valid separator between seconds and microseconds, preventing parsing of ISO 8601 dates. Contributed by @RyansS (gh issue #28, pr #106) - Fixed issue with tzwin encoding in locales with non-Latin alphabets (gh issue #92, pr #98) - Fixed an issue where tzwin was not being properly imported on Windows. Contributed by @labrys. (gh pr #134) - Fixed a problem causing issues importing zoneinfo in certain circumstances. Issue and solution contributed by @alexxv (gh issue #97, pr #99) - Fixed an issue where dateutil timezones were not compatible with basic time objects. One of many, many timezone related issues contributed and tested by @labrys. (gh issue #132, pr #181) - Fixed issue where tzwinlocal had an invalid utcoffset. (gh issue #135, pr #141, #142) - Fixed issue with tzwin and tzwinlocal where DST transitions were incorrectly parsed from the registry. (gh issue #143, pr #178) - updatezinfo.py no longer suppresses certain OSErrors. Contributed by @bjamesv (gh pr #164) - An issue that arose when timezone locale changes during runtime has been fixed by @carlosxl and @mjschultz (gh issue #100, prs #107, #109) - Python 3.5 was added to the supported platforms in the metadata (@tacaswell gh pr #159) and the test suites (@moreati gh pr #117). - An issue with tox failing without unittest2 installed in Python 2.6 was fixed by @moreati (gh pr #115) - Several deprecated functions were replaced in the tests by @moreati (gh pr #116) - Improved the logic in Travis and Appveyor to alleviate issues where builds were failing due to connection issues when downloading the IANA timezone files. In addition to adding our own mirror for the files (gh pr #183), the download is now retried a number of times (with a delay) (gh pr #177) - Many failing doctests were fixed by @moreati. (gh pr #120) - Many fixes to the documentation (gh pr #103, gh pr #87 from @radarhere, gh pr #154 from @gpoesia, gh pr #156 from @awsum, gh pr #168 from @ja8zyjits) - Added a code coverage tool to the CI to help improve the library. (gh pr #182) - We now have a mailing list - [email protected], graciously hosted by Python.org. Version 2.4.2 ------------- - Updated zoneinfo to 2015b. - Fixed issue with parsing of tzstr on Python 2.7.x; tzstr will now be decoded if not a unicode type. gh #51 (lp:1331576), gh pr #55. - Fix a parser issue where AM and PM tokens were showing up in fuzzy date stamps, triggering inappropriate errors. gh #56 (lp: 1428895), gh pr #63. - Missing function "setcachesize" removed from zoneinfo __all__ list by @RyansS, fixing an issue with wildcard imports of dateutil.zoneinfo. (gh pr #66). - (PyPi only) Fix an issue with source distributions not including the test suite. Version 2.4.1 ------------- - Added explicit check for valid hours if AM/PM is specified in parser. (gh pr #22, issue #21) - Fix bug in rrule introduced in 2.4.0 where byweekday parameter was not handled properly. (gh pr #35, issue #34) - Fix error where parser allowed some invalid dates, overwriting existing hours with the last 2-digit number in the string. (gh pr #32, issue #31) - Fix and add test for Python 2.x compatibility with boolean checking of relativedelta objects. Implemented by @nimasmi (gh pr #43) and Cédric Krier (lp: 1035038) - Replaced parse() calls with explicit datetime objects in unit tests unrelated to parser. (gh pr #36) - Changed private _byxxx from sets to sorted tuples and fixed one currently unreachable bug in _construct_byset. (gh pr #54) - Additional documentation for parser (gh pr #29, #33, #41) and rrule. - Formatting fixes to documentation of rrule and README.rst. - Updated zoneinfo to 2015a.

API: * Constructing a Query for a non-reference counted PostingSource object will now try to clone the PostingSource object (as happened in 1.3.4 and earlier). This clone code was removed as part of the changes in 1.3.5 to support optional reference counting of PostingSource objects, but that breaks the case when the PostingSource object is on the stack and goes out of scope before the Query object is used. Issue reported by Till Schäfer and analysed by Daniel Vrátil in a bug report against Akonadi: https://bugs.kde.org/show_bug.cgi?id=363741 * Add BM25PlusWeight class implementing the BM25+ weighting scheme, implemented by Vivek Pal (xapian/xapian#104). * Add PL2PlusWeight class implementing the PL2+ weighting scheme, implemented by Vivek Pal (xapian/xapian#108). * LMWeight: Implement Dir+ weighting scheme as DIRICHLET_PLUS_SMOOTHING. Patch from Vivek Pal. * Add CoordWeight class implementing coordinate matching. This can be useful for specialised uses - e.g. to implement sorting by the number of matching filters. * DLHWeight,DPHWeight,PL2Weight: With these weighting schemes, the formulae can give a negative weight contribution for a term in extreme cases. We used to try to handle this by calculating a per-term lower bound on the contribution and subtracting this from the contribution, but this idea is fundamentally flawed as the total offset it adds to a document depends on what combination of terms that document matches, meaning in general the offset isn't the same for every matching document. So instead we now clamp each term's weight contribution to be >= 0. * TfIdfWeight: Always scale term weight by wqf - this seems the logical approach as it matches the weighting we'd get if we weighted every non-unique term in the query, as well as being explicit in the Piv+ formula. * Fix OP_SCALE_WEIGHT to work with all weighting schemes - previously it was ignored when using PL2Weight and LMWeight. * PL2Weight: Greatly improve upper bound on weight: + Split the weight equation into two parts and maximise each separately as that gives an easily solvable problem, and in common cases the maximum is at the same value of wdfn for both parts. In a simple test, the upper bounds are now just over double the highest weight actually achieved - previously they were several hundred times. This approach was suggested by Aarsh Shah in: xapian/xapian#48 + Improve upper bound on normalised wdf (wdfn) - when wdf_upper_bound > doclength_lower_bound, we get a tighter bound by evaluating at wdf=wdf_upper_bound. In a simple test, this reduces the upper bound on wdfn by 36-64%, and the upper bound on the weight by 9-33%. * PL2Weight: Fix calculation of upper_bound when P2>0. P2 is typically negative, but for a very common term it can be positive and then we should use wdfn_lower not wdfn_upper to adjust P_max. * Weight::unserialise(): Check serialised form is empty when unserialising parameter-free schemes BoolWeight, DLHWeight and DPHWeight. * TermGenerator::set_stopper_strategy(): New method to control how the Stopper object is used. Patch from Arnav Jain. * QueryParser: Fix handling of CJK query over multiple prefixes. Previously all the n-gram terms were AND-ed together - now we AND together for each prefix, then OR the results. Fixes #719, reported by Aaron Li. * Add Database::get_revision() method which provides access to the database revision number for chert and glass, intended for use by xapiand. Marked as experimental, so we don't have to go through the usual deprecation cycle if this proves not to be the approach we want to take. Fixes #709, reported by German M. Bravo. * Mark RangeProcessor constructor as `explicit`. * Update to Unicode 9.0.0. * Reimplement ESet and ESetIterator as we did for MSet and MSetIterator in 1.3.5. ESetIterator internally now counts down to the end of the ESet, so the end test is now against 0, rather than against eset.size(). And more of the trivial methods are now inlined, which reduces the number of relocations needed to load the library, and should give faster code which is a very similar size to before. * MSetIterator and ESetIterator are now STL-compatible random_access_iterators (previously they were only bidirectional_iterators). * TfIdfWeight: Support freq and squared IDF normalisations. Patch from Vivek Pal. * New Xapian::Query::OP_INVALID to provide an "invalid" query object. * Reject OP_NEAR/OP_PHRASE with non-leaf subqueries early to avoid a potential segmentation fault if the non-leaf subquery decayed at just the wrong moment. See TritonDataCenter#508. * Reduce positional queries with a MatchAll or PostingSource subquery to MatchNothing (since these subqueries have no positional information, so the query can't match). * Deprecate ValueRangeProcessor and introduce new RangeProcessor class as a replacement. RangeProcessor()::operator()() method returns Xapian::Query, so a range can expand to any query. OP_INVALID is used to signal that a range is not recognised. Fixes #663. * Combining of ranges over the same quantity with OP_OR is now handled by an explicit "grouping" parameter, with a sensible default which works for value range queries. Boolean term prefixes and FieldProcessor now support "grouping" too, so ranges and other filters can now be grouped together. * Formally deprecate WritableDatabase::flush(). The replacement commit() method was added in 1.1.0, so code can be switched to use this and still work with 1.2.x. * Fix handling of a self-initialised PIMPL object (e.g. Xapian::Query q(q);). Previously the uninitialised pointer was copied to itself, resulting in undefined behaviour when the object was used to destroyed. This isn't something you'd see in normal code, but it's a cheap check which can probably be optimised away by the compiler (GCC 6 does). * The Snipper class has been replaced with a new MSet::snippet() method. The implementation has also been redone - the existing implementation was slower than ideal, and didn't directly consider the query so would sometimes selects a snippet which doesn't contain any of the query terms (which users quite reasonably found surprising). The new implementation is faster, will always prefer snippets containing query terms, and also understands exact phrases and wildcards. Fixes TritonDataCenter#211. * Add optional reference counting support for ErrorHandler, ExpandDecider, KeyMaker, PostingSource, Stopper and TermGenerator. Fixes TritonDataCenter#186, reported by Richard Boulton. (ErrorHandler's reference counting isn't actually used anywhere in xapian-core currently, but means we can hook it up in 1.4.x if ticket TritonDataCenter#3 gets addressed). * Deprecate public member variables of PostingSource. The new getters and/or setters added in 1.2.23 and 1.3.5 are preferred. Fixes TritonDataCenter#499, reported by Joost Cassee. * Reimplement MSet and MSetIterator. MSetIterator internally now counts down to the end of the MSet, so the end test is now against 0, rather than against mset.size(). And more of the trivial methods are now inlined, which reduces the number of relocations needed to load the library, and should give faster code which is a very similar size to before. * Only issue prefetch hints for documents if MSet::fetch() is called. It's not useful to send the prefetch hint right before the actual read, which was happening since the implementation of prefetch hints in 1.3.4. Fixes #671, reported by Will Greenberg. * Fix OP_ELITE_SET selection in multi-database case - we were selecting different sets for each subdatabase, but removing the special case check for termfreq_max == 0 solves that. * Remove "experimental" marker from FieldProcessor, since we're happy with the API as-is. Reported by David Bremner on xapian-discuss. * Remove "experimental" marker from Database::check(). We've not had any negative feedback on the current API. * Databse::check() now checks that doccount <= last_docid. * Database::compact() on a WritableDatabase with uncommitted changes could produce a corrupted output. We now throw Xapian::InvalidOperationError in this case, with a message suggesting you either commit() or open the database from disk to compact from. Reported by Will Greenberg on #xapian-discuss * Add Arabic stemmer. Patch from Assem Chelli in xapian/xapian#45 * Improve the Arabic stopword list. Patch from Assem Chelli. * Make functions defined in xapian/iterator.h 'inline'. * Don't force the user to specify the metric in the geospatial API - GreatCircleMetric is probably what most users will want, so a sensible default. * Xapian::DBCHECK_SHOW_BITMAP: This was added in 1.3.0 (so has never been in a stable release) and was superseded by Xapian::DBCHECK_SHOW_FREELIST in 1.3.2, so just remove it. * Make setting an ErrorHandler a no-op - this feature is deprecated and we're not aware of anyone using it. We're hoping to rework ErrorHandler in 1.4.x, which will be simpler without having to support the current behaviour as well as the new. See TritonDataCenter#3. * Update to Unicode 8.0.0. Fixes #680. * Overhaul database compaction API. Add a Xapian::Database::compact() method, with the Database object specifying the source database(s). Xapian::Compactor is now just a functor to use if you want to control progress reporting and/or the merging of user metadata. The existing API has been reimplemented using the new one, but is marked as deprecated. * Add support for a default value when sorting. Fixes TritonDataCenter#452, patch from Richard Boulton. * Make all functor objects non-copyable. Previously some were, some weren't, but it's hard to correctly make use of this ability. Fixes #681. * Fix use after free with WILDCARD_LIMIT_MOST_FREQUENT. If we tried to open a postlist after processing such a wildcard, the postlist hint could be pointing to a PostList object which had been deleted. Fixes #696, reported by coventry. * Add support for optional reference counting of MatchSpy objects. * Improve Document::get_description() - the output is now always valid UTF-8, doesn't contain implementation details like "Document::Internal", and more clearly reports if the document is linked to a database. * Remove XAPIAN_CONST_FUNCTION marker from sortable_serialise_() helper, as it writes to the passed in buffer, so it isn't const or pure. Fixes decvalwtsource2 testcase failure when compiled with clang. * Make PostingSource::set_maxweight() public - it's hard to wrap for the bindings as a protected method. Fixes TritonDataCenter#498, reported by Richard Boulton. * Database: + Add new flag Xapian::DB_RETRY_LOCK which allows opening a database for writing to wait until it can get a write lock. (Fixes TritonDataCenter#275, reported by Richard Boulton). + Fix Database::get_doclength_lower_bound() over multiple databases when some are empty or consist only of zero-length documents. Previously this would report a lower bound of zero, now it reports the same lowest bound as a single database containing all the same documents. + Database::check(): When checking a single table, handle the ".glass" extension on glass database tables, and use the extension to guide the decision of which backend the table is from. * Query: + Add new OP_WILDCARD query operator, which expands wildcards lazily, so now we create the PostList tree for a wildcard directly, rather than creating an intermediate Query tree. OP_WILDCARD offers a choice of ways to limit wildcard expansion (no limit, throw an exception, use the first N by term name, or use the most frequent N). (See tickets TritonDataCenter#48 and #608). * QueryParser: + Add new set_max_expansion() method which provides access to OP_WILDCARD's choice of ways to limit expansion and can set limits for partial terms as well as for wildcards. Partial terms now default to the 100 most frequent matching terms. (Completes #608, reported by boomboo). + Deprecate set_max_wildcard_expansion() in favour of set_max_expansion(). * Add support for optional reference counting of FieldProcessor and ValueRangeProcessor objects. * Update Unicode character database to Unicode 7.0.0. * New Xapian::Snipper class from Mihai Bivol's GSOC 2012 project. (mostly fixes TritonDataCenter#211) * Fix all get_description() methods to always return UTF-8 text. (fixes #620) * Database::check(): + Alter to take its "out" parameter as a pointer to std::ostream instead of a reference, and make passing NULL mean "do not produce output", and make the second and third parameters optional, defaulting to a quiet check. + Escape invalid UTF-8 data in keys and tags reported by xapian-check, using the same code we use to clean up strings returned by get_description() methods. + Correct failure message which talks above the root block when it's actually testing a leaf key. + Rename DBCHECK_SHOW_BITMAP to DBCHECK_SHOW_FREELIST (old name still provided for now, but flagged as deprecated - DBCHECK_SHOW_BITMAP was new in 1.3.0, so will likely be removed before 1.4.0). * Methods and functions which take a string to unserialise now consistently call that parameter "serialised". * Weight: Make number of distinct terms indexing each document and the collection frequency of the term available to subclasses. Patch from Gaurav Arora's Language Modelling branch. * WritableDatabase: Add support for multiple subdatabases, and support opening a stub database containing multiple subdatabases as a WritableDatabase. * WritableDatabase can now be constructed from just a pathname (defaulting to opening the database with DB_CREATE_OR_OPEN). * WritableDatabase: Add flags which can be bitwise OR-ed into the second argument when constructing: + Xapian::DB_NO_SYNC: to disable use of fsync, etc + Xapian::DB_DANGEROUS: to enable in-place updates + Xapian::DB_BACKEND_CHERT: if creating, create a chert database + Xapian::DB_BACKEND_GLASS: if creating, create a glass database + Xapian::DB_NO_TERMLIST: create a database without a termlist (see TritonDataCenter#181) + Xapian::DB_FULL_SYNC flag - if this is set for a database, we use the Mac OS X F_FULL_SYNC instead of fdatasync()/fsync()/etc on the version file when committing. * Database: Add optional flags argument to constructor - the following can be bitwise OR-ed into it: + Xapian::DB_BACKEND_CHERT (only open a chert database) + Xapian::DB_BACKEND_GLASS (only open a glass database) + Xapian::DB_BACKEND_STUB (only open a stub database) * Xapian::Auto::open_stub() and Xapian::Chert::open() are now deprecated in favour of these new flags. * Add LMWeight class, which implements the Unigram Language Modelling weighting scheme. Patch from Gaurav Arora. * Add implementations of a number of DfR weighting schemes (BB2, DLH, DPH, IfB2, IneB2, InL2, PL2). Patches from Aarsh Shah. * Add support for the Bo1 query expansion scheme. Patch from Aarsh Shah. * Add Enquire::set_time_limit() method which sets a timelimit after which check_at_least will be disabled. * Database: Trying to perform operations on a database with no subdatabases now throws InvalidOperationError not DocNotFoundError. * Query: Implement new OP_MAX query operator, which returns the maximum weight of any of its subqueries. (see TritonDataCenter#360) * Query: Add methods to allow introspection on Query objects - currently you can read the leaf type/operator, how many subqueries there are, and get a particular subquery. For a query which is a term, Query::get_terms_begin() allows you to get the term. (see TritonDataCenter#159) * Query: Only simplify OP_SYNONYM with a single subquery if that subquery is a term or MatchAll. * Avoid two vector copies when storing term positions in most common cases. * Reimplement version functions to use a single function in libxapian which returns a pointer to a static const struct containing the version information, with inline wrappers in the API header which call this. This means we only need one relocation instead of 4, reducing library load time a little. * Make TermGenerator flags an anonymous enum, and typedef TermGenerator::flags to int for backward compatibility with existing user code which uses it. * Stem: Fix incorrect Unicode codepoints for o-double-acute and u-double-acute in the Hungarian Snowball stemmer. Reported by Tom Lane to snowball-discuss. * Stem: Add an early english stemmer. * Provide the stopword lists from Snowball plus an Arabic one, installed in ${prefix}/share/xapian-core/stopwords/. Patch from Assem Chelli, fixes TritonDataCenter#269. * Improve check for direct inclusion of Xapian subheaders in user code to catch more cases. * Add simple API to help with creating language-idiomatic iterator wrappers in <xapian/iterator.h>. * Give an compilation error if user code tries to include API headers other than xapian.h directly - these other headers are an internal implementation detail, but experience has shown that some people try to include them directly. Please just use '#include <xapian.h>' instead. * Update Unicode character database to Unicode 6.2.0. * Add FieldProcessor class (ticket#128) - currently marked as an experimental API while we sort out how best to sort out exactly how it interacts with other QueryParser features. * Add implementation of several TF-IDF weighting schemes via a new TfIdfWeight class. * Add ExpandDeciderFilterPrefix class which only return terms with a particular prefix. (fixes TritonDataCenter#467) * QueryParser: Adjust handling of Unicode opening/closing double quotes - if a quoted boolean term was started with ASCII double quote, then only ASCII double quote can end it, as otherwise it's impossible to quote a term containing Unicode double quotes. * Database::check(): If the database can't be opened, don't emit a bogus warning about there being too many documents to cross-check doclens. * TradWeight,BM25Weight: Throw SerialisationError instead of NetworkError if unserialise() fails. * QueryParser: Change the default stemming strategy to STEM_SOME, to eliminate the API gotcha that setting a stemmer is ignored until you also set a strategy. * Deprecate Xapian::ErrorHandler. (ticket#3) * Stem: Generate a compact and efficient table to decode language names. This is both faster and smaller than the approach we were using, with the added benefit that the table is auto-generated. * xapian.h: + Add check for Qt headers being included before us and defining 'slots' as a macro - if they are, give a clear error advising how to work around this (previously compilation would fail with a confusing error). + Add a similar check for Wt headers which also define 'slots' as a macro by default. * Update Unicode character database to Unicode 6.1.0. (ticket#497) * TermIterator returned by Enquire::get_matching_terms_begin(), Query::get_terms_begin(), Database::synonyms_begin(), QueryParser::stoplist_begin(), and QueryParser::unstem_begin() now stores the list of terms to iterate much more compactly. * QueryParser: + Allow Unicode curly double quote characters to start and/or end phrases. + The set_default_op() method will now reject operators which don't make sense to set. The operators which are allowed are now explicitly documented in the API docs. * Query: The internals have been completely reimplemented (ticket#280). The notable changes are: + Query objects are smaller and should be faster. + More readable format for Query::get_description(). + More compact serialisation format for Query objects. + Query operators are no longer flattened as you build up a tree (but the query optimiser still combines groups of the same operator). This means that Query objects are truly immutable, and so we don't need to copy Query objects when composing them. This should also fix a few O(n*n) cases when building up an n-way query pair-wise. (ticket#273) + The Query optimiser can do a few extra optimisations. * There's now explicit support for geospatial search (this API is currently marked as experimental). (ticket#481) * There's now an API (currently experimental) for checking the integrity of databases (partly addresses ticket#238). * Database::reopen() now returns true if the database may have been reopened (previously it returned void). (ticket#548) * Deprecate Xapian::timeout in favour of POSIX type useconds_t. * Deprecate Xapian::percent and use int instead in the API and our own code. * Deprecate Xapian::weight typedef in favour of just using double and change all uses in the API and our own code. (ticket#560) * Rearrange members of Xapian::Error to reduce its size (from 48 to 40 bytes on x86-64 Linux). * Assignment operators for PositionIterator and TermIterator now return *this rather than void. * PositionIterator, PostingIterator, TermIterator and ValueIterator now handle their reference counts in hand-crafted code rather than using intrusive_ptr/RefCntPtr, which means the compiler can inline the destructor and default constructor, so a comparison to an end iterator should now optimise to a simple NULL pointer check, but without the issues which the ValueIteratorEnd_ proxy class approach had (such as not working in templates or some cases of overload resolution). * Enquire: + Previously, Enquire::get_matching_terms_begin() threw InvalidArgumentError if the query was empty. Now we just return an end iterator, which is more consistent with how empty queries behave elsewhere. + Remove the deprecated old-style match spy approach of using a MatchDecider. * Remove deprecated Sorter class and MultiValueSorter subclass. * Xapian::Stem: + Add stemmers for Armenian (hy), Basque (eu), and Catalan (ca). + Stem::operator= now returns a reference to the assigned-to object. testsuite: * OP_SCALE_WEIGHT: Check top weight is non-zero - if it is zero, tests which try to check that OP_SCALE_WEIGHT works will always pass. * testsuite: Check SerialisationError descriptions from Xapian::Weight subclasses mention the weighting scheme name. * Merge queryparsertest and termgentest into apitest. Their testcases now use the backend manager machinery in the testharness, so we don't have to hard-code use of inmemory and chert backends, but instead run them under all backends which support the required features. This fixes some test failures when both chert and glass are disabled due to trying to run spelling tests with the inmemory backend. * Avoid overflowing collection frequency in totaldoclen1. We're trying to test total document length doesn't wrap, so avoid collection freq overflowing in the process, as that triggers errors when running the testsuite under ubsan. We should handle collection frequency overflow better, but that's a separate issue. * Add some test coverage for ESet::get_ebound(). * Fix testcase notermlist1 to check correct table extension - ".glass" not ".DB" (chert doesn't support DB_NO_TERMLIST). * unittest: We can't use Assert() to unit test noexcept code as it throws an exception if it fails. Instead set up macros to set a variable and return if an assertion fails in a unittest testcase, and check that variable in the harness. * Add unit test for internal C_isupper(), etc functions. * If command line option --verbose/-v isn't specified, set the verbosity level from environmental variable VERBOSE. * Re-enable replicate3 for glass, as it no longer fails. * Add more test coverage for get_unique_terms(). * Don't leave an extra fd open when starting xapian-tcpsrv for remotetcp tests. * Extend checkstatsweight1 to check that Weight::get_collection_freq() returns the same number as Database::get_collection_freq(). * queryparsertest: Add testcase for FieldProcessor on boolean prefix with quoted contents. * queryparsertest: Enable some disabled cases which actually work (in some cases with slightly tweaked expected answers which are equivalent to those that were shown). * Make use of the new writable multidatabase feature to simplify the multi-database handling in the test harness. * Change querypairwise1_helper to repeat the query build 100 times, as with a fast modern machine we were sometimes trying with so many subqueries that we would run out of stack. * apitest: Use Xapian::Database::check() in cursordelbug1. (partly addresses TritonDataCenter#238) * apitest: Test Query ops with a single MatchAll subquery. * apitest: New testcase readonlyparentdir1 to ensure that commit works with a read-only parent directory. * tests/generate-api_generated: Test that the string returned by a get_description() method isn't empty. * Use git commit hash in title of test coverage reports generated from a git tree. * Make unittest use the test harness, so it gets all the valgrind and fd leak checks, and other handy features all the other tests have. * Improve test coverage in several places. * Compress generated HTML files in coverage report. matcher: * Fix stats passed to Weight with OP_SYNONYM. Previously the number of unique terms was never calculated, and a term which matched all documents would be optimised to an all-docs postlist, which fails to supply the correct wdf info. * Use floating point calculation for OR synonym freq estimates. The division was being done as an integer division, which means the result was always getting rounded down rather than rounded to the nearest integer. * Fix upper bound on matches for OP_XOR. Due to a reversed conditional, the estimate could be one too low in some cases where the XOR matched all the documents in the database. * Improve lower bound on matches for OP_XOR. Previously the lower bound was always set to 0, which is valid, but we can often do better. * Optimise value range which is a superset of the bounds. If the value frequency is equal to the doccount, such a range is equivalent to MatchAll, and we now avoid having to read the valuestream at all. * Optimise OP_VALUE_RANGE when the upper bound can't be exceeded. In this case, we now use ValueGePostList instead of ValueRangePostList. * Streamline collation of statistics for use by weighting schemes - tests show a 2% or so increase in speed in some cases. * If a term matches all documents and its weight doesn't depend on its wdf, we can optimise it to MatchAll (the previous requirement that maxpart == 0 was unnecessarily strict). * Fix the check for a term which matches all documents to use the sub-db termfreq, not the combined db termfreq. * When we optimise a postlist for a term which matches all documents to use MatchAll, we still need to set a weight object on it to get percentages calculated correctly. * Drop MatchNothing subqueries in OR-like situations in add_subquery() rather than adding them and then handling it later. * Handle the left side of AND_NOT and AND_MAYBE being MatchNothing in add_subquery() rather than in done(). * Handle QueryAndLike with a MatchNothing subquery in add_subquery() rather than done(). * Query: Multi-way operators now store their subquery pointers in a custom class rather than std::vector<Xapian::Query>. The custom class take the same amount of space, or often less. It's particularly efficient when there are two subqueries, which is very desirable as we no longer flatten a subtree of the same operator as we build the query. * Optimise an unweighted query term which matches all the documents in a subdatabase to use the "MatchAll" postlist. (ticket#387) glass backend: * Fix allterms with prefix on glass with uncommitted changes. Glass aims to flush just the relevant postlist changes in this case but the end of the range to flush was wrong, so we'd only actually flush changes for a term exactly matching the prefix. Fixes #721. * Fix Database::check() parsing of glass changes file header. In practice this was unlikely to actually cause problems. * Make glass the default backend. The format should now be stable, except perhaps in the unlikely event that a bug emerges which requires a format change to address. * Don't explicitly store the 2 byte "component_of" counter for the first component of every Btree entry in leaf blocks - instead use one of the upper bits of the length to store a "first component" flag. This directly saves 2 bytes per entry in the Btree, plus additional space due to fewer blocks and fewer levels being needed as a result. This particularly helps the position table, which has a lot of entries, many of them very small. The saving would be expected to be a little less than the saving from the change which shaved 2 bytes of every Btree item in 1.3.4 (since that saved 2 bytes multiple times for large entries which get split into multiple items). A simple test suggests a saving of several percent in total DB size, which fits that. This change reduces the maximum component size to 8194, which affects tables with a 64KB blocksize in normal use and tables with >= 16KB blocksize with full compaction. * Refactor glass backend key comparison - == and < operations are replaced by a compare() function returns negative, 0 or positive (like strcmp(), memcmp() and std::string::compare()). This allows us to avoid a final compare to check for equality when binary chopping, and to terminate early if the binary chop hits the exact entry. * If a cursor is moved to an entry which doesn't exist, we need to step back to the first component of previous entry before we can read its tag. However we often don't actually read its tag (e.g. if we only wanted the key), so make this stepping back lazy so we can avoid doing it when we don't want to read the tag. * Avoid creating std::string objects to hold data when compressing and decompressing tags with zlib. * Store minimum compression length per table in the version file, with 0 meaning "don't compress". Currently you can only change this setting with a hex editor on the file, but now it is there we can later make use of it without needing a database format change. * Database::check() now performs additional consistency checks for glass. Reported by Jean-Francois Dockes and Bob Cargill via xapian-discuss. * Database::check(): check docids don't exceed db_last_docid when checking a single glass table. * We now throw DatabaseCorruptError in a few cases where it's appropriate but we didn't previously, in particular in the case where all the files in a DB have been truncated to zero size (which makes handling of this case consistent with chert). * Fix compaction to a single file which already exists. This was hanging. Noted by Will Greenberg on #xapian. * Shave 2 bytes of every Btree item (which will probably typically reduce database size by several percent). * More compact item format for branch blocks - 2 bytes per item smaller. This means each branch block can branch more ways, reducing the number of Btree levels needed, which is especially helpful for cold-cache search times. * Track an upper bound on spelling word frequency. This isn't currently used, but will be useful for improving the spelling algorithm, and we want to stabilise the glass backend format. See TritonDataCenter#225, reported by Philip Neustrom. * Support 64-bit docids in the glass backend on-disk format. This changes the encoding used by pack_uint_preserving_sort() to one which supports 64 bit values, and is a byte smaller for values 16384-32767, and the same size for all other 32 bit values. Fixes #686, from original report by James Aylett. * Use memcpy() not memmove() when no risk of overlap. * Store length of just the key data itself, allowing keys to be up to 255 bytes long - the previous limit was 252. * Change glass to store DB stats in the version file. Previously we stored them in a special item in the postlist table, but putting them in the version file reduces the number of block reads required to open the database, is simpler to deal with, and means we can potentially recalculate tight upper and lower bounds for an existing database without having to commit a new revision. * Add support for a single-file variant for glass. Currently such databases can only be opened for reading - to create one you need to use xapian-compact (or its API equivalent). You can embed such databases within another file, and open them by passing in a file descriptor open on that file and positioned at the offset the database starts at). Database::check() also supports them. Fixes #666, reported by Will Greenberg (and previously suggested on xapian-discuss by Emmanuel Engelhart). * Avoid potential DB corruption with full-compaction when using 64K blocks. * Where posix_fadvise() is available, use it to prefetch postlist Btree blocks from the level below the root block which will be needed for postlists of terms in the query, and similarly for the docdata table when MSet::fetch() is called. Based on patch by Will Greenberg in #671. * When reporting freelist errors during a database check, distinguish between a block in use and in the freelist, and a block in the freelist more than once. * Fix compaction and database checking for the change to the format of keys in the positionlist table which happened in 1.3.2. * After splitting a block, we always insert the new block in the parent right after the block it was split from - there's no need to binary chop. * Avoid infinite recursion when we hit the end of the freelist block we're reading and the end of the block we're writing at the same time. * Fix freelist handling to allow for the newly loaded first block of the freelist being already used up. * 'brass' backend renamed to 'glass' - we decided to use names in ascending alphabetical order to make it easier to understand which backend is newest, and since 'flint' was used recently, we skipped over 'd', 'e' and 'f'. * Change positionlist keys to be ordered by term first rather than docid first, which helps phrase searching significantly. For more efficient indexing, positionlist changes are now batched up in memory and written out in key order. * Use a separate cursor for each position list - now we're ordering the position B-tree by term first, phrase matching would cause a single cursor to cycle between disparate areas of the B-tree and reread the same blocks repeatedly. * Reference count blocks in the btree cursor, so cursors can cheaply share blocks. This can significantly reduce the amount of memory used by cursors for queries which contain a lot of terms (e.g. wildcards which expand to a lot of terms). * Under glass, optimise the turning of a query into a postlist to reuse the cursor blocks which are the same as the previous term's postlist. This is particularly effective for a wildcard query which expands to a lot of terms. * Keep track of unused blocks in the Btrees using freelists rather than bitmaps. (fixes TritonDataCenter#40) * Eliminate the base files, and instead store the root block and freelist pointers in the "iamglass" file. * When compacting, sync all the tables together at the end. * In DB_DANGEROUS mode, update the version file in-place. * Only actually store the document data if it is non-empty. The table which holds the document data is now lazily created, so won't exist if you never set the document data. * Iterating positional data now decodes it lazily, which should speed up phrases which include common words. * Compress changesets in brass replication. Increments the changeset version. Ticket TritonDataCenter#348 * Restore two missing lines in database checking where we report a block with the wrong level. * When checking if a block was newly allocated in this revision, just look at its revision number rather than consulting the base file's bitmap. remote backend: * Improve handling of invalid remote stub entries: Entries without a colon now give an error rather than being quietly skipped; IPv6 isn't yet supported, but entries with IPv6 addresses now result in saner errors (previously the colons confused the code which looks for a port number). * Fix hook for remote support of user weighting schemes. The commented-out code used entirely the wrong class - now we use the server object we have access to, and forward the method to the class which needs it. * Avoid dividing zero by zero when calculating the average length for an empty database. * Bump remote protocol version to 38.0, due to extra statistics being tracked for weighting. * Make Weight::Internal track if any max_part values are set, so we don't need to serialise them when they've not been set. * Prefix compress list of terms and metadata keys in the remote protocol. This requires a remote protocol major version bump. * When propagating exceptions from a remote backend server, the protocol now sends a numeric code to represent which exception is being propagated, rather than the name of the type, as a number can be turned back into an exception with a simple switch statement and is also less data to transfer. (ticket#471) * Remote protocol (these changes require a protocol major version bump): + Unify REPLY_GREETING and REPLY_UPDATE. + Send (last_docid - doccount) instead of last_docid and (doclen_ubound - doclen_lbound) instead of doclen_ubound. * Remove special check which gives a more helpful error message when a modern client is used against a remote server running Xapian <= 0.9.6. chert backend: * When using 64-bit Xapian::docid, consistently use the actual maximum valid docid value rather instead of the maximum value the type can hold. * Where posix_fadvise() is available, use it to prefetch postlist Btree blocks from the level below the root block which will be needed for postlists of terms in the query, and similarly for the record table when MSet::fetch() is called. Based on patch by Will Greenberg in #671. * Fix problems with get_unique_terms() on a modified chert database. * Fix xapian-check on a single chert table, which seg faulted in 1.3.2. * Improve DBCHECK_FIX: + if fixing a whole database, we now take the revision from the first table we successfully look at, which should be correct in most cases, and is definitely better than trying to determine the revision of each broken table independently. + handle a zero-sized .DB file. + After we successfully regenerate baseA, remove any empty baseB file to prevent it causing problems. Tracked down with help from Phil Hands. * Iterating positional data now decodes it lazily, which should speed up phrases which include common words. flint backend: * Remove flint backend.

- Fix bug #181 ERR_RESPONSE_HEADERS with bypass - Fix bug #183 whitelist and identification - Fix bug #182 e2guardian systemd service does not support reload - Fix bug #178 (NTLM Messages)

bahamat closed this as completed Mar 3, 2014

jperkin pushed a commit that referenced this issue Mar 20, 2017

Update to 3.5.1. Changelog:

a53a2f6

- Fix bug #181 ERR_RESPONSE_HEADERS with bypass - Fix bug #183 whitelist and identification - Fix bug #182 e2guardian systemd service does not support reload - Fix bug #178 (NTLM Messages)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mod_wsgi error with base64 image and pkgsrc 2013Q3 #181

mod_wsgi error with base64 image and pkgsrc 2013Q3 #181

bahamat commented Mar 3, 2014

bahamat commented Mar 3, 2014

mod_wsgi error with base64 image and pkgsrc 2013Q3 #181

mod_wsgi error with base64 image and pkgsrc 2013Q3 #181

Comments

bahamat commented Mar 3, 2014

bahamat commented Mar 3, 2014