Eliminate Leaks by wrapping resources in contextlib with-blocks

.appveyor.yml

@@ -37,7 +37,7 @@ install:
         git config --global user.email "[email protected]"
         git config --global user.name "Travis Runner"
 
-    - pip install -e .
+    - pip install -r requirements.txt
 
 build: false
 

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "smmap"]
-	path = gitdb/ext/smmap
-	url = https://github.com/Byron/smmap.git

.travis.yml

@@ -12,6 +12,7 @@ git:
   depth: 1000
 install:
   - pip install coveralls
+  - pip install -r requirements.txt
 script: 
   - ulimit -n 48
   - ulimit -n

doc/source/changes.rst

@@ -2,42 +2,83 @@
 Changelog
 #########
 
-*****
+2.1.0
+* **BREAKING API:** retrofit streams and (internal) packers as context-managers.
+
+  Specifically if you are using directly the packers 
+  (``git.pack.PackIndexFile``, ``git.pack.PackFile`` & ``git.pack.PackEntity``) 
+  they must always be used from within a ``with ..`` block, or else 
+  you will get *mmap-cursor* missing errors.
+
+  .. Tip::
+
+    You can "enter" `PackIndexFile`` & ``PackFile`` multiple time, but ``PackEntity`` only once
+    to detect and avoid sloppy deep-nesting.
+    Since ``git.pack.PackEntity`` class just coalseces ``PackIndexFile`` & ``PackFile``,
+    you may "enter" either each internal packer separately, or the entity only once.
+
+* **BREAKING API:** some utilities moved between ``git.util``, ``git.const`` & ``git.utils.compat``.
+* Fix (probably) all leaks in Windows.  
+
+  .. Note::
+
+    The problem is that on Linux, any open files go unoticed, or collected by GC.
+    But on *Windows* (and specifically on PY3 where GC is not deterministic), 
+    the underlying files cannot delete due to *access violation*.
+
+    That's a Good-thing|copy|, because it is dangerous to *leak*  memory-mapped handles.
+    Actually *Windows* may leak them even after process who created them have died, 
+    needing a full restart(!) to clean them up (signing-out is not enough).
+
+
+* Stop importing *on runtime* *smmap* submodule - deleted completely submodule from sources.
+
+  .. Tip::
+
+      Developer now has to specify specific dependency to *smmap* in ``requirements.txt`` file, and
+      remember to updated it before a final release.
+
+* Run TCs also on Appveyor.  
+
+
 0.6.1
-*****
+=====
 
 * Fixed possibly critical error, see https://github.com/gitpython-developers/GitPython/issues/220
 
     - However, it only seems to occur on high-entropy data and didn't reoccour after the fix
 
-*****
+
 0.6.0
-*****
+=====
 
 * Added support got python 3.X
 * Removed all `async` dependencies and all `*_async` versions of methods with it.
 
-*****
+
 0.5.4
-*****
+=====
 * Adjusted implementation to use the SlidingMemoryManager by default in python 2.6 for efficiency reasons. In Python 2.4, the StaticMemoryManager will be used instead.
 
-*****
+
 0.5.3
-*****
+=====
 * Added support for smmap. SmartMMap allows resources to be managed and controlled. This brings the implementation closer to the way git handles memory maps, such that unused cached memory maps will automatically be freed once a resource limit is hit. The memory limit on 32 bit systems remains though as a sliding mmap implementation is not used for performance reasons. 
 
-*****
+
 0.5.2
-*****
+=====
 * Improved performance of the c implementation, which now uses reverse-delta-aggregation to make a memory bound operation CPU bound.
 
-*****
+
 0.5.1
-*****
+=====
 * Restored most basic python 2.4 compatibility, such that gitdb can be imported within python 2.4, pack access cannot work though. This at least allows Super-Projects to provide their own workarounds, or use everything but pack support.
 
-*****
+
 0.5.0
-*****
+=====
 Initial Release
+
+
+.. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN

doc/source/tutorial.rst

@@ -35,29 +35,28 @@ Databases support query and/or addition of objects using simple interfaces. They
 Both have two sets of methods, one of which allows interacting with single objects, the other one allowing to handle a stream of objects simultaneously and asynchronously.
 
 Acquiring information about an object from a database is easy if you have a SHA1 to refer to the object::
-    
-    
+
+
     ldb = LooseObjectDB(fixture_path("../../../.git/objects"))
-    
+
     for sha1 in ldb.sha_iter():
         oinfo = ldb.info(sha1)
-        ostream = ldb.stream(sha1)
-        assert oinfo[:3] == ostream[:3]
-
-        assert len(ostream.read()) == ostream.size
-    # END for each sha in database
-
+        with =ldb.stream(sha1) as ostream:
+            assert oinfo[:3] == ostream[:3]
+
+            assert len(ostream.read()) == ostream.size
+
 To store information, you prepare an *IStream* object with the required information. The provided stream will be read and converted into an object, and the respective 20 byte SHA1 identifier is stored in the IStream object::
-    
+
     data = "my data"
-    istream = IStream("blob", len(data), StringIO(data))
-    
-    # the object does not yet have a sha
-    assert istream.binsha is None
-    ldb.store(istream)
-    # now the sha is set
-    assert len(istream.binsha) == 20
-    assert ldb.has_object(istream.binsha)
+    with IStream("blob", len(data), StringIO(data)) as istream:
+
+        # the object does not yet have a sha
+        assert istream.binsha is None
+        ldb.store(istream)
+        # now the sha is set
+        assert len(istream.binsha) == 20
+        assert ldb.has_object(istream.binsha)
 
 **********************
 Asynchronous Operation
@@ -67,33 +66,33 @@ For each read or write method that allows a single-object to be handled, an *_as
 Using asynchronous operations is easy, but chaining multiple operations together to form a complex one would require you to read the docs of the *async* package. At the current time, due to the *GIL*, the *GitDB* can only achieve true concurrency during zlib compression and decompression if big objects, if the respective c modules where compiled in *async*.
 
 Asynchronous operations are scheduled by a *ThreadPool* which resides in the *gitdb.util* module::
-    
+
     from gitdb.util import pool
-    
+
     # set the pool to use two threads
     pool.set_size(2)
-    
+
     # synchronize the mode of operation
     pool.set_size(0)
-    
-    
+
+
 Use async methods with readers, which supply items to be processed. The result is given through readers as well::
-    
+
     from async import IteratorReader
-    
+
     # Create a reader from an iterator
     reader = IteratorReader(ldb.sha_iter())
-    
+
     # get reader for object streams
     info_reader = ldb.stream_async(reader)
-    
+
     # read one
     info = info_reader.read(1)[0]
-    
+
     # read all the rest until depletion
     ostreams = info_reader.read()
-    
-    
+
+
 
 *********
 Databases

gitdb/__init__.py

@@ -7,33 +7,14 @@
 import sys
 import os
 
-#{ Initialization
-
-
-def _init_externals():
-    """Initialize external projects by putting them into the path"""
-    for module in ('smmap',):
-        sys.path.append(os.path.join(os.path.dirname(__file__), 'ext', module))
-
-        try:
-            __import__(module)
-        except ImportError:
-            raise ImportError("'%s' could not be imported, assure it is located in your PYTHONPATH" % module)
-        # END verify import
-    # END handel imports
-
-#} END initialization
-
-_init_externals()
-
 __author__ = "Sebastian Thiel"
 __contact__ = "[email protected]"
 __homepage__ = "https://github.com/gitpython-developers/gitdb"
-version_info = (2, 0, 0)
+version_info = (2, 1, 0, 'dev3')
 __version__ = '.'.join(str(i) for i in version_info)
 
 
 # default imports
-from gitdb.base import *
-from gitdb.db import *
-from gitdb.stream import *
+from gitdb.base import *    # @IgnorePep8
+from gitdb.db import *      # @IgnorePep8
+from gitdb.stream import *  # @IgnorePep8