Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update coverage to 5.0 #230

Merged
merged 1 commit into from
Dec 14, 2019
Merged

Conversation

pyup-bot
Copy link
Contributor

This PR updates coverage from 4.5.4 to 5.0.

Changelog

5.0

--------------------------

Nothing new beyond 5.0b2.


.. _changes_50b2:

5.0b2

----------------------------

- An experimental ``[run] relative_files`` setting tells coverage to store
relative file names in the data file. This makes it easier to run tests in
one (or many) environments, and then report in another.  It has not had much
real-world testing, so it may change in incompatible ways in the future.

- When constructing a :class:`coverage.Coverage` object, `data_file` can be
specified as None to prevent writing any data file at all.  In previous
versions, an explicit `data_file=None` argument would use the default of
".coverage". Fixes `issue 871`_.

- Python files run with ``-m`` now have ``__spec__`` defined properly.  This
fixes `issue 745`_ (about not being able to run unittest tests that spawn
subprocesses), and `issue 838`_, which described the problem directly.

- The ``[paths]`` configuration section is now ordered. If you specify more
than one list of patterns, the first one that matches will be used.  Fixes
`issue 649`_.

- The :func:`.coverage.numbits.register_sqlite_functions` function now also
registers `numbits_to_nums` for use in SQLite queries.  Thanks, Simon
Willison.

- Python 3.9a1 is supported.

- Coverage.py has a mascot: :ref:`Sleepy Snake <sleepy>`.

.. _issue 649: https://github.com/nedbat/coveragepy/issues/649
.. _issue 745: https://github.com/nedbat/coveragepy/issues/745
.. _issue 838: https://github.com/nedbat/coveragepy/issues/838
.. _issue 871: https://github.com/nedbat/coveragepy/issues/871


.. _changes_50b1:

5.0b1

----------------------------

- The HTML and textual reports now have a ``--skip-empty`` option that skips
files with no statements, notably ``__init__.py`` files.  Thanks, Reya B.

- Configuration can now be read from `TOML`_ files.  This requires installing
coverage.py with the ``[toml]`` extra.  The standard "pyproject.toml" file
will be read automatically if no other configuration file is found, with
settings in the ``[tool.coverage.]`` namespace.  Thanks to Frazer McLean for
implementation and persistence.  Finishes `issue 664`_.

- The ``[run] note`` setting has been deprecated. Using it will result in a
warning, and the note will not be written to the data file.  The
corresponding :class:`.CoverageData` methods have been removed.

- The HTML report has been reimplemented (no more table around the source
code). This allowed for a better presentation of the context information,
hopefully resolving `issue 855`_.

- Added sqlite3 module version information to ``coverage debug sys`` output.

- Asking the HTML report to show contexts (``[html] show_contexts=True`` or
``coverage html --show-contexts``) will issue a warning if there were no
contexts measured (`issue 851`_).

.. _TOML: https://github.com/toml-lang/tomlreadme
.. _issue 664: https://github.com/nedbat/coveragepy/issues/664
.. _issue 851: https://github.com/nedbat/coveragepy/issues/851
.. _issue 855: https://github.com/nedbat/coveragepy/issues/855


.. _changes_50a8:

5.0a8

----------------------------

- The :class:`.CoverageData` API has changed how queries are limited to
specific contexts.  Now you use :meth:`.CoverageData.set_query_context` to
set a single exact-match string, or :meth:`.CoverageData.set_query_contexts`
to set a list of regular expressions to match contexts.  This changes the
command-line ``--contexts`` option to use regular expressions instead of
filename-style wildcards.


.. _changes_50a7:

5.0a7

----------------------------

- Data can now be "reported" in JSON format, for programmatic use, as requested
in `issue 720`_.  The new ``coverage json`` command writes raw and summarized
data to a JSON file.  Thanks, Matt Bachmann.

- Dynamic contexts are now supported in the Python tracer, which is important
for PyPy users.  Closes `issue 846`_.

- The compact line number representation introduced in 5.0a6 is called a
"numbits."  The :mod:`coverage.numbits` module provides functions for working
with them.

- The reporting methods used to permanently apply their arguments to the
configuration of the Coverage object.  Now they no longer do.  The arguments
affect the operation of the method, but do not persist.

- A class named "test_something" no longer confuses the ``test_function``
dynamic context setting.  Fixes `issue 829`_.

- Fixed an unusual tokenizing issue with backslashes in comments.  Fixes
`issue 822`_.

- ``debug=plugin`` didn't properly support configuration or dynamic context
plugins, but now it does, closing `issue 834`_.

.. _issue 720: https://github.com/nedbat/coveragepy/issues/720
.. _issue 822: https://github.com/nedbat/coveragepy/issues/822
.. _issue 834: https://github.com/nedbat/coveragepy/issues/834
.. _issue 829: https://github.com/nedbat/coveragepy/issues/829
.. _issue 846: https://github.com/nedbat/coveragepy/issues/846


.. _changes_50a6:

5.0a6

----------------------------

- Reporting on contexts. Big thanks to Stephan Richter and Albertas Agejevas
for the contribution.

- The ``--contexts`` option is available on the ``report`` and ``html``
 commands.  It's a comma-separated list of shell-style wildcards, selecting
 the contexts to report on.  Only contexts matching one of the wildcards
 will be included in the report.

- The ``--show-contexts`` option for the ``html`` command adds context
 information to each covered line.  Hovering over the "ctx" marker at the
 end of the line reveals a list of the contexts that covered the line.

- Database changes:

- Line numbers are now stored in a much more compact way.  For each file and
 context, a single binary string is stored with a bit per line number.  This
 greatly improves memory use, but makes ad-hoc use difficult.

- Dynamic contexts with no data are no longer written to the database.

- SQLite data storage is now faster.  There's no longer a reason to keep the
 JSON data file code, so it has been removed.

- Changes to the :class:`.CoverageData` interface:

- The new :meth:`.CoverageData.dumps` method serializes the data to a string,
 and a corresponding :meth:`.CoverageData.loads` method reconstitutes this
 data.  The format of the data string is subject to change at any time, and
 so should only be used between two installations of the same version of
 coverage.py.

- The :meth:`CoverageData constructor<.CoverageData.__init__>` has a new
 argument, `no_disk` (default: False).  Setting it to True prevents writing
 any data to the disk.  This is useful for transient data objects.

- Added the classmethod :meth:`.Coverage.current` to get the latest started
Coverage instance.

- Multiprocessing support in Python 3.8 was broken, but is now fixed.  Closes
`issue 828`_.

- Error handling during reporting has changed slightly.  All reporting methods
now behave the same.  The ``--ignore-errors`` option keeps errors from
stopping the reporting, but files that couldn't parse as Python will always
be reported as warnings.  As with other warnings, you can suppress them with
the ``[run] disable_warnings`` configuration setting.

- Coverage.py no longer fails if the user program deletes its current
directory. Fixes `issue 806`_.  Thanks, Dan Hemberger.

- The scrollbar markers in the HTML report now accurately show the highlighted
lines, regardless of what categories of line are highlighted.

- The hack to accommodate ShiningPanda_ looking for an obsolete internal data
file has been removed, since ShiningPanda 0.22 fixed it four years ago.

- The deprecated `Reporter.file_reporters` property has been removed.

.. _ShiningPanda: https://wiki.jenkins.io/display/JENKINS/ShiningPanda+Plugin
.. _issue 806: https://github.com/nedbat/coveragepy/pull/806
.. _issue 828: https://github.com/nedbat/coveragepy/issues/828


.. _changes_50a5:

5.0a5

----------------------------

- Drop support for Python 3.4

- Dynamic contexts can now be set two new ways, both thanks to Justas
Sadzevičius.

- A plugin can implement a ``dynamic_context`` method to check frames for
 whether a new context should be started.  See
 :ref:`dynamic_context_plugins` for more details.

- Another tool (such as a test runner) can use the new
 :meth:`.Coverage.switch_context` method to explicitly change the context.

- The ``dynamic_context = test_function`` setting now works with Python 2
old-style classes, though it only reports the method name, not the class it
was defined on.  Closes `issue 797`_.

- ``fail_under`` values more than 100 are reported as errors.  Thanks to Mike
Fiedler for closing `issue 746`_.

- The "missing" values in the text output are now sorted by line number, so
that missing branches are reported near the other lines they affect. The
values used to show all missing lines, and then all missing branches.

- Access to the SQLite database used for data storage is now thread-safe.
Thanks, Stephan Richter. This closes `issue 702`_.

- Combining data stored in SQLite is now about twice as fast, fixing `issue
761`_.  Thanks, Stephan Richter.

- The ``filename`` attribute on :class:`.CoverageData` objects has been made
private.  You can use the ``data_filename`` method to get the actual file
name being used to store data, and the ``base_filename`` method to get the
original filename before parallelizing suffixes were added.  This is part of
fixing `issue 708`_.

- Line numbers in the HTML report now align properly with source lines, even
when Chrome's minimum font size is set, fixing `issue 748`_.  Thanks Wen Ye.

.. _issue 702: https://github.com/nedbat/coveragepy/issues/702
.. _issue 708: https://github.com/nedbat/coveragepy/issues/708
.. _issue 746: https://github.com/nedbat/coveragepy/issues/746
.. _issue 748: https://github.com/nedbat/coveragepy/issues/748
.. _issue 761: https://github.com/nedbat/coveragepy/issues/761
.. _issue 797: https://github.com/nedbat/coveragepy/issues/797


.. _changes_50a4:

5.0a4

----------------------------

- You can specify the command line to run your program with the ``[run]
command_line`` configuration setting, as requested in `issue 695`_.

- Coverage will create directories as needed for the data file if they don't
exist, closing `issue 721`_.

- The ``coverage run`` command has always adjusted the first entry in sys.path,
to properly emulate how Python runs your program.  Now this adjustment is
skipped if sys.path[0] is already different than Python's default.  This
fixes `issue 715`_.

- Improvements to context support:

- The "no such table: meta" error is fixed.: `issue 716`_.

- Combining data files is now much faster.

- Python 3.8 (as of today!) passes all tests.

.. _issue 695: https://github.com/nedbat/coveragepy/issues/695
.. _issue 715: https://github.com/nedbat/coveragepy/issues/715
.. _issue 716: https://github.com/nedbat/coveragepy/issues/716
.. _issue 721: https://github.com/nedbat/coveragepy/issues/721


.. _changes_50a3:

5.0a3

----------------------------

- Context support: static contexts let you specify a label for a coverage run,
which is recorded in the data, and retained when you combine files.  See
:ref:`contexts` for more information.

- Dynamic contexts: specifying ``[run] dynamic_context = test_function`` in the
config file will record the test function name as a dynamic context during
execution.  This is the core of "Who Tests What" (`issue 170`_).  Things to
note:

- There is no reporting support yet.  Use SQLite to query the .coverage file
 for information.  Ideas are welcome about how reporting could be extended
 to use this data.

- There's a noticeable slow-down before any test is run.

- Data files will now be roughly N times larger, where N is the number of
 tests you have.  Combining data files is therefore also N times slower.

- No other values for ``dynamic_context`` are recognized yet.  Let me know
 what else would be useful.  I'd like to use a pytest plugin to get better
 information directly from pytest, for example.

.. _issue 170: https://github.com/nedbat/coveragepy/issues/170

- Environment variable substitution in configuration files now supports two
syntaxes for controlling the behavior of undefined variables: if ``VARNAME``
is not defined, ``${VARNAME?}`` will raise an error, and ``${VARNAME-default
value}`` will use "default value".

- Partial support for Python 3.8, which has not yet released an alpha. Fixes
`issue 707`_ and `issue 714`_.

.. _issue 707: https://github.com/nedbat/coveragepy/issues/707
.. _issue 714: https://github.com/nedbat/coveragepy/issues/714


.. _changes_50a2:

5.0a2

----------------------------

- Coverage's data storage has changed.  In version 4.x, .coverage files were
basically JSON.  Now, they are SQLite databases.  This means the data file
can be created earlier than it used to.  A large amount of code was
refactored to support this change.

- Because the data file is created differently than previous releases, you
 may need ``parallel=true`` where you didn't before.

- The old data format is still available (for now) by setting the environment
 variable COVERAGE_STORAGE=json. Please tell me if you think you need to
 keep the JSON format.

- The database schema is guaranteed to change in the future, to support new
 features.  I'm looking for opinions about making the schema part of the
 public API to coverage.py or not.

- Development moved from `Bitbucket`_ to `GitHub`_.

- HTML files no longer have trailing and extra whitespace.

- The sort order in the HTML report is stored in local storage rather than
cookies, closing `issue 611`_.  Thanks, Federico Bond.

- pickle2json, for converting v3 data files to v4 data files, has been removed.

.. _Bitbucket: https://bitbucket.org/ned/coveragepy
.. _GitHub: https://github.com/nedbat/coveragepy

.. _issue 611: https://github.com/nedbat/coveragepy/issues/611


.. _changes_50a1:

5.0a1

----------------------------

- Coverage.py no longer supports Python 2.6 or 3.3.

- The location of the configuration file can now be specified with a
``COVERAGE_RCFILE`` environment variable, as requested in `issue 650`_.

- Namespace packages are supported on Python 3.7, where they used to cause
TypeErrors about path being None. Fixes `issue 700`_.

- A new warning (``already-imported``) is issued if measurable files have
already been imported before coverage.py started measurement.  See
:ref:`cmd_warnings` for more information.

- Running coverage many times for small runs in a single process should be
faster, closing `issue 625`_.  Thanks, David MacIver.

- Large HTML report pages load faster.  Thanks, Pankaj Pandey.

.. _issue 625: https://bitbucket.org/ned/coveragepy/issues/625/lstat-dominates-in-the-case-of-small
.. _issue 650: https://bitbucket.org/ned/coveragepy/issues/650/allow-setting-configuration-file-location
.. _issue 700: https://github.com/nedbat/coveragepy/issues/700


.. _changes_454:
Links

@codecov
Copy link

codecov bot commented Dec 14, 2019

Codecov Report

Merging #230 into develop will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff            @@
##           develop     #230   +/-   ##
========================================
  Coverage    73.11%   73.11%           
========================================
  Files           70       70           
  Lines         2786     2786           
========================================
  Hits          2037     2037           
  Misses         749      749

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59325ae...a813a10. Read the comment docs.

@abhinavsingh abhinavsingh merged commit fee3af9 into develop Dec 14, 2019
@abhinavsingh abhinavsingh deleted the pyup-update-coverage-4.5.4-to-5.0 branch December 14, 2019 21:14
abhinavsingh added a commit that referenced this pull request Dec 26, 2019
* Always update latest tag for docker releases

* Update issue templates (#123)

* Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125)

* Add tests to verify certificate generation

* Separate out tests for ProtocolHandler and WebServerPlugin

* Keep-alive connections for web server.

TODO: Only keep-alivei if HTTP/1.1

* Add request.path to avoid build_url repeatedly whose name is also slightly misleading

* Fix example usage of request.path

* Pipeline only for HTTP/1.1

* Lint fix

* Teardown HTTP/1.1 keep-alive request when Connection: close header is sent

* Add instructions on how to build docker image locally

* Move access_log to separate function for pretty logging

* Reduce docker image size

* Ensure teardown is always accompanied with Connection: close header

Fix tests

* Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128)

* Add tests for is_http_1_1_keep_alive

* Add ModifyPostDataPlugin in README

* Fixes #126

* Refactor HttpProxyBasePlugin API

* before_upstream_connection too can drop request by returning None

* Remove HTTP Server startup during tests, no longer used

* Removed unused imports

* Simplify load_plugins

* Add --timeout flag with default value of 10 second. (#129)

* Add --timeout flag with default value of 5.  This value was previously hardcoded to 30

* --timeout=10 by default

* Dispatch 408 timeout when connection is dropped due to inactivity

* Add httpStatusCodes named tuple

* Update plugin client connection reference after TLS connection upgrade

* Test plugin examples (#130)

* Add tests for plugin_examples.* to ensure we never break functionality

* Add tests for plugin_examples.*

* Test man in the middle

* Lint fixes

* Checkin

* Add tests for plugin examples with TLS encryption enabled

* Threadless execution using coroutines (#134)

* Workers need not register/unregister sock for every loop

* No need of explicit socket.settimeout(0) which is same as socket.setblocking(False)

* Remove settimeout assertion

* Only store sender side of Pipe().  Also ensure both end of the Pipe() are closed on shutdown

* Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes

* Use time.time throughout.  Remove incomplete test_cache_responses_plugin to avoid resource leak in tests

* Remove unused

* Wrap selector register/unregister within a context manager

* Refactor in preparation of threadless request handling

* MyPy generator fix

* Add --threadless flag

* Internally call them acceptors

* Internally use acceptors

* Add Threadless class.

Also no need to pass family over pipe to acceptors.

* Make threadless work for a single client :)

* Threadless is soon be our default

* Close client queue

* Use context manager for register/unregister

* Fix Acceptor tests broken after refactoring

* Use asyncio tasks to invoke ProtocolHandle.handle_events

This gives all client threads a chance to respond without
waiting for other handlers to return.

* Explicitly initialize event loop per Threadless process

* Mypy fixes

* Add ThreadlessWork abstract class implemented by ProtocolHandler

* Add benchmark.py

Avoid TIME_WAIT by properly shutting down the connection.

* Add benchmark.py as part of testing workflow

* When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state

* MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence

* Unwrap if wrapped before shutdown

* Unwrap if wrapped before shutdown

* socket.SHUT_RDWR will cause leaks

* MyPy

* Add instructions for monitor.sh

* Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized

* Add Fast & Scalable section

* Update internal classes section

* Dont print out local dir path in help text :)

* Refactor

* Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang

* Add chrome_with_proxy.sh helper script

* Handle OSError during client.flush which can happen due to invalid protocol type for socket error

* Remove redundant e

* Add classmethods to quickly construct a parser object

* Don't raise from TcpConnection abstract class.

This allows both client/socket side of communication to handle
exceptions as necessary. We might refactor this again later to remove
redundant code :)

* Disable response parsing when TLS interception is enabled. See issue #127

* remove unused imports

* Within webserver parse pipelined requests only if we have a route

* Add ShortLinkPlugin plugin

* Add more shortlinks

* Add ShortLinkPlugin to README.md

* Add path forwarding too instead of leaving as excercise ;)

* Add shortlink to TOC

* Ensure no socket leaks

* Ensure no leaks

* Naming

* Default number of clients 1

* Avoid shortlinking localhost

* Stress more

* Remove pip upgrade for windows which seems to be failing on travis (#136)

* Remove pip upgrade for windows which seems to be failing on travis

* Remove windows testing on Travis, pip install is failing

* Add pipeline response parsing tests (#137)

* Add pipeline response parsing tests

* build_http_response now only adds content-length if transfer-encoding is
not provided.

Also return pending raw chunks from ChunkParser so that we can parse
pipelined chunk responses.

* os.close only for threadless (#138)

* os.close only for Threadless to avoid fd leaks

* Remove os.close mock which is only called for threadless

* Update pytest from 5.2.1 to 5.2.2 (#142)

* Update setuptools from 41.4.0 to 41.5.0 (#145)

* Update typing-extensions from 3.7.4 to 3.7.4.1 (#147)

* Update flake8 from 3.7.8 to 3.7.9 (#148)

* Update setuptools from 41.5.0 to 41.5.1 (#149)

* Update py-spy from 0.2.2 to 0.3.0 (#144)

* Proxy.py Dashboard (#141)

* Remove redundant variables

* Initialize frontend dashboard app (written in typescript)

* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection

* Add read_and_build_static_file_response method for reusability in plugins

* teardown websocket connection when opcode CONNECTION_CLOSE is received

* First draft of proxy.py dashboard

* Remove uglify, obfuscator is superb enough

* Correct generic V

* First draft of dashboard

* ProtocolConfig is now Flags

* First big refactor toward no-single-file-module

* Working tests

* Update dashboard for refactored imports

* Remove proxy.py as now we can just call python -m proxy -h

* Fix setup.py for refactored code

* Banner update

* Lint check

* Fix dashboard static serving and no UNDER_TEST constant necessary

* Add support for plugin imports when specified in path/to/module.MyPlugin

* Update README with instructions to run proxy.py after refactor

* Move dashboard under /dashboard path

* Rename to devtools.ts

* remove unused

* Update github workflow for new directory structure

* Update test command too

* Fix coverage generation

* *.py is an invalid syntax on windows

* No * on windows

* Enable execution via github zip downloads

* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory

* Forget zip runs for now

* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load

* Enforce eslint with standard style

* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)

* Remove extra empty line

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Fix tests

* Move common code under common sub-module

* Move flags under common module

* Move acceptor under core

* Move connection under core submodule

* Move chunk_parser under http

* Move http_parser as http/parser

* Move http_methods as http/methods

* Move http_proxy as http/proxy

* Move web_server as http/server

* Move status_codes as http/codes

* move websocket as http/websocket

* Move exception under http/exception, also move http/proxy exceptions under http/exceptions

* move protocol_handler as http/handler

* move devtools as http/devtools

* Move version under common/version

* Lifecycle if now core Event

* autopep8

* Add core event queue

* Register / unregister handler

* Enable inspection support for frontend dashboard

* Dont give an illusion of exception for HttpProtocolExceptions

* Update readme for refactored codebase

* DictQueueType everywhere

* Move all websocket API related code under WebsocketApi class

* Inspection enabled on tab switch.

1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.

* Corresponding ends of the work queues can be closed immediately.

Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.

* No need of a manager for shared multiprocess Lock.

This unnecessarily creates additional manager process.

* Move threadless into its own module

* Merge acceptor and acceptor_pool tests

* Defer os.close

* Change content display with tab clicks.

Also ensure relay manager shutdown.

* Remove --cov flags

* Use right type for SyncManager

* Ensure coverage again

* Print help to discover flags, --cov certainly not available on Travis for some reason

* Add pytest-cov to requirements-testing

* Re-add windows on .travis also add changelog to readme

* Use 3.7 and no pip upgrade since it fails on travis windows

* Attempt to fix pip install on windows

* Disable windows on travis, it fails and uses 3.8.  Try reporting coverage from github actions

* Move away from coveralls, use codecov

* Codecov app installation either didnt work or token still needs to be passed

* Remove travis CI

* Use https://github.com/codecov/codecov-action for coverage uploads

* Remove run codecov

* Ha, codecov action only works on linux, what a mess

* Add cookie.js though unable to use it with es5/es6 modules yet

* Enable testing for python 3.8 also Build dashboard during testing

* No python 3.8 on github actions yet

* Autopep8

* Add separate workflows for library (python) and dashboard (node) app

* Type jobs not job

* Add checkout

* Fix parsing node version

* Fix dashboard build on windows

* Show codecov instead of coveralls

* Update mypy==0.740 (#151)

* Update README.md (#152)

* Update flags

* Update debugging instructions and run instructions for develops

* Update references to plugins directory

* For readability add sections for run from command line using pip

* Move internal doc under developer section

* Add option to pass fully-qualified plugin path

* Update setuptools from 41.5.1 to 41.6.0 (#153)

* Test refactor + Docker image CI (#154)

* Move tests into individual modules too

* Ensure one test class per file

* Fix docker image after refactoring

* Add github actions workflow for building docker image

* Fix image name

* Setup python required for extracting proxy version

* Version will also require deps

* Separate packages for Dashboard (#157)

* Refactor Makefile and add dashboard setup.py

* Package dashboard as proxy.py-dashboard pip package

* Give dashboard releases its own version

* Fix lib-package reference

* Add non-blocking embedded mode feature (#159)

* Fixes #158

* mypy fixes

* Instructions for non-blocking embed mode

* Toggle running flag before shutdown

* Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160)

* Add utilities to generate private key and public keys with alternate cnames

* Add separate package proxy.py-plugins, fixes #156

* Generate certificates to comply with Mac requirements.

* Add utility for CSR generation and signing

* Fixes #161

* Add initial pki tests

* Give structure to dashboard app (#163)

* Separate out files for different responsibilities.

1. Add src/plugins directory.  This directory holds one typescript file
   per plugin.  Each plugin is optionally can be displayed as a tab on
   the UI.
2. Move WebsocketApi to ws.ts.  This file contains all websocket APIs
   provided by dashboard.py backend.

* Make dashboard pluggable

* Move devtools under core too

* Register tabs dynamically

* Typescript fixes for abstract interfaces

* Initialize plugin app body skeleton

* Call activated / deactivated on tab change

* Move plugin name within plugin classes and initialize plugin within proxy dashboard constructor

* templatize api development plugin

* eslint fixes

* use globs

* Remove useless constructors

* Move traffic_control outside of core plugin, it maps to several plugin examples like redirectToUpstreamHost, filterByUpstreamHost plugins (#165)

* Introduce sendMessage websocket api which allows for callbacks (#166)

* Introduce sendMessage websocket api which allows for callbacks, deprecate lastPingId in favor of callbacks

* Let InspectTrafficPlugin handle all pushed inspection events

* Add proxy.main.TestCase for unit testing Python application with proxy.py (#167)

* Add demonstration of how to use proxy.py within Python application unittests

* mypy fixes

* test_with_proxy example

* Add docs for proxy.main.TestCase.

Also wait for proxy.py server to come up before running the tests.

* Consistent dashboard look and feel across plugins (#169)

* Explicitly link version changelog in TOC

* Separate out app header body builder

* Ensure unsubscribe when disabling inspection.  Fixes #164

* Avoid creation of new manager per dashboard instance.

* Add UI header for all plugins (tabs)

* Ensure app body for all plugin skeleton

* Move app-header and app-body within core for consistent dashboard look and feel

* Consistent UI header body for plugins

* autopep8

* Dashboard Inspect traffic tab + devtools (#170)

* Explicitly link version changelog in TOC

* Separate out app header body builder

* Ensure unsubscribe when disabling inspection.  Fixes #164

* Avoid creation of new manager per dashboard instance.

* Add UI header for all plugins (tabs)

* Ensure app body for all plugin skeleton

* Move app-header and app-body within core for consistent dashboard look and feel

* Consistent UI header body for plugins

* autopep8

* make devtools

* convert to es6

* Add inspect_traffic plugin devtools app

* trigger re-build, github UI is stuck

* Dynamically load devtools within inspect traffic view

* Just copy devtools into public/dashboard folder

* Works but not how we wanted, devtools takes over entire body and doesnt contain itself within a div

* Load devtools within iframe

* Load devtools within iframe (#171)

* Allow to pass flags as kwargs too in embed mode (#172)

* Dynamically load devtools instead of on page load

* Add support for passing flags as kwargs to main / start methods.

* Fix tests for refactored code

* Allow proxy.main, proxy.start, proxy.TestCase.

Also update README.md to reflect the same.

* Use Any for **opts

* Move main as __init__ to avoid name conflicts

* Fix tests

* Update setup.py entry_point

* Explicitly install requirements before setup.py

* Explicitly mention packages of interest

* ipv6 fails on ubuntu, use ipv4

* Make typing-extensions optional

* Instead of putting it all under __init__.py, move main.py to proxy.py

* Simply make setup.py module free

* autopep8

* Devtools Protocol (#174)

* Refine docs

* Decouple relay from dashboard.

Will be re-used by devtools protocol plugin.

* Just have a single manager for all eventing

* Ofcourse managers cant be shared across processes

* Remove unused

* Add DevtoolsProtocolPlugin

* Emit REQUEST_COMPLETE core event

* Emit only if --enable-events used

* Add event emitter for response cycle

* Fill up core events to devtools protocol expectations

* Serve static content with Cache-Control header and gzip compression

* Add PWA manifest.json and icons from sample PWA apps (replace later)

* Catch any exception and be ssl agnostic

* Add CSP headers and avoid inline scripts

* Re-enable iframe and deobfuscation

* Embed plugins within <section/> block

* Make tab switching agnostic of block name

* Add support for browser history on tab change

* Default hash to #home

* Switch to tab if hash is already set

* Expand canvas to fill screen even without content

* Remove inline css for embedded devtools

* Make dashboard backend websocket API pluggable

* doc

* Move dashboard backend within proxy module, now ships via same pip package (#177)

* Allow resources to load from http and ws when running w/o https

* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py

* Update ref to dashboard backend in github workflows

* Add git-pre-commit hook file.

Enable it by symlinking as .git/hooks/pre-commit

* Also enable static server for dashboard serving

* Move plugin_examples/ as proxy.plugin and update readme (#179)

* Update dev guide

* Move plugin_examples/ as proxy.plugin

* Update proxy.plugin ref path in readme

* Remove unnecessary port flag

* Remove plugin_examples from github workflows

* dashboard folder is a npm package not python package anymore

* Plugins can now be tried using Docker image

* Move benchmark module within proxy (#181)

* Move benchmark within proxy module

* chmod 0644 for benchmark.py which was executable till now

* Turn utilities into its own section

* Update pytest from 5.2.3 to 5.2.4 (#180)

* Doc & Banner update to match GitHub (#182)

* Update doc and banner

* Update banner to match GitHub

* Update older banners too

* Add update_desc to .gitignore

* Update banner for dashboard to match github

* also update html, js, css

* Update twine from 2.0.0 to 3.0.0 (#183)

* Update pytest from 5.2.4 to 5.3.0 (#186)

* Testing support improvements (#185)

* Introduce proxy.Proxy context manager.

This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.

* Gracefully shutdown threadless processes

* Update tests and add a VCR method. See #184

* Refactor routes

* Add Proxy to __all__

* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder

* Add a base cache plugin class which can be customized for custom cache behaviors

* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...

* Make cache plugin pluggable + make cache storage pluggable

* Make dashboard npm module agnostic of top level directory

* Symlink dashboard public folder

* Dump devtools within dashboard public folder

* Remove unused 3rd party js

* Initialize Menubar (#188)

* Initialize MacOS Menubar application

* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin

* Add menu bar icon

* Add respective test directories

* Sync test banners

* Move plugin tests under its own package

* Enable daemon for threads, other this wont shutdown cleanly

* Update twine from 3.0.0 to 3.1.0 (#190)

* Update setuptools from 41.6.0 to 42.0.0 (#191)

* Memory optimizations (#189)

* Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187

* Clarity in test comments

* Update setuptools from 42.0.0 to 42.0.1 (#193)

* Make connection queue / recv work with memoryview to avoid copies (#192)

* connection.recv now returns a memoryview

* Make connection.queue also memoryview compliant

* autopep8

* wrap in memoryview as necessary

* Add default timeout for socket_connection and test_embed urllib

* Fix tests

* Skip TestProxyPyEmbedded for now, verifying GitHub actions

* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set

* Verify if GitHub Action fails due to wait_for_server spinning forever

* Add test for wait_for_server timeout error exception

* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions

* Cleanup (#194)

* Add basic README description for dashboard

* Use spaces for all except makefile

* enable tests for py 3.5

* Python 3.5 support label

* Avoid clash of names

* Add py3.8 support and bump node to 12.x (#195)

* Add py3.8 support and bump node to 12.x

* Add 10.x, 11.x, 12.x matrix for dashboard testing

* Add Python 3.8 support label

* Single tested with label

* autopep8 (#196)

* autopep8

* Update TestCase section

* Update pytest from 5.3.0 to 5.3.1 (#197)

* Update twine from 3.1.0 to 3.1.1 (#200)

* Add reverse proxy example (#201)

* Add reverse proxy example

* Add separate sections for http proxy and web server plugins

* Add doc

* Add proxy over ssh tunnel functionality (#198)

* update mypy to 0.750 (#204)

* Test Core Eventing (#205)

* Add core event tests

* Update .gitignore with coverage

* Add shortlink gif

* Add event dispatcher test

* Test event subscriber

* Test Dashboard backend (#206)

* Update shortlink gif name

* Conditionally run workflows as necessary

* Use pytest

* It works but github workflow is not reporting any status :(

* Separate out badges

* Add python_requires to setup.py

* Update setuptools from 42.0.1 to 42.0.2 (#207)

* Add tox.ini (#208)

* Homebrew formula (#209)

* Add homebrew formula

* Build PyPi package and Homebrew installation verification

* Check develop

* bdist_wheel reported as error: invalid command "bdist_wheel"

* Move under stable/develop folders to keep Proxy class name same

* uff

* develop installs proxy not proxy.py binary

* Prepend site-packages

* Install typing-extensions explicitly with brew

* Use find_packages

* Most likely failing due to lack of find_packages in current develop branch

* Fix windows setup.py build

* test_static_web_server_serves seems flaky on Ubuntu python 3.8

* Add instructions to install using homebrew

* Disable test_static_web_server_serves on GitHub actions, seems flaky

* Packaging (#210)

* Move docker installation steps above

* Try brewing with virtualenv

* depends on python

* Update homebrew formula for stable release

* Just test brewing on latest python

* Add support for regex based routing. Fixes #203 (#211)

* Remove public folder references (#212)

* Refactor (#213)

* Add DEFAULT_HTTP_PORT constant

* Use DEFAULT_HTTP_PORT in tests

* Refactor into exception module

* Refactor into inspector module

* Refactor into server module

* Refactor into proxy module

* Build docker of Python 3.8 (#214)

* Move homebrew under helper (#215)

* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216)

* Catch TimeoutError and OSError (host unreachable)

* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET

* Enable mccabe (#217)

* No need of per day or week stats (#218)

* Make HTTP handler constructor free of socket file number (#219)

* Refactor into acceptor module

* Add tunnel doc

* Make fileno free

* Autopep8

* Response parser now reaches COMPLETE even when no body is expected (#220)

* Stash current changes

* Refactor into connection module

* Response parser state complete when no body expect

* Raise NotImplementedError if invalid state reached within parser

* Update tox from 3.14.1 to 3.14.2 (#221)

* Update paramiko from 2.6.0 to 2.7.0 (#225)

* Update paramiko from 2.7.0 to 2.7.1 (#227)

* Proxy Pool Plugin (#228)

* Add proxy pool example. See #226

* Add ProxyPoolPlugin to doc

* Update pytest from 5.3.1 to 5.3.2 (#229)

* Update coverage from 4.5.4 to 5.0 (#230)

* Update mypy from 0.750 to 0.760 (#232)

* Update mypy from 0.760 to 0.761 (#235)

* Move manager initialization outside of top level scope.  Fixes #233 (#236)

* Share lock to acceptors via pool (#238)

* Move manager initialization outside of top level scope.  Fixes #233

* Share lock to acceptor via pool

* Optionally initialize manager in main thread and use the same for EventQueue initialization (#239)

* Highlight language syntax (#240)

* Highlight lang syntax

* zsh prompt

* Update coverage from 5.0 to 5.0.1 (#241)

* Integration testing (#243)

* Add tests for public/private/csr generation

* Add integration testing skeleton for mac and ubuntu

* Merge integration within lib test to avoid too many workflows

* Disable integration testing on windows for now

* Use sudo to start integration test script as lsof fails on MacOS.

lsof: WARNING: can't stat() vmhgfs file system

* Add basic integration testing for now to assert proxy works as expected when started out of develop branch

* Add a call to inbuilt http server to verify it works

* wait for server to accept requests

Co-authored-by: pyup.io bot <[email protected]>
abhinavsingh added a commit that referenced this pull request Dec 26, 2019
* v2.x (#173)

* Always update latest tag for docker releases

* Update issue templates (#123)

* Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125)

* Add tests to verify certificate generation

* Separate out tests for ProtocolHandler and WebServerPlugin

* Keep-alive connections for web server.

TODO: Only keep-alivei if HTTP/1.1

* Add request.path to avoid build_url repeatedly whose name is also slightly misleading

* Fix example usage of request.path

* Pipeline only for HTTP/1.1

* Lint fix

* Teardown HTTP/1.1 keep-alive request when Connection: close header is sent

* Add instructions on how to build docker image locally

* Move access_log to separate function for pretty logging

* Reduce docker image size

* Ensure teardown is always accompanied with Connection: close header

Fix tests

* Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128)

* Add tests for is_http_1_1_keep_alive

* Add ModifyPostDataPlugin in README

* Fixes #126

* Refactor HttpProxyBasePlugin API

* before_upstream_connection too can drop request by returning None

* Remove HTTP Server startup during tests, no longer used

* Removed unused imports

* Simplify load_plugins

* Add --timeout flag with default value of 10 second. (#129)

* Add --timeout flag with default value of 5.  This value was previously hardcoded to 30

* --timeout=10 by default

* Dispatch 408 timeout when connection is dropped due to inactivity

* Add httpStatusCodes named tuple

* Update plugin client connection reference after TLS connection upgrade

* Test plugin examples (#130)

* Add tests for plugin_examples.* to ensure we never break functionality

* Add tests for plugin_examples.*

* Test man in the middle

* Lint fixes

* Checkin

* Add tests for plugin examples with TLS encryption enabled

* Threadless execution using coroutines (#134)

* Workers need not register/unregister sock for every loop

* No need of explicit socket.settimeout(0) which is same as socket.setblocking(False)

* Remove settimeout assertion

* Only store sender side of Pipe().  Also ensure both end of the Pipe() are closed on shutdown

* Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes

* Use time.time throughout.  Remove incomplete test_cache_responses_plugin to avoid resource leak in tests

* Remove unused

* Wrap selector register/unregister within a context manager

* Refactor in preparation of threadless request handling

* MyPy generator fix

* Add --threadless flag

* Internally call them acceptors

* Internally use acceptors

* Add Threadless class.

Also no need to pass family over pipe to acceptors.

* Make threadless work for a single client :)

* Threadless is soon be our default

* Close client queue

* Use context manager for register/unregister

* Fix Acceptor tests broken after refactoring

* Use asyncio tasks to invoke ProtocolHandle.handle_events

This gives all client threads a chance to respond without
waiting for other handlers to return.

* Explicitly initialize event loop per Threadless process

* Mypy fixes

* Add ThreadlessWork abstract class implemented by ProtocolHandler

* Add benchmark.py

Avoid TIME_WAIT by properly shutting down the connection.

* Add benchmark.py as part of testing workflow

* When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state

* MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence

* Unwrap if wrapped before shutdown

* Unwrap if wrapped before shutdown

* socket.SHUT_RDWR will cause leaks

* MyPy

* Add instructions for monitor.sh

* Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized

* Add Fast & Scalable section

* Update internal classes section

* Dont print out local dir path in help text :)

* Refactor

* Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang

* Add chrome_with_proxy.sh helper script

* Handle OSError during client.flush which can happen due to invalid protocol type for socket error

* Remove redundant e

* Add classmethods to quickly construct a parser object

* Don't raise from TcpConnection abstract class.

This allows both client/socket side of communication to handle
exceptions as necessary. We might refactor this again later to remove
redundant code :)

* Disable response parsing when TLS interception is enabled. See issue #127

* remove unused imports

* Within webserver parse pipelined requests only if we have a route

* Add ShortLinkPlugin plugin

* Add more shortlinks

* Add ShortLinkPlugin to README.md

* Add path forwarding too instead of leaving as excercise ;)

* Add shortlink to TOC

* Ensure no socket leaks

* Ensure no leaks

* Naming

* Default number of clients 1

* Avoid shortlinking localhost

* Stress more

* Remove pip upgrade for windows which seems to be failing on travis (#136)

* Remove pip upgrade for windows which seems to be failing on travis

* Remove windows testing on Travis, pip install is failing

* Add pipeline response parsing tests (#137)

* Add pipeline response parsing tests

* build_http_response now only adds content-length if transfer-encoding is
not provided.

Also return pending raw chunks from ChunkParser so that we can parse
pipelined chunk responses.

* os.close only for threadless (#138)

* os.close only for Threadless to avoid fd leaks

* Remove os.close mock which is only called for threadless

* Update pytest from 5.2.1 to 5.2.2 (#142)

* Update setuptools from 41.4.0 to 41.5.0 (#145)

* Update typing-extensions from 3.7.4 to 3.7.4.1 (#147)

* Update flake8 from 3.7.8 to 3.7.9 (#148)

* Update setuptools from 41.5.0 to 41.5.1 (#149)

* Update py-spy from 0.2.2 to 0.3.0 (#144)

* Proxy.py Dashboard (#141)

* Remove redundant variables

* Initialize frontend dashboard app (written in typescript)

* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection

* Add read_and_build_static_file_response method for reusability in plugins

* teardown websocket connection when opcode CONNECTION_CLOSE is received

* First draft of proxy.py dashboard

* Remove uglify, obfuscator is superb enough

* Correct generic V

* First draft of dashboard

* ProtocolConfig is now Flags

* First big refactor toward no-single-file-module

* Working tests

* Update dashboard for refactored imports

* Remove proxy.py as now we can just call python -m proxy -h

* Fix setup.py for refactored code

* Banner update

* Lint check

* Fix dashboard static serving and no UNDER_TEST constant necessary

* Add support for plugin imports when specified in path/to/module.MyPlugin

* Update README with instructions to run proxy.py after refactor

* Move dashboard under /dashboard path

* Rename to devtools.ts

* remove unused

* Update github workflow for new directory structure

* Update test command too

* Fix coverage generation

* *.py is an invalid syntax on windows

* No * on windows

* Enable execution via github zip downloads

* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory

* Forget zip runs for now

* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load

* Enforce eslint with standard style

* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)

* Remove extra empty line

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Fix tests

* Move common code under common sub-module

* Move flags under common module

* Move acceptor under core

* Move connection under core submodule

* Move chunk_parser under http

* Move http_parser as http/parser

* Move http_methods as http/methods

* Move http_proxy as http/proxy

* Move web_server as http/server

* Move status_codes as http/codes

* move websocket as http/websocket

* Move exception under http/exception, also move http/proxy exceptions under http/exceptions

* move protocol_handler as http/handler

* move devtools as http/devtools

* Move version under common/version

* Lifecycle if now core Event

* autopep8

* Add core event queue

* Register / unregister handler

* Enable inspection support for frontend dashboard

* Dont give an illusion of exception for HttpProtocolExceptions

* Update readme for refactored codebase

* DictQueueType everywhere

* Move all websocket API related code under WebsocketApi class

* Inspection enabled on tab switch.

1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.

* Corresponding ends of the work queues can be closed immediately.

Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.

* No need of a manager for shared multiprocess Lock.

This unnecessarily creates additional manager process.

* Move threadless into its own module

* Merge acceptor and acceptor_pool tests

* Defer os.close

* Change content display with tab clicks.

Also ensure relay manager shutdown.

* Remove --cov flags

* Use right type for SyncManager

* Ensure coverage again

* Print help to discover flags, --cov certainly not available on Travis for some reason

* Add pytest-cov to requirements-testing

* Re-add windows on .travis also add changelog to readme

* Use 3.7 and no pip upgrade since it fails on travis windows

* Attempt to fix pip install on windows

* Disable windows on travis, it fails and uses 3.8.  Try reporting coverage from github actions

* Move away from coveralls, use codecov

* Codecov app installation either didnt work or token still needs to be passed

* Remove travis CI

* Use https://github.com/codecov/codecov-action for coverage uploads

* Remove run codecov

* Ha, codecov action only works on linux, what a mess

* Add cookie.js though unable to use it with es5/es6 modules yet

* Enable testing for python 3.8 also Build dashboard during testing

* No python 3.8 on github actions yet

* Autopep8

* Add separate workflows for library (python) and dashboard (node) app

* Type jobs not job

* Add checkout

* Fix parsing node version

* Fix dashboard build on windows

* Show codecov instead of coveralls

* Update mypy==0.740 (#151)

* Update README.md (#152)

* Update flags

* Update debugging instructions and run instructions for develops

* Update references to plugins directory

* For readability add sections for run from command line using pip

* Move internal doc under developer section

* Add option to pass fully-qualified plugin path

* Update setuptools from 41.5.1 to 41.6.0 (#153)

* Test refactor + Docker image CI (#154)

* Move tests into individual modules too

* Ensure one test class per file

* Fix docker image after refactoring

* Add github actions workflow for building docker image

* Fix image name

* Setup python required for extracting proxy version

* Version will also require deps

* Separate packages for Dashboard (#157)

* Refactor Makefile and add dashboard setup.py

* Package dashboard as proxy.py-dashboard pip package

* Give dashboard releases its own version

* Fix lib-package reference

* Add non-blocking embedded mode feature (#159)

* Fixes #158

* mypy fixes

* Instructions for non-blocking embed mode

* Toggle running flag before shutdown

* Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160)

* Add utilities to generate private key and public keys with alternate cnames

* Add separate package proxy.py-plugins, fixes #156

* Generate certificates to comply with Mac requirements.

* Add utility for CSR generation and signing

* Fixes #161

* Add initial pki tests

* Give structure to dashboard app (#163)

* Separate out files for different responsibilities.

1. Add src/plugins directory.  This directory holds one typescript file
   per plugin.  Each plugin is optionally can be displayed as a tab on
   the UI.
2. Move WebsocketApi to ws.ts.  This file contains all websocket APIs
   provided by dashboard.py backend.

* Make dashboard pluggable

* Move devtools under core too

* Register tabs dynamically

* Typescript fixes for abstract interfaces

* Initialize plugin app body skeleton

* Call activated / deactivated on tab change

* Move plugin name within plugin classes and initialize plugin within proxy dashboard constructor

* templatize api development plugin

* eslint fixes

* use globs

* Remove useless constructors

* Move traffic_control outside of core plugin, it maps to several plugin examples like redirectToUpstreamHost, filterByUpstreamHost plugins (#165)

* Introduce sendMessage websocket api which allows for callbacks (#166)

* Introduce sendMessage websocket api which allows for callbacks, deprecate lastPingId in favor of callbacks

* Let InspectTrafficPlugin handle all pushed inspection events

* Add proxy.main.TestCase for unit testing Python application with proxy.py (#167)

* Add demonstration of how to use proxy.py within Python application unittests

* mypy fixes

* test_with_proxy example

* Add docs for proxy.main.TestCase.

Also wait for proxy.py server to come up before running the tests.

* Consistent dashboard look and feel across plugins (#169)

* Explicitly link version changelog in TOC

* Separate out app header body builder

* Ensure unsubscribe when disabling inspection.  Fixes #164

* Avoid creation of new manager per dashboard instance.

* Add UI header for all plugins (tabs)

* Ensure app body for all plugin skeleton

* Move app-header and app-body within core for consistent dashboard look and feel

* Consistent UI header body for plugins

* autopep8

* Dashboard Inspect traffic tab + devtools (#170)

* Explicitly link version changelog in TOC

* Separate out app header body builder

* Ensure unsubscribe when disabling inspection.  Fixes #164

* Avoid creation of new manager per dashboard instance.

* Add UI header for all plugins (tabs)

* Ensure app body for all plugin skeleton

* Move app-header and app-body within core for consistent dashboard look and feel

* Consistent UI header body for plugins

* autopep8

* make devtools

* convert to es6

* Add inspect_traffic plugin devtools app

* trigger re-build, github UI is stuck

* Dynamically load devtools within inspect traffic view

* Just copy devtools into public/dashboard folder

* Works but not how we wanted, devtools takes over entire body and doesnt contain itself within a div

* Load devtools within iframe

* Load devtools within iframe (#171)

* Allow to pass flags as kwargs too in embed mode (#172)

* Dynamically load devtools instead of on page load

* Add support for passing flags as kwargs to main / start methods.

* Fix tests for refactored code

* Allow proxy.main, proxy.start, proxy.TestCase.

Also update README.md to reflect the same.

* Use Any for **opts

* Move main as __init__ to avoid name conflicts

* Fix tests

* Update setup.py entry_point

* Explicitly install requirements before setup.py

* Explicitly mention packages of interest

* ipv6 fails on ubuntu, use ipv4

* Make typing-extensions optional

* Instead of putting it all under __init__.py, move main.py to proxy.py

* Simply make setup.py module free

* autopep8

* Devtools Protocol (#174)

* Refine docs

* Decouple relay from dashboard.

Will be re-used by devtools protocol plugin.

* Just have a single manager for all eventing

* Ofcourse managers cant be shared across processes

* Remove unused

* Add DevtoolsProtocolPlugin

* Emit REQUEST_COMPLETE core event

* Emit only if --enable-events used

* Add event emitter for response cycle

* Fill up core events to devtools protocol expectations

* Serve static content with Cache-Control header and gzip compression

* Add PWA manifest.json and icons from sample PWA apps (replace later)

* Catch any exception and be ssl agnostic

* Add CSP headers and avoid inline scripts

* Re-enable iframe and deobfuscation

* Embed plugins within <section/> block

* Make tab switching agnostic of block name

* Add support for browser history on tab change

* Default hash to #home

* Switch to tab if hash is already set

* Expand canvas to fill screen even without content

* Remove inline css for embedded devtools

* Make dashboard backend websocket API pluggable

* doc

* Move dashboard backend within proxy module, now ships via same pip package (#177)

* Allow resources to load from http and ws when running w/o https

* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py

* Update ref to dashboard backend in github workflows

* Add git-pre-commit hook file.

Enable it by symlinking as .git/hooks/pre-commit

* Also enable static server for dashboard serving

* Move plugin_examples/ as proxy.plugin and update readme (#179)

* Update dev guide

* Move plugin_examples/ as proxy.plugin

* Update proxy.plugin ref path in readme

* Remove unnecessary port flag

* Remove plugin_examples from github workflows

* dashboard folder is a npm package not python package anymore

* Plugins can now be tried using Docker image

* Move benchmark module within proxy (#181)

* Move benchmark within proxy module

* chmod 0644 for benchmark.py which was executable till now

* Turn utilities into its own section

* Update pytest from 5.2.3 to 5.2.4 (#180)

* Doc & Banner update to match GitHub (#182)

* Update doc and banner

* Update banner to match GitHub

* Update older banners too

* Add update_desc to .gitignore

* Update banner for dashboard to match github

* also update html, js, css

* Update twine from 2.0.0 to 3.0.0 (#183)

* Update pytest from 5.2.4 to 5.3.0 (#186)

* Testing support improvements (#185)

* Introduce proxy.Proxy context manager.

This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.

* Gracefully shutdown threadless processes

* Update tests and add a VCR method. See #184

* Refactor routes

* Add Proxy to __all__

* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder

* Add a base cache plugin class which can be customized for custom cache behaviors

* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...

* Make cache plugin pluggable + make cache storage pluggable

* Make dashboard npm module agnostic of top level directory

* Symlink dashboard public folder

* Dump devtools within dashboard public folder

* Remove unused 3rd party js

* Initialize Menubar (#188)

* Initialize MacOS Menubar application

* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin

* Add menu bar icon

* Add respective test directories

* Sync test banners

* Move plugin tests under its own package

* Enable daemon for threads, other this wont shutdown cleanly

* Update twine from 3.0.0 to 3.1.0 (#190)

* Update setuptools from 41.6.0 to 42.0.0 (#191)

* Memory optimizations (#189)

* Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187

* Clarity in test comments

* Update setuptools from 42.0.0 to 42.0.1 (#193)

* Make connection queue / recv work with memoryview to avoid copies (#192)

* connection.recv now returns a memoryview

* Make connection.queue also memoryview compliant

* autopep8

* wrap in memoryview as necessary

* Add default timeout for socket_connection and test_embed urllib

* Fix tests

* Skip TestProxyPyEmbedded for now, verifying GitHub actions

* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set

* Verify if GitHub Action fails due to wait_for_server spinning forever

* Add test for wait_for_server timeout error exception

* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions

* Cleanup (#194)

* Add basic README description for dashboard

* Use spaces for all except makefile

* enable tests for py 3.5

* Python 3.5 support label

* Avoid clash of names

* Add py3.8 support and bump node to 12.x (#195)

* Add py3.8 support and bump node to 12.x

* Add 10.x, 11.x, 12.x matrix for dashboard testing

* Add Python 3.8 support label

* Single tested with label

* autopep8 (#196)

* autopep8

* Update TestCase section

* Update pytest from 5.3.0 to 5.3.1 (#197)

* Update twine from 3.1.0 to 3.1.1 (#200)

* Add reverse proxy example (#201)

* Add reverse proxy example

* Add separate sections for http proxy and web server plugins

* Add doc

* Add proxy over ssh tunnel functionality (#198)

* update mypy to 0.750 (#204)

* Test Core Eventing (#205)

* Add core event tests

* Update .gitignore with coverage

* Add shortlink gif

* Add event dispatcher test

* Test event subscriber

* Test Dashboard backend (#206)

* Update shortlink gif name

* Conditionally run workflows as necessary

* Use pytest

* It works but github workflow is not reporting any status :(

* Separate out badges

* Add python_requires to setup.py

* Update setuptools from 42.0.1 to 42.0.2 (#207)

* Add tox.ini (#208)

* Homebrew formula (#209)

* Add homebrew formula

* Build PyPi package and Homebrew installation verification

* Check develop

* bdist_wheel reported as error: invalid command "bdist_wheel"

* Move under stable/develop folders to keep Proxy class name same

* uff

* develop installs proxy not proxy.py binary

* Prepend site-packages

* Install typing-extensions explicitly with brew

* Use find_packages

* Most likely failing due to lack of find_packages in current develop branch

* Fix windows setup.py build

* test_static_web_server_serves seems flaky on Ubuntu python 3.8

* Add instructions to install using homebrew

* Disable test_static_web_server_serves on GitHub actions, seems flaky

* Packaging (#210)

* Move docker installation steps above

* Try brewing with virtualenv

* depends on python

* Update homebrew formula for stable release

* Just test brewing on latest python

* Add support for regex based routing. Fixes #203 (#211)

* Remove public folder references (#212)

* Refactor (#213)

* Add DEFAULT_HTTP_PORT constant

* Use DEFAULT_HTTP_PORT in tests

* Refactor into exception module

* Refactor into inspector module

* Refactor into server module

* Refactor into proxy module

* Build docker of Python 3.8 (#214)

* Move homebrew under helper (#215)

* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216)

* Catch TimeoutError and OSError (host unreachable)

* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET

* Enable mccabe (#217)

* No need of per day or week stats (#218)

* Make HTTP handler constructor free of socket file number (#219)

* Refactor into acceptor module

* Add tunnel doc

* Make fileno free

* Autopep8

* Response parser now reaches COMPLETE even when no body is expected (#220)

* Stash current changes

* Refactor into connection module

* Response parser state complete when no body expect

* Raise NotImplementedError if invalid state reached within parser

* Update tox from 3.14.1 to 3.14.2 (#221)

* Update paramiko from 2.6.0 to 2.7.0 (#225)

* Update paramiko from 2.7.0 to 2.7.1 (#227)

* Proxy Pool Plugin (#228)

* Add proxy pool example. See #226

* Add ProxyPoolPlugin to doc

* Update pytest from 5.3.1 to 5.3.2 (#229)

* Update coverage from 4.5.4 to 5.0 (#230)

* Update mypy from 0.750 to 0.760 (#232)

* Update mypy from 0.760 to 0.761 (#235)

* Move manager initialization outside of top level scope.  Fixes #233 (#236)

* Share lock to acceptors via pool (#238)

* Move manager initialization outside of top level scope.  Fixes #233

* Share lock to acceptor via pool

* Optionally initialize manager in main thread and use the same for EventQueue initialization (#239)

* Highlight language syntax (#240)

* Highlight lang syntax

* zsh prompt

* Update coverage from 5.0 to 5.0.1 (#241)

* Integration testing (#243)

* Add tests for public/private/csr generation

* Add integration testing skeleton for mac and ubuntu

* Merge integration within lib test to avoid too many workflows

* Disable integration testing on windows for now

* Use sudo to start integration test script as lsof fails on MacOS.

lsof: WARNING: can't stat() vmhgfs file system

* Add basic integration testing for now to assert proxy works as expected when started out of develop branch

* Add a call to inbuilt http server to verify it works

* wait for server to accept requests

Co-authored-by: pyup.io bot <[email protected]>

* Add github workflow badges

Co-authored-by: pyup.io bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants