TODO
====

* Remove genshi module? Last release in January 2014!
* upload:

  - add Q1 and Q3: keys 'q1' and 'q3'
  - "description"

* bm_python_startup: add again python metadata, to get back the Python version
  in benchmark suite metadata. Same for bm_hg_startup.
* Remove compare command or reimplement it using pyperf compare_to
* Add a --log option to create a log file. Use the logging module
  and replace print() with logger.error().
* Write a test to ensure that benchmarks listed in groups exist
* Decide if sqlalchemy benchmarks should only benchmark SELECT ALL
  or INSERT+SELECT?
* Run pep8 on Travis
* compile: add metadata on compiler option, especially LTO and PGO flags


Bugs on old CPython version
===========================

* Python 3.5.0beta2: SyntaxError in parse() of chameleon/astutil.py,
  on parsing "def func(target):". Failed commits:

  * CPython 8a8f453c5a6d

* Skip bm_django_template on Python 3 older than 3.4:
  Python 3.3.6 final: django_template: django/utils/module_loading.py:
  "from importlib.util import find_spec as importlib_find"
  ImportError: cannot import name find_spec
  CPython 75e3630c6071

* sympy uses deprecated inspect.getargspec(), function removed from 3.6a0 then
  reintroduced. TODO: fix sympy upstream?
  sympy: sympy/core/function.py
  "inspect.getargspec(cls.eval)"
  module 'inspect' has no attribute 'getargspec'
  CPython 0d30940dd256


numpy benchmarks?
=================

* https://morepypy.blogspot.fr/2016/11/vectorization-extended-powerpc-and-s390x.html
* https://bitbucket.org/plan_rich/numpy-benchmark
  fork of https://bitbucket.org/mikefc/numpy-benchmark/src to use pyperf


Port PyPy benchmarks
====================

Repository: https://bitbucket.org/pypy/benchmarks/

Different from pyperformance?

* json_bench

Todo:

* pypy_interp
* pyxl_bench

  PyPI version 1.0: https://pypi.python.org/pypi/pyxl (2012)
  Old repo: https://github.com/awable/pyxl (last commit: Feb 2013)
  Python2: https://github.com/dropbox/pyxl (latest commit: Aug 2016)
  Python3: https://github.com/gvanrossum/pyxl3 (latest commit: Apr 2016)

* sphinx

  benchmarks.py: invoke sphinx-build.py on lib/cpython-doc/
  cpython-doc/: 24 MB

* trans2_annotate
* trans2_backendopt
* trans2_database
* trans2_rtype
* trans2_source
* twisted_iteration
* twisted_names
* twisted_pb
* twisted_tcp

Deliberate choice to not add it:

* eparse: https://pypi.python.org/pypi/Monte 0.0.11 was released in 2013,
  no tarball on PyPI, only on SourceForge
* krakatau: https://github.com/Storyyeller/Krakatau is not on PyPI, but it
  seems actively developed
* slowspitfire, spitfire, spitfire_cstringio: not on PyPI
* rietveld: not on PyPy

Done:

* ai (called bm_nqueens in pyperformance)
* bm_chameleon
* bm_mako
* chaos
* crypto_pyaes
* deltablue
* django (called django_template in pyperformance)
* dulwich_log
* fannkuch
* float
* genshi_text
* genshi_xml
* go
* hexiom2
* html5lib
* mdp
* meteor-contest
* nbody_modified (called nbody in pyperformance)
* nqueens
* pidigits
* pyflate-fast (called pyflate in pyperformance)
* raytrace-simple (called raytrace in pyperformance)
* richards
* scimark_fft
* scimark_lu
* scimark_montecarlo
* scimark_sor
* scimark_sparsematmult
* spectral-norm
* sqlalchemy_declarative
* sqlalchemy_imperative
* sqlitesynth (called pyflate in sqlite_synth)
* sympy_expand
* sympy_integrate
* sympy_str
* sympy_sum
* telco


pyston benchmarks
=================

Add benchmarks from the Pyston benchmark suite:
https://github.com/dropbox/pyston-perf
and convince Pyston to use pyperformance :-)

TODO:

- django_lexing
- django_migrate
- django_template2
- django_template3_10x
- django_template3
- django_template
- fasta (it's different than pyperformance "regex_dna")
- interp2
- pyxl_bench_10x
- pyxl_bench2_10x
- pyxl_bench2
- pyxl_bench
- sre_parse_parse
- virtualenv_bench2
- virtualenv_bench

Done:

- chaos
- deltablue
- fannkuch, fannkuch_med
- nbody
- pidigits: pyston has a flat implementation, single function
- raytrace, raytrace_small: use "--width=80 --height=60" cmdline option to get
  raytrace_small profile
- richards
- sqlalchemy_imperative, sqlalchemy_imperative2, sqlalchemy_imperative2_10x:
  use --rows cmdline option to control the number of SQL rows
- sre_compile_ubench: pyperformance has a much more complete benchmark on regex