Order/Trade/Position API

[kernc]

refs #8
fixes #28
closes #77
closes #83
closes #92

New features:

• Add order sizing, pyramiding, order types (stop-limit order) (fixes Option to limit order size/exposure #3, fixes How to use backtesting.py for forex trading #10, closes Feat/order size #24, closes Is there a way to config the position size of each trade> #76, closes How to use the size parameter while buying and selling? #106),
• Strategy.orders list of active orders,
• Strategy.trades list of active trades,
• Strategy.closed_trades list of settled trades,
• Order class, representing an active order,
• Trade class, representing an active or already settled trade,
• auto-downsample plots when too much data (fixes Error when plotting a large amount of data #35),
• New stats keys (fixes Way to access equity values, individual trade returns, ... #4, fixes Track the actions of the trade strategy #29):
  • _equity_curve, equity, drawdown, and drawdown duration,
  • _trades, DataFrame of executed trades.

Breakages:

• Need to call explicit Strategy.position.close() between subsequent Strategy.buy/sell() calls to approximate previous behavior.
• Strategy.buy(price=) → Strategy.buy(limit=)
• Strategy.buy/sell() only takes keyword parameters.
• Position.open_price → Trade.entry_price
• Position.open_time → Trade.entry_time
• Strategy.orders → Strategy.orders[i]
• Backtest.plot(omit_missing=) is gone.
• Revised backtesting.lib.SignalStrategy.

[qacollective]

Awesome framework! This looks like a great PR too.

If I wanted to disable the NFA compliant FIFO functionality, would that be as simple as commenting out lines 824 to 842 in backtesting.py?

This rule only applies to brokers in the USA, so perhaps this could even be a configuration option?

[sdmovie]

one bug found:
I set cash to 10000 and strategy for NQmain whcih is 9200 to 9700 price range in Feb.Strategy check indicator and position - when sell indicator and position>=0,sell ; when buy inidicator and position<=0, buy.
several normal unsuccessful trades executed then equity fall down to 9400$.
startegy try to place an sell order whenprice is 96xx$, it's failed as expected due to insufficient cash.Then there are 3 consequent sell requests in and all fail. However the failed order si not killed but resides in self.orders.
when NQmain price fall, one order executed but other 3 still in self.orders. When a buy order in ,the short position covered.But the order in queue automatically executed and make a mess.
in _broker -- _process_order:

# If we don't have enough liquidity to cover for the order, skip it
if abs(need_size) * adjusted_price > self.margin_available * self._leverage:
    continue

should the order be removed as it fails fro insufficient cash?

# If we don't have enough liquidity to cover for the order, skip it
if abs(need_size) * adjusted_price > self.margin_available * self._leverage:
    self.orders.remove(order)
    continue

[sdmovie]

for the previous margin_available calculation prohibit buyback of short position ( and I think vice versa for sell long position). The opposite order (buy for short position, sell for long position) should not be constrained by margin_available. Opposite orders bypass margin_available check, only open (long or short) orders do.
thus avoid dealing with the "frozen cash" concept which might be more complex (issue #77).
correct me if this assumption is too simple to be practical.
temporarily I do as below:

size = order.size
if -1 < size < 1:
    #close order do not check margin_available
    if np.sign(self.position.size) + np.sign(order.size) == 0:
        size=np.sign(size)*round(abs(size)*abs(self.position.size))
    else:
        size = np.sign(size) * int((self.margin_available * self._leverage * abs(size))
                                   // adjusted_price)
        # Not enough cash/margin even for a single unit
        if not size:
            continue

[kernc]

    if np.sign(self.position.size) + np.sign(order.size) == 0:
        size=np.sign(size)*round(abs(size)*abs(self.position.size))

size= computation overrides the size/portion set by the user (could be e.g.

self.buy(size=.1)
# ...
self.sell(size=.05)  # order.size = 0.5

This also likely wouldn't work for hedging traders that run long and short trades in parallel (see #47 (comment)).

I think a better approach would be to fix margin_available() to take into account order direction.

[sdmovie]

would it be more clear to distiguish order as open(long or short) orders and close orders(long or short)? an open order leverages available cash(percentage) , and close orders count on holding positions( percentage)?

[sdmovie]

how does resample work to comply various open hours (start time unalign and segmented hours)?
US stock 9:30-16:00 - should be 9:30-10:30,10:30-11:30,...15:00-16:00
US future 17:00 -16:15 should be 17:00-18:00,... 16:00-16:15
HK stock: 9:30-12:00,13:00-16:00 -- should be 9:30-10:30,10:30-11:30,11:30-12:00,13:00-14:00 .. 15:00-16:00

[sdmovie]

what if adding optional data1,data2,data3 to Strategy, cause only strategy might use extra timeframe data (also Backtest class to pass data to strategy). brokers are not impacted.

class Strategy(metaclass=ABCMeta):
    """
    A trading strategy base class. Extend this class and
    override methods
    `backtesting.backtesting.Strategy.init` and
    `backtesting.backtesting.Strategy.next` to define
    your own strategy.
    """
    def __init__(self, broker, data, params,*,data1=None,data2=None,data3=None):
        self._indicators = []
        self._broker = broker  # type: _Broker
        self._data = data   # type: _Data
        self._params = self._check_params(params)
        self._data1=data1
        self._data2=data2
        self._data3=data3

and self.I should reindex the extra timeframe to base timeframe.
resample still used if timeframe <30min (assume Exchanges start boundary minimum :30 as far as I can see) or base timeframe >=day

in Backtest.run(), steps still count on data(base timeframe), when data's time meet data1..data3's time, signal strategy e.g. strategy.next_on_data1( )
demons below(not actual code)

for i in range(start, len(self._data)):
       # Prepare data and indicators for `next` call
       data._set_length(i + 1)
        . . .
       # Next tick, a moment before bar close
       strategy.next()
       if data1 is not None and self._data1.index[len(data1)]<=data.index[len(data)-1]:
               data1._set_length(len(data1) + 1)
               strategy.next_on_data1()
  

[sdmovie]

one more change is to enable strategy choose brokers ,thus can split backtest for future/us stock /forex etc to standalone brokers, instead of mixing up everrthing in one process/function, expect to be easier for implementation

[sdmovie]

@kernc hi, are you going to use dataframe for trades and maybe orders,positions?As I am building realtrading , would like to align with you to get minimum difference from the backtest

New stats keys (fixes #4, fixes #29):
_equity_curve, equity, drawdown, and drawdown duration,
_trades, DataFrame of executed trades.

[kernc]

how does resample work to comply various open hours (start time unalign and segmented hours)?
US stock 9:30-16:00 - should be 9:30-10:30,10:30-11:30,...15:00-16:00
US future 17:00 -16:15 should be 17:00-18:00,... 16:00-16:15
HK stock: 9:30-12:00,13:00-16:00 -- should be 9:30-10:30,10:30-11:30,11:30-12:00,13:00-14:00 .. 15:00-16:00

The only place where Backtesting.py does resampling is in Backtest.plot(superimpose=...), i.e. when plotting superimposed OHLC overlay.

backtesting.py/backtesting/_plotting.py

Line 367 in b460092

df2 = orig_df.resample(resample_rule, label='left').agg(dict(OHLCV_AGG, _width='count'))

Everywhere else we just use whatever bars are available in user-provided data and interpret them as bars in sequence.

[kernc]

what if adding optional data1,data2,data3 to Strategy, cause only strategy might use extra timeframe data (also Backtest class to pass data to strategy). brokers are not impacted.

Not in favor of extra data parameters. Resampling and data shaping is the domain of and easy enough in pandas. Note, data passed into Backtest can already contain any extra data columns (#64), and there's a recipe published for strategy testing on multiple timeframes simultaneously (backtesting.lib.resample_apply() also doesn't care about workday time offsets of different exchanges but just passes on the provided pandas' datetime offset string (or object, e.g. pd.offsets.CustomBusinessHour)).

[kernc]

@kernc hi, are you going to use dataframe for trades and maybe orders,positions?

If you mean the new mentioned dataframes stats['_equity_curve'] and stats['_trades'], then yes, they are intended as public API despite the underscore. Positions can be inferred from the _trades data. The internal orders I'd prefer to keep a list (sequence) of our backtesting.Order or Order-like objects.

[kernc]

orders/trades/positions from real broker are often dataframes or easy convertable to df

@sdmovie And dataframes, compared to simple dicts, namedtuples, or thin dataclasses are also painfully slow. I was considering it for a time, but the few advantages are minor. 😅

[kernc]

@arunavo4 Thanks. I fixed the geometric mean of negative numbers by adding +1 to contain all returns around 1 instead of 0. Seems to work and is the correct approach as returns of 1 (× 1.0 × 1.0 × ...) maintain the identity.

backtesting.py/backtesting/backtesting.py

Line 1288 in e56f758

mean_return = np.exp(np.log(1 + returns).sum() / (len(returns) or np.nan)) - 1

[kernc]

@sdmovie I fixed the stale orders issue, but the margin_available issue (i.e. #77) I ended up disagreeing with. At that point in time that is the remaining available margin. If one needs to close existing trades before placing new ones, one can call self.position.close() or run Backtest(..., exclusive_orders=True) which does it automatically.

[kernc]

@qacollective I added Backtest(..., hedging=True) switch that disables NFA FIFO behavior.

[kernc]

Anybody happen to have an idea why this fails on Python 3.5? 😕

Traceback (most recent call last):
  File "/opt/python/3.5.6/lib/python3.5/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/python/3.5.6/lib/python3.5/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/travis/build/kernc/backtesting.py/backtesting/__init__.py", line 52, in <module>
    from .backtesting import Backtest, Strategy  # noqa: F401
  File "/home/travis/build/kernc/backtesting.py/backtesting/backtesting.py", line 940
    ):
    ^
SyntaxError: invalid syntax

backtesting.py/backtesting/backtesting.py

Lines 925 to 935 in b123fba

	def __init__(self,
	data: pd.DataFrame,
	strategy: Type[Strategy],
	*,
	cash: float = 10000,
	commission: float = .0,
	margin: float = 1.,
	trade_on_close=False,
	hedging=False,
	exclusive_orders=False,
	):

[qacollective]

@kernc

Trailing commas in function declarations in combination with * were only allowed from Python version 3.6 onwards.

https://bugs.python.org/issue9232

[qacollective]

❤️

[kernc]

As a simple performance benchmark, 0.1.8 CI tests take 2min 40sec versus this PR 3min 1sec. It's the same ballpark figure, so its fine.

[kernc]

Much thanks to @qacollective, @sdmovie, and @arunavo4 for reviews and help!

[Benouare]

Hi,
About the first post of this issue : "Position.open_time → Trade.entry_time"
You should change this line, the entry_time doesnt not exist and you have to do something like this
self.data.index[self.trades[-1].entry_bar]
To get the entry time.

It confused me at the begining.
Cheers
FYI : #117