'Purity Test' unexpected results / flawed expectations

[qacollective]

As I mentioned in #47 , I've been beginning by conducting some 'purity tests' on that pull request.

Note that I have disabled FIFO by commenting out the lines I referred to earlier on in the comments on that PR.

Expected Behavior

It may be my expectations are flawed, but that is part of this test 😜

Generally speaking my expectations are that with a set of 20 orders already placed (in the init() function) and one oscillation of the market, all orders should result in positions being taken and take profit ... ending with a neat little bow on top.

I'm also expecting that the numbers that result from print(output) somewhat reveal the overly simple and pure scenario that preceded it - with a lot of figures that may have even numbers and numbers with relatively clear relationships to one another.

The code:

from backtesting import Backtest, Strategy

from backtesting.lib import crossover
from backtesting.test import SMA, PureSimpleData

from backtesting.lib import crossover

from pprint import pformat

class PureSimple(Strategy):
    
    def init(self):
        tradeCount=0
        for price in range(1,12,1):
            if price < 11:
                self.buy(size=1,limit=float(price),tp=price+1.0)
                print(f" BUY@{price}, tp@{price+1}")
                tradeCount+=1

            if price > 1:
                self.sell(size=1,limit=float(price),tp=price-1.0)
                print(f"SELL@{price}, tp@{price-1}")
                tradeCount+=1
        print(f"{tradeCount} trades placed.")

    def next(self):
        pass

bt = Backtest(PureSimpleData, PureSimple, cash=10000, commission=.002)
output = bt.run()
bt.plot()
print(output)

That should place the following trade orders before the first OHLC is processed:

• BUY@1, tp@2
• BUY@2, tp@3
• SELL@2, tp@1
• BUY@3, tp@4
• SELL@3, tp@2
• BUY@4, tp@5
• SELL@4, tp@3
• BUY@5, tp@6
• SELL@5, tp@4
• BUY@6, tp@7
• SELL@6, tp@5
• BUY@7, tp@8
• SELL@7, tp@6
• BUY@8, tp@9
• SELL@8, tp@7
• BUY@9, tp@10
• SELL@9, tp@8
• BUY@10, tp@11
• SELL@10, tp@9
• SELL@11, tp@10

The data being used for market movements is:

,Open,High,Low,Close
1/01/2020 12:01,1.25,1.99,0.99,1.75
1/01/2020 12:02,2.25,2.99,1.99,2.75
1/01/2020 12:03,3.25,3.99,2.99,3.75
1/01/2020 12:04,4.25,4.99,3.99,4.75
1/01/2020 12:05,5.25,5.99,4.99,5.75
1/01/2020 12:06,6.25,6.99,5.99,6.75
1/01/2020 12:07,7.25,7.99,6.99,7.75
1/01/2020 12:08,8.25,8.99,7.99,8.75
1/01/2020 12:09,9.25,9.99,8.99,9.75
1/01/2020 12:10,10.25,10.99,9.99,10.75
1/01/2020 12:11,9.25,9.99,8.99,9.75
1/01/2020 12:12,8.25,8.99,7.99,8.75
1/01/2020 12:13,7.25,7.99,6.99,7.75
1/01/2020 12:14,6.25,6.99,5.99,6.75
1/01/2020 12:15,5.25,5.99,4.99,5.75
1/01/2020 12:16,4.25,4.99,3.99,4.75
1/01/2020 12:17,3.25,3.99,2.99,3.75
1/01/2020 12:18,2.25,2.99,1.99,2.75
1/01/2020 12:19,1.25,1.99,0.99,1.75

Actual Behavior

1. It is not clear to me why a position is not taken from the very first candle - I'm assuming that the dotted lines are deals closed?
2. It is not entirely clear to me why only 16 trades are shown. I expected at least 18 and as you can see from my code and the trade request list it generates, I was aiming for 20 orders placed and 20 deals closed. Again, I'd not be surprised if I've missed something here.
3. The numbers printed at the end aren't as clear cut as I'd have expected ... but again, perhaps I've expected too much simplicity, perhaps I've indeed picked up on something or perhaps the results could do with some more fundamentals in there too (potentially more absolute values rather than %ages).

Zip file of the chart produced:
PureSimple.zip

This is the output that is printed:

Start                     2020-01-01 12:01:00
End                       2020-01-01 12:19:00
Duration                      0 days 00:18:00
Exposure Time [%]                     89.4737
Equity Final [$]                      10046.1
Equity Peak [$]                       10046.1
Return [%]                           0.418744
Buy & Hold Return [%]                       0
Max. Drawdown [%]                   -0.160377
Avg. Drawdown [%]                   -0.160377
Max. Drawdown Duration        0 days 00:06:00
Avg. Drawdown Duration        0 days 00:06:00
# Trades                                   16
Win Rate [%]                              100
Best Trade [%]                        354.646
Worst Trade [%]                       12.0192
Avg. Trade [%]                        111.359
Max. Trade Duration           0 days 00:15:00
Avg. Trade Duration           0 days 00:07:00
Expectancy [%]                            NaN
SQN                                   4.88235
Sharpe Ratio                         0.944904
Sortino Ratio                             NaN
Calmar Ratio                          694.362
_strategy                          PureSimple
_equity_curve                             ...
_trades                       Size  EntryB...
dtype: object

Steps to Reproduce

1. Run the code
2. With the data
3. Have the same possibly flawed expectations! 🌝

Additional info

• Backtesting version: Order/Trade/Position API #47 with lines 824 to 842 commented out
• More tests to come soon hopefully!

[kernc]

Apologies for the delayed response. I much appreciate your close inspection!

1. It is not clear to me why a position is not taken from the very first candle - I'm assuming that the dotted lines are deals closed?

Seems to be due to these lines in Backtest.run():

backtesting.py/backtesting/backtesting.py

Lines 699 to 702 in 1715c91

	# Skip first few candles where indicators are still "warming up"
	# +1 to have at least two entries available
	start = 1 + max((np.isnan(indicator.astype(float)).argmin(axis=-1).max()
	for _, indicator in indicator_attrs), default=0)

I think the "+1 to have at least two entries available" is there as a simple workaround to have "previous close" certainly accessible:

backtesting.py/backtesting/backtesting.py

Line 484 in 1715c91

price = self._data.Close[-2]

The dotted lines indeed mark the duration of trades.

2. It is not entirely clear to me why only 16 trades are shown. I expected at least 18 and as you can see from my code and the trade request list it generates, I was aiming for 20 orders placed and 20 deals closed. Again, I'd not be surprised if I've missed something here.

In my slightly updated branch (force-pushed), 17 trades are closed. At the end, the remaining two trades and three orders are:

• trade BUY@10, tp@11 since its TP price is never hit,
• order SELL@11, tp@10 since its limit price is never hit,
• trade BUY@1, tp@2 which was just entered on the last bar instead of the first (see previous answer).

3. The numbers printed at the end aren't as clear cut as I'd have expected ... but again, perhaps I've expected too much simplicity, perhaps I've indeed picked up on something or perhaps the results could do with some more fundamentals in there too (potentially more absolute values rather than %ages).

You can dive into trades details by accessing output._trades dataframe:

    Size  EntryBar  ExitBar  EntryPrice  ExitPrice     PnL  ReturnPct           EntryTime            ExitTime Duration
0      1         1        2      2.0040       3.25  1.2460   0.621756 2020-01-01 12:02:00 2020-01-01 12:03:00 00:01:00
1      1         1        3      2.2545       4.25  1.9955   0.885119 2020-01-01 12:02:00 2020-01-01 12:04:00 00:02:00
2      1         1        4      2.2545       5.25  2.9955   1.328676 2020-01-01 12:02:00 2020-01-01 12:05:00 00:03:00
3      1         1        5      2.2545       6.25  3.9955   1.772233 2020-01-01 12:02:00 2020-01-01 12:06:00 00:04:00
4      1         1        6      2.2545       7.25  4.9955   2.215791 2020-01-01 12:02:00 2020-01-01 12:07:00 00:05:00
5      1         1        7      2.2545       8.25  5.9955   2.659348 2020-01-01 12:02:00 2020-01-01 12:08:00 00:06:00
6      1         1        8      2.2545       9.25  6.9955   3.102905 2020-01-01 12:02:00 2020-01-01 12:09:00 00:07:00
7      1         1        9      2.2545      10.25  7.9955   3.546463 2020-01-01 12:02:00 2020-01-01 12:10:00 00:08:00
8     -1         9       10     10.2295       9.00  1.2295   0.120192 2020-01-01 12:10:00 2020-01-01 12:11:00 00:01:00
9     -1         8       11      9.2315       8.00  1.2315   0.133402 2020-01-01 12:09:00 2020-01-01 12:12:00 00:03:00
10    -1         7       12      8.2335       7.00  1.2335   0.149815 2020-01-01 12:08:00 2020-01-01 12:13:00 00:05:00
11    -1         6       13      7.2355       6.00  1.2355   0.170755 2020-01-01 12:07:00 2020-01-01 12:14:00 00:07:00
12    -1         5       14      6.2375       5.00  1.2375   0.198397 2020-01-01 12:06:00 2020-01-01 12:15:00 00:09:00
13    -1         4       15      5.2395       4.00  1.2395   0.236568 2020-01-01 12:05:00 2020-01-01 12:16:00 00:11:00
14    -1         3       16      4.2415       3.00  1.2415   0.292703 2020-01-01 12:04:00 2020-01-01 12:17:00 00:13:00
15    -1         2       17      3.2435       2.00  1.2435   0.383382 2020-01-01 12:03:00 2020-01-01 12:18:00 00:15:00
16    -1         1       18      2.2455       1.00  1.2455   0.554665 2020-01-01 12:02:00 2020-01-01 12:19:00 00:17:00

[qacollective]

Hi Again! This time, its my turn to apologise for my response time - as you can probably now tell, the amount of time I get to spend doing the things I WANT to do is highly variable 😉

Thanks muchly for the explanation on why that trade wasn't getting picked up on the first candle - that 'warm up' period seems fair enough.

Mostly clear for now except one thing - in that dataframe at the end of your reply, why is the EntryPrice for rows indexed 1-7 all set at 2.2545? My expectation on all those trades would be that the EntryPrice was one dollar less than the ExitPrice at dollar increments, much like the sell orders but in reverse or course. This means that I'm making more profit than I'd expect and may explain why some of the final figures coming out in the summary don't look as 'round' as I was expecting.

One final ponderance on the resultant chart: would it be at all possible to have mouseover information showing the trades executed on those dotted lines? I only ask because I instinctively mouse over them to see more detail but don't get any 😜 I had a quick look at Bokeh documentation and couldn't see obviously how to do this, but that's literally my first look at Bokeh. If you point me in the right direction, I'm happy to look into doing that.

[kernc]

Yeah, busy year for everyone. 😃

in that dataframe at the end of your reply, why is the EntryPrice for rows indexed 1-7 all set at 2.2545?

All the orders are placed in advance in init() (a case I didn't know was supported) and they are limit orders, so they get filled on first matching bar.

would it be at all possible to have mouseover information showing the trades executed

Does the tooltip over triangles in the PL chart, aligned with dotted lines' ends, maybe do for you?

backtesting.py/backtesting/_plotting.py

Lines 373 to 399 in ff97188

	def _plot_pl_section():
	"""Profit/Loss markers section"""
	fig = new_indicator_figure(y_axis_label="Profit / Loss")
	fig.add_layout(Span(location=0, dimension='width', line_color='#666666',
	line_dash='dashed', line_width=1))
	returns_long = np.where(trades['Size'] > 0, trades['ReturnPct'], np.nan)
	returns_short = np.where(trades['Size'] < 0, trades['ReturnPct'], np.nan)
	size = trades['Size'].abs()
	size = np.interp(size, (size.min(), size.max()), (8, 20))
	trade_source.add(returns_long, 'returns_long')
	trade_source.add(returns_short, 'returns_short')
	trade_source.add(size, 'marker_size')
	if 'count' in trades:
	trade_source.add(trades['count'], 'count')
	r1 = fig.scatter('index', 'returns_long', source=trade_source, fill_color=cmap,
	marker='triangle', line_color='black', size='marker_size')
	r2 = fig.scatter('index', 'returns_short', source=trade_source, fill_color=cmap,
	marker='inverted_triangle', line_color='black', size='marker_size')
	tooltips = [("Size", "@size{0,0}")]
	if 'count' in trades:
	tooltips.append(("Count", "@count{0,0}"))
	set_tooltips(fig, tooltips + [("P/L", "@returns_long{+0.[000]%}")],
	vline=False, renderers=[r1])
	set_tooltips(fig, tooltips + [("P/L", "@returns_short{+0.[000]%}")],
	vline=False, renderers=[r2])
	fig.yaxis.formatter = NumeralTickFormatter(format="0.[00]%")
	return fig

Could add a few more fields like EntryTime and Duration. 🤔

[qacollective]

All the orders are placed in advance in init() (a case I didn't know was supported) and they are limit orders, so they get filled on first matching bar.

Ah, okay - I just looked up the textbook definition of a limit order and that seems right: instructs your broker to fill your buy or sell order at a specific price or better. I gather that's the or better part. It raises a question with my broker as to why they fill a limit order only ever at the exact price I request - rather than what I'm seeing as the textbook definition.

So I just tried changing the code to place a stop limit order in stead and that produced the set of trades I was originally expecting:

    Size  EntryBar  ExitBar  EntryPrice  ExitPrice     PnL  ReturnPct           EntryTime            ExitTime Duration
0      1         1        2      2.0040       3.25  1.2460   0.621756 2020-01-01 12:02:00 2020-01-01 12:03:00 00:01:00
1      1         2        3      3.0060       4.25  1.2440   0.413839 2020-01-01 12:03:00 2020-01-01 12:04:00 00:01:00
2      1         3        4      4.0080       5.25  1.2420   0.309880 2020-01-01 12:04:00 2020-01-01 12:05:00 00:01:00
3      1         4        5      5.0100       6.25  1.2400   0.247505 2020-01-01 12:05:00 2020-01-01 12:06:00 00:01:00
4      1         5        6      6.0120       7.25  1.2380   0.205921 2020-01-01 12:06:00 2020-01-01 12:07:00 00:01:00
5      1         6        7      7.0140       8.25  1.2360   0.176219 2020-01-01 12:07:00 2020-01-01 12:08:00 00:01:00
6      1         7        8      8.0160       9.25  1.2340   0.153942 2020-01-01 12:08:00 2020-01-01 12:09:00 00:01:00
7      1         8        9      9.0180      10.25  1.2320   0.136616 2020-01-01 12:09:00 2020-01-01 12:10:00 00:01:00
8     -1         9       10     10.2295       9.00  1.2295   0.120192 2020-01-01 12:10:00 2020-01-01 12:11:00 00:01:00
9     -1         8       11      9.2315       8.00  1.2315   0.133402 2020-01-01 12:09:00 2020-01-01 12:12:00 00:03:00
10    -1         7       12      8.2335       7.00  1.2335   0.149815 2020-01-01 12:08:00 2020-01-01 12:13:00 00:05:00
11    -1         6       13      7.2355       6.00  1.2355   0.170755 2020-01-01 12:07:00 2020-01-01 12:14:00 00:07:00
12    -1         5       14      6.2375       5.00  1.2375   0.198397 2020-01-01 12:06:00 2020-01-01 12:15:00 00:09:00
13    -1         4       15      5.2395       4.00  1.2395   0.236568 2020-01-01 12:05:00 2020-01-01 12:16:00 00:11:00
14    -1         3       16      4.2415       3.00  1.2415   0.292703 2020-01-01 12:04:00 2020-01-01 12:17:00 00:13:00
15    -1         2       17      3.2435       2.00  1.2435   0.383382 2020-01-01 12:03:00 2020-01-01 12:18:00 00:15:00
16    -1         1       18      2.2455       1.00  1.2455   0.554665 2020-01-01 12:02:00 2020-01-01 12:19:00 00:17:00

Does the tooltip over triangles in the PL chart, aligned with dotted lines' ends, maybe do for you?

Ahh yes they work fine, thankyou! That should be okay and will be interesting to see how the chart looks when we start testing it with 1 minute candles with multiple trades per minute.

Could add a few more fields like EntryTime and Duration. ::thinking::

Potentially some additional info in there would be handy, duration yes, perhaps EntryPrice, ExitPrice, PnL? Or maybe there could be an option to simply include a linked DataTable which seems to automatically synchronise click events between plot and table?
https://docs.bokeh.org/en/latest/docs/user_guide/interaction/widgets.html?highlight=data%20table#datatable

In a little while, I hope to test backtesting.py using some real world tick data (rolled up to OHLC of course) and real world trades and see how closely backtesting.py reproduces how we've actually seen our trades pan out! 👍 💪 😄

[qacollective]

@kernc unless you disagree, I figure this issue can be closed? If there are any feature suggestions you'd like me to write up as a result of the discussion here, I'm happy to write them up. I may even attempt a few myself with time!