Refactor timing tests

[iritkatriel]

The idea here is to make the tests more configurable, so we can see how they scale.

These are results from a Mac. On the windows box the current test chokes to computer, I think there's a bug somewhere because these tests are not huge.

Sorted by load+exec time ratio:
                                                                  load+exec   steady state
 shared call locals consts 100 funcs, 90 vars          2.187          1.619
 shared call locals 100 funcs, 10 vars            2.166          1.277
 shared call locals 100 funcs, 90 vars            1.897          1.790
 unique call locals 100 funcs, 10 vars            1.858          1.171
 shared call locals consts 100 funcs, 10 vars          1.758          1.121
 shared call locals consts 100 funcs, 50 vars          1.638          1.301
 shared call locals 100 funcs, 50 vars            1.630          1.427
 unique call locals consts 100 funcs, 10 vars          1.497          1.126
 unique call globals consts 100 funcs, 10 vars          1.455          1.005
 shared call globals 100 funcs, 90 vars           1.304          1.011
 shared call globals 100 funcs, 50 vars           1.279          1.035
 shared call globals consts 100 funcs, 90 vars          1.256          1.014
 unique call locals consts 100 funcs, 90 vars          1.217          1.628
 unique call locals consts 100 funcs, 50 vars          1.197          1.374
 shared globals 100 funcs, 50 vars                1.193          1.014
 unique call locals 100 funcs, 50 vars            1.177          1.504
 shared call globals consts 100 funcs, 50 vars          1.174          0.997
 shared call globals consts 100 funcs, 10 vars          1.149          1.066
 shared call globals 100 funcs, 10 vars           1.129          1.043
 unique call locals 100 funcs, 90 vars            1.106          1.799
 shared globals consts 100 funcs, 90 vars          1.102          0.873
 shared globals consts 100 funcs, 50 vars          1.072          1.023
 shared globals 100 funcs, 90 vars                1.070          1.030
 unique call globals 100 funcs, 10 vars           1.070          1.069
 unique call globals consts 100 funcs, 50 vars          1.063          0.985
 shared locals consts 100 funcs, 90 vars          1.058          1.014
 unique call globals consts 100 funcs, 90 vars          1.056          0.988
 unique call globals 100 funcs, 50 vars           1.034          1.037
 unique call globals 100 funcs, 90 vars           1.011          1.007
 unique globals 100 funcs, 90 vars                0.969          1.094
 unique globals consts 100 funcs, 50 vars          0.945          1.156
 shared globals 100 funcs, 10 vars                0.942          1.088
 shared locals consts 100 funcs, 10 vars          0.932          1.090
 unique globals consts 100 funcs, 90 vars          0.914          0.979
 shared globals consts 100 funcs, 10 vars          0.911          1.016
 unique globals 100 funcs, 50 vars                0.910          0.986
 shared locals consts 100 funcs, 50 vars          0.908          1.262
 unique globals consts 100 funcs, 10 vars          0.869          0.982
 shared locals 100 funcs, 10 vars                 0.856          0.978
 unique globals 100 funcs, 10 vars                0.799          1.015
 shared locals 100 funcs, 50 vars                 0.588          1.304
 shared locals 100 funcs, 90 vars                 0.496          1.041
 unique locals 100 funcs, 10 vars                 0.371          1.159
 unique locals consts 100 funcs, 10 vars          0.347          1.224
 unique locals consts 100 funcs, 50 vars          0.207          0.995
 unique locals 100 funcs, 50 vars                 0.162          1.048
 unique locals consts 100 funcs, 90 vars          0.159          1.431
 unique locals 100 funcs, 90 vars                 0.112          1.172


Sorted by test name:
                                                                  load+exec   steady state
 shared call globals 100 funcs, 10 vars           1.129          1.043
 shared call globals 100 funcs, 50 vars           1.279          1.035
 shared call globals 100 funcs, 90 vars           1.304          1.011
 shared call globals consts 100 funcs, 10 vars          1.149          1.066
 shared call globals consts 100 funcs, 50 vars          1.174          0.997
 shared call globals consts 100 funcs, 90 vars          1.256          1.014
 shared call locals 100 funcs, 10 vars            2.166          1.277
 shared call locals 100 funcs, 50 vars            1.630          1.427
 shared call locals 100 funcs, 90 vars            1.897          1.790
 shared call locals consts 100 funcs, 10 vars          1.758          1.121
 shared call locals consts 100 funcs, 50 vars          1.638          1.301
 shared call locals consts 100 funcs, 90 vars          2.187          1.619
 shared globals 100 funcs, 10 vars                0.942          1.088
 shared globals 100 funcs, 50 vars                1.193          1.014
 shared globals 100 funcs, 90 vars                1.070          1.030
 shared globals consts 100 funcs, 10 vars          0.911          1.016
 shared globals consts 100 funcs, 50 vars          1.072          1.023
 shared globals consts 100 funcs, 90 vars          1.102          0.873
 shared locals 100 funcs, 10 vars                 0.856          0.978
 shared locals 100 funcs, 50 vars                 0.588          1.304
 shared locals 100 funcs, 90 vars                 0.496          1.041
 shared locals consts 100 funcs, 10 vars          0.932          1.090
 shared locals consts 100 funcs, 50 vars          0.908          1.262
 shared locals consts 100 funcs, 90 vars          1.058          1.014
 unique call globals 100 funcs, 10 vars           1.070          1.069
 unique call globals 100 funcs, 50 vars           1.034          1.037
 unique call globals 100 funcs, 90 vars           1.011          1.007
 unique call globals consts 100 funcs, 10 vars          1.455          1.005
 unique call globals consts 100 funcs, 50 vars          1.063          0.985
 unique call globals consts 100 funcs, 90 vars          1.056          0.988
 unique call locals 100 funcs, 10 vars            1.858          1.171
 unique call locals 100 funcs, 50 vars            1.177          1.504
 unique call locals 100 funcs, 90 vars            1.106          1.799
 unique call locals consts 100 funcs, 10 vars          1.497          1.126
 unique call locals consts 100 funcs, 50 vars          1.197          1.374
 unique call locals consts 100 funcs, 90 vars          1.217          1.628
 unique globals 100 funcs, 10 vars                0.799          1.015
 unique globals 100 funcs, 50 vars                0.910          0.986
 unique globals 100 funcs, 90 vars                0.969          1.094
 unique globals consts 100 funcs, 10 vars          0.869          0.982
 unique globals consts 100 funcs, 50 vars          0.945          1.156
 unique globals consts 100 funcs, 90 vars          0.914          0.979
 unique locals 100 funcs, 10 vars                 0.371          1.159
 unique locals 100 funcs, 50 vars                 0.162          1.048
 unique locals 100 funcs, 90 vars                 0.112          1.172
 unique locals consts 100 funcs, 10 vars          0.347          1.224
 unique locals consts 100 funcs, 50 vars          0.207          0.995
 unique locals consts 100 funcs, 90 vars          0.159          1.431

[iritkatriel]

Task manager shows Python taking up 5GB of memory with these tests running, we probably have a leak.

[gvanrossum]

Undoubtably. I left a lot of the cleanup as a TODO. :-(

How does this relate to #41? Should I merge that first or is this the same?

[iritkatriel]

This builds on top of #41. I'd merge that one because it's ready. This one I'll mark as draft, I'm not as sure about it.

[gvanrossum]

Okay, I pushed #41, so this one then reduces to just the last commit, right? Can you rebase and push?

[gvanrossum]

Maybe make the column align? I've also wondered if this should live in its own file under Tools/pyco, rather than in the tests -- we don't really want speed tests to be run as part of a typical test run.

[iritkatriel]

Yes, I'll make it a separate script and we can export a csv and then look at the number how we want.

[gvanrossum]

That would be much appreciated.

That said, I discussed the status of my branch with Mark this morning, and we agreed to pause development and try some other experiments before we go further down this road. See this comment for details.

[iritkatriel]

This leaves the unit tests intact and add a Tools script that creates a csv like this:

num_funcs,func_length,num_vars,is_locals,is_unique_names,is_vary_constants,is_call,load+exec,steady-state
100,100,10,True,True,True,False,0.48208001265181794,1.8006031261949376
100,100,10,True,True,False,False,0.6247775052002594,1.117697033467494
100,100,10,True,False,True,False,1.3275500510221705,0.5142171649652352
100,100,10,True,False,False,False,1.488203579213147,1.236673018122803
100,100,10,False,True,True,False,1.0964298904159986,1.1144771096037227
100,100,10,False,True,False,False,1.1869445891311527,1.2383130228793535
100,100,10,False,False,True,False,2.1040627265924807,3.2148957848476702
100,100,10,False,False,False,False,2.1927223029602048,1.0373691114118782
100,100,50,True,True,True,False,0.36557221939430096,1.0328338244836868
100,100,50,True,True,False,False,0.13919623326056121,0.9858244760072785
100,100,50,True,False,True,False,2.4282341610807805,1.1397928706287759
100,100,50,True,False,False,False,2.359500217657294,1.1053358397692061
100,100,50,False,True,True,False,0.953743863574252,1.0624800152760991
100,100,50,False,True,False,False,1.0122546139461015,1.1394324449555224
100,100,50,False,False,True,False,2.3998853557350706,1.056263172073379
100,100,50,False,False,False,False,2.6153347776174116,1.0890903666477845

[iritkatriel]

I see where you're coming from with the other directions. My takeaway so far is that we haven't squeezed what we can from this branch, but it would take quite a lot to, essentially, replicate what marshal does in a lazy way.

[gvanrossum]

Why not just delete the speed tests from test_pyc_new.py?

And maybe add a flag to perf_micro.py to keep the old formatting? (I'm not a data scientist and I don't want to become one -- I have no idea what people do with CSV these days. :-)

[gvanrossum]

I see where you're coming from with the other directions. My takeaway so far is that we haven't squeezed what we can from this branch, but it would take quite a lot to, essentially, replicate what marshal does in a lazy way.

We certainly could squeeze more (and research more), but I don't know if it would be sufficient, and I'd prefer not to have to much more work when we're not sure that this is the direction we want to take. So I'd like to explore a few other things before deciding to pick this back up.

[iritkatriel]

Now it's all in the Tools/ script, writes like before to the screen and optionally also a csv to a file.

[gvanrossum]

Success!