Investigate slow EVALs

[hannahwhy]

https://github.com/ArchiveTeam/universal-tracker/blob/master/models/tracker/transactions.rb#L114 is part of a code path that's executed on every item request.

In the swipnet project, the tracker is experiencing loads of ~16700 req/min. INFO ALL on the Redis server indicates that EVAL is being called quite a lot, and it's also one of the slowest operations:

# Commandstats
cmdstat_hexists:calls=2591116,usec=4303236,usec_per_call=1.66
cmdstat_zcard:calls=369383,usec=763072,usec_per_call=2.07
cmdstat_sadd:calls=7041202,usec=15939036,usec_per_call=2.26
cmdstat_scard:calls=4538658,usec=11051608,usec_per_call=2.43
cmdstat_get:calls=13678184,usec=34801535,usec_per_call=2.54
cmdstat_spop:calls=3293991,usec=9114550,usec_per_call=2.77
cmdstat_rpush:calls=368895,usec=1138723,usec_per_call=3.09
cmdstat_hdel:calls=373647,usec=1442246,usec_per_call=3.86
cmdstat_llen:calls=309,usec=1213,usec_per_call=3.93
cmdstat_lrange:calls=243,usec=967,usec_per_call=3.98
cmdstat_smove:calls=1,usec=4,usec_per_call=4.00
cmdstat_monitor:calls=1,usec=4,usec_per_call=4.00
cmdstat_select:calls=103,usec=425,usec_per_call=4.13
cmdstat_incr:calls=3117545,usec=13831391,usec_per_call=4.44
cmdstat_expire:calls=2771164,usec=12423965,usec_per_call=4.48
cmdstat_hget:calls=8486772,usec=41284817,usec_per_call=4.86
cmdstat_srandmember:calls=378908,usec=2124885,usec_per_call=5.61
cmdstat_sismember:calls=13931606,usec=81017579,usec_per_call=5.82
cmdstat_hset:calls=869270,usec=5247782,usec_per_call=6.04
cmdstat_publish:calls=368895,usec=2429089,usec_per_call=6.58
cmdstat_lpop:calls=370478,usec=2699604,usec_per_call=7.29
cmdstat_set:calls=531,usec=3911,usec_per_call=7.37
cmdstat_subscribe:calls=2,usec=16,usec_per_call=8.00
cmdstat_hincrby:calls=1874314,usec=16261654,usec_per_call=8.68
cmdstat_srem:calls=2559767,usec=32941778,usec_per_call=12.87
cmdstat_zrem:calls=373643,usec=5019324,usec_per_call=13.43
cmdstat_zadd:calls=397964,usec=6712297,usec_per_call=16.87
cmdstat_append:calls=14503,usec=316858,usec_per_call=21.85
cmdstat_hkeys:calls=492,usec=11773,usec_per_call=23.93
cmdstat_slowlog:calls=9,usec=312,usec_per_call=34.67
cmdstat_smembers:calls=672,usec=27011,usec_per_call=40.19

# here
cmdstat_eval:calls=3443578,usec=287902307,usec_per_call=83.61

cmdstat_hvals:calls=711,usec=67559,usec_per_call=95.02
cmdstat_info:calls=268,usec=92237,usec_per_call=344.17
cmdstat_hgetall:calls=1416,usec=529464,usec_per_call=373.92
cmdstat_keys:calls=2608,usec=1246060,usec_per_call=477.78
cmdstat_zrangebyscore:calls=1,usec=3433,usec_per_call=3433.00
cmdstat_del:calls=354,usec=1853681,usec_per_call=5236.39
cmdstat_zrange:calls=52,usec=713398,usec_per_call=13719.19

There are slower operations, but they are not called nearly as often.

Some possibilities:

1. The scripts are slow.
2. Continuously EVALing the script is slow. In this case we might be able to speed things up via EVALSHA.