HIGH: [Performance] Stop logging huge ID lists when nothing changes

[quinthar]

Problem:

When merging an Onyx value, it can sometimes log huge lines -- even if nothing changes. This is not just confusing when looking at the logs, but is probably slowing down the app (because logging is in general slow, and logging a 15KB logline pointlessly isn't great, especially when using "local logging" in mobile, which stores everything to Onyx, crippling the app after a few minutes).

Solution:

Only log the actual keys that change. I think @marcaaron might have already done this as mentioned here, but tracking it in this issue.

[melvin-bot]

Auto-assigning issues to engineers is no longer supported. If you think this issue should receive engineering attention, please raise it in #whatsnext.

[quinthar]

We've been talking about the client, but I see a similar enormous log line on the server, basically all the time:

2024-03-25T07:22:15.751472+00:00 staging-www1.lax php-fpm: 869d26180dc3680b-SEA /api.php [email protected] !ecash1.4.56-2! ?api? [info] [App] 2024-03-25T07:22:04.043Z - [info] [Onyx] merge() called for key: personalDetailsList properties: 1,1540,26328,29512,163559,193895,262345,262521,281396,290555,297463,308305,623727,665613,666702,673555,704993,778531,908447,968366,1054392,1075588,1109065,1130031,1145365,1146648,1153269,1202885,1214769,1296494,1309498,1310739,1330212,1346672,1350085,1353644,1377139,1381556,1520269,1526891,1547322,1556965,1658455,1773169,1773174,1773704,1812400,1812411,1944125,1958521,2012843,2074551,2082326,2125302,2129354,2147098,2147106,2147108,2147109,2147608,2147610,2150159,2230455,2234069,2254404,2257929,2257951,2257955,2257956,2257971,2258735,2267637,2278174,2285584,2285592,2286227,2288376,2312936,2333569,2339290,2366079,2380688,2380691,2392416,2407434,2419878,2421277,2421296,2455152,2462715,2462987,2473134,2474028,2495229,2536717,2553884,2553893,2554348,2554384,2562137,2568593,2568747,2568767,2598123,2600072,2600113,2622494,2622527,2630849,2630854,2636505,2638074,2638076,2638079,2638084,2638088,2642308,2644832,2644835,2644838,2644839,2663922,2663923,2663925,2671946,2675048,2682632,2682641,2682647,2697693,2746214,2746215,2746219,2746220,2746223,2746226,2746397,2761578,2761580,2761581,2761583,2761588,2761589,2762690,2762691,2762692,2777229,2779336,2782966,2782976,2782978,2782980,2820722,2820725,2832690,2832694,2832697,2832700,2833492,2840332,2842284,2842286,2842291,2842292,2875893,2875905,2890011,2890015,2890020,2890021,2890026,2890029,2890030,2890032,2912685,2921878,2921880,2921881,2921884,2921885,2921893,2921895,2921896,2937132,3000672,3003963,3007366,3008390,3064822,3067863,3079111,3079112,3092322,3107470,3126513,3149650,3155852,3171124,3216967,3243510,3275542,3284413,3299225,3300485,3371395,3371396,3431989,3432223,3453161,3459014,3472166,3472167,3554321,3560518,3560520,3560522,3560523,3560524,3563764,3571282,3592969,3624311,3624313,3624316,3624318,3624319,3624364,3627820,3657342,3691204,3724545,3737330,3742716,3750411,3772464,3806748,3857556,3895429,3917107,3955465,3987530,4024333,4037741,4045531,4048249,4054862,4054864,4054867,4054872,4054873,4055588,4057821,4083363,4083424,4087621,4087623,4102607,4102609,4111825,4111830,4111834,4111836,4121216,4124718,4124719,4124721,4124722,4124724,4124725,4124729,4124734,4129170,4129445,4129489,4132519,4132522,4135687,4139013,4149574,4149576,4152127,4153074,4168739,4168743,4168747,4169270,4173774,4177157,4182323,4187104,4187530,4187660,4196858,4197777,4197778,4197782,4197784,4197785,4197786,4197789,4197791,4203957,4212090,4212885,4212917,4217718,4226839,4241105,4241106,4241109,4252732,4274992,4274993,4274998,4275001,4275002,4304005,4308155,4308156,4308157,4308159,4314163,4314164,4314165,4314166,4314330,4319448,4340379,4374026,4419606,4451796,4456173,4490297,4519931,4536616,4536618,4674600,4784620,4787773,4795409,4797160,4883492,4906852,4974958,4982620,5000634,5000635,5000637,5000639,5000640,5000641,5000645,5013164,5024251,5024253,5027672,5044284,5070537,5081772,5081775,5081781,5090992,5094346,5127323,5127325,5129322,5162183,5185278,5213911,5213915,5213916,5222380,5268244,5307064,5307067,5307068,5312385,5318345,5340414,5356024,5356092,5360919,5361806,5361811,5361817,5361821,5361823,5361825,5365698,5382669,5450325,5450330,5450490,5452713,5456332,5456478,5461563,5461678,5474970,5477629,5477916,5488640,5488747,5492979,5497918,5521769,5532429,5534662,5535718,5542376,5543529,5546797,5548257,5574985,5574988,5574993,5601540,5613175,5614582,5622000,5622001,5622005,5622611,5622942,5625161,5626032,5626033,5626036,5630312,5635513,5651834,5651866,5676965,5676967,5676968,5676970,5688485,5689352,5689382,5697051,5726973,5733301,5735188,5757820,5761760,5771100,5801774,5827150,5857685,5957886,5975923,6029384,6042925,6147370,6179491,6211978,6224991,6242578,6317681,6427408,6427835,6433008,6433015,6433017,6433019,6433021,6433022,6455346,6533726,6626667,6641964,6692819,6722215,6722224,6746318,6762694,6783249,6783250,6783261,6789156,6794290,6795270,6795273,6795274,6795286,6795288,6798658,6810103,6810118,6851710,6857321,7141697,7172565,7182988,7193407,7259014,7267435,7365200,7464428,7467869,7494755,7507267,7510573,7520197,7545812,7612310,7643215,7648879,7663412,7697132,7753800,7760601,7811534,7811560,7868906,7875768,7906268,7912970,8033800,8077239,8125084,8164232,8205996,8218766,8264813,8279983,8322850,8347405,8392101,8435593,8437429,8440563,8443973,8505565,8536674,8572718,8595733,8668412,8706477,8762744,8788142,8803681,8827839,8829116,8838175,8843294,8844271,8846861,8889303,8889320,8889323,8892974,8892975,8900169,8903736,8913736,8931541,8932118,8959101,9065528,9161529,9292530,9301990,9339753,9365603,9375152,9606781,9641191,9645353,9662119,9663820,9663824,9663825,9663826,9666842,9666844,9666846,9670219,9675014,9679724,9683670,9693329,9713816,9718513,9722304,9756185,9847009,9852915,9861825,9872152,9897723,9913826,9928877,9941005,9943207,9943210,9951125,9979114,10009871,10013133,10023049,10027416,10032677,10063272,10085857,10123814,10145166,10160771,10185791,10192797,10199254,10199263,10205731,10229124,10243096,10252523,10265087,10267258,10345041,10362760,10370920,10398656,10431810,10449177,10476956,10494364,10513888,10521045,10564832,10580317,10588864,10612505,10645089,10692374,10708766,10742289,10751130,10763682,10784430,10797322,10888063,10899789,10903701,11007914,11011338,11017746,11023767,11028350,11037775,11059009,11099123,11123323,11146056,11162460,11181824,11192813,11226634,11253557,11268064,11275076,11310352,11321336,11341541,11399964,11411519,11411625,11443900,11447813,11462583,11467882,11610921,11665625,11685583,11700721,11923112,11923438,11923508,11924214,11939857,11944253,11944375,11944413,11944472,12006592,12009975,12025014,12032067,12032073,12068359,12100914,12100925,12107120,12107126,12123908,12140471,12140568,12140584,12142019,12142024,12142046,12142087,12142093,12142101,12142122,12142129,12142133,12142145,12142153,12142162,12142174,12142179,12142191,12142215,12142233,12142337,12142340,12142348,12142362,12142370,12144259,12169077,12182129,12198804,12213987,12255306,12281353,12284283,12297508,12343542,12358160,12371319,12415483,12425982,12521911,12551045,12552541,12717247,12739478,12741034,12743033,12743090,12753033,12771701,12782882,12784939,12807549,12807563,12807585,12807597,12807690,12843258,12843261,12849223,12861117,12862931,12863805,12867196,12875748,12879666,12887476,12892581,12892587,12892594,12892604,12892609,12892614,12936603,12963576,12975514,12979964,12980512,13033106,13040813,13040823,13043885,13050342,13058496,13061917,13064762,13066249,14248724,14257381,14258384,14262424,14262553,14266263,14266364,14266390,14272494,14274821,14280554,14281886,14281893,14284420,14301011,14303991,14306122,14307210,14325128,14331906,14331908,14335339,14340877,14341247,14341248,14341249,14341250,14341251,14355615,14357819,14360437,14362248,14362250,14362260,14362357,14364505,14369837,14370273,14389556,14390071,14401615,14409945,14412040,14416162,14417642,14418251,14418913,14418914,14419704,14422525,14422575,14423185,14423242,14425592,14443274,14456780,14467745,14467746,14476127,14477200,14479293,14488113,14495662,14497695,14
2024-03-25T07:22:15.751500+00:00 staging-www1.lax php-fpm: 869d26180dc3680b-SEA /api.php [email protected] !ecash1.4.56-2! ?api? [info] [App] 500310,14522155,14523381,14526867,14526868,14527240,14527607,14527610,14527719,14537178,14539755,14549682,14549683,14549684,14549686,14549687,14549688,14549689,14549690,14549691,14549693,14549694,14549696,14560708,14560788,14565944,14574511,14575437,14575438,14575439,14580972,14585848,14585852,14590729,14600511,14601612,14602540,14602541,14602542,14602543,14602544,14602545,14603032,14607191,14608598,14610081,14610890,14613843,14614768,14614885,14619292,14620019,14620042,14620047,14620051,14620078,14622223,14622224,14622225,14622226,14627311,14627312,14627314,14627315,14627316,14628264,14628495,14631765,14632875,14636344,14637016,14640421,14649441,14654993,14656282,14656465,14658592,14674740,14679198,14680732,14681712,14685188,14685639,14688511,14689143,14691139,14703795,14709426,14709969,14710663,14713055,14716344,14732838,14738502,14753322,14755457,14755994,14762342,14764682,14766183,14766716,14767095,14768317,14779826,14798973,14799011,14803613,14811477,14826314,14826403,14826554,14828775,14829202,14832164,14835384,14845926,14846769,14846809,14855957,14856005,14856179,14857502,14859205,14860679,14861492,14863651,14867798,14869493,14872500,14876225,14880239,14880240,14881873,14882824,14885424,14886071,14886139,14886316,14886327,14886444,14887069,14887165,14887626,14891057,14891189,14891516,14892658,14893170,14895064,14895108,14895243,14895385,14896150,14896331,14896420,14896425,14897021,14898196,14898463,14898617,14899126,14899161,14899517,14899707,14900844,14903052,14904734,14906492,14907653,14908000,14908098,14908606,14908969,14909041,14912267,14913876,14913882,14913970,14917062,14918357,14920991,14921139,14928563,14929194,14930801,14932568,14932783,14936719,14937632,14937664,14937669,14944892,14952446,14961336,14974237,15011064,15016557,15025599,15046794,15051278,15070634,15075484,15075601,15089790,15102661,15107571,15110652,15123469,15127242,15154446,15162084,15191541,15192136,15193474,15194729,15194930,15218865,15221531,15221569,15223335,15236576,15236708,15244952,15250571,15252830,15261989,15267275,15304044,15305309,15307184,15310455,15313597,15320017,15320018,15320019,15320020,15320021,15320022,15320023,15320024,15320025,15320026,15320028,15320034,15320035,15320036,15320037,15320038,15320039,15320040,15320041,15320042,15320079,15543627,15544884,15549194,15568553,15586896,15595185,15602814,15602835,15602839,15602841,15602842,15602847,15602852,15628387,15628663,15630219,15661216,15667563,15693640,15724048,15737155,15737921,15737923,15738796,15739022,15739052,15742040,15742048,15742097,15742112,15742888,15742896,15765012,15802250,15819632,15839088,15852820,15853042,15855081,15855089,15856026,15856082,15856090,15856133,15856145,15856146,15856172,15856183,15856223,15856264,15856273,15856315,15856322,15857523,15857659,15861944,15861993,15863418,15864555,15869738,15870681,15870686,15870695,15870701,15870704,15870715,15870731,15889203,15889295,15900278,15901453,15902172,15903602,15909161,15915203,15915207,15915209,15915390,15915908,15915995,15917315,15918674,15918818,15919781,15924805,15930100,15939596,15949907,15955276,15963635,15963637,15963657,15970235,15979536,15981793,15983228,15984522,15986173,15989886,15992545,15992894,15997169,16009344,16009362,16009389,16009413,16009483,16009503,16009531,16009561,16009595,16012621,16013156,16013858,16021408,16032013,16040059,16059166,16061227,16067348,16130894,16155948,16175199,16204002,16204005,16215968,16241556,16355733,16360852,16380854,16398726,16408064,16421508,16461184,16461194,16461196,16461202,16461207,16464559,16492628,16493062,16518810,16553371,16553408,16553957 ~~ parameters: '' userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36'

[quinthar]

Is that also misleading? It really looks like it's returning an enormous (99% unchanged) dataset, are you sure it's not?

[quinthar]

Being worked on here: Expensify/react-native-onyx#517

[marcaaron]

yes, they are the same log. The client logs in the JS console also get sent to the server.

[marcaaron]

This should have gone out with #38114 so on staging, but not production yet. Checked staging and see this:

[info] [Onyx] merge called for key: personalDetailsList properties: 7520197 hasChanged: false - ""

So - seems to be working 🎉

[quinthar]

is this done? Can it be closed?

[marcaaron]

Yep, let's close it out.