"Fast" timelord logging fix, "slow" timelord process termination fix, and configurable "plot quality lookup" warning for harvesters #18524
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… stderr in timelord_launcher.py
|
Added a check on the streamreaders to ensure they are initialized. Should help pass the CI checks. |
…king up plot qualities. It was hard coded to 8 and is now handled by latency_warning_threshold in config.yaml. Default is still 8 seconds.
thesemaphoreslim
changed the title
…ool executor are not terminated when the service is stopped.
thesemaphoreslim
changed the title
…ifically "black" hook
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose:
Resolve formatting issues with the debug.log file as it pertains to timelord/VDF Client logging. The previous method was capturing stdout/stderr using proc.communicate, which blocks output until the process is complete and then writes all data to the log. The new method will write lines as they are received. This will make it easier to visually monitor timelords as well as import the log into monitoring tools.
Added ".vs" to gitignore for Visual Studio IDEs.
Made the warning configurable regarding harvester plot lookup quality delays. The current warning threshold was hard-coded at 8 seconds. A new value called "latency_warning_threshold" was added to the harvester section of config.yaml. Farmers can increase/decrease this value as they see fit to tune their farms. Provides more configuration options for the upcoming plot format.
Fixed an issue with the "slow" bluebox timelord on Linux failing to terminate chia_timelord_worker processes when the service is shut down. The ProcessPoolExecutor (bluebox_pool) does not terminate spawned processes when the shutdown() command is issued - it only prevents new processes from being created and closes the pool. To resolve this, we kill each individual process in the pool before shutting it down.
Current Behavior:
Old Log Output from timelord_launcher:
2024-08-22T12:17:20.054 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Discriminant = -0xa9a84f534427ab8785349baeb2d044c2dba9fc7bd56102c61abca8151e8ecfa3cfca496656e29517b99f9e2902e352ae2f17253aa7ae6da640c42b68be4a5c4812a099b4e26cbdeee7e24162e852668bb490e795019732860115f9b64fca631f72c55a46984d1d800253525e27cedf932315ce1a1625e6e7cf8e80317732cae7
Got simple weso proof: 0300cc24fdc4c84d4ad80271a645f623265724fb9d7f886877e42dbdeb1ffe871871b42cc14175bf338f432688e69fed8221cda8a1d51de2ca3bdb487afbe91c892bc3b1f6f57f071e62c860c72aef7c2b8d8580a08aa7c59afd06898649bbf4a04801000000329a8dc420602632cbeff1141a66320f0e5d4611d2d0ad6c9dc6a63a73f895a7323904d31d6bc213e97d919af8336cc18ecd9ce329ddd4d5e96e082135c4024dcf98a64846cb4c9a828ee567bda8507529505aa06cb7c2d38515d2907ea3e5270100
VDF Client: Sending proof
VDF Client: Sent proof
VDF loop finished. Total iters: 14379442
VDF Client: Stopped everything! Ready for the next challenge.
2024-08-22T12:19:56.603 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: VDF Client: Discriminant = -0x88ed19d269a1e921f07da3ab94a31cd0d27c4d0ab00647ae690c2250e2b6b4fe9823c276bacb24ef5ed27769544a318dde71e9014731718fd0cb2d8dd50d2d97c6a8f574eb66519f78ccac217640fa5a1b51f21f4db114aeba0f8094f11d45b31c81cc23d5a16428c6aa2d744290c9afb9593f025d4ad0452121320b2fbb409f
Got simple weso proof: 0100d10c887b9dc57e053762ccfd34da11e6017af1dd8c8179482baa602cea77c347d73a648518831ce3b31f31bd514dad3b58eefaadf320147e3b180769e140bc0e1f1284dbe710e3eccb2260f8bf421c83a22660145f8dda0f22d38b66e85fbb1b01000200caca8e5916e70cd12d4e25cee3e73e91da056354542c4f3d5900277fe47ceec2d568946864ccae49bc7eaad6177d0d019abe755be4cdeb223e4fd0d7cfa5430c4504e59f1f0b9255f67a950195859af934cf3d1a1b3a3b8465056e16389b0c360100
VDF Client: Sending proof
VDF Client: Sent proof
VDF loop finished. Total iters: 36283122
VDF Client: Stopped everything! Ready for the next challenge.
2024-08-22T12:20:31.346 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF Client: Discriminant = -0x9088b94965debcf44946a97ae29a677cbafa9be82409c5537d060cfcbd9dd0c0090b54378d869dc14e55554ef8d22beeb8a76eac448030dc80ead5cafa445dfaa5d8c42f56ec592e4c2899ecc349427ba129993894a1d87359fd166344748ae9367ea8354cf59b8dcb8c5eaf86199fcedd3ee8c9dc19afa8772d8c98aff3181f
Got simple weso proof: 03008999008d91be61aa6f624caea08e5c2c896cf1c8535752ed94be672997e0d61b3d26d1057db0007e184ea2b01c4daaf7c0790e2127d6e64c3665fa614b01b1160b2157030d261dde001b23fd2baf8033ba0a36dc8db37291bc3ad67fc9aba31c020100008a4676af3d334a8c745a667c9078cff1353e43474a1982256628156ca83db84a24efd7c0669601e8fe14caa2759c56e14a8e00d20a74a9c0c36ef2bf308bad2129cee25adf6140564c2d249dcfd8e7aa8d15b7a05e31b53c4489c1aa880d54170100
VDF Client: Sending proof
VDF Client: Sent proof
VDF loop finished. Total iters: 40378893
VDF Client: Stopped everything! Ready for the next challenge.
New Behavior:
New Log Output from timelord_launcher:
2024-08-24T05:11:10.932 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF Client: Discriminant = -0xd07400b6fe4bb1a4d3b4b8044508bb26892b016319595c36df707fb39947473e969f66b8ccb7ca35a926febbc4fb300a9275a488c00a5397167b37bf0342cea47c94bfea13c07e3a75d1d0ec02bb2474591fe012f383e8b445528953b50c2127fc3569ff86066e1fd8491697179110de890aa19862b91cb5ca9a6398245d35e7
2024-08-24T05:11:10.935 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: VDF Client: Discriminant = -0x85c7587b2701e2c972c96d42a3595bf70f16f8411f3b645628c81ed6e8d485a00394c1594f321083b9914d253a745de1cfbf8c2b6891529835592d879289154ecf9b00f20b8489241fe6066c7a0c94e5c3d140b439d5aea0bb06b899c63039fb3779176b377629b70fa6a0496dc455abb5672da63c3df49b6712c380c83abc17
2024-08-24T05:11:10.939 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Discriminant = -0xd07400b6fe4bb1a4d3b4b8044508bb26892b016319595c36df707fb39947473e969f66b8ccb7ca35a926febbc4fb300a9275a488c00a5397167b37bf0342cea47c94bfea13c07e3a75d1d0ec02bb2474591fe012f383e8b445528953b50c2127fc3569ff86066e1fd8491697179110de890aa19862b91cb5ca9a6398245d35e7
2024-08-24T05:17:04.432 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: Got simple weso proof: 0100a262d41e0e6548645fdb3fed02d2467752235ee9be84065185429764f561ce4bd93bafa4d2fee2e5bf09f974e8a315bcc959d0ccd515d509b3d2955ccd77e300a5b5eba6f1bbd9f5f2121189fe9974b719c32ad713f3a2d3566e5bf856971600181302007ee2792129e292f0116b008febcade7d445d7e78a7eb74f53ad2d5c455d6b740ea4ad5e6359f4eaef86e64203acfc1636c8d8fbcfa60aeb2419c2cfd302f5e42b90b5034c773b16bf4503f070b7ff3ca57e3747d9e996ff83e3169dee688882c0100
2024-08-24T05:17:04.533 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF Client: Sending proof
2024-08-24T05:17:04.633 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF Client: Sent proof
2024-08-24T05:17:04.734 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF loop finished. Total iters: 49473707
2024-08-24T05:17:04.835 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF Client: Stopped everything! Ready for the next challenge.
2024-08-24T05:17:04.986 TLauncher chia.timelord.timelord_launcher: INFO VDF client 0: VDF Client: Discriminant = -0x95bafb45e4694594b57e5673d5a9663aa73e59289f8d6fbf0fc52af6cd09b96430fe79cb020959757f4fb2b2fc389ac3fda2f7e5a93a6a7f3b88e68cf8f078385ff6920d5fff7420e819e679faf79c54894551ff0ab6372c9be8c76a45f1b0c91cc4149b1bd86902127be818fb620ed694e280bd868111e80ea83e612b975617
2024-08-24T05:18:09.563 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: Got simple weso proof: 00007e2e97f4b20590e71b7935ac271ebc86c1ecad6d12e7392489ec8c4dce6f2edee97998bb7e7ad8dce0dfcba7d329e5e9952c0326e94bcd178fdbc9c8efd2a22fdf92aafdc8e7ab0fac4d0c1a2adf844047cdddd29faf1fab893430e8fb935433010001001c757a8ee6113c9ea57d7873c8f779371ef898e9846d0b168394bf15f46d991c604137f651892a4d937f678ee2ab2c9ff1612ad6d61990114ef8460bd0e8400fa15c9a953214f70ff399e9a85cbea97bfa4df3a5429a4e8d577ba11a4c5b74060400
2024-08-24T05:18:09.664 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Sending proof
2024-08-24T05:18:09.764 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Sent proof
2024-08-24T05:18:09.865 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF loop finished. Total iters: 58846227
2024-08-24T05:18:09.966 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Stopped everything! Ready for the next challenge.
2024-08-24T05:18:10.089 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Discriminant = -0xbed7f968547e67a6dcbde8692344789e18bf440dc5791d92490d5e7e5842f1a5bd48566dfcf66ad35797de1ceb82c85c1bf120f3daaf5310c238cc48b86e738746a39e81dee1fb0562ee4af509fadc071c8f5857a0cd1f4fa58956e0202938afc59ad7ea9fa140ba578b913a0be27c166d37ad252b6d4d6deda721b0f9d7b8e7
2024-08-24T05:32:30.695 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: Got simple weso proof: 03002960ebc7cc189e601d1d322bdc6012485ad4ed25069e724bc81bb5ad0afb5b21c63d6308be4628f98fedac9688f0d03d719d5ee43f4a8a3a8c5f3fb90b89431b02263f0f7290838792d8156998b72abea177852409057f5502b796aba63c2d1b01000100ff213ad86ff40963a184720ae9692ef492742820198c60d6067acec775d4c024cc20e5555a220bd00f42806e1b8c460409fa38eaa8801712b8ce53c0362f216ef56e8b11f28ff5788ad05025c950bfe048a44d15f9be7953e249f0356aaa08480100
2024-08-24T05:32:30.795 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Sending proof
2024-08-24T05:32:30.896 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Sent proof
2024-08-24T05:32:30.996 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF loop finished. Total iters: 117855141
2024-08-24T05:32:31.097 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Stopped everything! Ready for the next challenge.
2024-08-24T05:32:31.207 TLauncher chia.timelord.timelord_launcher: INFO VDF client 1: VDF Client: Discriminant = -0xe159a5d6e2a62a1bb3267bdcdf8e5ef5bfc3e560bc95b61a631d7b62dd9ac110ad838ea67e1c818eec35d82d0fd8b107f763613f08c6f62cf5dee4302e32fd65ad02dacbe3285574ed945e25427598b9ccdca70a26399e28ad26b8cb8b0008f23fedd50463a0efb0c4b9eae681349ef00cb7947c40a7a79f70e92c6e0199c31f
2024-08-24T05:36:41.464 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: Got simple weso proof: 00000561ef6c155a231591ae6c193b8a97a8a87b6557a8ad236c2ea96abf354653b9a9d749eb4fa06110396ffca4d1b8340881ca5163df53523c9fb3ec9ef5934206533709f1364e847a42b522578fac3b148a191a6489889209efba548cd4c9641f010001004ee17e8215dff27f48a1f6ab6e2fbb919075a302fe2c1bcbd3dab810c7c2926a8f9404b232d74817d726d695647b6390456d3564939f488ee26272240d0b715295ee1a93bd4bca860d9d46a05cfed21ed5d0704f92b0b956c34a1f39a36ecb3e0100
2024-08-24T05:36:41.565 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: VDF Client: Sending proof
2024-08-24T05:36:41.665 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: VDF Client: Sent proof
2024-08-24T05:36:41.766 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: VDF loop finished. Total iters: 210289493
2024-08-24T05:36:41.867 TLauncher chia.timelord.timelord_launcher: INFO VDF client 2: VDF Client: Stopped everything! Ready for the next challenge.
Old Log Output from harvester_api.py:
2024-08-27T12:23:07.882 harvester chia.harvester.harvester: WARNING Looking up qualities on C:\plots\plot-k32-c06-2023-09-09-02-53-xxx.plot took: 8.127413272857666. This should be below 8 seconds to minimize risk of losing rewards.
New Log Output from harvester_api.py (latency_warning_threshold = 15):
2024-08-27T10:45:32.785 harvester chia.harvester.harvester: WARNING Looking up qualities on C:\plots\plot-k32-c06-2023-11-19-08-25-xxx.plot took: 16.766469717025757. This should be below 15 seconds to minimize risk of losing rewards.
2024-08-27T10:45:32.787 harvester chia.harvester.harvester: INFO 30 plots were eligible for farming 42b40a7d22... Found 0 proofs. Time: 16.76647 s. Total 8056 plots
2024-08-27T10:45:33.823 harvester chia.harvester.harvester: INFO 31 plots were eligible for farming 42b40a7d22... Found 0 proofs. Time: 8.69389 s. Total 8056 plots
2024-08-27T10:45:39.764 harvester chia.harvester.harvester: INFO 24 plots were eligible for farming 42b40a7d22... Found 0 proofs. Time: 5.72729 s. Total 8056 plots
"chia_timelord_worker" processes remaining alive after running "chia stop all -d" command on slow bluebox timelord
"chia_timelord_worker" terminated properly
Testing Notes:
See log outputs above.