Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic invalid memory address after reconnect a node #139

Closed
davidroman0O opened this issue Jan 6, 2023 · 6 comments
Closed

Panic invalid memory address after reconnect a node #139

davidroman0O opened this issue Jan 6, 2023 · 6 comments

Comments

@davidroman0O
Copy link

Describe the bug
Using the example genstage, change it to have two nodes, connect one to the other, disconnect, reconnect and panic

It kills the producer process directly, the entire node does NOT crash

  • Open two terminals
  • First one: go run ./producer
  • Second one: go run ./consumer
  • They should communicate and trigger events
  • CTRL + C on consumer
  • Restart it again
  • The error should show up
2023/01/06 17:06:25 WARNING! Server terminated <336B493D.0.1011>["producer"]. Panic reason: "invalid memory address or nil

To Reproduce
Steps to reproduce the behavior:

I've made a gist with all the code and a readme there https://gist.github.com/davidroman0O/db04fa224c710a4ae33d53d28304e88b

Expected behavior
Maybe a better error why it has a nil pointer? why? where? stacktrace?

Or maybe it should just not panic

Environment (please complete the following information):

  • OS: Tested on Windows 10 and QubeOS 4.11
  • Framework Version v1.999.220
  • Number of CPU 12

Additional context

I was just experimenting with the examples, trying to plug a bit of my codebase of some side projects and end up with this issue
I came back on the examples to test it again with very little modification and the error appear!
@halturin
Copy link
Collaborator

halturin commented Jan 6, 2023

Hello, thanks for the report. May I ask you to run with -ergo.norecover?

@davidroman0O
Copy link
Author

davidroman0O commented Jan 6, 2023 •
edited
Loading

Of course

I will take a look soon

Either it's missing a condition or there is a desync on some state (since disconnection of a node, trying to interact with previous demand)

https://github.com/ergo-services/ergo/blob/master/gen/stage_dispatcher.go#L479

> go run .\producer --ergo.norecover --ergo.trace --ergo.warning

to stop press Ctrl-C

Starting nodes 'producer@localhost'
2023/01/06 17:27:44 Start node with name "producer@localhost" and cookie "cookies"
2023/01/06 17:27:44 Node listener[0] port range: 15000...65000
2023/01/06 17:27:44 Started embedded EMPD service and listen port: 4369
2023/01/06 17:27:44 EPMD accepted new connection from [::1]:64323
2023/01/06 17:27:44 Request from EPMD client: [0 26 120 58 152 77 0 0 6 0 5 0 8 112 114 111 100 117 99 101 114 0 4 17 59 1 0 0]
2023/01/06 17:27:44 [producer@localhost] EPMD client: node registered
2023/01/06 17:27:44 SYSTEM: Application load
2023/01/06 17:27:44 [producer@localhost] CORE registering behavior "system_app" in group "ergo:applications"
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1001>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1001> (registered name: "")
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1002>): system_app_sup
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1002>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1002> (registered name: "system_app_sup")
2023/01/06 17:27:44 [producer@localhost] SUPERVISOR "system_app_sup" with restart strategy: one_for_one[permanent]
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1003>): system_metrics
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1003>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1003> (registered name: "system_metrics")
2023/01/06 17:27:44 [producer@localhost] SYSTEM_METRICS: Init: []etf.Term{node.System{DisableAnonMetrics:false}}
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1002> => <336B493D.0.1003>
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1001> => <336B493D.0.1002>
2023/01/06 17:27:44 [producer@localhost] SYSTEM: Application started
2023/01/06 17:27:44 [producer@localhost] CORE registering behavior "erlang" in group "ergo:applications"
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1004>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1004> (registered name: "")
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1005>): net_kernel_sup
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1005>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1005> (registered name: "net_kernel_sup")
2023/01/06 17:27:44 [producer@localhost] SUPERVISOR "net_kernel_sup" with restart strategy: one_for_one[permanent]
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1006>): net_kernel
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1006>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1006> (registered name: "net_kernel")
2023/01/06 17:27:44 NET_KERNEL: Init: []etf.Term(nil)
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1005> => <336B493D.0.1006>
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1007>): global_name_server
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1007>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1007> (registered name: "global_name_server")
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1005> => <336B493D.0.1007>
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1008>): rex
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1008>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1008> (registered name: "rex")
2023/01/06 17:27:44 REX: Init: []etf.Term(nil)
2023/01/06 17:27:44 [producer@localhost] CORE registering behavior "erpc" in group "ergo:remote"
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1005> => <336B493D.0.1008>
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1009>): observer_backend
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1009>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1009> (registered name: "observer_backend")
2023/01/06 17:27:44 OBSERVER: Init: []etf.Term(nil)
2023/01/06 17:27:44 [producer@localhost] RPC provide: proc_lib:translate_initial_call (gen.RPC)(0x64be20)
2023/01/06 17:27:44 [producer@localhost] RPC provide: appmon_info:start_link2 (gen.RPC)(0x64c060)
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1005> => <336B493D.0.1009>
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1010>): erlang
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1010>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1010> (registered name: "erlang")
2023/01/06 17:27:44 ERLANG: Init: []etf.Term(nil)
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1005> => <336B493D.0.1010>
2023/01/06 17:27:44 [producer@localhost] LINK process: <336B493D.0.1004> => <336B493D.0.1005>
Spawn producer on 'producer@localhost'
2023/01/06 17:27:44 [producer@localhost] CORE registering name (<336B493D.0.1011>): producer
2023/01/06 17:27:44 [producer@localhost] CORE registering process: <336B493D.0.1011>
2023/01/06 17:27:44 [producer@localhost] CORE spawn a new process <336B493D.0.1011> (registered name: "producer")
2023/01/06 17:27:52 EPMD accepted new connection from [::1]:64336
2023/01/06 17:27:52 Request from EPMD client: [0 26 120 58 153 77 0 0 6 0 5 0 8 99 111 110 115 117 109 101 114 0 4 17 59 1 0 0]
2023/01/06 17:27:52 EPMD accepted new connection from [::1]:64337
2023/01/06 17:27:52 Request from EPMD client: [0 9 122 112 114 111 100 117 99 101 114]
2023/01/06 17:27:52 [producer@localhost] NETWORK accepted new connection from 127.0.0.1:64339
2023/01/06 17:27:52 [producer@localhost] NETWORK registering peer "consumer@localhost"
2023/01/06 17:27:52 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:52 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:52 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:52 [producer@localhost] CONTROL MONITOR [from consumer@localhost]: etf.Tuple{19, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, "producer", etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}
2023/01/06 17:27:52 [producer@localhost] CONTROL MONITOR [from consumer@localhost]: etf.Tuple{19, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, "producer", etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23b, 0x5cdf, 0x0, 0x0, 0x0}}}
2023/01/06 17:27:52 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:52 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:52 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:52 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1011>
2023/01/06 17:27:52 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23b, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"subscribe", "nil", etf.List{etf.Tuple{"min_demand", 1}, etf.Tuple{"max_demand", 2}, etf.Tuple{"cancel", 0}, etf.Tuple{"manual", false}, etf.Tuple{"partition", 0}}}}
New subscription from: <A8F228F6.0.1011> with min: 1 and max: 2
2023/01/06 17:27:52 [producer@localhost] MONITOR process: <336B493D.0.1011> => <A8F228F6.0.1011>
2023/01/06 17:27:52 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:52 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1011>
2023/01/06 17:27:52 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23b, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 2}}
Producer: just got demand for 2 event(s) from <A8F228F6.0.1011>
2023/01/06 17:27:52 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, "", "producer"}
Producer. Generate random numbers and send them to consumers... [41 96 58 14 77]
2023/01/06 17:27:52 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:52 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:52 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:52 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:52 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 4}}
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"subscribe", "nil", etf.List{etf.Tuple{"min_demand", 2}, etf.Tuple{"max_demand", 4}, etf.Tuple{"cancel", 0}, etf.Tuple{"manual", false}, etf.Tuple{"partition", 1}}}}
New subscription from: <A8F228F6.0.1012> with min: 2 and max: 4
2023/01/06 17:27:53 [producer@localhost] MONITOR process: <336B493D.0.1011> => <A8F228F6.0.1012>
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:53 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23b, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 2}}
Producer: just got demand for 2 event(s) from <A8F228F6.0.1011>
Producer. Generate random numbers and send them to consumers... [10 31 10 20 88]
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:53 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23b, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 2}}
Producer: just got demand for 2 event(s) from <A8F228F6.0.1011>
Producer. Generate random numbers and send them to consumers... [28 60 49 37 48]
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:53 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:53 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1011>
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:54 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <336B493D.0.1011>
2023/01/06 17:27:54 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"retry-ask", 0x4}}
Producer: just got demand for 4 event(s) from <A8F228F6.0.1012>
Producer. Generate random numbers and send them to consumers... [9 61 38 49 40 9 74]
2023/01/06 17:27:54 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1012>
2023/01/06 17:27:54 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1012>
2023/01/06 17:27:54 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:54 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:54 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:27:54 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 4}}
2023/01/06 17:27:54 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, "", "producer"}
Producer: just got demand for 4 event(s) from <A8F228F6.0.1012>
2023/01/06 17:27:54 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
Producer. Generate random numbers and send them to consumers... [82 57 0 58 60 97 59]
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:54 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1012>
2023/01/06 17:27:54 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, "", "producer"}
2023/01/06 17:27:54 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:27:54 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:55 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:27:55 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 4}}
Producer: just got demand for 4 event(s) from <A8F228F6.0.1012>
Producer. Generate random numbers and send them to consumers... [95 61 88 36 60 98 26]
2023/01/06 17:27:55 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:55 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1012>
2023/01/06 17:27:55 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:27:55 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a068, ID:[5]uint32{0x3a23d, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 4}}
Producer: just got demand for 4 event(s) from <A8F228F6.0.1012>
Producer. Generate random numbers and send them to consumers... [47 96 46 5 55 66 39]
2023/01/06 17:27:55 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:27:55 [producer@localhost] CORE route message by pid (remote) <A8F228F6.0.1012>
2023/01/06 17:27:57 Request from EPMD client: []
2023/01/06 17:27:57 EPMD unregistering node: 'consumer'
2023/01/06 17:27:57 [producer@localhost] NETWORK unregistering peer consumer@localhost
2023/01/06 17:27:57 [producer@localhost] MONITOR NODE  down: consumer@localhost
2023/01/06 17:27:57 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:57 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:27:57 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <336B493D.0.1011>
2023/01/06 17:27:57 m: gen.MessageDown{Ref:etf.Ref{Node:"producer@localhost", Creation:0x63b8a060, ID:[5]uint32{0x33227, 0x5cdf, 0x0, 0x0, 0x0}}, ProcessID:gen.ProcessID{Name:"", Node:""}, Pid:etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a068}, Reason:"noconnection"}
2023/01/06 17:27:57 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <336B493D.0.1011>
2023/01/06 17:27:57 m: gen.MessageDown{Ref:etf.Ref{Node:"producer@localhost", Creation:0x63b8a060, ID:[5]uint32{0x33228, 0x5cdf, 0x0, 0x0, 0x0}}, ProcessID:gen.ProcessID{Name:"", Node:""}, Pid:etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a068}, Reason:"noconnection"}
2023/01/06 17:27:59 EPMD accepted new connection from [::1]:64344
2023/01/06 17:27:59 Request from EPMD client: [0 26 120 58 153 77 0 0 6 0 5 0 8 99 111 110 115 117 109 101 114 0 4 17 59 1 0 0]
2023/01/06 17:27:59 EPMD accepted new connection from [::1]:64345
2023/01/06 17:27:59 Request from EPMD client: [0 9 122 112 114 111 100 117 99 101 114]
2023/01/06 17:28:00 [producer@localhost] NETWORK accepted new connection from 127.0.0.1:64347
2023/01/06 17:28:00 [producer@localhost] NETWORK registering peer "consumer@localhost"
2023/01/06 17:28:00 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a06f}, "", "producer"}
2023/01/06 17:28:00 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:28:00 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:28:00 [producer@localhost] CONTROL MONITOR [from consumer@localhost]: etf.Tuple{19, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a06f}, "producer", etf.Ref{Node:"consumer@localhost", Creation:0x63b8a06f, ID:[5]uint32{0x5283, 0x5cdf, 0x0, 0x0, 0x0}}}
2023/01/06 17:28:00 [producer@localhost] CONTROL MONITOR [from consumer@localhost]: etf.Tuple{19, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a06f}, "producer", etf.Ref{Node:"consumer@localhost", Creation:0x63b8a06f, ID:[5]uint32{0x5285, 0x5cdf, 0x0, 0x0, 0x0}}}
2023/01/06 17:28:00 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a06f}, "", "producer"}
2023/01/06 17:28:00 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:28:00 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:28:00 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a06f}, "", "producer"}
2023/01/06 17:28:00 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:28:00 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1011>
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a06f}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a06f, ID:[5]uint32{0x5283, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 2}}
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1011>
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f3, Creation:0x63b8a06f}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a06f, ID:[5]uint32{0x5283, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"subscribe", "nil", etf.List{etf.Tuple{"min_demand", 1}, etf.Tuple{"max_demand", 2}, etf.Tuple{"cancel", 0}, etf.Tuple{"manual", false}, etf.Tuple{"partition", 0}}}}
New subscription from: <A8F228F6.0.1011> with min: 1 and max: 2
2023/01/06 17:28:00 [producer@localhost] MONITOR process: <336B493D.0.1011> => <A8F228F6.0.1011>
2023/01/06 17:28:00 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a06f}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a06f, ID:[5]uint32{0x5285, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"subscribe", "nil", etf.List{etf.Tuple{"min_demand", 2}, etf.Tuple{"max_demand", 4}, etf.Tuple{"cancel", 0}, etf.Tuple{"manual", false}, etf.Tuple{"partition", 1}}}}
New subscription from: <A8F228F6.0.1012> with min: 2 and max: 4
2023/01/06 17:28:00 [producer@localhost] MONITOR process: <336B493D.0.1011> => <A8F228F6.0.1012>
2023/01/06 17:28:00 [producer@localhost] NETWORK found active connection with consumer@localhost
2023/01/06 17:28:00 [producer@localhost] CONTROL REG_SEND [from consumer@localhost]: etf.Tuple{6, etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a06f}, "", "producer"}
2023/01/06 17:28:00 [producer@localhost] CORE route message by gen.ProcessID (local) <producer:producer@localhost>
2023/01/06 17:28:00 [producer@localhost] CORE route message by pid (local) <336B493D.0.1011>
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER <336B493D.0.1011> got message from <A8F228F6.0.1012>
2023/01/06 17:28:00 [producer@localhost] GEN_SERVER etf.Pid{Node:"producer@localhost", ID:0x3f3, Creation:0x63b8a060} got simple message etf.Tuple{"$gen_producer", etf.Tuple{etf.Pid{Node:"consumer@localhost", ID:0x3f4, Creation:0x63b8a06f}, etf.Ref{Node:"consumer@localhost", Creation:0x63b8a06f, ID:[5]uint32{0x5285, 0x5cdf, 0x0, 0x0, 0x0}}}, etf.Tuple{"ask", 4}}
Producer: just got demand for 4 event(s) from <A8F228F6.0.1012>
Producer. Generate random numbers and send them to consumers... [95 69 87 59 44 97 27]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x58 pc=0x5e46bb]

goroutine 122 [running]:
github.com/ergo-services/ergo/gen.(*dispatcherPartition).Dispatch(0xc0000982f0, {0x689ae0?, 0xc00008d140}, {0xc000164280, 0x7, 0x46b8ce?})
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/stage_dispatcher.go:479 +0x4db
github.com/ergo-services/ergo/gen.(*StageProcess).SendEvents(0xc0000ea000, {0xc000164280?, 0xc0000f5888?, 0x2?})
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/stage.go:352 +0x74
main.(*Producer).HandleDemand(0x0?, 0x0?, {{{0xc00047a060, 0x12}, 0x3f4, 0x63b8a06f}, {{0xc00047a078, 0x12}, 0x63b8a06f, {0x5285, ...}}}, ...)
        D:/ergoex/producer/producer.go:40 +0x185
github.com/ergo-services/ergo/gen.(*StageProcess).handleProducer(0xc0000ea000, {{{0xc00047a060, 0x12}, 0x3f4, 0x63b8a06f}, {{0xc00047a078, 0x12}, 0x63b8a06f, {0x5285, 0x5cdf, ...}}}, ...)
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/stage.go:914 +0x604
github.com/ergo-services/ergo/gen.(*StageProcess).handleStageRequest(0x100000001?, {{0xc000212090, 0xd}, {{{0xc00047a060, 0x12}, 0x3f4, 0x63b8a06f}, {{0xc00047a078, 0x12}, 0x63b8a06f, ...}}, ...})
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/stage.go:612 +0x3bb
github.com/ergo-services/ergo/gen.(*Stage).HandleInfo(0x0?, 0xc0000ac300, {0x6a2460?, 0xc000238288})
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/stage.go:497 +0x2c5
github.com/ergo-services/ergo/gen.(*ServerProcess).handleInfo(0xc0000ac300, {{0x6a2460?, 0xc000238288?}})
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/server.go:571 +0xd5
github.com/ergo-services/ergo/gen.(*ServerProcess).waitCallbackOrDeferr.func3()
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/server.go:463 +0x2e
created by github.com/ergo-services/ergo/gen.(*ServerProcess).waitCallbackOrDeferr
        desktop/go/pkg/mod/github.com/ergo-services/[email protected]/gen/server.go:461 +0x32a
exit status 2

@halturin
Copy link
Collaborator

halturin commented Jan 9, 2023

Thanks. It helped. Nice catch.

PS: I wouldn't recommend to use this approach https://gist.github.com/davidroman0O/db04fa224c710a4ae33d53d28304e88b#file-consumer-go-L88

for ergoNode.IsAlive() {
		if len(previousNodes) != len(ergoNode.Nodes()) {
			isConnected = false
		}
		if !isConnected {
			if err := ergoNode.Connect("producer@localhost"); err != nil {
				isConnected = false
			}
			if !strings.Contains(strings.Join(ergoNode.Nodes(), " "), "producer@localhost") {
				isConnected = false
			} else {
				isConnected = true
			}
		}
		previousNodes = ergoNode.Nodes()
	}

This code burns the CPU.

The better way is using ergoNode.Wait() to lock the main goroutine and use one of the approaches to monitor connections

  • gen.Process.MonitorNode(...) for the existing connection
  • gen.Process.MonitorEvent(node.EventNetwork) - you will receive node.MessageEventNetwork{...} message on connection/disconnection node.

@davidroman0O
Copy link
Author

This code burns the CPU.

Yeah I know, it was just to play around quickly 😄

halturin added a commit that referenced this issue Jan 27, 2023
@halturin
Fixes #139 - caused stage dispatcher process termination
326c102
@halturin
Copy link
Collaborator

Fixed. In the v221 branch.

@davidroman0O
Copy link
Author

Damn I didn't even had the time to look at it
Bravo!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@halturin @davidroman0O