nsqd: persisted data repeatedly consumed

[mreiferson]

hi，
i have a question about nsqd persisted meta file update.
my nsqd version:

nsqd v0.3.7 (built w/go1.6)

when nsqd start up, it load persisted data from disk, and then subscriber consumed the msg, but the nsqd not update the persist meta file, and then if nsqd crash when it restart, my subscriber will consume the same messages again. I know nsqd guarantees "messages are delivered at least onceAnchor link for: messages are delivered at least once"， but is this scene can improve？

[ploxiln]

The metadata file should have (eventually) been updated. It may not have been synced to disk until some time and/or more messages were processed ... but it would indeed be a bug if it could wait indefinitely before being synced. See the --sync options:

  -sync-every int
        number of messages per diskqueue fsync (default 2500)
  -sync-timeout duration
        duration of time per diskqueue fsync (default 2s)

Looks like, by default, the metadata file should be written and fsynced every 2 seconds. If you can reproduce a case where that does not happen, it's a bug :)

[jinhao]

@ploxiln it seems option -sync-timeout duration not work as our expected , the case can reproduce every time, you can try my scene.

[mreiferson]

@jinhao can you list the steps to reproduce?

[jinhao]

@mreiferson

my topic is uploadMsg channel is default
my meta fileuploadMsg:default.diskqueue.meta.dat

1. publish msgA to topicA (no consumer consumed the msg)

2. kill -2 nsqdpid (nsqd will update the meta file and msgA of topicA to disk)
   my meta file

   1
   0,2622
   0,2760

3. restart the nsqd(reload the msgA)

4. start nsq_tail consumed msgA

5. see meta file not updated

6. kill -9 nsqdpid

7. restart the nsqd and will see it reload msgA again
   meta file is still

1
0,2622
0,2760

[mreiferson]

@jinhao what are the command line flags you're using for nsqd?

[jinhao]

@mreiferson
first time:

./nsqd -tcp-address=:4250 -http-address=:4251 --lookupd-tcp-address=172.16.154.105:4160 -broadcast-address=172.16.154.105

second:

./nsqd -sync-every=1 -tcp-address=:4250 -http-address=:4251 --lookupd-tcp-address=172.16.154.105:4160 -broadcast-address=172.16.154.105

both not work.

[mreiferson]

OK, I understand what's happening.

nsqd will only sync metadata when writes have occurred, see: https://github.com/nsqio/nsq/blob/master/nsqd/diskqueue.go#L618-L622

This feels wrong, for this exact reason. We can still avoid "unnecessary" syncs, which is what I think the code was intended to guard against, by counting reads too.

Thoughts @jehiah ?

[jehiah]

@mreiferson yeah i agree. it should count reads and writes equally

[mreiferson]

RFR

[jehiah]

don't we want (wcount + rcount) == d.syncEvery ?

And more generally, do we need to distinguish between read and write counts, can we always use a single count var?

[mreiferson]

can we always use a single count var?

That should simplify things.

[mreiferson]

ready @jehiah

[jinhao]

@mreiferson
can release a new version of nsq

I try to get the latest code from github, and rebuild the nsq
but the nsqd cannot communicate with nsqlookupd
it seems new nsqd use protocol 'V2', but nsqlookupd still use 'V1'
nsqlookupd log

[nsqlookupd] 2016/04/11 10:01:01.732021 CLIENT(127.0.0.1:63929): desired protocol magic '  V1'
[nsqlookupd] 2016/04/11 10:01:01.732123 ERROR: [127.0.0.1:63929] - E_BAD_BODY IDENTIFY    missing fields
[nsqlookupd] 2016/04/11 10:01:01.732179 CLIENT(127.0.0.1:63929): closing
[nsqlookupd] 2016/04/11 10:01:01.732192 ERROR: client(127.0.0.1:63929) - E_BAD_BODY IDENTIFY missing fields

[judwhite]

@jinhao That looks like you have nsqd -lookupd-tcp-address set to connect to the nsqlookupd HTTP port (4161). Can you try 4160 instead? If that's not the case can you paste your nsqd and nsqlookupd args?