some questions in dev-sessions about replications

[httaotao]

in dev-sessions vedio 14# and 15# , there are some things I'm not sure.

1. in read transaction part, what is refresh ? Does it mean that get the new attr information about this inode? and I don't know when use the refresh .
2. what is skip pre-op operation for metadata/entry transactions? it means that if we do the pre-op operations successfully, then if there are some other same operations in the same times, we don't need to do this pre-op operations again, right ?
3. in afr-transaction.c file, there has comment

The changelog of a file (in the xattr of the file on the server)
       stores information (pending count) about the state of the file
       on the OTHER server. This changelog is blindly trusted, and must
       therefore be updated in such a way it remains trustworthy. This
       implies that decrementing the pending count (essentially "clearing
       the dirty flag") must be done STRICTLY after we are sure that the
       operation on the other server has reached stable storage.

       While the backend filesystem on that server will eventually flush
       it to stable storage, we (being in userspace) have no mechanism
       to get notified when the write became "stable".

       This means we need take matter into our own hands and issue an
       fsync() EVEN IF THE APPLICATION WAS PERFORMING UNSTABLE WRITES,
       and get an acknowledgement for it. And we need to wait for the
       fsync() acknowledgement before initiating the hard POST-OP.

       However if the FD itself was opened in O_SYNC or O_DSYNC then
       we are already guaranteed that the writes were made stable as
       part of the FOP itself. The same holds true for NFS stable
       writes which happen on an anonymous FD with O_DSYNC or O_SYNC
       flag set in the writev() @flags param. For all other write types,
       mark a flag in the fdctx whenever an unstable write is witnessed.

I don't know how to understand this comment, it means that if client knows one brick is broken or can't connect ,perhaps other bricks can connect this broken brick ,so it's not a stable situation to deal and write the dirty information, right ? for this reason, how to judge this brick is broken in truth ? is that use the quorum numbers to judge ?

thanks.

[pranithk]

in dev-sessions vedio 14# and 15# , there are some things I'm not sure.

1. in read transaction part, what is refresh ? Does it mean that get the new attr information about this inode? and I don't know when use the refresh .

Yes, refresh means getting the new attr information about the inode. At the moment refresh is triggered when

• Brick went down and came back up from last refresh
• there was an operation on the file/directory which experienced failure

2. what is skip pre-op operation for metadata/entry transactions? it means that if we do the pre-op operations successfully, then if there are some other same operations in the same times, we don't need to do this pre-op operations again, right ?

Unlike data operations detecting the changes for an entry/metadata can be done with one LOOKUP. For metadata and entry transactions pre-op operation is not done by default because any failures will be fixed on next LOOKUP operation anyway.

3. in afr-transaction.c file, there has comment

The changelog of a file (in the xattr of the file on the server)
       stores information (pending count) about the state of the file
       on the OTHER server. This changelog is blindly trusted, and must
       therefore be updated in such a way it remains trustworthy. This
       implies that decrementing the pending count (essentially "clearing
       the dirty flag") must be done STRICTLY after we are sure that the
       operation on the other server has reached stable storage.

       While the backend filesystem on that server will eventually flush
       it to stable storage, we (being in userspace) have no mechanism
       to get notified when the write became "stable".

       This means we need take matter into our own hands and issue an
       fsync() EVEN IF THE APPLICATION WAS PERFORMING UNSTABLE WRITES,
       and get an acknowledgement for it. And we need to wait for the
       fsync() acknowledgement before initiating the hard POST-OP.

       However if the FD itself was opened in O_SYNC or O_DSYNC then
       we are already guaranteed that the writes were made stable as
       part of the FOP itself. The same holds true for NFS stable
       writes which happen on an anonymous FD with O_DSYNC or O_SYNC
       flag set in the writev() @flags param. For all other write types,
       mark a flag in the fdctx whenever an unstable write is witnessed.

I don't know how to understand this comment, it means that if client knows one brick is broken or can't connect ,perhaps other bricks can connect this broken brick ,so it's not a stable situation to deal and write the dirty information, right ? for this reason, how to judge this brick is broken in truth ? is that use the quorum numbers to judge ?

This is to avoid filesystem syncing metadata before syncing data in which case the data that AFR things to be good can be bad.

thanks.

[httaotao]

thanks for answer. but i am not sure , how to make sure this brick is broken or doesn't connect ? for example , there are one client and three brick in replica volume. then if one of the brick is broken ,then client can't connect, so could client can judge this brick is borken ? or it should let other brick to connect and then use quorum to judge , right ? thanks.

[pranithk]

When a brick is broken/client can't connect then AFR gets a notification called CHILD_DOWN check afr_notify() function. In that function it will mark this brick as down. As long quorum number of bricks are up, and at least one of them is good copy it will proceed with the modification operation.