Unlikelihood Agent

[mvh57]

Hi @hadasah and @stephenroller, following up on this issue from September (#2966), I am wondering if you've had a chance to update the unlikelihood model so that the reward parameter can be non-binary (ie, take on integer or float value, including 0). Thanks for your help with this. I would also be happy to try to help with this, but would need some guidance as to where to begin.

[stephenroller]

#3517

[mvh57]

@stephenroller, thank you for making these changes! One remaining question we have is: will these changes carry though to the loss function for the unlikelihood agent (for example --> will a reward: -0.9 be more down weighted than a reward: -0.5; also, will reward: 0 be parsed as such and be effectively ignored by the agent?)

[stephenroller]

No, it will not.

You'll need to add some multipliers for the value here:

ParlAI/projects/dialogue_unlikelihood/agents.py

Line 105 in 637ee92

* mle_notnull.float()

[mvh57]

@stephenroller thank you!

[jakeane]

Hi @stephenroller. I am working with @mvh57, and I am currently writing the additions to allow the weights to be nonbinary. I have some questions before I go through the process detailed in CONTRIBUTING.md:

• Currently, these additions would entail copying the RewardUnlikelihoodAgentTrait class and simply modifying the compute_loss method, specifically 'mle_loss' and 'ul_loss'. Then an agent will take in this modified trait. Is this the approach you would prefer in the codebase?
• As for naming this new trait and agent, how would you like them named? Is something like RewardWeightedUnlikelihoodAgentTrait and TransformerWeightedUnlikelihoodAgent solid or too verbose?
• I noticed this comment here

  ParlAI/projects/dialogue_unlikelihood/agents.py

  Line 96 in 637ee92

  # note it's >= because convai2 and other teachers all provide a 0 reward

  and I was wondering if my modified trait class should handle such instances, where the teacher only provides rewards of 0. My current implementation would cause the mle_loss = 0 in such cases. Is this fine? as this trait should not be used with such teachers.

[jakeane]

I suppose I should also ping @hadasah on this, as it seems that she also contributed to this unlikelihood trait.

I also I have one more question about this line:

ParlAI/projects/dialogue_unlikelihood/agents.py

Line 139 in 637ee92

loss = mle_loss + self.opt['alpha'] * ul_loss

Why is ul_loss multiplied by alpha but mle_loss is not?

This is more of a curiosity question, as I am still learning about this stuff. Why does the calculation of mle_loss use nll_loss while the calculation of ul_loss just use log?

[stephenroller]

If you look at the definition of NLL loss it's similar.

As far as alpha, it doesn't really matter. We could do it convex (one term gets 1-alpha and one gets alpha). We just chose to implement with just the one scalar and tune it the same.

[github-actions]

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.