TF model cards

[Rocketknight1]

• Creating model cards for models trained with Keras!
• Model cards will automatically be created by the PushToHubCallback - the callback will peek at the metrics and accumulate information during the training run to enable this.
• Calling model.push_to_hub() will not, by default, create a model card. This method is cross-platform (it comes from the PushToHubMixin), and changing its behaviour would negatively affect Trainer users.
• Instead, Keras models now have a create_model_card() method if users want to create a model card outside of the PushToHubCallback. Because this method can't peek at the training history like the callback can, you need to pass the History object returned by model.fit() to create the card.

[sgugger]

Thanks for adding this! Let's avoid storing a token that users may then share by mistake if we can :-)

[sgugger]

If you store it, make sure it doesn't appear in the repr (we had that surprise in the TrainingArguments) or something saved automatically.

[Rocketknight1]

Is there any way to make this work without storing the hub_token? I could move the repo creation to __init__, which would avoid the need to store the token as a property, but that might surprise users when the repo is made when the callback is made, rather than when training begins.

[sgugger]

I think it's better to do it at init and have a potential error sooner rather than later.

[Rocketknight1]

I set it to clear the hub_token attribute as soon as the repo is created, and made sure it wasn't being printed in the __repr__, so I think this is fairly safe.

[sgugger]

But why not create the repo at init? Is there some infor we only get later?

[Rocketknight1]

The main reason is that it's surprising for Keras users - Callbacks usually don't do anything visible at init, because they exist to trigger actions at certain points during training. If you think the problems with moving initialization to on_train_begin are too annoying, though, I can create the repo there, I'm sure it's fine!

[sgugger]

In this case I find it weird to not do it at init (the Trainer does it there FYI): this method clones the repo where we will work, which is something you do before you execute your script in normal life. If there is an error because the repo already exists and doesn't match the clone_from it will be more obvious if the error is raised at the callback creation rather when launching the training.

[Rocketknight1]

That makes sense, yes. I'll move it!

[Rocketknight1]

Done!

[Rocketknight1]

@sgugger all comments should be addressed and testing looks good. My main remaining concern is that model.push_to_hub does not generate a model card by default, because that method comes from the cross-platform PushToHubMixin. If you want, I can edit that method so that it creates a model card if you pass it a Keras model history? This would leave the behaviour for Trainer unchanged.

[sgugger]

I'm fine with both options, so it's really up to what you think is best @Rocketknight1
Also, if you could rebase on master to fix the CI that would be great :-)

[Rocketknight1]

Will do both!

[sgugger]

Perfect, thanks a lot for all your work on this!

[LysandreJik]

This is great! Do you have an example of what a training would look like, that we could link in the release and in the docs?

[LysandreJik]

Cool addition!

[LysandreJik]

Nice! This will be super useful to track!

[Rocketknight1]

@LysandreJik I'm hoping to post an example with it when it's ready, but right now I'm having some issues with the method generating malformed YAML and they're a pain to track down!

[sgugger]

Nice new test!

[Rocketknight1]

@sgugger @LysandreJik This should be ready to go now - some tests are failing even after rebasing but they have nothing to do with this PR. Okay if I merge?

[sgugger]

Yes you can!