SoftmaxDerivation

Softmax Derivation

Derivation

• Softmax Classification Back Propagation with Cross Entropy

Implementation

Single Layer & Binary Classification

python3 Softmax_Derivation.py

Multi-Layer & Multi-class Classification

python3 Softmax_Derivation_Multiple.py

Notes

• Softmax - Activation Function
• Cross Entropy - Loss Function

Resources

• Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and all those confusing names
  • calculate cross-entropy with for loop way (two sigma formula) - for each batch & each category
• Classification and Loss Evaluation - Softmax and Cross Entropy Loss
  • explain the stable version of softmax
  • calculate cross-entropy with matrix way (don't need one-hot label)