How to implement the Softmax derivative independently from any loss function?

Mathematically, the derivative of Softmax σ(j) with respect to the logit Zi (for example, Wi*X) is

Image for post
Image for post

where the red delta is a Kronecker delta.

If you implement iteratively:

Let’s test.

If you implement it in a vectorized version:

I’m an Engineering Manager at Scale AI and this is my notepad for Applied Math / CS / Deep Learning topics. Follow me on Twitter for more!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store