From Abhishek Thakur
Chief Data Scientist @ boost.ai
"Mish" is a new activation function that seems to be beating most of the state-of-the-art benchmarks!
In this kernel, I used Mish for the two dense layers of my previous kernel on using categorical embeddings: https://lnkd.in/d8p-NUB
 |
Entity embeddings to handle categories using Mish Using data from Categorical Feature Encoding Challenge lnkd.in |
In comparison to my previous kernel which used ReLU, Mish shows a considerable improvement with a 5 fold score 0.80544 (AUC) and 20 fold score of 0.80743 (AUC).
The best part about Mish is that it has been created by a budding researcher who is currently an undergraduate student! Kudos to Diganta Misra
Read the Mish paper here: https://lnkd.in/diZ_UP6
Github repo: https://lnkd.in/djFqKtC
| digantamisra98/Mish Mish: A Self Regularized Non-Monotonic Neural Activation Function – digantamisra98/Mish lnkd.in |