激活函数

名稱	函數圖形	方程式	導數	區間	连续性	單調	一阶导数单调	原点近似恒等
恆等函數		${\displaystyle f(x)=x}$	${\displaystyle f'(x)=1}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{\infty }}$	是	是	是
單位階躍函數		${\displaystyle f(x)={\begin{cases}0&{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(x)={\begin{cases}0&{\text{for }}x\neq 0\\{\text{不 存 在}}&{\text{for }}x=0\end{cases}}}$	${\displaystyle \{0,1\}}$	${\displaystyle C^{-1}}$	是	否	否
邏輯函數 (S函數的一种)		${\displaystyle f(x)=\sigma (x)={\frac {1}{1+e^{-x}}}}$	${\displaystyle f'(x)=f(x)(1-f(x))}$	${\displaystyle (0,1)}$	${\displaystyle C^{\infty }}$	是	否	否
雙曲正切函數		${\displaystyle f(x)=\tanh(x)={\frac {(e^{x}-e^{-x})}{(e^{x}+e^{-x})}}}$	${\displaystyle f'(x)=1-f(x)^{2}}$	${\displaystyle (-1,1)}$	${\displaystyle C^{\infty }}$	是	否	是
反正切函數		${\displaystyle f(x)=\tan ^{-1}(x)}$	${\displaystyle f'(x)={\frac {1}{x^{2}+1}}}$	${\displaystyle \left(-{\frac {\pi }{2}},{\frac {\pi }{2}}\right)}$	${\displaystyle C^{\infty }}$	是	否	是
Softsign 函數[1][2]		${\displaystyle f(x)={\frac {x}{1+|x|}}}$	${\displaystyle f'(x)={\frac {1}{(1+|x|)^{2}}}}$	${\displaystyle (-1,1)}$	${\displaystyle C^{1}}$	是	否	是
反平方根函數 (ISRU)[3]		${\displaystyle f(x)={\frac {x}{\sqrt {1+\alpha x^{2}}}}}$	${\displaystyle f'(x)=\left({\frac {1}{\sqrt {1+\alpha x^{2}}}}\right)^{3}}$	${\displaystyle \left(-{\frac {1}{\sqrt {\alpha }}},{\frac {1}{\sqrt {\alpha }}}\right)}$	${\displaystyle C^{\infty }}$	是	否	是
線性整流函數 (ReLU)		${\displaystyle f(x)={\begin{cases}0&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(x)={\begin{cases}0&{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle [0,\infty )}$	${\displaystyle C^{0}}$	是	是	否
帶泄露線性整流函數 (Leaky ReLU)		${\displaystyle f(x)={\begin{cases}0.01x&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(x)={\begin{cases}0.01&{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	是	是	否
參數化線性整流函數 (PReLU)[4]		${\displaystyle f(\alpha ,x)={\begin{cases}\alpha x&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(\alpha ,x)={\begin{cases}\alpha &{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	Yes iff ${\displaystyle \alpha \geq 0}$	是	Yes iff ${\displaystyle \alpha =1}$
帶泄露隨機線性整流函數 (RReLU)[5]		${\displaystyle f(\alpha ,x)={\begin{cases}\alpha x&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(\alpha ,x)={\begin{cases}\alpha &{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	是	是	否
指數線性函數 (ELU)[6]		${\displaystyle f(\alpha ,x)={\begin{cases}\alpha (e^{x}-1)&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(\alpha ,x)={\begin{cases}f(\alpha ,x)+\alpha &{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle (-\alpha ,\infty )}$	${\displaystyle {\begin{cases}C_{1}&{\text{when }}\alpha =1\\C_{0}&{\text{otherwise }}\end{cases}}}$	Yes iff ${\displaystyle \alpha \geq 0}$	Yes iff ${\displaystyle 0\leq \alpha \leq 1}$	Yes iff ${\displaystyle \alpha =1}$
擴展指數線性函數 (SELU)[7]		${\displaystyle f(\alpha ,x)=\lambda {\begin{cases}\alpha (e^{x}-1)&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$ with ${\displaystyle \lambda =1.0507}$ and ${\displaystyle \alpha =1.67326}$	${\displaystyle f'(\alpha ,x)=\lambda {\begin{cases}\alpha (e^{x})&{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle (-\lambda \alpha ,\infty )}$	${\displaystyle C^{0}}$	是	否	否
S 型線性整流激活函數 (SReLU)[8]		${\displaystyle f_{t_{l},a_{l},t_{r},a_{r}}(x)={\begin{cases}t_{l}+a_{l}(x-t_{l})&{\text{for }}x\leq t_{l}\\x&{\text{for }}t_{l}<x<t_{r}\\t_{r}+a_{r}(x-t_{r})&{\text{for }}x\geq t_{r}\end{cases}}}$ ${\displaystyle t_{l},a_{l},t_{r},a_{r}}$ are parameters.	${\displaystyle f'_{t_{l},a_{l},t_{r},a_{r}}(x)={\begin{cases}a_{l}&{\text{for }}x\leq t_{l}\\1&{\text{for }}t_{l}<x<t_{r}\\a_{r}&{\text{for }}x\geq t_{r}\end{cases}}}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	否	否	否
反平方根線性函數 (ISRLU)[3]		${\displaystyle f(x)={\begin{cases}{\frac {x}{\sqrt {1+\alpha x^{2}}}}&{\text{for }}x<0\\x&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle f'(x)={\begin{cases}\left({\frac {1}{\sqrt {1+\alpha x^{2}}}}\right)^{3}&{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$	${\displaystyle \left(-{\frac {1}{\sqrt {\alpha }}},\infty \right)}$	${\displaystyle C^{2}}$	是	是	是
自適應分段線性函數 (APL)[9]		${\displaystyle f(x)=\max(0,x)+\sum _{s=1}^{S}a_{i}^{s}\max(0,-x+b_{i}^{s})}$	${\displaystyle f'(x)=H(x)-\sum _{s=1}^{S}a_{i}^{s}H(-x+b_{i}^{s})}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	否	否	否
SoftPlus 函數[10]		${\displaystyle f(x)=\ln(1+e^{x})}$	${\displaystyle f'(x)={\frac {1}{1+e^{-x}}}}$	${\displaystyle (0,\infty )}$	${\displaystyle C^{\infty }}$	是	是	否
彎曲恆等函數		${\displaystyle f(x)={\frac {{\sqrt {x^{2}+1}}-1}{2}}+x}$	${\displaystyle f'(x)={\frac {x}{2{\sqrt {x^{2}+1}}}}+1}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{\infty }}$	是	是	是
S 型线性加权函数 (SiLU)[11] (也被稱為Swish[12])		${\displaystyle f(x)=x\cdot \sigma (x)}$	${\displaystyle f'(x)=f(x)+\sigma (x)(1-f(x))}$	${\displaystyle [\approx -0.28,\infty )}$	${\displaystyle C^{\infty }}$	否	否	否
软指数函數[13]		${\displaystyle f(\alpha ,x)={\begin{cases}-{\frac {\ln(1-\alpha (x+\alpha ))}{\alpha }}&{\text{for }}\alpha <0\\x&{\text{for }}\alpha =0\\{\frac {e^{\alpha x}-1}{\alpha }}+\alpha &{\text{for }}\alpha >0\end{cases}}}$	${\displaystyle f'(\alpha ,x)={\begin{cases}{\frac {1}{1-\alpha (\alpha +x)}}&{\text{for }}\alpha <0\\e^{\alpha x}&{\text{for }}\alpha \geq 0\end{cases}}}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{\infty }}$	是	是	Yes iff ${\displaystyle \alpha =0}$
正弦函數		${\displaystyle f(x)=\sin(x)}$	${\displaystyle f'(x)=\cos(x)}$	${\displaystyle [-1,1]}$	${\displaystyle C^{\infty }}$	否	否	是
Sinc 函數		${\displaystyle f(x)={\begin{cases}1&{\text{for }}x=0\\{\frac {\sin(x)}{x}}&{\text{for }}x\neq 0\end{cases}}}$	${\displaystyle f'(x)={\begin{cases}0&{\text{for }}x=0\\{\frac {\cos(x)}{x}}-{\frac {\sin(x)}{x^{2}}}&{\text{for }}x\neq 0\end{cases}}}$	${\displaystyle [\approx -0.217234,1]}$	${\displaystyle C^{\infty }}$	否	否	否
高斯函數		${\displaystyle f(x)=e^{-x^{2}}}$	${\displaystyle f'(x)=-2xe^{-x^{2}}}$	${\displaystyle (0,1]}$	${\displaystyle C^{\infty }}$	否	否	否

说明

^ 若一函数是连续的，则称其为${\displaystyle C^{0}}$函数；若一函数${\displaystyle n}$阶可导，并且其${\displaystyle n}$阶导函数连续，则为${\displaystyle C^{n}}$函数（${\displaystyle n\geq 1}$）；若一函数对于所有${\displaystyle n}$都属于${\displaystyle C^{n}}$函数，则称其为${\displaystyle C^{\infty }}$函数，也称光滑函数。

^ 此處H是單位階躍函數。

^ α是在訓練時間從均勻分佈中抽取的隨機變量，並且在測試時間固定為分佈的期望值。

^ ^ ^ 此處${\displaystyle \sigma }$是邏輯函數。

名稱	方程式	導數	區間	光滑性
Softmax函數	${\displaystyle f_{i}({\vec {x}})={\frac {e^{x_{i}}}{\sum _{j=1}^{J}e^{x_{j}}}}}$    for i = 1, …, J	${\displaystyle {\frac {\partial f_{i}({\vec {x}})}{\partial x_{j}}}=f_{i}({\vec {x}})(\delta _{ij}-f_{j}({\vec {x}}))}$	${\displaystyle (0,1)}$	${\displaystyle C^{\infty }}$
Maxout函數[14]	${\displaystyle f({\vec {x}})=\max _{i}x_{i}}$	${\displaystyle {\frac {\partial f}{\partial x_{j}}}={\begin{cases}1&{\text{for }}j={\underset {i}{\operatorname {argmax} }}\,x_{i}\\0&{\text{for }}j\neq {\underset {i}{\operatorname {argmax} }}\,x_{i}\end{cases}}}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$

说明

^ 此處δ是克羅內克δ函數。

• 邏輯函數
• 線性整流函數
• Softmax函數
• 人工神經網路
• 深度學習

1. Bergstra, James; Desjardins, Guillaume; Lamblin, Pascal; Bengio, Yoshua. . Département d’Informatique et de Recherche Opérationnelle, Université de Montréal. 2009. （原始内容存档于2018-09-25）.
2. Glorot, Xavier; Bengio, Yoshua, (PDF), , Society for Artificial Intelligence and Statistics, 2010, （原始内容存档 (PDF)于2017-04-01）
3. Carlile, Brad; Delamarter, Guy; Kinney, Paul; Marti, Akiko; Whitney, Brian. . 2017-11-09. arXiv:1710.09967  [cs.LG].
4. He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian. . 2015-02-06. arXiv:1502.01852  [cs.CV].
5. Xu, Bing; Wang, Naiyan; Chen, Tianqi; Li, Mu. . 2015-05-04. arXiv:1505.00853  [cs.LG].
6. Clevert, Djork-Arné; Unterthiner, Thomas; Hochreiter, Sepp. . 2015-11-23. arXiv:1511.07289  [cs.LG].
7. Klambauer, Günter; Unterthiner, Thomas; Mayr, Andreas; Hochreiter, Sepp. . 2017-06-08. arXiv:1706.02515  [cs.LG].
8. Jin, Xiaojie; Xu, Chunyan; Feng, Jiashi; Wei, Yunchao; Xiong, Junjun; Yan, Shuicheng. . 2015-12-22. arXiv:1512.07030  [cs.CV].
9. Forest Agostinelli; Matthew Hoffman; Peter Sadowski; Pierre Baldi. . 21 Dec 2014. arXiv:1412.6830  [cs.NE].
10. Glorot, Xavier; Bordes, Antoine; Bengio, Yoshua. (PDF). International Conference on Artificial Intelligence and Statistics. 2011. （原始内容存档 (PDF)于2018-06-19）.
11. . [2018-06-13]. （原始内容存档于2018-06-13）.
12. . [2018-06-13]. （原始内容存档于2018-06-13）.
13. Godfrey, Luke B.; Gashler, Michael S. . 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management: KDIR. 2016-02-03, 1602: 481–486. Bibcode:2016arXiv160201321G. arXiv:1602.01321 .
14. Goodfellow, Ian J.; Warde-Farley, David; Mirza, Mehdi; Courville, Aaron; Bengio, Yoshua. . JMLR WCP. 2013-02-18, 28 (3): 1319–1327. Bibcode:2013arXiv1302.4389G. arXiv:1302.4389 .