ReLU's zero gradient for negative inputs can kill neurons. Implement three fixes and their derivatives with respect to the input x:
leaky_relu(x, a=0.01): x if x > 0 else a * xelu(x, alpha=1.0): x if x > 0 else alpha * (exp(x) - 1)selu(x): scale * elu(x, alpha) with alpha = 1.6732632423543772 and scale = 1.0507009873554805leaky_relu_grad, elu_grad and selu_grad (use the right-hand derivative at 0, i.e. treat x = 0 as negative)