Short answers, pulled from the story.
The universal approximation theorem states that neural networks with a certain structure can approximate any continuous function to any desired degree of accuracy. The best-known version applies to feedforward networks with a single hidden layer using a non-polynomial activation function such as sigmoid or ReLU.
Ken-ichi Funahashi published the first results in May 1989, followed by Stinchcombe and Halbert White in July 1989, and George Cybenko independently in December 1989. Cybenko had also published a technical report version in 1988. All three groups worked on the arbitrary-width case.
The arbitrary-width version achieves universality by adding more neurons to a single hidden layer, making the network wider. The arbitrary-depth version keeps the width bounded but allows the number of layers to increase, making the network deeper. Zhou Lu and colleagues established in 2017 that width n plus 4 is sufficient for the depth case using ReLU activation.
No. The universal approximation theorems are existence theorems. They confirm that a network with the right structure exists but do not specify how to find its parameters through training, nor do they state exactly how large the network must be for a given function. Training is a practical challenge typically addressed with optimization algorithms like backpropagation.
The activation function must be nonpolynomial. Moshe Leshno and colleagues established this equivalence in 1993, and Allan Pinkus confirmed it in 1999. Hornik showed in 1991 that the choice of specific activation function is less important than the multilayer feedforward architecture itself.
Two hidden layers are sufficient, as Maiorov and Pinkus showed in 1999. Guliyev and Ismailov later constructed a smooth sigmoidal activation function in 2018 that achieves universal approximation with two hidden layers using fewer units than previous bounds required.