Foundations of Recurrent Neural Networks and Training Challenges

Apuntes

Recurrent Neural Networks and Sequence to Sequence Recurrent Neural Networks (RNN) Motivation • Until now the ANNs have been use to map an input to a single output x y Motivation • Until now the ANNs have been use to map an input to a single output • What if we want to predict a new (the next) ? x y y The ball is moving. What is its next position? ? Prediction problem The next word in a sentence? This morning I took ... my cat for a walk. Practical problem The ball is moving. What is its next position? Now we know something about the past... Prediction problem past The next word in a sentence? Can we do this with a multilayer feedforward neural net? x ̂y W xh W hy Feedforward Neural Net We have to deal with variable size text! Use a fixed size window We loose long-term dependencies France is where I grew up, but now I live in Boston. I speak fluent ___ The next word in a sentence? Can we do this with a multilayer feedforward neural net? x ̂y W xh W hy Feedforward Neural Net Maintain information about order The food was good, not bad at all. The food was bad, not good at all. If we transform this in a bag of words... Motivation • ResNets showed skip connections and the impact of dif...