Attention mechanism

An attention mechanism lets a neural network decide, for each piece of output, which parts of the input to focus on. When translating a sentence, for example, the network scores every input word for relevance to the word it is about to write and takes a weighted mix of them. It replaced the older approach of squeezing a whole sentence into one fixed summary.