What does keras 'accuracy' mean in seq2seq model?

Question

I'm trying to build a seq2seq model to predict sequence. The most basic model was built, but I'm having trouble with understanding what 'metric=['accuracy']' means here. Below is the link that is very similar to the model I built.

https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html

But when I compiled, I added 'metric=['accuracy']', like this.

model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['acc'])

The accuracy graph was fine, but since what I'm trying to get is prediction of a sequences, I'm not sure how the accuracy is calculated here. Does it count a sequence to be 'accurate' only when output and label are completely same? For instance, if decoder output is 'ABCDE' and label(the correct answer) is 'JKCDE', since only three characters are correct, is it counted as 'inaccurate'?

Any help would be appreciated.

Answer 1

At the training time, the seq2seq model is treating the output as a sequence labeling problem. This means each token in the target sentence gets classified with what token should follow. This allows computing the accuracy in the same way for any sequence labeling tasks, i.e., as the proportion of correctly labeled tokens.

The interpretation is, however, a bit tricky. It says how many times a correct (i.e., exactly the same as in the reference) is generated given the correct prefix. This does not say much about the generation quality, i.e., a model that always messes up the beginning and ends the sequence prematurely can have high accuracy of this sort, but a low BLEU score or high edit distance from the desired output. Proper validation with beam search or greedy search is needed to reliably estimate the quality of the generated outputs.