Fix parsing bug with unecognized token at EOS

In get_token(), we could read one byte past the end of the
region. This is only a big problem in the case where the region
itself is (a) potentially hostile, and (b) not explicitly
nul-terminated.

This patch fixes the underlying bug, and also makes sure that the
one remaining case of not-NUL-terminated potentially hostile data
gets NUL-terminated.

Fix for bug 21018, TROVE-2016-12-002, and CVE-2016-1254