We have had this error in the past now its doing it again with AME 18.104.22.168
This not an Encore issue but rather AME.
There is something wrong with the encoded file. (Blueray H.264 preset)
Earlier versions of 2020 (as in 14.0 and a few up)) dont produce this error.
Two latest versions 14.5 and 14.4 produce the error also and the beta.
Its old fashioned I know but I still use Encore every so often.