The FMServer installer will look for the "fmserver" user on the host machine and will let the install almost complete but hangs up at the end and never finish. Checked for a mismatch on names.
"fmserver user" permissions may get set when updating or installing FileMaker Server 16. The word "user" in the actual name will cause problems.
Add "fmserver" to the permissions group and giving it read/write permissions.