The installer for FileMaker Server 16 refuses to install on a Mac if Apache is running. It doesn’t check what ports Apache has grabbed, just the fact that Apache is running is enough for the installer to bail out.
If you have Server.app running on your Mac, even if all web services are turned off or disabled, you will still have Apache running.
This situation is laughable. Apple’s subsidiary, FileMaker Pro, is making software that with a few tweaks could co-exist perfectly with macOS Server.app and it’s Apache reverse proxy, but they don’t.
A quick hack to at least get you through the installer is to run the following command in the Terminal before running the installer, and then killing it when your’e done.
while sleep 5; do sudo killall -9 httpd; done
Just make sure you don’t then enable FileMaker Server’s web services, or if you do that you don’t use a port that Server.app uses (namely 80 and 443 and a few others) and you should be good to go.