Making the following update to your Nginx .conf file will prevent your website from being accessed directly by it's IP address

You'll need to create a new server block and configure it such that any requests to the servers IP address will instead be redirected to the domain name.

server {
    listen 80;
    # Listen to your server ip address
    # Redirect all traffic comming from your-server-ip to your domain
    return 301 $scheme://$request_uri;

Now, when a request is made to it will be redirected to