How to drop packets to port 5901 other than localhost with nftables

I was trying to block VNC port 5901 from internet but accessible from localhost. I have a docker container running on the server, which works fine. After the following 2 commands, I still can access my VNC port from remote. What did I do wrong? Thanks in advance.

sudo nft add chain filter INPUT

sudo nft add rule ip filter INPUT ip saddr != 127.0.0.1 tcp dport 5901 drop

i.sstatic.net/31g2Q.jpg