You need several things, one of them is the use of custom encryption, change the way values are stored in LUA files to not expose RSA related values, connect with a special Key on your client and server can be with a encrypted binary or hexadecimal values, so even if some things are read in memory, it would make it difficult to read or at least obtain that data. You can also weekly update these Keys or values with weekly "patches" to your client, so you make it even more difficult in any of the cases that this encryption may be Broken, you can use simple keys in HEXcode and then encrypt them with something very complicated like AES256. Change the calls in the client for the most basic functions, so you avoid that they can use the same client against you.
Keep in mind that things like the host IP can only be hidden with large proxies like using Cloudflare Spectrum. Or with some related changes, apart from all that with the key between client and server you would already be avoiding connecting from other clients to your server in any way.
I also remember saying that the practice of cryptography is very complicated, and you have to know how to execute it to avoid misunderstandings such as false virus detections. Thus providing in the same way a good experience with the client for the players, in loading times and so on.
I use google translate so I apologize for any mistakes.