TFS 1.1 is only a engine. The map and scripts and players use RAM. not the engine. WibbenZ correct me if i'm wrong.
The server utilizes RAM even at idle state, but after that the server should not eat memory (memory leaks etc), not even a script should increase the ram more then what should be noticed (and should be deleted right after it has been loaded).
Without really knowing anything im gonna say the issue here is memory leaks, or a bad TFS version that eats memory at start up.
And about players taking up memory, well they do but not as this rate. I highly doubt even shadowcores runs at 10gb tbh.
There is a good formula here on OTLand somewhere that gives you a pointer (not the actual number) of how much a "normal" player would use, if you run a high exp server that formula is useless due to spamming spells, with a big radius aswell as higher damage values that has to be read in, handled and then change the health state of the creatures.
That is why most hexp servers crash after a while - bad written scripts aswell as that TFS and the client is not designed to handle that high integers and spamming of outgoing and incoming packages.
What I normally start with in these cases is trying another TFS, mainly since it can be a quick fix.
If the problem is still there, start monitoring the RAM usage, is it at: startup, intevals, time stamps, players online (this is not fun to look for, it can be pretty much anything) etc etc.
And what I can say is that if you don't know how to fix these things you should expect a high price for fixing it, I never even start without a payment on things like this if the TFS has been replaced and compiled correctly.
Mainly since you might have to rescript ALOT of things aswell as trying to debug what functions are crashing it etc etc
I can go on and on haha..