TFS 1.X+ Optimize RAM

Fortera Global

Well-Known Member
Joined
Nov 20, 2015
Messages
1,076
Best answers
0
Reaction score
52
What can I do to optimize the use of RAM? Script examples that can take up too much RAM?

tfs 1.x+

thanks
 

Gesior.pl

Mega Noob&LOL 2012
Joined
Sep 18, 2007
Messages
2,031
Best answers
19
Reaction score
1,087
Location
PLand
In 2018:
Get better dedic.

Script that consume too much RAM? Scripts with memory leak.
In many cases it's better to leave small memory leak in LUA to reduce CPU usage.

Example:
Some temporary table with NPC state.
PHP:
local function creatureSayCallback(cid, type, msg)
   if not npcHandler:isFocused(cid) then
      return false
   end
   local player = Player(cid)
   if isInArray({"soft", "boots", "repair", "soft boots"}, msg) then
      npcHandler:say("Do you want to repair your worn soft boots for 10000 gold coins?", cid)
      npcHandler.topic[cid] = 1
   elseif msgcontains(msg, 'yes') and npcHandler.topic[cid] == 1 then
...
There is table 'npcHandler.topic' indexed by 'cid'. 'cid' changes every relog, so after player logout there stays something like:
PHP:
npcHandler.topic[12343234] = 1
There are 2 ways to fix it:

1. In onDisappear event of NPC remove key from table.
Change:
PHP:
function onCreatureDisappear(cid)
    npcHandler:onCreatureDisappear(cid)
end
To:
PHP:
function onCreatureDisappear(cid)
    npcHandler:onCreatureDisappear(cid)
    npcHandler.topic[cid] = nil // = nil deletes value from LUA table
end
2. Use GUID as table index, not 'cid'. It does not change when player relog, but it will stay in RAM after logout anyway. It will reduce RAM usage to 1 key=value pair per player:
PHP:
local function creatureSayCallback(cid, type, msg)
   if not npcHandler:isFocused(cid) then
      return false
   end
   local player = Player(cid)
   if isInArray({"soft", "boots", "repair", "soft boots"}, msg) then
      npcHandler:say("Do you want to repair your worn soft boots for 10000 gold coins?", cid)
      npcHandler.topic[player:getGuid()] = 1
   elseif msgcontains(msg, 'yes') and npcHandler.topic[player:getGuid()] == 1 then
...
Both fixes require some programmer who would analyse every NPC and add changes in proper places. Programmers are expensive, hardware is cheap.
Both fixes will increase CPU usage (call to :getGuid() or remove table value) and reduce RAM usage by few MB per day.

Can you buy faster CPU? Fastest you can get in OVH is 4.7GHz. Soon there will be 5.0GHz, but you can't pay extra 100 euro and get 9.5GHz.
Can you buy more RAM? Of course. 4.7GHz dedic has 64GB RAM. Server will small memory leaks after 24 hours online with 1000 online will use less then 8GB.
In case of serious applications that must run 24/7 without restarts - memory leaks are not acceptable. In case of OTS - it's not a big problem.
 

eldera

Active Member
Joined
Oct 27, 2012
Messages
161
Best answers
1
Reaction score
25
In this example you aren't showcasing memory leak but application level cache. *just saying*
 
OP
Fortera Global

Fortera Global

Well-Known Member
Joined
Nov 20, 2015
Messages
1,076
Best answers
0
Reaction score
52
In 2018:
Get better dedic.

Script that consume too much RAM? Scripts with memory leak.
In many cases it's better to leave small memory leak in LUA to reduce CPU usage.

Example:
Some temporary table with NPC state.
PHP:
local function creatureSayCallback(cid, type, msg)
   if not npcHandler:isFocused(cid) then
      return false
   end
   local player = Player(cid)
   if isInArray({"soft", "boots", "repair", "soft boots"}, msg) then
      npcHandler:say("Do you want to repair your worn soft boots for 10000 gold coins?", cid)
      npcHandler.topic[cid] = 1
   elseif msgcontains(msg, 'yes') and npcHandler.topic[cid] == 1 then
...
There is table 'npcHandler.topic' indexed by 'cid'. 'cid' changes every relog, so after player logout there stays something like:
PHP:
npcHandler.topic[12343234] = 1
There are 2 ways to fix it:

1. In onDisappear event of NPC remove key from table.
Change:
PHP:
function onCreatureDisappear(cid)
    npcHandler:onCreatureDisappear(cid)
end
To:
PHP:
function onCreatureDisappear(cid)
    npcHandler:onCreatureDisappear(cid)
    npcHandler.topic[cid] = nil // = nil deletes value from LUA table
end
2. Use GUID as table index, not 'cid'. It does not change when player relog, but it will stay in RAM after logout anyway. It will reduce RAM usage to 1 key=value pair per player:
PHP:
local function creatureSayCallback(cid, type, msg)
   if not npcHandler:isFocused(cid) then
      return false
   end
   local player = Player(cid)
   if isInArray({"soft", "boots", "repair", "soft boots"}, msg) then
      npcHandler:say("Do you want to repair your worn soft boots for 10000 gold coins?", cid)
      npcHandler.topic[player:getGuid()] = 1
   elseif msgcontains(msg, 'yes') and npcHandler.topic[player:getGuid()] == 1 then
...
Both fixes require some programmer who would analyse every NPC and add changes in proper places. Programmers are expensive, hardware is cheap.
Both fixes will increase CPU usage (call to :getGuid() or remove table value) and reduce RAM usage by few MB per day.

Can you buy faster CPU? Fastest you can get in OVH is 4.7GHz. Soon there will be 5.0GHz, but you can't pay extra 100 euro and get 9.5GHz.
Can you buy more RAM? Of course. 4.7GHz dedic has 64GB RAM. Server will small memory leaks after 24 hours online with 1000 online will use less then 8GB.
In case of serious applications that must run 24/7 without restarts - memory leaks are not acceptable. In case of OTS - it's not a big problem.

o_O Well, thank you I will mark your answer as the best later, but I will keep the topic so that more people can give more ideas, very good indeed!

PS. Do you advise making the both modifications or choosing only 1 of them? The onCreatureDisappear part seems to be easier and more practical, but if both can be applied together I'll try apply
 
Last edited by a moderator:

Gesior.pl

Mega Noob&LOL 2012
Joined
Sep 18, 2007
Messages
2,031
Best answers
19
Reaction score
1,087
Location
PLand
In this example you aren't showcasing memory leak but application level cache. *just saying*
After player logout, data connected with player is not removed from memory = memory leaks.
We got table that only grows and some parts of it won't be accessed anymore by program (every relog player gets new unique 'cid').

It's not 'c++ style' memory leak. Where you lose pointer to some object and it's memory is not accessible anymore.
It's LUA, script language with garbage collector, so by definition, it should be not possible to create object that is not accessible from anywhere.

I wouldn't say it's cache. It's table with information needed to execute script. You can't store it any other way. Only problem with this script is that, it does not remove table records after last possible use of given key.

@eldera
If you got some better example. I'm pretty sure a lot of people would like to see it and learn more about LUA optimization problems. That's what we do on forum :cool:
 

Madzix

Well-Known Member
Joined
Sep 8, 2016
Messages
62
Best answers
12
Reaction score
70
In this example you aren't showcasing memory leak but application level cache. *just saying*
As gesior say it's just simple variable that you store in table and it reside in the ram this is because this table is linked with npc so until you remove this npc the garbage collector will not get rid of the data inside the table.

In the example gesior provide the data save as creatureId and with how cipsoft designed their engine there should be limit of 805306368 unique ids of every players, monsters, npcs(TFS actually probably don't have that limit) so the whole table can get as big as 6442450944 bytes which is 6gb for one npc!(it will take a lot of time to get to that big table but it still possible)
There's also to say that fetching data witch brackets from table still iterate through some of the keys so it hurts cpu too but probably not as much as ram(depends of how tables are implemented in Lua).
 
OP
Fortera Global

Fortera Global

Well-Known Member
Joined
Nov 20, 2015
Messages
1,076
Best answers
0
Reaction score
52
Is there more examples guys? thank you so much @Gesior.pl I'll apply this "npcHandler.topic[cid] = nil // = nil deletes value from LUA table" to all npcs and will check later if changes
 

StreamSide

Joseluis Gonzalez
Support Team
Joined
Aug 31, 2007
Messages
3,350
Best answers
34
Reaction score
851
Location
Arica - Chile
Is there more examples guys? thank you so much @Gesior.pl I'll apply this "npcHandler.topic[cid] = nil // = nil deletes value from LUA table" to all npcs and will check later if changes
you dont need to add on all of them. just the ones that use table values indexed as player id
 
OP
Fortera Global

Fortera Global

Well-Known Member
Joined
Nov 20, 2015
Messages
1,076
Best answers
0
Reaction score
52
you dont need to add on all of them. just the ones that use table values indexed as player id
What you mean? I dont need add in these all npcs?



all npcs I have here works like this:
hastebin

so I need change all? looking this example
 

tokenzz

:thinking:
Joined
Feb 2, 2013
Messages
783
Best answers
2
Reaction score
313
PHP:
function onCreatureDisappear(cid)
    npcHandler:onCreatureDisappear(cid)
    npcHandler.topic[cid] = nil // = nil deletes value from LUA table
end
also worth mentioning that the line

npcHandler.topic[cid] = nil

actually doesn't "delete" the value from the Lua table until the garbage collector is ran
 

eldera

Active Member
Joined
Oct 27, 2012
Messages
161
Best answers
1
Reaction score
25
After player logout, data connected with player is not removed from memory = memory leaks.
We got table that only grows and some parts of it won't be accessed anymore by program (every relog player gets new unique 'cid').

It's not 'c++ style' memory leak. Where you lose pointer to some object and it's memory is not accessible anymore.
It's LUA, script language with garbage collector, so by definition, it should be not possible to create object that is not accessible from anywhere.

I wouldn't say it's cache. It's table with information needed to execute script. You can't store it any other way. Only problem with this script is that, it does not remove table records after last possible use of given key.

@eldera
If you got some better example. I'm pretty sure a lot of people would like to see it and learn more about LUA optimization problems. That's what we do on forum :cool:
Actually before writing my post I only briefly looked at the example you provided and misunderstood some things.

I thought you were doing:

PHP:
npcHandler.topic[cid] = someExpensiveWork();
and the reason of not deleting it from memory was that it wouldn't have to be recalculated upon next user login (saving CPU by caching function results) . In this case it would be what I called "application level cache".

So I take back my previous statement :)
 

Delusion

Divine Intellect
Support Team
Joined
Feb 14, 2015
Messages
5,340
Best answers
528
Reaction score
3,068
more than likely you aren't going to fix your memory leaks by optimizing tiny parts of code such as a few table indexes or unneeded variables
if you're really having that big of an issue with ram and truly believe it's from lua you can set the garbage collector to run faster and more aggressively by doing:
Lua:
collectgarbage("setpause", 100)
and maybe setting "setstepmul" to higher than 200
the likely issue at hand is you have a memory leak in your sources, which would be a lot bigger in terms of memory usage than a leak in lua
if you're running linux on your server you can use valgrind to detect memory leaks in your sources
 
Last edited:

gudan garam

Well-Known Member
Joined
Feb 13, 2011
Messages
278
Best answers
13
Reaction score
97
Location
Tibialand
more than likely you aren't going to fix your memory leaks by optimizing tiny parts of code such as a few table indexes or unneeded variables
if you're really having that big of an issue with ram and truly believe it's from lua you can set the garbage collector to run faster and more aggressively by doing:
Lua:
collectgarbage("setstepmul", 100)
collectgarbage("setpause", 100)
the likely issue at hand is you have a memory leak in your sources, which would be a lot bigger in terms of memory usage than a leak in lua
if you're running linux on your server you can use valgrind to detect memory leaks in your sources
What are the defaults on thoose values?
 

Delusion

Divine Intellect
Support Team
Joined
Feb 14, 2015
Messages
5,340
Best answers
528
Reaction score
3,068
What are the defaults on thoose values?
i was wrong with setstepmul, lower values makes it slower
but theyre both defaulted to 200
setpause: each value after 100 is 1% more memory until garbage collector is ran again
so by default it's 200, meaning it runs a new cycle when the total memory is doubled
100 meaning it doesn't wait to run a new cycle
setstepmul defaulted to 200, runs the collector twice as fast as memory allocation
 

Madzix

Well-Known Member
Joined
Sep 8, 2016
Messages
62
Best answers
12
Reaction score
70
It actually don't change anything at all since it's a table inside the npchandler which is handled by npcsystem when release focus "self.topic[focus] = nil".
It was only example of how Lua can leak memory but it's only useful in npcs that uses external tables and don't "free" the variables.
 
OP
Fortera Global

Fortera Global

Well-Known Member
Joined
Nov 20, 2015
Messages
1,076
Best answers
0
Reaction score
52
I'm trying here to see how much LUA is consuming the RAM here, but its returning in Kbytes (I think), how to return in MB? Can someone help?

Code:
20:43 Memory Using: 36848.040039062.
Lua:
local textMem = collectgarbage("count")
    player:sendTextMessage(MESSAGE_STATUS_CONSOLE_BLUE, "Memory Using: " .. textMem .. ".")
I have 8GB in my VPS.
its showing here:


my screen show in htop:


Its seems my ram is consumed by C++ then? :O o_O
 
Last edited by a moderator:

Delusion

Divine Intellect
Support Team
Joined
Feb 14, 2015
Messages
5,340
Best answers
528
Reaction score
3,068
just do collectgarbage("count") / 1024 for MB
and yes, if your process is using ~4.4 gigs of memory it's your sources, not Lua
have you restarted your server and watched memory as time goes on?
generally if you have a memory leak over time you'll see it keep growing and growing
 
Top