A generic framework for programmatic modification of OTBM files

[elao]

I don't think that this library was heavily tested.
I think you should report it here Inconcessus/OTBM2JSON (https://github.com/Inconcessus/OTBM2JSON/issues) or fix it yourself, if you know JS and then make PR.
Can you post your .otbm and client version? Maybe it's just not compatible with that tool.

It seems that it's not working for clients < 8.0.


I created one for 7.4, the attribute count was not created on the .json file. These are the versions:

"itemsMajorVersion": 1,
"itemsMinorVersion": 1,

One for 7.6, the attribute count was not created on the .json file. These are the versions:

"itemsMajorVersion": 1,
"itemsMinorVersion": 3,

One for 8.0, the attribute count was created on the .json file. These are the versions:

"itemsMajorVersion": 2,
"itemsMinorVersion": 7,

One for 8.1, the attribute count was created on the .json file. These are the versions:

"itemsMajorVersion": 2,
"itemsMinorVersion": 8,

So I guess itemsMajorVersion: 1 is saving the attributes on a different bytes order?

 

[Gesior.pl]

So I guess itemsMajorVersion: 1 is saving the attributes on a different bytes order?

There is a comment in TFS code that describes problem with OTBM header version 0 and stackable items ( forgottenserver/src/iomap.cpp at 1e040a334a7aa51e7ab1d25741e463188a17bfe1 · otland/forgottenserver (https://github.com/otland/forgottenserver/blob/1e040a334a7aa51e7ab1d25741e463188a17bfe1/src/iomap.cpp#L73-L79) ):

// In otbm version 1 the count variable after splashes/fluidcontainers and stackables are saved as
// attributes instead, this solves a lot of problems with items that are changed
// (stackable/charges/fluidcontainer/splash) during an update.

IDK how it's related to itemsMajorVersion version.

You may try to update OTBM2JSON code related to reading/writing items:

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

and reading/writing item attributes:

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

or just use 8.0 map with your 7.4/7.6 server. It should be easy to edit OTS code loading map to use new version of OTBM.

 

[elao]

There is a comment in TFS code that describes problem with OTBM header version 0 and stackable items ( forgottenserver/src/iomap.cpp at 1e040a334a7aa51e7ab1d25741e463188a17bfe1 · otland/forgottenserver (https://github.com/otland/forgottenserver/blob/1e040a334a7aa51e7ab1d25741e463188a17bfe1/src/iomap.cpp#L73-L79) ):

// In otbm version 1 the count variable after splashes/fluidcontainers and stackables are saved as
// attributes instead, this solves a lot of problems with items that are changed
// (stackable/charges/fluidcontainer/splash) during an update.

IDK how it's related to itemsMajorVersion version.

You may try to update OTBM2JSON code related to reading/writing items:

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

and reading/writing item attributes:

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

OTBM2JSON/otbm2json.js at master · Inconcessus/OTBM2JSON

OTBM2JSON - A generic framework for programmatic modification of OTBM files. - Inconcessus/OTBM2JSON

github.com

or just use 8.0 map with your 7.4/7.6 server. It should be easy to edit OTS code loading map to use new version of OTBM.

Thank you for the insights. I think loading as 8.0 should work for me.

I managed to create a 2.8gb .json file.
Apparently Node's V8 is not able to open large .json files like that. I rewrote whole otbm2json to python and managed to convert it to .otbm (100mb) just fine.
And it is opening in latest RME without any error or warning.

I should do more testings tomorrow

 

[Gesior.pl]

Apparently Node's V8 is not able to open large .json files like that.

It should work, if you increase RAM limit:

JavaScript heap out of memory - with bigger maps · Issue #7 · Inconcessus/OTBM2JSON

i was testing it and im in dead end right now cause i cant convert it somehow... map size is 136394KB in size (its a huge map) Error Message: FATAL ERROR: Ineffective mark-compacts near heap limit ...

github.com

 

[elao]

It should work, if you increase RAM limit:

JavaScript heap out of memory - with bigger maps · Issue #7 · Inconcessus/OTBM2JSON

i was testing it and im in dead end right now cause i cant convert it somehow... map size is 136394KB in size (its a huge map) Error Message: FATAL ERROR: Ineffective mark-compacts near heap limit ...

github.com

I tried that, and also using 16gb as size, but it didn't work (I have 32gb RAM).

This is what I got from ChatGPT:

If Python can read and parse the giant JSON without error (ok), but Node.js crashes with v8::ToLocalChecked Empty MaybeLocal even with a 16GB heap, this confirms:

1. The problem is with V8/Node, not the file.
Your JSON is valid and can be read in other environments/languages (Python, jq, etc.).

Node is hitting an internal V8 limit for parsing, stack, or nested/extended objects — even on systems with a lot of RAM.

2. Node.js's object model is the bottleneck.
JS arrays and objects are very memory-intensive (especially deep objects or large arrays).

V8 can crash if a single object/array is huge (e.g., an array with tens of millions of tiles or features).

Anyway, it worked with Python and is probably even faster than Node. Once I get time to do more testings I might open a pull request there with Python's version.



Btw, the attribute issue I mentioned before is not related to the itemsMajorVersion: 1. But the version: 0. I just forced it to version: 1 while still using itemsMajorVersion: 1

 

[elao]

Not sure if it will get merged, but the PR is open in case someone is interested.

Converting otbm2json to Python with to_json and to_otbm example scripts by brzuck · Pull Request #9 · Inconcessus/OTBM2JSON

Ran into some issues (apparently Node limitation) when trying to handle a huge .json file (2.8gb). I converted the script to Python in order to make the conversion while gaining some performance. T...

github.com