OpenTibia OTClient that unpack map.otbm to .png files

[Gesior.pl]

generateMap(25, 45, 0, 555, 699, 15, 1)
25, 45, 0 - start position
555, 699, 15 - end position
RL map positions are around 30000,30000,0 to 35000, 35000, 15. Your are much too low.

 

[Mokerhamer]

generateMap(25, 45, 0, 555, 699, 15, 1)
25, 45, 0 - start position
555, 699, 15 - end position
RL map positions are around 30000,30000,0 to 35000, 35000, 15. Your are much too low.

I've tested all kinds of variables, it keeps crashing, despite if i use 1, 4 or more threads

 

[Gesior.pl]

Error after 32629 'threads' looks like it reach operating system limit of threads number [32768?]. There must be some leak in thread destroying.

Maybe someone got idea how to destroy thread after it finish? I though that it destroys thread automatically after execution of function and only leave pointer to some object that keeps status of thread [which is obvious memory leak, but not big and this application is not designed to work 24/7].
I start threads here:
https://github.com/gesior/otclient/blob/master/src/client/mapio.cpp#L112

Function overwrite existing finished threads pointers in array.

 

[Damc]

@Gesior.pl
https://github.com/gesior/otclient/pull/1
As for the threading scheme you're using - it's quite inefficient to create a thread for each task (at least on some platforms). And you're leaking the thread objects themselves (you should also use std::thread, because boost::thread has some weird quirks for people inexperienced with concurrency).

 

[Gesior.pl]

@Gesior.pl
https://github.com/gesior/otclient/pull/1
As for the threading scheme you're using - it's quite inefficient to create a thread for each task (at least on some platforms). And you're leaking the thread objects themselves (you should also use std::thread, because boost::thread has some weird quirks for people inexperienced with concurrency).

Thank you very much for fix. I will test it tommorow (if it compile with default Makefile).
It's inefficient, but it's still much faster then 1 core application and that's only way to balance tasks for threads [for me, idk anything about C++ threads except 'mutex' and 'posix thread' from studies, I'm sure that there are some 'pools' of tasks with balance and progress reporting, but I don't know any ).
I can't start 4 threads and give to each 1/4 of map, because there are many 'black' parts that generate in 0.0001, so one threads would finish much faster then others.

 

[Mokerhamer]

@Gesior.pl
https://github.com/gesior/otclient/pull/1
As for the threading scheme you're using - it's quite inefficient to create a thread for each task (at least on some platforms). And you're leaking the thread objects themselves (you should also use std::thread, because boost::thread has some weird quirks for people inexperienced with concurrency).

Still crashing :S

@Gesior.pl
Can u upload a full package (orts map?) of it with gui support?

 

[Gesior.pl]

Still crashing :S

@Gesior.pl
Can u upload a full package (orts map?) of it with gui support?

You mean all .jpg images of it?

JPG 100% quality:
722.052 files
7.598 folders
34.2 GB

I'm compressing it right now with JPG 75% quality. It should take 7-10GB then. I can upload it somewhere, would be nice, if you give me some FTP or SSH passwordto server take can handle so big file (I can split it to few .zips, if you want), because my fastest host with enough free HDD got download speed only 8-15 mb/s.

 

[Mokerhamer]

You mean all .jpg images of it?

JPG 100% quality:
722.052 files
7.598 folders
34.2 GB

I'm compressing it right now with JPG 75% quality. It should take 7-10GB then. I can upload it somewhere, would be nice, if you give me some FTP or SSH passwordto server take can handle so big file (I can split it to few .zips, if you want), because my fastest host with enough free HDD got download speed only 8-15 mb/s.

My current ftp host is slow, Dropbox is a option tough. Can u add the webgui preconfigured to it to?

 

[Gesior.pl]

My current ftp host is slow, Dropbox is a option tough. Can u add the webgui preconfigured to it to?

JPG images, 75% quality, 6.5GB in zip:
http://5.135.146.219/ortslevels75_all.zip

Configured script (+ stream script for TFS 1.1), I got some problems with 'Follow player' option (followed player icon disappear and camera move a bit weird, but I got no time to test and fix it):
http://5.135.146.219/ORTS_website_map_view_html_JS_lua_for_TFS_1_1.zip

This link will work for few next days.

Unzip both files in 1 folder (like /var/www/map/)

 

[Mokerhamer]

JPG images, 75% quality, 6.5GB in zip:
http://5.135.146.219/ortslevels75_all.zip

Configured script (+ stream script for TFS 1.1), I got some problems with 'Follow player' option (followed player icon disappear and camera move a bit weird, but I got no time to test and fix it):
http://5.135.146.219/ORTS_website_map_view_html_JS_lua_for_TFS_1_1.zip

This link will work for few next days.

Unzip both files in 1 folder (like /var/www/map/)

got it, downloading and installing it now

-- Double Post Merged, at Sat Aug 8, 2015 21:24 PM --

JPG images, 75% quality, 6.5GB in zip:
http://5.135.146.219/ortslevels75_all.zip

Configured script (+ stream script for TFS 1.1), I got some problems with 'Follow player' option (followed player icon disappear and camera move a bit weird, but I got no time to test and fix it):
http://5.135.146.219/ORTS_website_map_view_html_JS_lua_for_TFS_1_1.zip

This link will work for few next days.

Unzip both files in 1 folder (like /var/www/map/)

Oke it's up and running, i just have a small question. (already tryt to solve it myself )

How can i goto direct coordinates?
i'm currently using this, -> https://tibia.xyz/32903,32095,6:1 Is there a way to do it the same way?

-- Triple Post Merged, at Sat Aug 8, 2015 21:24 PM --

Oke it's up and running, i just have a small question. (already tryt to solve it myself )

How can i goto direct coordinates?
i'm currently using this, -> https://tibia.xyz/32903,32095,6:1 Is there a way to do it the same way?

#%$@%#! Cant edit posts!!!

Got it working what i requested above

@When is the map "decompiler" comming out, since i wish to decompile my own 10.80 map (Based on ORTS) since we have new houses and areas

 

[Mokerhamer]

@Gesior.pl
@Damc

Any progress in the thread leak? I've been messing around with the code for the past few days, no success yet in generating full map on 1 thread

 

[Gesior.pl]

@Damc
Got any idea why threads don't close after execution?

@Mokerhamer
Can you check in 'top' how many threads are running when you generate map [is it growing]? What operating system do you use?

 

[Gesior.pl]

@Gesior.pl
@Damc

Any progress in the thread leak? I've been messing around with the code for the past few days, no success yet in generating full map on 1 thread

Try to edit:

https://github.com/gesior/otclient/blob/master/otclientrc.lua
Lines:

local areaSizeY = 25
local areaSizeX = 25

To some higher values like 300 (increase it and restart rendering until it shows '...X of YYY generated..' and YYY will be lower then 10000).

 

[Damc]

The proper way to cleanup a thread is:

thread.join();
//make sure thread destructor is called, preferably through RAII

That is, assuming you want the thread to complete before `this_thread`. If you want the thread to run in the background, possibly forever, like a daemon, you need to call detach() on it instead of join().

https://github.com/djarek/otclient_mapgen/commits/master
I've implemented a wait-free(wait-free as long as the system is in a steady-state, in other words, the queue is not full but consumers are not starved) single-producer, multi consumer Work Queue with Round-Robin work item scheduling. It's thread safe (though the queue itself can only be touched by the producer thread, workers don't touch shared state). I didn't have time to change the interface and implement progress reporting so it's a little wonky ATM, however it works correctly. As for performance - I'm actually quite disappointed, because the IOPS of your storage will be the primary limiting factor.
Surprisingly, most time is wasted opening files and creating/opening directories. Dumping all images into one directory and using std:fstream sped up IO by approximately a factor of 5-10 on my machine (really weird).

BTW considering how we're going to use these images it probably makes more sense to implement some sort of on-demand generator with a short-lived cache(though that's a little more work).
Another alternative is to design(or use an already available) aggregate file format. Sequential writes will improve generation speed considerably, especially on HDDs.

 

[Gesior.pl]

@Mokerhamer
Can you test if this works fine on your system?
https://github.com/djarek/otclient_mapgen

 

[OZiRY]

JPG images, 75% quality, 6.5GB in zip:
http://5.135.146.219/ortslevels75_all.zip

Configured script (+ stream script for TFS 1.1), I got some problems with 'Follow player' option (followed player icon disappear and camera move a bit weird, but I got no time to test and fix it):
http://5.135.146.219/ORTS_website_map_view_html_JS_lua_for_TFS_1_1.zip

This link will work for few next days.

Unzip both files in 1 folder (like /var/www/map/)

Could you let me download them aswell please? Love this!

 

[whitevo]

wow, didn't know something so epic has been made.

Would you know how to make website map interactive.
When mouseover certain area it will highlight it. if clicked then it zooms in.
In zoomed in state all the marks I have placed down manually, can be seen by players who use the map.
Mouse overing mark will give small information panel (if its npc mark then short npc info, if monster then monster info etc.)
If the mark is ladder, hole, teleport or spot where can climb up, then if clicked, map moves to correct new place (player cant move up and down their own, gotta kind of move trough clicking marks)
if just mouseover then it will say where it will take you.

 

[OZiRY]

For me, step 2 removes the folders that I need to copy in step 3?
Then in step 4 it just runs alooong time but doesnt do anything with the folders, just creates a folder for each floor but nothing inside these folders (named: 1/ 2/ 3/ ... 13/ 14/ 15/)

.....

1. Put generated map folders in 'map1Unpacked'
2. Run: 1removeEmptyFolders.php
3. Copy folders [floors] '7' and '15' from 'map1Unpacked' to 'map2Composed'
4. Run: 2composeFloors.php
5. Run: 3shader.php
5. Copy or rename 'map3Shaded' to 'map4Tiled/12'
6. Run: 4tileGenerator.php
7. Run: 5compressor.php

but I want make it automatic with JS script. You just open page, click 'start' and watch 'progress'.

 

[Gesior.pl]

wow, didn't know something so epic has been made.

Would you know how to make website map interactive.
When mouseover certain area it will highlight it. if clicked then it zooms in.
In zoomed in state all the marks I have placed down manually, can be seen by players who use the map.
Mouse overing mark will give small information panel (if its npc mark then short npc info, if monster then monster info etc.)
If the mark is ladder, hole, teleport or spot where can climb up, then if clicked, map moves to correct new place (player cant move up and down their own, gotta kind of move trough clicking marks)
if just mouseover then it will say where it will take you.

It's possible, You just need to add these points to JavaScript arrays [you need to unpack stair/ladder items positions and assign to them their 'teleport actions' with OTClient].
There are already functions for it in Leaflet Maps:
http://leafletjs.com/examples/quick-start.html // on first image is example of 'polygons' [highlight map - in example blue triangle/red circle]

How to draw polygons in Javascript (highlight part of map):

var polygon = L.polygon([
  [51.509, -0.08],
  [51.503, -0.06],
  [51.51, -0.047]
]).addTo(map);

You can set any number of points - create complex shapes.

When user click on that area [polygon] ( http://stackoverflow.com/questions/11421127/how-to-trigger-events-on-leaflet-map-polygons ) you just need to start this code:

map.fitBounds(polygon.getBounds())

And map will zoom and set center of polygon as center of map.

I don't know anyone who could make it for you [I don't have time].

 

[whitevo]

It's possible, You just need to add these points to JavaScript arrays [you need to unpack stair/ladder items positions and assign to them their 'teleport actions' with OTClient].

what OTC has anything to do about it? i was talking about website.
Also to use Javascripts it means website must be build with flash?