C++ Map::saveMapItems()

[Itutorial]

Okay, there aren't any compile errors and the code runs through (stable enough). The issue is no data is being saved into the database. I must not be saving or creating data in the correct way for this to work. Here is the code involved.

Its possible my loop to go through each tile on the map is incorrect. (seen in Map::getAllTiles)

Save tiles and populate database query

bool IOMapSerialize::saveMapItems()
{
    int64_t start = OTSYS_TIME();
    Database& db = Database::getInstance();

    //Start the transaction
    DBTransaction transaction;
    if (!transaction.begin()) {
        return false;
    }

    //clear old tile data
    if (!db.executeQuery("DELETE FROM `map_tiles`")) {
        return false;
    }

    // Create database query to save tile information
    DBInsert stmt("INSERT INTO `map_tiles` (`data`) VALUES ");

    PropWriteStream stream;

    // Loop through tile vector and add tile information to query
    for (Tile* tile : g_game.map.getAllTiles(0, 15)) {
        if (tile) {
            saveTile(stream, tile);


            size_t attributesSize;
            const char* attributes = stream.getStream(attributesSize);
            if (attributesSize > 0) {
                if (!stmt.addRow(fmt::format("{:s}", db.escapeBlob(attributes, attributesSize)))) {
                    return false;
                }
                stream.clear();
            }
        }
    }

    if (!stmt.execute()) {
        return false;
    }

    //End the transaction
    bool success = transaction.commit();
    std::cout << "> Saved map items in: " <<
        (OTSYS_TIME() - start) / (1000.) << " s" << std::endl;
    return success;
}

Map::getAllTiles()

std::vector<Tile*> Map::getAllTiles(uint8_t floorMin, uint8_t floorMax) const
{
    if (floorMax >= MAP_MAX_LAYERS) {
        floorMax = MAP_MAX_LAYERS;
    }

    if (floorMin < 0) {
        floorMin = 0;
    }

    uint16_t maxX = 5000;
    uint16_t maxY = 5000;
    uint8_t currentZ = 0;
    uint8_t cFloorMax = 0;

    if (floorMin) {
        currentZ = floorMin;
    }

    if (floorMax) {
        cFloorMax = floorMax;
    }

    std::vector<Tile*> tileVec;

    for (currentZ; currentZ <= cFloorMax; currentZ++) {
        for (uint16_t currentY = 0; currentY < maxY; currentY++) {
            for (uint16_t currentX = 0; currentX < maxX; currentX++) {
                if (currentX == maxX && currentY == maxY && currentZ == cFloorMax) {
                    break;
                }

                if (currentX > maxX) {
                    currentX = maxX;
                }

                if (currentY > maxY) {
                    currentY = maxY;
                }

                const QTreeLeafNode* leaf = QTreeNode::getLeafStatic<const QTreeLeafNode*, const QTreeNode*>(&root, currentX, currentY);
                if (leaf) {
                    const Floor* floor = leaf->getFloor(currentZ);
                    if (floor) {
                        tileVec.push_back(floor->tiles[currentX & FLOOR_MASK][currentY & FLOOR_MASK]);
                    }
                }
            }
        }
    }
    return tileVec;
}

Would even be willing to pay for the solution.

Post automatically merged: Jan 29, 2022

I GOT IT SAVING... okay, now its not loading the items in when the server starts up.

void IOMapSerialize::loadMapItems(Map* map)
{
    int64_t start = OTSYS_TIME();

    DBResult_ptr result = Database::getInstance().storeQuery("SELECT `data` FROM `map_tiles`");
    if (!result) {
        return;
    }

    do {
        unsigned long attrSize;
        const char* attr = result->getStream("data", attrSize);

        PropStream propStream;
        propStream.init(attr, attrSize);

        uint16_t x, y;
        uint8_t z;
        if (!propStream.read<uint16_t>(x) || !propStream.read<uint16_t>(y) || !propStream.read<uint8_t>(z)) {
            continue;
        }

        Tile* tile = map->getTile(x, y, z);
        if (!tile) {
            continue;
        }

        uint32_t item_count;
        if (!propStream.read<uint32_t>(item_count)) {
            continue;
        }

        while (item_count--) {
            loadItem(propStream, tile);
        }
    } while (result->next());
    std::cout << "> Loaded map items in: " << (OTSYS_TIME() - start) / (1000.) << " s" << std::endl;
}

Post automatically merged: Jan 29, 2022

I think the issue is in the loadItems function. All items should be treated the same.

bool IOMapSerialize::loadItem(PropStream& propStream, Cylinder* parent)
{
    uint16_t id;
    if (!propStream.read<uint16_t>(id)) {
        return false;
    }

    Tile* tile = nullptr;
    if (parent->getParent() == nullptr) {
        tile = parent->getTile();
    }

    const ItemType& iType = Item::items[id];
    if (iType.moveable || iType.forceSerialize || !tile) {
        //create a new item
        Item* item = Item::CreateItem(id);
        if (item) {
            if (item->unserializeAttr(propStream)) {
                Container* container = item->getContainer();
                if (container && !loadContainer(propStream, container)) {
                    delete item;
                    return false;
                }

                parent->internalAddThing(item);
                item->startDecaying();
            } else {
                std::cout << "WARNING: Unserialization error in IOMapSerialize::loadItem()" << id << std::endl;
                delete item;
                return false;
            }
        }
    } else {
        // Stationary items like doors/beds/blackboards/bookcases
        Item* item = nullptr;
        if (const TileItemVector* items = tile->getItemList()) {
            for (Item* findItem : *items) {
                if (findItem->getID() == id) {
                    item = findItem;
                    break;
                } else if (iType.isDoor() && findItem->getDoor()) {
                    item = findItem;
                    break;
                } else if (iType.isBed() && findItem->getBed()) {
                    item = findItem;
                    break;
                }
            }
        }

        if (item) {
            if (item->unserializeAttr(propStream)) {
                Container* container = item->getContainer();
                if (container && !loadContainer(propStream, container)) {
                    return false;
                }

                g_game.transformItem(item, id);
            } else {
                std::cout << "WARNING: Unserialization error in IOMapSerialize::loadItem()" << id << std::endl;
            }
        } else {
            //The map changed since the last save, just read the attributes
            std::unique_ptr<Item> dummy(Item::CreateItem(id));
            if (dummy) {
                dummy->unserializeAttr(propStream);
                Container* container = dummy->getContainer();
                if (container) {
                    if (!loadContainer(propStream, container)) {
                        return false;
                    }
                } else if (BedItem* bedItem = dynamic_cast<BedItem*>(dummy.get())) {
                    uint32_t sleeperGUID = bedItem->getSleeper();
                    if (sleeperGUID != 0) {
                        g_game.removeBedSleeper(sleeperGUID);
                    }
                }
            }
        }
    }
    return true;
}

 

[pink_panther]

This save your blackboards/lights/torches states and writing (without stacking the blackboards/lights, eetc)?

 

[Itutorial]

I am still working out some kinks before I can start implementing the structure of how it will work. As of now it is saving the first item on each tile and loading it when the server starts. Once I get it saving all items on the tile it will:

1) Remove all previous item data from the database
2) Save all new item data on the map
3) Load the item data from database when the server starts

So no, it will not duplicate items on the tile each save period. It deletes all old data. HOWEVER. If you have an item created on the map from the map editor those items will be duplicated as of now.

I am working on a fix for that and some type of work around to only save specific items, any items in house tiles, and a way to create house tiles and save them on the fly. This way players can create their own houses and everything will save correctly but any trash on the map will be removed/ignored. I will also need to figure out a system to determine where the code needs to scan on the map because as of now it is looping through them all and its taking 1.2s to complete at 375 million iterations. That is a map size of 5000x5000x15 (x,y,z) that is with very few items on the tiles.

Post automatically merged: Jan 29, 2022

If anyone can figure out why it only saves the first item.... This is pretty much the same exact code for house tiles. House tiles save all items.

bool IOMapSerialize::saveMapItems()
{
    int64_t start = OTSYS_TIME();
    Database& db = Database::getInstance();

    //Start the transaction
    DBTransaction transaction;
    if (!transaction.begin()) {
        return false;
    }

    //clear old tile data
    if (!db.executeQuery("DELETE FROM `map_tiles`")) {
        return false;
    }

    PropWriteStream stream;

    // Create database query to save tile information
    DBInsert stmt("INSERT INTO `map_tiles` (`data`) VALUES ");

    // Loop through tile vector and add tile information to query
    for (Tile* tile : g_game.map.getAllTiles(0, 15)) {
        if (tile) {
            saveTile(stream, tile);

            size_t attributesSize;
            const char* attributes = stream.getStream(attributesSize);
            if (attributesSize > 0) {
                if (!stmt.addRow(fmt::format("{:s}", db.escapeBlob(attributes, attributesSize)))) {
                    return false;
                }
                stream.clear();
            }
        }
    }

    if (!stmt.execute()) {
        return false;
    }

    //End the transaction
    bool success = transaction.commit();
    std::cout << "> Saved map items in: " <<
        (OTSYS_TIME() - start) / (1000.) << " s" << std::endl;
    return success;
}

Post automatically merged: Jan 29, 2022

I did it boys! All items on the map will save (with some exceptions) and I created a way to clean the map and rid it of suspected trashed items.

 

[pink_panther]

Like the save house items function, make a new table for world data, basically a copy of tile_store table that house uses...

Make a copy of these functions

bool IOMapSerialize::saveHouseItems()
   
void IOMapSerialize::loadHouseItems(Map* map)

Change the loop to loop all tiles that are NOT house tiles

for (const auto& it : g_game.map.houses.getHouses()) {
        //save house items
        House* house = it.second;
        for (HouseTile* tile : house->getTiles()) {
           
            // etc etc etc

Replace

    for (const auto& it : g_game.map.houses.getHouses()) {
        //save house items
        House* house = it.second;
        for (HouseTile* tile : house->getTiles()) {

With something like this

for (uint8_t z = 0; z < MAP_MAX_LAYERS; ++z) {
                Floor* floor = leafNode->getFloor(z);
                if (!floor) {
                    continue;
                }

                for (auto& row : floor->tiles) {
                    for (auto tile : row) {
                        if (!tile || tile->hasFlag(TILESTATE_PROTECTIONZONE)) {
                            continue;
                        }
                       
                        // Save tile funtion part here, same as how it saves house tiles.

For the Load part, hope the way LoadHouse tiles works, except select from the world_tile_store table instead of the tile_store house items table.


You will also want to do a check when saving world items that it does not save items loaded in from the map, othewise you will get duplicate shit everywhere each time the server restarts and loads everything in. IE map loads in all boxes and crap, this will save them and load them twice from the map file and from the world items table.

Once you have the functions for saving and loading world items, just locate where the save/load house items are and add it there so it runs.

 

[Itutorial]

Like the save house items function, make a new table for world data, basically a copy of tile_store table that house uses...

Make a copy of these functions

bool IOMapSerialize::saveHouseItems()
  
void IOMapSerialize::loadHouseItems(Map* map)

Change the loop to loop all tiles that are NOT house tiles

for (const auto& it : g_game.map.houses.getHouses()) {
        //save house items
        House* house = it.second;
        for (HouseTile* tile : house->getTiles()) {
          
            // etc etc etc

Replace

    for (const auto& it : g_game.map.houses.getHouses()) {
        //save house items
        House* house = it.second;
        for (HouseTile* tile : house->getTiles()) {

With something like this

for (uint8_t z = 0; z < MAP_MAX_LAYERS; ++z) {
                Floor* floor = leafNode->getFloor(z);
                if (!floor) {
                    continue;
                }

                for (auto& row : floor->tiles) {
                    for (auto tile : row) {
                        if (!tile || tile->hasFlag(TILESTATE_PROTECTIONZONE)) {
                            continue;
                        }
                      
                        // Save tile funtion part here, same as how it saves house tiles.

For the Load part, hope the way LoadHouse tiles works, except select from the world_tile_store table instead of the tile_store house items table.


You will also want to do a check when saving world items that it does not save items loaded in from the map, othewise you will get duplicate shit everywhere each time the server restarts and loads everything in. IE map loads in all boxes and crap, this will save them and load them twice from the map file and from the world items table.

Once you have the functions for saving and loading world items, just locate where the save/load house items are and add it there so it runs.

Already got it all done.

 

[pink_panther]

Nice.

Does it save light/torch states and blackboard texts?

 

[Itutorial]

Yes, I had to create the ability to tell the server items that are spawned on the map but should still be saved. The item is deleted from the database then saved before the server shutdowns, or anytime I tell it to save. So when the server loads up it will populate the torches, ect. with their states saved. Instead of creating the item it just updates it (which was already coded for something else in the sources)

 

[Felipe93]

Yes, I had to create the ability to tell the server items that are spawned on the map but should still be saved. The item is deleted from the database then saved before the server shutdowns, or anytime I tell it to save. So when the server loads up it will populate the torches, ect. with their states saved. Instead of creating the item it just updates it (which was already coded for something else in the sources)

Could you share the code pls? To save items on map

 

[pink_panther]

I am still working out some kinks before I can start implementing the structure of how it will work. As of now it is saving the first item on each tile and loading it when the server starts. Once I get it saving all items on the tile it will:

1) Remove all previous item data from the database
2) Save all new item data on the map
3) Load the item data from database when the server starts

So no, it will not duplicate items on the tile each save period. It deletes all old data. HOWEVER. If you have an item created on the map from the map editor those items will be duplicated as of now.

Here is a bool on items loadedFromMap, so when saving everything, you can skip saving any items that were loaded in by the map in the first place. If you're using the old original map, you can also skip saving any tiles that have the refresh tile flag.

For lamps/lights/torches, I used the lua script here: TFS 1.X+ - Wall torches don't save their stat (https://otland.net/threads/wall-torches-dont-save-their-stat.265702/page-2#post-2568509)

 

[highsanta]

f5 why rewrite something if already multiple people written it. Let the creativity do the moar.