Swords artificially generated (GAN)

[Night Wolf]

Dear all,

I would like to showoff a glimpse of what will be the future of pixel art:


All the swords above were drawn from a generative adversarial network (GAN) called StyleGAN2 that was recently released by NVIDIA (article, github). I've basically extracted some of the tibia sprites and trainned the network with them. You can check what was the learning in each iteration through this dope gif:

Animated GIF

An animated gif. Make your own gifs with our Animated Gif Maker.

imgflip.com

Below I'll go explore a little on a few examples (first 16):
Initially it was basically drawing noises randomly


It first understood that the blackground had black colors


then it for some reason made this very strange shape


And HEAVILY insisted on it


Then the first swords came out


By iteration 9 they were still very 'conceptual' but we could see they improving alot. 5 hours have passed.


By iteration 12 they were already beginning to feel like real swords, in 32x32 they are very alike the real tibia ones, but in 64x64 we can still see some blur:


This is the 16th generation, 8 hours have passed and they are already looking pretty realistic in 32x32.



So far what I have done and what I have used:

• Open tibia Library to extract all sprites from Tibia.
• Some lua code to parse weapons.xml and extract the items by property (sword, axes, clubs)
• Used a python code to move files that have the same name as the id extracted in previous step to put all weapons in their own folders.
• Initially I thought it would nice to use Waifu2x to enlarge the sprites up to 128x128 in order to train with more degrees of freedom, but it turned out I was wrong. Even with high denoise the images still have some blur that is 'learned' by the GAN and inherited with the learning. I'm going to train for one more day and then train in 32x32 to make a comparison. This is the result using waifu2x up to 128x128 and then training with the results.
• I had to adapt StyleGAN2 to train with 128x128. Initially it was made to train with 512x512 and 1024x1024.

TO DO's:
[ ] Test if it's more effective to train weapons separately or all together. We can potentially have hammers made of swords but who cares?
[ ] Insert some sprites from the forum to make our results better than the tibia style.
[ ] Once I have finalized the learning, provide the files so people can just run to generate infinite sprites in tibia style.
At first weapons but then equipments, outfits, borders, walls, floors. The sky is the limit!
[ ] There's a very recent article (it was published around a month ago) about a neural network called 'GANSpace' that 'attach' to GANs and allow you to control the outputs. I want to try that out and potentially make a website (if it works for me) where you can slide down to create your own custom sprite. Things like type, element, color and so on. @Gesior.pl, perhaps we can insert this in your current open tibia library page?
[ ] Find a quick way to make the alpha compositing (for transparent background) back, otherwise I'll need to retrain those in magenta background

Tell me what you thought of it!

 

[pink_panther]

 

[Morrison]

Now do in real 32 x 32 haha!
Nice post

 

[Night Wolf]

resizing down is pretty simple and can be done massively very quickly, this would be an example of 32x32 versions:

 

[totolol123]

Looks much better than current cipsoft swords

 

[Night Wolf]

Looks much better than current cipsoft swords

What really got me thinking was when I noted that the GAN was choosing to drop some unique concepts in order to fit the Tibia Style. By iteration 9 we had some swords made of plants and ice and by those last iterations they were all replaced by silver blade.

Post automatically merged: May 26, 2020

Iteration 10:


Iteration 16:


They are the same swords, you can note how the style have changed to fit the 'tibia one'

 

[totolol123]

What really got me thinking was when I noted that the GAN was choosing to drop some unique concepts in order to fit the Tibia Style. By iteration 9 we had some swords made of plants and ice and by those last iterations they were all replaced by silver blade.

Post automatically merged: May 26, 2020

Iteration 10:
View attachment 45985

Iteration 16:
View attachment 45986

They are the same swords, you can note how the style have changed to fit the 'tibia one'

Did you try with anything else? Like phobos or zezenia swords. Just to see if it'll look better than the swords from tibia

 

[Night Wolf]

Did you try with anything else? Like phobos or zezenia swords. Just to see if it'll look better than the swords from tibia

Unfortunately I do not have those sprites, have they leaked? I was even considering asking for friends of mine to draw some new concepts to balance the learning of the algorithm but let's see how it progress

 

[Morrison]

what happens if you add monster look types.. lets see tha shtt xD

 

[Night Wolf]

what happens if you add monster look types.. lets see tha shtt xD

Looktypes will take me a while because I need to figure out how to train them in batches, specially the player ones that have the color layer. @Iryont managed to apply a super resolution (SR) Neural Network (NN) to batch resizing sprites of Medivia. If he tell me what he used perhaps I can try to apply the same technique on Looktypes for GANs

 

[Dip Set]

I mean absolutely genius. You could easily sell these sprites for 5-10$ each, and they would be 100% custom. Really really impressive job.

 

[Night Wolf]

I mean absolutely genius. You could easily sell these sprites for 5-10$ each, and they would be 100% custom. Really really impressive job.

I'm more interested in having free copyright sprites to replace the Tibia ones and that players would accept (similarity). Hopefully by the time I managed to have generators for all types of sprites the Tibia has we will have @slavi unity client released and fully working with those babies

 

[Iryont]

Looktypes will take me a while because I need to figure out how to train them in batches, specially the player ones that have the color layer. @Iryont managed to apply a super resolution (SR) Neural Network (NN) to batch resizing sprites of Medivia. If he tell me what he used perhaps I can try to apply the same technique on Looktypes for GANs

I am unsure how it will help you in this case, but base outfits and masks are enlarged separately and before that masks are cut into sub-layers for each body part (it is not a single mask image with different colors anymore, but rather a white layer for each part).

Perhaps in the future we will try this technique out so I can learn how to approach it myself, but before that we also wanted to increase animation frames for certain objects.

 

[Morrison]

Looktypes will take me a while because I need to figure out how to train them in batches, specially the player ones that have the color layer. @Iryont managed to apply a super resolution (SR) Neural Network (NN) to batch resizing sprites of Medivia. If he tell me what he used perhaps I can try to apply the same technique on Looktypes for GANs

i never look in medivia, looks amazin custom sprites

Post automatically merged: May 26, 2020

Or maybe artificial sprites? xD

 

[Leshrot]

[ ] Find a quick way to make the alpha compositing (for transparent background) back, otherwise I'll need to retrain those in magenta background

Tell me what you thought of it!

It's an interesting thing to see, let's see how far it goes.

Select all pixels that aren't pure black, expand selection by one pixel thickness, add black outline, delete outside pixels. I think you should test magenta anyway.

I think 32x will give you better results.

 

[Wirox]

Currently it's definitely not impressive as all of the examples are just worse-looking versions of the source images. You can clearly see a bunch of weird poison daggers, weird slayers of carving, weird bright swords, etc.

 

[Leshrot]

Currently it's definitely not impressive as all of the examples are just worse-looking versions of the source images. You can clearly see a bunch of weird poison daggers, weird slayers of carving, weird bright swords, etc.

Basically, it's because of lack of data.

As far as I know, GANs basically regurgitate patterns that were inserted as data with an "extra particle" called entropy (that changes a bit the values, colors, shapes & etc).

While one Neural Network behaves as a discriminator and assesses whether the results conform to the standards, another NN called generator tries to produce results similar to the data and "deceive" discriminator.
If the data is insufficient it becomes more obvious after a few training iterations.
And if he trains it too much, it will be exactly similar to Tibia sprites.

 

[Night Wolf]

Basically, it's because of lack of data.

As far as I know, GANs basically regurgitate patterns that were inserted as data with an "extra particle" called entropy (that changes a bit the values, colors, shapes & etc).

While one Neural Network behaves as a discriminator and assesses whether the results conform to the standards, another NN called generator tries to produce results similar to the data and "deceive" discriminator.
If the data is insufficient it becomes more obvious after a few training iterations.
And if he trains it too much, it will be exactly similar to Tibia sprites.

There's also another factor, the sword of carving (as example) are more common in the training data (because there are around 6 versions of it?) so it make the generator be biased towards its pattern.

But do not fear, this is just a small test that I was performing, I'm still to filter the sprites and try find what is the best way to make such extractions and trainings. I'll keep you all posted on the results.

For those contacting me to see if you can help, the best way to help me right now would be to filter some of good looking sprites. The more data we have the better would be the results

 

[Lessaire]

This has a lot of potential, but if you want really usable results you're going to have to expand your source pool well beyond Tibia, and pre-categorize your pools. Keep pools for things like "bloody, dripping with liquid, or glowing" separated from those which are not.

Using neural net scaler, such as waifu2x, is a perfectly valid tactic here, but you must also post-process with unsharp mask. Work at double your intended size, and downscale using lohalo.

This will absolutely be your friend here: libvips/libvips (https://github.com/libvips/libvips)

So source pool -> neural net scaler -> unsharp mask -> GAN -> lohalo 2x shrink -> final products

Nice work!

 

[Night Wolf]

This has a lot of potential, but if you want really usable results you're going to have to expand your source pool well beyond Tibia, and pre-categorize your pools. Keep pools for things like "bloody, dripping with liquid, or glowing" separated from those which are not.

Using neural net scaler, such as waifu2x, is a perfectly valid tactic here, but you must also post-process with unsharp mask. Work at double your intended size, and downscale using lohalo.

This will absolutely be your friend here: libvips/libvips (https://github.com/libvips/libvips)

So source pool -> neural net scaler -> unsharp mask -> GAN -> lohalo 2x shrink -> final products

Nice work!

actually by the number of samples I think I'll try to adapt the GAN to train directly in 32x32 and try to give even more data instead of filtering too much in each pool. I'll probably add tibia weapons (all kinds) along with other weapons from good spriters here. That can actually make a better result that we can later use scalers and unsharp mask on to SR.


I'll make a comparison after I reach the same number of epochs and will update here later with the results

Post automatically merged: May 26, 2020

The main problem I'm facing right now is that I'm using a GAN that usually requires 10k+ samples and people usually try with millions of images with 140 low-quality super resolutioned sprites. I'll try to extract this time not only until 10.98 sprites but to consider also new models of 12.x+
The only filter I'm planning to apply are in shitty models like knife and variations of the same swords (like the sword of carving there).

I'll probably update this post with results in about 2 days from now.