PyOT2 (private project, open to trade code/release project for contributions 1:1)

Status
Not open for further replies.

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Background:
Sometime around mid August I started working on an OTS project mostly for fun that is written entirely in Python.
I've named it PyOT2 after the now defunct and abandoned PyOT project simply because they're both projects written in Python, even though they share no codebase beyond PyOT's C implementation of XTEA encryption/decryption, which I modified slightly anyways for a 2x performance gain.

Vision:
Rather than implementing some traditional Otserv engine in Python I'm working on a simpler and completely modular event-driven system which should allow for much easier customization and perhaps a plugin/P&P-like system.

How PyOT2 compares with traditional OTS engines coded in C++:
PyOT2 Pros:
* Easier to read and modify the engine's source code since the server is 100% coded in Python and Python only.
* Easier to customize (simpler and more transparent/better organized code design).
* Python code has much smaller margins of errors and better exception handling than C++, meaning that instead of the server ever unexpectedly crashing in various circumstances and you having to attach a debugger to it and recreate the crash cause to figure out what went wrong, Python will gracefully handle every exception that's thrown at it without crashing and makes it very clear to the user what line of code in what file caused the exception without being overly verbose or overdoing it.
* Way fewer lines of code needed to accomplish something, which imo. makes coding way more fun.
* Still able to pretty much catch up to the speed of traditional engines in most cases with some effort put into using 3rd party modules or by writing your own.
* Single-threaded asynchronous model. No race conditions, no locks or mutexes, no deadlocks or other forms of multithreaded performance degradation, yet still completely asynchronous.
* No need to ever compile or re-compile since Python is an interpreted language. You simply run the server like any other application through Python and the server is up and running within fractions of a second.
* Very cross-platform. Should be able to run on pretty much any platform with Python3.5+ installed without having to worry about setting up and compiling anything first.

PyOT2 Cons:
* Offers weaker performance out of the box compared to traditional engines (but has the potential to pretty much catch up in every way with a little bit of effort).
* High-performance libraries and algorithms developed on my own would not be included for free (except for any donated by volunteers or taken directly from any open source projects).
* Python apparently does not release freed up memory back to the OS while still running, but it will still re-use freed up memory, so won't turn into a memory-hog as long as you have some basic common sense not to deep-copying a full real map in memory on a live/public server for example.

Progress:
As of the 24th of October the server is currently as good as stateless, everything is statically defined, there's no multiplayer interaction available, no map loading, and no saving of any states even between a single login and logout.
Features like this will be added along the way after the initial experimental phase is over.

Current client actions working as of the 1st of October:
* Logging in.
* Logging out.
* Moving around in all directions.
* Turning.
* In-game commands (currently only !pos).
* Normal talking in default.
* Looking at items and tiles which outputs formatted network data in-game.
* Throwing items which just outputs formatted network data in-game.

Screenshot from early October:


Current technical features:
* Less than 500 lines of code and much of it is simply formatted text (will obviously grow over time).
* Recognizes and names all standard 7.7 opcodes.
* Single threaded application with separate game and network logic.
* Lightning-speed network code (100k+ connections per second according to 3rd party benchmarks).
* Near C++ speed RSA encryption/decryption (0.7ms on my 8 yrs old laptop vs 0.4-0.7ms with C++) implemented by myself (no 3rd party dependencies).
* XTEA encryption/decryption 10x slower than C++ (PyOT2 speed: 10µs for small packets on my old laptop without any 3rd party dependencies. Will likely be able to achieve near C++ speeds later on as I haven't tried implementing it yet myself).
* Current CPU consumption on my old laptop running around/holding down key or map clicking with max character speed: 0.3-0.7% of one core.
* Items (objects.srv) parsing.
* Partial map (sec format) parsing - work in progress.

Interests / Looking for:
I'm interested in contact and possibly collaborating or exchanging favors with other positive-minded and helpful developers who are interested either in this project or other creative OT-related projects (and discussing or talking about these interests in the latter case).
Voluntary code-contributions are very much appreciated as well, and I will always offer my services in return for valuable contributions, though to be clear: any actual expectations of rewards or services should be formally and clearly discussed with me before-hand rather than assumed on the go.

Tasks open for contributions (evolving list):
* OTBM map format/parsing.
* OTB items format/parsing.
* SEC map parsing of recursive "Content={}" blocks using mainly a primitive C++ toolset. The rest of the parsing is already done.
* Comparing different protocol versions, in particular: 6.4, 7.1, 7.4, 8.0 and 8.6 with 7.7.
* Algorithm for XTEA encryption/decryption in pure-python.

Contact & development info:
Git: Fibrosis/PyOT2
Discord: Add e.e#0480 and msg me at least once so I know what it's about or goto OTWorld's Public Discord Server.
 
Last edited:

Ochman

Premium User
Premium User
Joined
Feb 27, 2016
Messages
201
Reaction score
128
Is there any github/bitbucket fork of this; or is it private?
 

Nekiro

7375727072697365643F203A29
Support Team
Joined
Sep 7, 2015
Messages
1,783
Reaction score
552
Is there any github/bitbucket fork of this; or is it private?
Background:
So I've been working on a new private OT project mostly for fun recently that I've simply named PyOT2 after the old abandoned project PyOT.
Prior to this I tried to get in touch with the couple of guys involved with the original but completely abandoned PyOT project that are still actively chatting in their PyOT discord channel if nothing else, however after being rudely treated by Amir out of nowhere I left and started my own little project from scratch instead which doesn't throw a syntax error on execution due to broken dependencies and garbage spaghetti code as is the case with the original PyOT :p
Also PyOT2 supports my favorite client version 7.7! :)
 

StreamSide

Joseluis Gonzalez
Support Team
Joined
Aug 31, 2007
Messages
3,233
Reaction score
754
Location
Arica - Chile
I believe in closed source projects cus people who work on it care about his project and its not waiting for someone else to do his job. I like it and I wish I know more python to help you in some stuff. Good luck on this.
 
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
I believe in closed source projects cus people who work on it care about his project and its not waiting for someone else to do his job. I like it and I wish I know more python to help you in some stuff. Good luck on this.
Thank you.
I believe those to be true words in particular for this community in this time and age.

Look very fine, but closed-source really?
It's really neither closed nor open source, it's something sort of in between like conditionally open source.
I'll make it open source if I have an absolute guarantee that I'll receive code contributions or financial support equal to or greater to what I've put in, but the only way to get that guarantee is by receiving said contributions or support first.
This is kind of similar to what Fenrisus is doing with his Unity3D client project, with the main difference being that he's set up a Patreon with $600 pledged already and he has promised that he's going to release most of the code, and I have not (at least yet).

Edit: Deleted rant.
 
Last edited:

Acubens

Old Penguin
Joined
May 6, 2008
Messages
973
Reaction score
70
Location
Venezuela
Thank you very much :)
I think those are some true words especially for this community in this time and age!


Thanks, but it's really neither closed nor open source, it's something sort of in between like conditionally open source.
I'll make it open source if I have an absolute guarantee that I'll receive code contributions or financial support equal to or greater to what I've put in it, but the only way to get that guarantee is by receiving said contributions or support first.
This is kind of similar to what Fenrisus is doing with his Unity3D client project, with the main difference being that he's set up a Patreon with $600 pledged already and he has promised that he's going to release most of the code (I don't understand why he doesn't make a git so other people can contribute if he's definitely going to release the code like he's said, but that's a completely different topic).

So to be honest I don't really understand why you're shocked in the first place about me deciding not to immediately release the source code to this.
To recap:
* It's the only OTS to my knowledge that exists (and for that matter that is in active development) in this forum that's written in a scripting language and that actually runs/works.
* It doesn't have a lot of features yet, but on the flipside it's not restrained to old and rigid conventions of coding and functionality (namely trying to replicate exactly how cip made their game with only moderate levels of success).
* It's the best/imo. perfect OTS engine in current existence for small teams/solo developers that doesn't know how to or doesn't want to write hundreds to thousands of tedious lines of C++ code to add a new feature, or for anyone relatively new to OTS engines to learn how OTS engines actually works on the inside.
* The codebase is uncomplicated, consists of less than 500 lines, is easy to read, easy to modify, and has endless potential.
And if you don't think such a project deserves to be closed sourced until someone are willing to contribute to it, which is the new Open Tibia standard: demand more free stuff and contribute less, then that's really just too bad for you ¯\_(ツ)_/¯
Either way my plan is to continue enjoying myself developing this whenever I feel like it and launch my own games in the future whether or not anyone here are interested, it's just a matter of how much time and fun will be involved along the way and how many people will get their hands on something like this ✌
Excuse me, you are in the correct way, my comment wasn't made correctly i hope see this grow.
 
  • Like
Reactions: e.e
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Excuse me, you are in the correct way, my comment wasn't made correctly i hope see this grow.
It's ok. Thanks for the post and feel free to ask more questions in the future if you have any, I get maybe a bit defensive in the start of new projects because I'm used to a lot of criticism from people that doesn't know or understand what I'm doing or what I'm trying to do since I'm not very conventional, but this phase is pretty much over now, so I'll try to and probably will be able to be more calm in the future.


@Everyone
I'm still looking for collaborators / contributors particular for small and basic tasks at the moment, such as translating protocolgame.cpp functions to pythonic pseudo-functions.
For example: ProtocolGame:: parseLookAt original code:
Code:
void ProtocolGame::parseLookAt(NetworkMessage& msg)
{
	Position pos = msg.GetPosition();
	uint16_t spriteId = msg.GetSpriteId();
	uint8_t stackpos = msg.GetByte();

	addGameTaskTimed(DISPATCHER_TASK_EXPIRATION, &Game::playerLookAt, player->getID(), pos, spriteId, stackpos);
}
Python translation:
Code:
def parseLookAt(net):
	x = net.getShort()
	y = net.getShort()
	z = net.getByte()
	spriteid = net.getShort()
	stack = net.getByte()
	print(x, y, z, spriteid, stack)
Pretty basic.

Hopefully this will be a good and comfortable start for anyone new to the project who may be interested to contribute, although you may contribute any way you wish if you're more interested in other tasks.
I have moved on to the code organization and conceptual design of the server for now.

Edit: NO LONGER APPLIES (DONE).
 
Last edited:

Ochman

Premium User
Premium User
Joined
Feb 27, 2016
Messages
201
Reaction score
128
duh duh duh, going with python, taking a look...
 
Last edited:

waterson

New Member
Joined
Jul 9, 2010
Messages
9
Reaction score
1
Still looking for someone ???

I did some AI coding in Python at University, so I do know a little about it.
If you are still looking for someone to help you, I will be glad to help :)

If so, contact me at reyhammer::at::gmail::dot::com (Bot prevention)
Best wishes :)
 
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Update:
* Cleaned up the thread a bit and added some more information.
* Added most of the protocolsend functions allowing for more basic single-player functionality such as setting character outfit, opening channels, etc, and fixed various "minor" bugs, such as not being able to log back into the server after logging out without restarting the server.

There will likely not be any flashy updates for at least another month or so as I will be focusing on the core design for a while.

@waterson
Yeah, I'm still open, but would really prefer texting over Discord, PM or any of the most common IM protocols such as aim, jabber, msn, etc if possible, since I do not have a work e-mail or my own IMAP server atm for incoming mail.
Cheers.
 
Last edited:
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Minor update:
* Added statefulness - outfit and other player data is now stored and loaded again from memory across login sessions.
* Basic map/world setup, loading, and drawing.
* Added items (objects.dat) loading which is used by the LookAt protocol function.
Demo:


This simple map was generated in 0.5ms with the following demo code:
Code:
t = [103, 2110]
tile = []
for itemid in t:
	if itemid in items.items:
		tile.append(items.items[itemid])
for x in range(32000, 32020):
	for y in range(32000, 32020):
		setTile(x, y, 7, tile)
Printing the full item structure/data of item 2110 (coal basin) with print(items.items[2110]):
Code:
{'Name': 'a coal basin', 'Flags': ['Unpass', 'Unmove', 'Light', 'Height'], 'Attributes': {'Brightness': 8, 'LightColor': 207, 'Elevation': 8}}
 
Last edited:

Ochman

Premium User
Premium User
Joined
Feb 27, 2016
Messages
201
Reaction score
128
Do I understand it correctly, that it'll be only server-side? Or are there any plans on doing also client-side in python?

Good job so far, glad that you share screenshots / progress with us. Keep it up mate ;)
 
  • Like
Reactions: e.e
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Do I understand it correctly, that it'll be only server-side? Or are there any plans on doing also client-side in python?
Yes, I was toying with the idea of trying to make a client like a month ago, but realized very quickly that this would certainly be too much work for most solo developers, and especially someone like me with no UI dev experience.
I think cleaning up OTClient or praying for Fenrisus to save us with his Unity3d project are our best bets on the client side :)

Good job so far, glad that you share screenshots / progress with us. Keep it up mate ;)
Thank you, it's nice that someone is interested :)
More updates should be coming soon as I'm taking a break from working on the more complicated scheduling and event systems.
 
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
News:
* Got a #pyot2 channel on the OpenTibia discord server thanks to Amiroslo.
* Made my own discord server: OTWorld as well to bridge the communication gap between my three main projects: Realots.org, PyOT2 and OTWorld.
Feel free to drop by with questions or comments in either discord server.
You can also read old and new news about the Unity3D Client and many other projects in the OpenTibia discord server (first link).

Update:
* Added throwing/moving items.
* Fixed some protocolsend functions.
* Fixed and improved world structure and functions (had some bugs and couldn't fully operate with stackable items previously).
* Added a simple nosetup noconfig server commands-system, simply create a new function and call it in-game with !funcname optargs.
* Added basic otclient support + fixed afk disconnect issue that is specific to otclient.
* Completed memory benchmarks both in pure Python and C++ import.

Moving items & server commands demo:

Old recording of set outfit + save states on relog + walk/turn demo from a few weeks back:
 
Last edited:
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Added a public git repository since people keep asking for this: Fibrosis/PyOT2.
For those not already aware, this/PyOT2 is not an open source project yet.
You can help make it open source by contributing code and in return I will share parts of my codebase (in addition to your contribution) equal to the value of the contribution itself.
 

Ochman

Premium User
Premium User
Joined
Feb 27, 2016
Messages
201
Reaction score
128
Oh, well, this is a very big step.

I'm observing this project from the start, and I can really tell you, that none of OT projects before got me as excited as this one.
It would help to contribute if you could supplement issue tab in github, but don't feel pressured to do so :)

Fingers crossed!
 
  • Like
Reactions: e.e
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
I'm observing this project from the start, and I can really tell you, that none of OT projects before got me as excited as this one.
Nice to hear :)

It would help to contribute if you could supplement issue tab in github, but don't feel pressured to do so :)
I'm not sure about that git feature for this project, especially while there's no public codebase yet.
Did you have any sort of special or particular use of this feature in mind?
 
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Added "developer spaces" as github "issues".
This gives developers their own github space (under the issue tab) to optionally introduce themselves and/or tell us what skills you bring to the table, what you'd like to work on more specifically, and what you'd like in return (defaults to me/e.e sharing part of his personal sources/codebase in equal proportions to the contribution made if you don't specify).
You may also ask questions in your developer space either as you introduce yourself or as you go along the way.

Update in the background:
Been working a lot on bigger but less visible things again.
I may have completely finished the basics of the world structure now in C++.
Both creating/storing new tiles with a simple function and retrieving tiles seems to be working, appropriately converting python object to/from c++ in normal use cases.
Memory benchmarks for a full 7.7 real tibia map with creatures, containers, items, attributes and everything else using real map statistics indicates around 1.5GB of memory use.
Will try to finish the sec parser in the upcoming days and and see whether I'm able to load a full 7.7 real map this week or no.
 
OP
e.e

e.e

Divine Intellect
Joined
Sep 16, 2016
Messages
457
Reaction score
194
Location
/dev/urandom
Update:
Map progress:
Finished the sec parser, map loading, and fixed all the map bugs I've found so far except one last smaller one which will likely be fixed in the upcoming days.
Screenshots: before and after in just 2 weeks, try PyOT2 now only €19999.99! ;)
There was a lot of bugs to begin with such as black tiles, client crashes, not being able to go underground at all, not being able to walk on many different types of tiles and items (stackpos bug), and roofs and mountains not being at the right position among others.
Now it's not only possible to walk anywhere on the map as usual, but also teleporting inside and walking out of mountains, walls, etc, go up/down anywhere even if there's no ground there, and go inside and out again of black tiles without any bugs or crashes, example screenshot.

Memory footprint and performance:
To my surprise the memory footprint came at only slightly below 900MB.
The only server that I believe can actually compare itself to this footprint is leaked closed-sourced original files (which loads at slightly below 1GB). OTHire takes a massive 1.7GB of memory to load the same map (actually a smaller version of the original real map as a lot of water-tiles has been removed in the OTBM version).
I am also incredibly happy with the CPU and network performance of my server so far.
By messing around with the protocol and not being restricted to old systems, threading and hard-coded limitations I've found a way to make player movement incredibly fast which can not be easily done on e.g. OTHire.
I achieved this a while back while using the old map shown in the old coal basin screenshots from months ago before messing with real map loading, but I'm very happy to see that it's just as fast with a real map loaded!
Here's a short description of the speed:
Autowalking/map clicking between large distances such as from POH to Dwarven Bridge or Carlin to Ab as it turns out only takes fractions of a second (about 0.1-0.2 seconds) currently without any teleportation involved and seemingly only uses 3% of the CPU, this is on a 10 years old laptop with an old Intel Duo Core CPU running on a single core!
I kind of assume nobody is going to believe this even if I were to make a gif or video recording, I would've been skeptical myself, so I've decided to take steps towards hosting the server for a short period of time so you can see for yourself instead.
I will post the domain name/ip here once it's up (probably in about a week time or less).
That's about it for news I suppose.
Still looking for someone to help me out with OTBM parsing/reading in particular - just to be able to read the file format properly as a whole without having to study the whole file format on my own. Willing to pay $$$.
And also still looking for some positive people to chat and chill with, maybe co-op on some silly OT project for fun, play some free Linux-compatible games (I have steam), or just chat.
 
Status
Not open for further replies.
Top