Server development

[while]

To be pretty blunt, I would love to work on Ultima's server code. Is there a non-zero chance of that happening? Whether you're looking for people or not, it would be nice to get a discussion going. Hopefully some talk will start up at the very least. Here are a few driving forces behind me wanting to work on the code:

First off, it's sad to see threads about character resets and inventory corruption. Ever since I registered on the forum here in mid January, those types of threads keep on popping up. Is this stuff being looked into? I don't recall hearing about preventative action being taken, only cleaning up problems after the fact. It sounds like the community has just fallen into dulled acceptance of it, which is a little disappointing. Please do not take this as "u suck admins" or a negative, personal attack like that. The reality is that it is a ongoing, highly visible issue. If there is a solution in the works then the solution should be just as visible as the originating issue, which doesn't seem to be the case. At any rate, I offer my time towards this and any other work.

Secondly, I saw that pioneer2.net had forum posts late last year about starting a new server source base from the ground up. Soly (thanks again!) kindly prodded the admins at their forum so I could get an account activated there and ask them if I could help with development. It's been about two and a half months since I first asked, but that forum feels too dead. Nobody ever responded in all that time. There wasn't even a "No, we don't want your help." As unlikely as it would've been, the initial hope was to later attempt to have Ultima migrate to fresher code, but I'm flat out giving up on those guys. After re-reading some of their ideas and seeing so little activity, I think their new server is not going to happen. And if it manages to happen, I believe some of their ideas are steps in the wrong direction. Plus I'm guessing you guys wouldn't have gone along with migrating anyway and I do very much wish to stay with Ultima.

Lastly, my day job is to mess around with code. I guess it's in my blood. Sometimes digging through source is more enjoyable than playing games for me. butnotformycurrentprojectatworkholycrapthatoneissobadihateitugh

I realize I keep very quiet and nobody here knows me at all. As such, you admins have absolutely no reason to trust me. But I've got to start somewhere. Even if you start thinking about any of this, it's a win for me. I've been turning it all over in my mind for the past couple of months and couldn't keep silent any longer. I want to help. May I? What would it take to convince you I could or should? is this thing working yet?

Thanks for your time. Please add your thoughts, everyone. I'm very curious as to what'll happen here.

[makaveli]

I like your idea and that it is awesome you want to contribute your knowledge to better the server. However I doubt they will accept any help, as many people have offered their services before even during times of a ddos crisis, only to be turned down their assistance. Best of luck though hopefully they will allow you to do something.

[Cyane]

It is being worked on, that lv reset issue/Items vanishing etc, at least the problem is located.

It's an old game with an old code, anyone could mess it up easy as it is now. The issue lies on having an old code if somebody with enough time would recreate it in a more safer way (there's no way to avoid hacking there's always somewhere some way to exploit), recent or old all games have their own issues.

It's something even without anyone (accepting help or saying they want it), if you are able and got the time to work on, making a few patches to fix the exploits on your own private server and sharing, it's a great help hand for all the other servers.

If pioneer2.net had a art side i would make many stuffs available there too. There needs to be somebody there asking for more and keeping all of them motivated.

It's hard to get that trust at start but start your own thing, share with the community and we will surely progress.

hope to hear more stuffs from you

[while]

Thanks for the kinds words. The intention is not to start a brand new server, new community, or new competition, but to work with what is in use. I'm concerned about the stability of the server and integrity of the data here.

I would look into creating patches if the code was available to me. Is it open source here? I haven't seen anything pointing to it being open source. pioneer2's latest code I could find looks 5 years out of date and I don't even know if Ultima forked from it. I had been looking through it anyways, however, there's no point in suggesting changes unless it's actually compatible with Ultima. If I do find anything to share, I really would want it to be something that could be put into use.

[Stark 1]

[while]

I just won't shut up about this stuff and go away!

I've been looking more through the Tethealla source here and there since last weekend. Here are my thoughts on it. Does this all apply to Ultima's code at all? I don't know. At minimum, I'd like to show that I am serious about contributing my time. Anyway, here's what I've been poking through.

The login server allocates memory up front for the maximum possible client connections. Each connection is about 2Mb. That is far too much for what it's used for. The majority doesn't need to be held onto. In main()'s socket loop, each client has its own set of data buffers which are populated and later processed. Memory usage could be cut down by reusing a single buffer for all clients: populate the buffer and process the data immediately, then who cares about holding onto that buffered data? Just move on to the next connection. The setup and the processing are pointlessly disconnected from one another and wasting all that memory.

The login server has 46 unique MySQL queries in its source file. The queries are all for the most part formatted by sprintf() and directly executed. This method sends the query string to the server, which has to parse it every time. Prepared statements could help improve this situation, at the expense of some of the MySQL server's memory. Prepared statements are parsed by the MySQL server a single time, data is bound to placeholders in the query, and only the data itself needs to be transferred. This would benefit queries that are executed many times. Drawbacks would be setting up the prepared statements in the server code and the fact that the MySQL server has to store the statement handle.

MySQL transactions might worth using for certain database actions (they are not used in the Tethealla source I'm looking at). These can make a series of data changes atomic. Performing multiple steps like "remove an item from the player's bank" followed by "place the item in the player's inventory" would be guaranteed to never perform one action without the other; either both happen or neither happen. The item is properly moved or safely stays in the bank. Despite how it sounds, this is the least important of these things I'm bringing up.

Numerous chunks of code are copied and pasted in multiple places throughout the source files. This is a nightmare to maintain. Better be sure that if you update one, you update all the other instances of the same function! Help keep earth.c clean and recycle. There's no excuse for slapping multiple copies of the same code everywhere and pointlessly bulking up the source files. Also, there are many massive functions that should be broken down into several smaller functions.

Existing libraries could be used to cut down on the server's code. Two prime choices for removal are the MD5 and the Mersenne Twister implementations. MD5 can be done for a Windows platform through the Cryptography API or for linux through libgcrypt. The Mersenne Twister is available from <random> in C++11. Both the crypto and the twister libraries are available in Visual Studio 2012.

If using C++, the MySQL C++ connector library could be used in place of the C API. This would greatly simplify the MySQL interface, especially for prepared statements. I will always be pushing for C++ here. The STL would be hella sweet to be using: easier to manage containers of objects like the client connections, easier to parse the configuration files, easier to break peoples' spirit by using templates, etc. C++, awwwyissss. As a side note, fuzziqer's newserv is in "C++" but makes absolutely no use of anything C++ that I could see when browsing through that project. Kind of depressing.

Aside from the Tethealla source, I've been thinking about the PSO control panel, which can easily be done from the database viewpoint. Honestly, the only hard part would be the user interface. Exactly as chuk stated in that original thread about the panel, knowing if an account is online is the single most important part of this. And to be a complete jerk about it, I will tell you right now that I am keeping the specifics to myself to attempt to gain leverage for myself.

I also began looking into the client's disassembly but am not yet in a position to report anything since I just started looking at a few functions of interest.

So yeah, the whole point of this nonsense is to get the admins' attention and let them know I do mean business. If you say no to me, I will stop whining about wanting to do development stuff and just chill. I will be quite disappointed if I only get silence from the admins here, like at how it seems to be at pioneer2. If you say yes, I will try my best.

[Soly]

I'm by no means an expert in programming.. but i like the points you stated

from some of them

Memory... i can't test with 70-100 clients at a time... as far as that goes i think memory is not really an issue (but is always good to keep everything in shape).

Queries ... indeed... in my database manager i was using simple statements... later changed everything for prepared ones for security/simplicity/etc .

Reusing code... i have been improving this in my projects a lot.. i don't like to have 2 or more things that do the same (i mean like irl or my pc) same goes for the code.

SQL Connector.. i made my database manager in java and i use the connector for java... it would be awesome integrating this into the server.

(I'm not admin... but i like it, hope larva looks at it )

[chuk]

Hey, first of all it's nice to see someone as enthusiastic about helping out with the development part of this old but lovely game. As you guessed it, the ultima source is indeed based on tethealla with quite a bunch of tweaks and fixes.

We're absolutely aware that the code is far from perfect and could use some polishing up, or even be recreated from scratch with a more OO approach (working slowly on that in C# in my scarce spare free time, but that's another story). So, moving on to your points :

The login server allocates memory up front for the maximum possible client connections. Each connection is about 2Mb. That is far too much for what it's used for. The majority doesn't need to be held onto. In main()'s socket loop, each client has its own set of data buffers which are populated and later processed. Memory usage could be cut down by reusing a single buffer for all clients: populate the buffer and process the data immediately, then who cares about holding onto that buffered data? Just move on to the next connection. The setup and the processing are pointlessly disconnected from one another and wasting all that memory.

The only benefit I see from this approach, is that this allows for a smoother debug, and making sure that one buffer doesn't interfere with another client. The buffer should be completely thread-safe to guarantee that no garbage data is being passed on from one client to another. But it's indeed more common nowadays to see one buffer doing the work for all clients.

The login server has 46 unique MySQL queries in its source file. The queries are all for the most part formatted by sprintf() and directly executed. This method sends the query string to the server, which has to parse it every time. Prepared statements could help improve this situation, at the expense of some of the MySQL server's memory. Prepared statements are parsed by the MySQL server a single time, data is bound to placeholders in the query, and only the data itself needs to be transferred. This would benefit queries that are executed many times. Drawbacks would be setting up the prepared statements in the server code and the fact that the MySQL server has to store the statement handle.

I fully agree that prepared statements are a safer and more efficient approach, together with a strong filtering system on passing on the arguments to it. Another option would be more stored procedures, but the drawback of those is that they need to be implemented on the actual SQL server and that it would slightly compromise the flexibillity to be able to use a different db type in the future. (for example Postgresql). But since the syntax has its 5% difference between several SQL db's, I guess it's not as flexible now anyways. (PDO ftw). But yes, prepared statements have great benefits in regards to SQL injections or even coding errors.

MySQL transactions might worth using for certain database actions (they are not used in the Tethealla source I'm looking at). These can make a series of data changes atomic. Performing multiple steps like "remove an item from the player's bank" followed by "place the item in the player's inventory" would be guaranteed to never perform one action without the other; either both happen or neither happen. The item is properly moved or safely stays in the bank. Despite how it sounds, this is the least important of these things I'm bringing up.

I made a post on pioneer2.net with a new proposal for the database architecture where I also aim for a more granular approach in saving data to the database. This allows again for more flexibility for the programmer, less network overheat and overall a cleaner way to store data. Problem is, the client expects one big blob of data anyway, so we would have to parse and re-assemble the blob in the server on the fly. (Which is not really a problem actually lol)

Numerous chunks of code are copied and pasted in multiple places throughout the source files. This is a nightmare to maintain. Better be sure that if you update one, you update all the other instances of the same function! Help keep earth.c clean and recycle. There's no excuse for slapping multiple copies of the same code everywhere and pointlessly bulking up the source files. Also, there are many massive functions that should be broken down into several smaller functions.

Hence I'm trying to remake it with an OO approach, so much easier to expand and maintain. But it takes time to work on it, time I don't really have with my fulltime job... Would be great to get help on this. If you're interested I could throw it in a GIT repository so we could work together on it. Was planning to make it opensource anyways, but it's still in a very early stage, so don't expect anything functional soon.

Existing libraries could be used to cut down on the server's code. Two prime choices for removal are the MD5 and the Mersenne Twister implementations. MD5 can be done for a Windows platform through the Cryptography API or for linux through libgcrypt. The Mersenne Twister is available from <random> in C++11. Both the crypto and the twister libraries are available in Visual Studio 2012.

Yeah, some libraries are outdated/obsolete with nowadays technology. Dusting those off would be a good idea.

If using C++, the MySQL C++ connector library could be used in place of the C API. This would greatly simplify the MySQL interface, especially for prepared statements. I will always be pushing for C++ here. The STL would be hella sweet to be using: easier to manage containers of objects like the client connections, easier to parse the configuration files, easier to break peoples' spirit by using templates, etc. C++, awwwyissss. As a side note, fuzziqer's newserv is in "C++" but makes absolutely no use of anything C++ that I could see when browsing through that project. Kind of depressing.

Personally I don't like C/C++ all that much, I would have chosen any other OO language over those. I know OO can be done with C++ but still, jsut not my cup of tea.

Aside from the Tethealla source, I've been thinking about the PSO control panel, which can easily be done from the database viewpoint. Honestly, the only hard part would be the user interface. Exactly as chuk stated in that original thread about the panel, knowing if an account is online is the single most important part of this. And to be a complete jerk about it, I will tell you right now that I am keeping the specifics to myself to attempt to gain leverage for myself.

When working on the mana client, I found a way to check if a player is online or not. Even better, it shows a list of all online players with some basic information. Just hasn't been a priority to fix the CPanel for now.

I also began looking into the client's disassembly but am not yet in a position to report anything since I just started looking at a few functions of interest.

So yeah, the whole point of this nonsense is to get the admins' attention and let them know I do mean business. If you say no to me, I will stop whining about wanting to do development stuff and just chill. I will be quite disappointed if I only get silence from the admins here, like at how it seems to be at pioneer2. If you say yes, I will try my best.

On the contrary, you bring up some valid points that most of us already talked about but never really got motivated to fix. Ofcourse we won't just give you access to the server code rightaway, but if you can make a solid proof of concept with the tethalla source, we'd be glad to add fixes to our source. And if it all goes smooth, over time you might be able to work your way into the ultima developer ranks

PS: I'd like to go deeper over some subjects, but I'm at work now, so can't really spend too much time on this now.

[while]

Thanks, both of you. I think the biggest problem I was having was with simply trying to approach you guys. I don't fully know what you've already discussed amongst yourselves since I'm late trying to get in on this.

It sounds like you like Java a lot, Soly. And chuk doesn't care for C++? Where's the C++ love? Regardless, I'd be glad to check out whatever new things you guys are putting together. What languages are you actively using the most now? chuk, If you do bother with a git repo, please let me know. I hope it's not C# though... not a fan.

I know the Tethealla source that's available is old but it was all I could get access to. And I know that I really shouldn't have access to anything for Ultima. Indirectly making contributions that are filtered through a git repo or somthing like that would be totally fine by me. I shouldn't be allowed to charge in and ruin everything forever anyway.

You made my day responding though. Please keep me informed on whatever may be going on.

And I'm sticking to my opinion of the main() loop reusing the buffers!

[chuk]

Bad news for you, It is in C# I just like that language a lot

=p