Linux Version

Want to discuss changes to the UOX3 source code? Got a code-snippet you'd like to post? Anything related to coding/programming goes here!
Post Reply
Devlin
UOX3 Newbie
Posts: 3
Joined: Wed Mar 09, 2005 2:18 pm
Has thanked: 0
Been thanked: 0

Linux Version

Post by Devlin »

Hi,

I'm running a small shard, which should be converted to UOX3, main problem was, that the shard needs to be running on a Linuxsystem. The linux-installer doesn't work for me, the Makefile created bei automake and friend was completly broken.

I downloaded latest CVS, and build a new Makefile for UOX3, so that this baby can be easily build under Linux (any BSD Folks here to test?). I will make a diff against the current CVS and post it here or send it to someone (who?).

Hopefully you agree, that having a *nix version couldn't be that bad :D

Stay tuned, I will run some stresstests to see if it's working, if it is, the patches will follow.

Devlin
p.S.: SlaveOnDope on Freenode.
Devlin
UOX3 Newbie
Posts: 3
Joined: Wed Mar 09, 2005 2:18 pm
Has thanked: 0
Been thanked: 0

Post by Devlin »

I found some problems with the dos-formation of the files. uox.ini must be in unix format, if it isn't no pathes are found, because they are expanded to something like:
/home/devlin/UO/uox-all/^M/

I resaved uox.ini in unix-format and UOX starts up, all other scriptfiles are problematic too, the jse_* Files are read the way, so my question to the devs is, where to add a Filter against this, I looked as ustring.cpp, but this would slow down the linux version massivly, on the other hand I could write a shell-script which converts all file to unix-format before startup.

What's your opinion?
giwo
Developer
Posts: 1780
Joined: Fri Jun 18, 2004 4:17 pm
Location: California
Has thanked: 0
Been thanked: 0

Post by giwo »

Hmm, quite interesting...

I am actually working on getting UOX3 functioning properly on gcc 3.4.2 using MinGW for Windows. So interested in any issues it is having on Linux / Unix systems.

I would say unless there is a good way to get the files in a format that will work on both Win32 and Posix systems, some form of script that automatically converts them for linux users would be the best way. I'd be hesitant to carry two lines of scripts as it would no doubt cause one to eventually become outdated (relying on someone with a linux/unix OS to update them, where most of our active people use Win32).

I take it since you got this far, you managed to get it compiling successfully? Out of curiosity, what compiler did you use, and were there any areas of the code you needed to fix?
Scott
Devlin
UOX3 Newbie
Posts: 3
Joined: Wed Mar 09, 2005 2:18 pm
Has thanked: 0
Been thanked: 0

Post by Devlin »

Ok here are the latest news:
UOX3 compiles fine, the resulting binary is 3,1 MB in size, packed with upx we got 1,1 MB. Sourcebase was the latest CVS Version.

Some .c_str()'s were missing, because the were not missing in the Windows-Define-Section nobody noticed. I will do a diff tonight or tomorrow and will contribute the changes.

Are the auto* tools includes in minGW? If so we could share this buildsystem between minGW for Win32 and gcc under any Posix-OS.

My main problem at the moment is the RAM Usage, 73 MB without a single player connected is too much in my opinion, maybe this is a STL Problem ... another way could be to staticly link the binary.

Some more details:

Code: Select all

seb@mushroomhead:~$ gcc -v
Lese Spezifikationen von /usr/lib/gcc-lib/i486-linux/3.3.5/specs
Konfiguriert mit: ../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared --with-system-zlib --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread-Modell: posix
gcc-Version 3.3.5 (Debian 1:3.3.5-8)

Code: Select all

seb@mushroomhead:~$ automake --version
automake (GNU automake) 1.4-p6
giwo
Developer
Posts: 1780
Joined: Fri Jun 18, 2004 4:17 pm
Location: California
Has thanked: 0
Been thanked: 0

Post by giwo »

Well, I'm still new to mingw, but I'd have to say, at least from a Windows users perspective, there's nothing auto about it. :)

It does have a make function, which can utilize a properly formatted makefile to generate the executable, but the makefile had to be generated first (either manually or [and what was done in this case] by a program like tmake).

No suprise that there were some... lacking... things in the specifically defined sections, I'll be the first to admit that we (including myself) have a bad habit of "fix it when the compiler pukes" :)

As for the RAM usage, it sounds like you built a debug build (based both on size and RAM usage). I generate an exe ~ 1600kb that uses about 30-40mb of RAM. Still alot, but I haven't found a good way around that, currently (as much of that is the JS Engine & Scripts). Still, any way to bring down what we can I'm more than open to. :)

As for compiling with MinGW, I'm currently getting a compiler error when building a specific line of code (namely the array internalRegions[][][] in cMapRegion). Perhaps this is a new issue in gcc 3.4.2??
Scott
Maarc
Developer
Posts: 576
Joined: Sat Mar 27, 2004 6:22 am
Location: Fleet, UK
Has thanked: 0
Been thanked: 0
Contact:

Post by Maarc »

The JS subsystem reserves 16MB at startup (configurable via engine.dat, put in a single line of text which is the size you want, however it can't be less than 16MB). Each script reserves 8KB. The resource system (logs and wood and stuff) reserves 250,100 resourceEntry (so about 3MB, prolly more like 4MB with structure padding, but prolly needs to be updated for multi-world stuff). There are 24,576 SubRegions (which are mostly two vector pointer lists, and a couple of funcs, so probably 30 or 40 bytes each, so another MB there). All the DFNs get loaded into RAM, and remain largely as text (barring items/npcs, they get tokenized down to smaller byte based stuff), so there's a chunk of RAM there. TileData gets chunked into memory in some fashion, as does multi.mul/multi.idx. Depending on your caching selection, the maps might make it in as well :) Given how much game data gets stored in memory, 30-40MB at startup isn't unexpected.

so... 16 + 4 + 1 + 1 (for DFNs) + 1.5 (multis/tiles) and I'm sure I'm missing other things, but that gets you to 23.5MB. Not that big a stretch to get to 30 with any real data load.
Sydius
UOX3 Apprentice
Posts: 171
Joined: Thu Mar 25, 2004 3:22 am
Has thanked: 0
Been thanked: 0

Post by Sydius »

Having the maps in memory would probably be worth the expense in RAM, I would say, and that would obviously increase consumption dramatically.
Maarc
Developer
Posts: 576
Joined: Sat Mar 27, 2004 6:22 am
Location: Fleet, UK
Has thanked: 0
Been thanked: 0
Contact:

Post by Maarc »

Yes, and I believe you'll find static/map caching is controlled via an entry in the config file, the support is already there, you just have to turn it on. Much more useful for dedicated servers, greatly reduces a lot of IO.
giwo
Developer
Posts: 1780
Joined: Fri Jun 18, 2004 4:17 pm
Location: California
Has thanked: 0
Been thanked: 0

Post by giwo »

So, having worked more on this region issue in gcc 3.4.2, I have seen a few other things.

the current size of the array is this (once all the math is done)

internalRegions[192][32][4] or 24,576 regions.

Now the breakpoint for this to work is here

internalRegions[127][32][4] Which works out to 16,256 regions.

As soon as any of those numbers rise even by 1, it pukes.

IE internalRegions[128][32][4] Which works out to 16,384 regions.

Now, to be honest, I think this is a bit much. I can admit that I'm not entirely familiar with the special handling needed for Ilshenar and Malas, but is it actually the same as the world itself (Do we need roughly 4,000 regions for each)?

Here is how I understand the worlds
Felucca
  • T2A
    Malas
    Ilshenar
Trammel
  • T2A
    Malas
    Ilshenar
Which would basically put each of those on the same plane as the regular Brittania map, in each world. (Obviously they are physically down in the corner on the map, but does that matter?)

Anyhow, as it stands, without revamping the entire system (perhaps using a double array for each world) this is a problem that will hinder the use of MinGW to compile UOX3. Hopefully when gcc 3.4.3 is released for use with MinGW it will fix the issue, but I'd like to consider nailing the excess subregions, if we can.
Scott
Maarc
Developer
Posts: 576
Joined: Sat Mar 27, 2004 6:22 am
Location: Fleet, UK
Has thanked: 0
Been thanked: 0
Contact:

Post by Maarc »

Few quick thoughts, some of which we've already discussed giwo, via MSN. Unfortunately, the extra lands (including the Samurai one, which we don't currently reflect, but another point :)) are not simply stuffed into a corner, but their own map files. I guess that, conceptually, we could just make a 12,288 x 8,192 sized internal map (that's 2x2 of the Brittania map, for those that care), and just plonk them beside each other. It would get away from the third dimension, but it would also introduce a layer of positional logic that we would have to care about, and doesn't make it easy for the 5th world.

Another option is to tweak with the size of the SubRegions. The SubRegion is there for smaller location based checks, so we avoid the entire world whenever we want to look for something. If you doubled the width, for instance, it would immediately solve the maximum number issue. Alternatively, the GCC/Cygwin/Mingw people could just fix their compiler, as I don't think there's anything there that breaks C++ rules.

As an aside, we might want to look at reevaluating the block size anyway. We do a 3x3 grid search (so effectively, 96 x 384) whenever we need to activate certain things. Perhaps changing to 64x64 might be more efficient, but it all depends on what we have, and where the bottlenecks are. Benchmarks might not be a bad idea (I've been thinking of benchmarking/load testing for a while, say, writing a dummy client that you can spawn many instances of, that thrash the server with certain inputs and other things, so we can see how things cope under load, and a way of testing for odd/bad inputs).

Just some thoughts.
punt
VIP
Posts: 244
Joined: Wed Mar 24, 2004 7:46 pm
Has thanked: 0
Been thanked: 9 times

Post by punt »

Maarc wrote:
Alternatively, the GCC/Cygwin/Mingw people could just fix their compiler, as I don't think there's anything there that breaks C++ rules.
Hehe, that is always true, for every compilier by the way! One fundementally decides to select a set of compiliers they plan on supporting, and then work around their issues (as was done with MSVC 6). Why some consider "ports" to be those environments that are not directly supported!

Of course, one can always choose. But unfortunately I have seen one project internally at my work get axed, for it took the purist view. Did a project that just insisted on a perfect C++ compilers. Unfortunately, there wasn't enough of those, and the one they finally did code to, left them too small an environment for the intended release.

So a purist view, a great place to start, but rarely a good place to finish at.
giwo
Developer
Posts: 1780
Joined: Fri Jun 18, 2004 4:17 pm
Location: California
Has thanked: 0
Been thanked: 0

Post by giwo »

Well since the release of 0.95.x, which had that array exactly as it stands now, I've never heard a complaint of the compiler puking on including regions.h. That being the case, I still hold the opinion that it's probably a quirk in gcc 3.4.2, possibly later. I know it has been compiled in gcc 3.3.x and never heard any issues with it.

As for changing the size of internalRegions, my main worry in this case is changing the worldfile... again.. :) My world is mostly an empty bit of nothing. Xuri, however, is working on quite the nice reproduction of an OSI world, and I'm sure others have their treasures. Disturbing peoples worlds is something I try to avoid whenever I possibly can.
Scott
Maarc
Developer
Posts: 576
Joined: Sat Mar 27, 2004 6:22 am
Location: Fleet, UK
Has thanked: 0
Been thanked: 0
Contact:

Post by Maarc »

Sure Punt, I can understand that, it's always nice to try and keep things pure and clean, it's just unfortunate that the real world intervenes and forces everything to muddy up :)

Sorry to hear about that work project, from the sounds of it, it could have been quite interesting (or at least, you seem to have been interested in it, so I imagine it was interesting).

I realise my comment was a bit off the cuff, though, I should have specified more details (which giwo has done). It's certainly nothing new to the 0.95.x code base, and it only seems to be on one variant of the GCC compiler, so it all seems a little odd.

As for giwo... yes, I fully understand the dilemma, you hardly want to worry about worldfile changes and everything, that's for certain :) However, the syntax wouldn't change at all, nor would the way of parsing. We'd just churn out files with slightly different names (ie the second number might go higher, the first lower). But that could be relatively easy to deal with, especially if we moved to portable directory code, as you could then just load all WSC files that aren't reserved (guilds, effects, house, jails, overflow, regions). It's only the number of and naming of files that would change, nothing else. So to all intents and purposes, it should be simple to support.

But as always, these things are never simple to decide upon, I agree. Such changes are never to be undertaken lightly without fully checking the consequences :) Just throwing out ideas, that's all, I can certainly understand how we hate the juggling act!
punt
VIP
Posts: 244
Joined: Wed Mar 24, 2004 7:46 pm
Has thanked: 0
Been thanked: 9 times

Post by punt »

I am never clear in these forums.

Ok, I believe the project (any project) chooses the environment it is commited to having it work in.


What I have heard has been windows/linux, msvc (7 and 6), gcc, mingw.


Ok, one can assume (and I realize that is not necessarly a great thing on my part) that it means the current available or readly available versions on those environments.


In this case, mingw 3.4.2. I don't' disagree nor dispute the conclusion gathered that it is a mingw error. Ok. Reality is one normally must work around compilier errors, and they get introducted with any release.


I am not lobbying that it be changed to support mingw 3.4.2 either. I have imposed an approach that one can consistently applye, and lets users have some understanding of where they will be in support. Clearly it may not be this projects, and their are other ways to do things.

I offer however that making users start using a tool just for your project, makes one take additional effort. Normally if one starts taking additioanl effort, they are more inclined to look for alternatives.

As for the project example, yes, it was rather interesting. But unfortunately it was managed naively. It clung to the fact it was in the right, and the tools where not. They won the moral arugment, but it made no difference in getting our product out for the environment we had commited to.

As for the magnitude of making changes, I agree as well, changes should be considered and thoughtful. But that is always true for any change. What we are talking here is the stimulus for the change, not the change itself. One is considering if the stimulus for a change is valid. So if MSVC 2005 (coming out this fall) has the same kind of error, or the next service pack for 2004 Net introduced a similar bug, they would just be dropped? That is the way to validate the stimulus for change. Or validate if it is a "supported" environment.
giwo
Developer
Posts: 1780
Joined: Fri Jun 18, 2004 4:17 pm
Location: California
Has thanked: 0
Been thanked: 0

Post by giwo »

My thoughts are that if MS went and broke something significant (IE not just a small change, or a few small changes, but possibly causing the need for a whole subsystem rewrite), I wouldn't initially support their new compiler. Now of course, with time we would work towards support of the latest and greatest, but it certainly wouldn't be an immediate thing.


The same goes for gcc 3.4.2, at this point. I'm more hoping for an update to MinGW than planning on a change to internalRegions. There have, though, been some good thoughts for changing the structure of internalRegions, and thus fixing this issue, which I am not against.

Ultimately it's a matter of time and priorities. I have heard very little call for support on MinGW specifically. I can see the need for it, and would like UOX3 to compile properly on it, but the primary reason I took on this endeavor was to fix any possible gcc issues (that of course, aren't in the #ifdef'd areas).
Scott
punt
VIP
Posts: 244
Joined: Wed Mar 24, 2004 7:46 pm
Has thanked: 0
Been thanked: 9 times

Post by punt »

I think it goes more to your statement, of wanting uox to go from a personal project to a wider, estalished project. One of the aspects is standard ways to support things.

Again, I am not lobbying for a change to support mingw 3.4.2. I only offer a perspective that starts one to consider what happens if this was on the compilier you where using (and wanted the update for other things you where doing). Would you keep two compiliers (and if multiple projects, then possible more then two), fix it, or ignore it. Clearly the past hasn't ignored it for MSVC 6. Mainly because that was the compiler one was using. A project that moves beyond a personal one (and I again, do not say it is bad being a personal project, but going on what I understood a dev to express) to something more, one establishes an approach and uses it consistently. Exceptions are considered with tijme frames for expriations (or stimulus for change).

This is true for the library discussion as well. I personally have nothing vested in mingw working, what libraries are used, etc. So I am not lobying for any of those. Only to consider , based on a stated desire, to consider aspects that may influence the probablity of reaching that desire.
giwo
Developer
Posts: 1780
Joined: Fri Jun 18, 2004 4:17 pm
Location: California
Has thanked: 0
Been thanked: 0

Post by giwo »

Well, I actually use 3, currently... MSVC 6, MSVC 7, MinGW. So I suppose adding on a 4th compiler without replacing the ones before it would be no big deal to me. :)

As for the scope of the project (being personal or a group project). It really is a chicken-egg conundrum, as you mentioned earlier. To have a team project, you must act like a team, but to truly act like a team, you typically need a team.

Whatever the case, time is always an object, in the case of UOX, it's a very important object, as one usually has very little free time, and such a large-scale project is quite good at taking up alot of it. Having said that, issues not seen as major by myself will need to be accomplished by others or will be gotten around to when I desire. :)
Scott
Post Reply