mapclasses.h, mapstuff.h, and mapstuff.cpp

[lingo]

I am beginning to document uox3 sources. The first one I choose is the map and tile class.

There are comment dating back to 1999, 'fur' said the loading of *.mul at the startup is slow, so the develop teams wrote caching scheme to avoid loading the whole thing. But if I understand correctly, the biggest data file is only 77MB. So we could load the whole *.mul to the memory. today's hard disk is much faster and main memory is vastly more plentiful. A few seconds more in startup won't matter that much,

Even if we worry about memory usage, which I doubt, we could use memory-mapped file. Memory-mapped file's caching scheme is going to be as good if not better than our implementation. That will simplify the class tremendously and eliminated those hard to debug caching code, and we will get better performance as bonus. By the way, wolfpack use the old caching logic but prettify it a lot.

[giwo]

You picked quite the doozy to start with.

In complete honesty, with more than 7 years working on/around UOX3, I've done my best to avoid those files at all costs.

I am quite sure there is alot that could be done to improve our caching and handling of the mul files. I am not quite so fond of the idea of putting them in memory entirely (the current UOX3 footprint is already well over 30mb when last I looked, with a small worldfile). But if someone is interested in giving that area of the code some love, they would have my support.

[Xuri]

Is there a chance that Line of Sight being broken (I think) through walls in custom statics-files is related to the caching of the statics-files?

[lingo]

I am quite sure there is alot that could be done to improve our caching and handling of the mul files. I am not quite so fond of the idea of putting them in memory entirely (the current UOX3 footprint is already well over 30mb when last I looked, with a small worldfile). But if someone is interested in giving that area of the code some love, they would have my support.

ok, got it. I'll try a memory-mapped file implementation.

Is there a chance that Line of Sight being broken (I think) through walls in custom statics-files is related to the caching of the statics-files?

Don't know, but after I finished the memory-mapped file we can test it together to see what really happenend.

[Sydius]

While I fully agree that memory mapping the files would be the best solution, are you planning to use Windows-specific code to do so? I wrote a memory-mapping class once, and it relied heavily on the Windows API, which simply will not do for UOX. I am not sure, but perhaps there are cross-platform implementations of it that might work?

If not, I would rather see the files stored entirely in RAM – caching is great and all, but useless with modern hardware. A 30MB footprint is nothing to worry about, nor would a 200MB one. Anyone serious about running a server will have much more RAM and anyone not serious will have it handled by paging anyway, at no major cost in performance.

The real reason I would like to see it entirely in memory (or access via memory map) is for simplicity, not performance, though. While the potential for UOX to out-shine the competition via performance is real, right now I believe stability is the key, and simplicity would go fathoms towards that end.

[Sydius]

Oh, and as for it breaking LOS, that may be the cause – but only if the caching code is not working properly (which is not that unlikely!). Perhaps we should test it. Maybe ask for statics/tiles randomly, comparing it to actual values, and see if any discrepancies arise.

The whole unit-test thing we were talking about, basically.

[lingo]

While I fully agree that memory mapping the files would be the best solution, are you planning to use Windows-specific code to do so? I wrote a memory-mapping class once, and it relied heavily on the Windows API, which simply will not do for UOX. I am not sure, but perhaps there are cross-platform implementations of it that might work?

Both Windows and *nix (posix rather) have memory-mapped files, it not portable, but a few #ifdef will solved it rather nicely. After we memory-mapped the files, we can access it like it was in memory, so the code is really simple and straightforward. Only a few functions calls, wrote it in 15 minutes, but googling it for 2 hours.

[lingo]

Anybody still got 'verdata.mul' in their UO client's directory? wolfpack's source code said it does not exist anymore, not for a long time. My UO client doesn't have it, but it very new. I will remove the relevant code if confirmed.

It makes sense since it was designed for patching multis and tiles data. But why goes roundabout, just patches the relevant files, multi.mul and tiledata.mul, directly.

[Sydius]

I think it has been gone for some time.

[Maarc]

I don't think that a server *has* to have a verdata file, but the support is there in case it is.

Reason being, that, by and large, we support any client from 2.0 onwards, I believe, though some things have started to get sent that may break that.

The question, of course, is what level of client is to be supported. Not necessarily an easy question to answer.

As far as caching goes... while I'm one of the first people to say that we have too many code based options (the INI file is too big, really), I don't think it's really *necessary* to force everyone to cache these files. I think shifting tiledata over to a cache compulsory would be good (it's a 1MB memory hit, and used fairly regularly). But statics and map data itself doesn't really need to be.

Remember, one of the things that UOX supports is 4 maps, and (unlike the client) up to Brittania size on all of them. Start throwing those around, and you're talking over 300MB of memory, where the vast, VAST, majority of it will be unnecessary. You'll also find that there is data in some of those files we just don't read or care about (the extra field in statics, for example), so what we actually store and use in memory is less than what is stored on disk. Just chunking it all into memory isn't necessarily the smartest solution.

I *know* that some of this stuff needs to be rethought out, but other classes external to those files depend on it to some degree (LoS, as you've seen, as well as MapRegions as another big one).

The question of responsibility of work becomes raised. Where do you put all of this code? Does all elevation/blocking code get put into a single class (which is a potential bottleneck, if ever threaded)? Do you distribute the logic wider (into, say, MapRegions)? Does the MapStuff become just a dumb store or does it do more (mapdiffs, for instance, I don't think are supported yet)?

Not trying to force people into any one solution, but just thought I'd mention some things. It's very easy (how well do I know this!) to get stuck on an idea and go at it, all guns blazing, without a huge degree of forethought.

[lingo]

I think 'verdata.mul' is deprecated, but I don't know when. We can only support it if I have the file to test against it, no testing no robustness. So anybody still using 'verdata.mul', please comment about it, and sent the file to me. lingo # ningo dot com dot tw.

Yes I am moving multiitems and tiledata to main memory directly. They are use often enough and small enough,less than 2 megabytes combines, to do so. statics and maps data I memory mapped them. They are around 150 megabytes combines, not insignificant. I think Sydius meant that if memory-mapped file is not support everywhere then we should seriously think about using main memory directly. Since memory-mapped file is support everywhere, the one we care about anywhere, so it not a problem now.

I *know* that some of this stuff needs to be rethought out, but other classes external to those files depend on it to some degree (LoS, as you've seen, as well as MapRegions as another big one).

The question of responsibility of work becomes raised. Where do you put all of this code? Does all elevation/blocking code get put into a single class (which is a potential bottleneck, if ever threaded)? Do you distribute the logic wider (into, say, MapRegions)? Does the MapStuff become just a dumb store or does it do more (mapdiffs, for instance, I don't think are supported yet)?

I don't think mapdiffs exist. uox3 only patch tiledata and runuo only patch multiitem. But threading issues, that real problems, I didn't thought about it. Originally, I was just trying to document and testing uox3's source files. But since the map class is obtuse enough, I thought rewriting the caching part then test for it would work much faster. I didn't change the API of cMapStuff, I just rewrote the caching code. I understand this is part of the questions that you want people to think about. Currently I think MapRegion's interdepency with cMapStuff looks problematic too me. I don't have an solution yet, as I just mentioned, currently I am more of documenting and testing phase. I don't understand it enough to have a good judgement. I am in a simplify the code phase and understand the code phase right now. But I think you (Abaddon?) have some solution to this Map problem. Care to share some some thought? I am all ears . we can work together on this.

Other Design I have an opinon now
Generic Properties
don't have the book. (Gave away when I decided to quit the Game Industry 2 years ago ). But if it involved a lot of template magic, I would have my concerned. See the boost library, they use template to implement lambda, closure, and metaprogramming, they gone too far, they are hijacking c++ to do something it was not meant to do. If I needs those, I would be using Lisp or Ocaml right now. which in fact I am learning some now.

Databases
Sqlite. Definitely, especially the ease of use in Windows environment.

Testing and Maintenance
Unit Testing all the way. I saw good result when apply religiously.

Other?
I am reading 'Massively Multiplayer Game Development' now. Honestly I am more of coding type, so I can help in coding and structure, but play design is more of a problem for me.

[Sydius]

I see absolutely nothing wrong with template meta-programming – in fact, I think it can go a long way. Just because it was not an intended capability does not detract from the usefulness of it – it well-defined, and happens to be perfect for meta-programming as an accident. When you really get into it, it is just so amazing that I have a hard time seeing why anyone could not like it (except, perhaps, because it is a little awkward to learn).

At any rate, if memory-mapping is an option, that would be the best of both worlds – fast access and little memory requirement (less even than caching). I wrote one of my map-editing utilities so that it took advantage of memory-maps, and it increased access speeds by more than 20% without increasing memory usage much at all. It only helps on big files like the maps, though.

As for verdata, I believe support should continue for it.

[lingo]

I see absolutely nothing wrong with template meta-programming – in fact, I think it can go a long way. Just because it was not an intended capability does not detract from the usefulness of it – it well-defined, and happens to be perfect for meta-programming as an accident. When you really get into it, it is just so amazing that I have a hard time seeing why anyone could not like it (except, perhaps, because it is a little awkward to learn).

Nothing against metaprogramming, but after reading "C++ Template Metaprogramming : Concepts, Tools, and Techniques from Boost and Beyond" I got scared. The way they manipulate the template mechanism!!! the clever tricks they employ., it makes my head hurt. I just don't think c++ should be use this way. I think metaprogramming should be support inside the language if it a desireable feature. I believe those generic programming concept is just not just well support by the template mechanism as it is. It very difficult to write and debugged those feature. Look at OCaml, the way they support type inference (generic) is just nature and easy for programmer to use. But, but, but ,may be I just got scared by that book, I really should go back and look again.

As for verdata, I believe support should continue for it.

I have no problem with that. I just need the data file to support it.

[Grimson]

Databases
Sqlite. Definitely, especially the ease of use in Windows environment.

I would rather see generic SQL support, because I already have a MySQL DB running on my server machine.

But IMHO the bigger bugs, like the not working books and bulletin boards, should be fixed first before you start to implement new features.

[lingo]

I would rather see generic SQL support, because I already have a MySQL DB running on my server machine.

But IMHO the bigger bugs, like the not working books and bulletin boards, should be fixed first before you start to implement new features.

Definitely, get the existing functionality to work is the first priority. Database is just some ideas, nothing set in stone. Keep the opinion coming

[Maarc]

It's a bit late here, and don't have a lot of time. I'll come back to this tomorrow and give you my thoughts on the MapRegion/mapping stuff, takes a bit to write out (though you can probably see a lot of it from the UML diagram I posted on the design page).

In a nutshell, though, it is basically just portioning out some of the logic (line of sight, statics/multis, "caching", even things like speech and other calcs) into a MapRegion's responsibility, with the occasional communication between MapRegion's (ie at a border).

[lingo]

After 2 days of struggling, I decided Ubuntu just don't want us to install Java . Give up, use my windows machine instead. Now I finally saw your uml diagram.

Nice work, the separation is very clear, you must spend a lot of time on this. Now a personal nitpick, cBaseObject, too much XXObject already, maybe cBaseThing, or cBaseStuff, I am not too imaginative . Also I think I might not like cPlayerChar, and cNpcChar, inherited from cBaseChar, I want a cChar, and it contains a controller, playerController, or npController. That just my current opinion, it will changes after I understand more of uox3.

Just got cMapStuff and UOXFile done, no change in API, just change the caching scheme. Still thinking about the separation of cMapStuff, it seems multis should go with cMulits? Want to join uox3 irc? could exchange ideas faster , I think, I will stay there as much as I can.