(Proposal) UOX3 Data Versioning System and Update Command

Forum where anything UOX3-related goes - including, but not limited to: newbie-support, ideas, general questions, comments, etc and-so-forth.
Post Reply
User avatar
Xuri
Site Admin
Posts: 3704
Joined: Mon Jun 02, 2003 9:11 am
Location: Norway
Has thanked: 48 times
Been thanked: 8 times
Contact:

(Proposal) UOX3 Data Versioning System and Update Command

Post by Xuri »

This is a proposal for a system to handle updating of world-file object data after upgrading to a newer version of DFNs. Any feedback to this proposal appreciated!
  • A version number is added to the INI file, which defines the "data version" for the current DFNs. This version number is applied to all new objects added from DFNs. Whenever some major updates are made to DFNs that need to be reflected in existing objects in the world-files, this version number is increased. Ex: DATAVERSION=3
  • When UOX3 starts up, it loads existing objects from the world-files and then checks the version number. If the objects have no version number yet (the world-files were created before the data versioning system was added), the items are given a dataVersion number of 0.
  • A special command - 'updateWorldData - usable by server administrators only, iterates through all objects in the world-file, and applies any version-specific updates to those objects until they are up-to-date with current DFNs, then updates their dataVersion to match the current one. This command would need to be updated whenever any big changes to the DFNs are done that need to be reflected in already existing objects in-game, with any new changes added towards the end of the script.
Example of what such an 'updateWorldData command would do:
var currentDataVersion = GetServerSetting( "CurrentDataVersion" );

// Iterate through all items/characters, and for each such object, run checks like...

if( object.dataVersion < 1)
{
    // ... update some data property here that changed between version 0 and 1
    // Example: Layer of weapon X changed from layer 1 to layer 2
    if( object.isItem && object.id == 0x### )
        object.layer = 2;
}

if( object.dataVersion < 2 )
{
    // ... update some data property here that changed between version 1 and 2
}

if( object.dataVersion < 3 )
{
    // ... update some data property here that changed between version 2 and 3
    // Example: Shopkeepers are made vulnerable
    if( object.isChar && object.isShop )
        object.vulnerable = true;
}

// Finally, the dataVersion on the object is updated to the latest version:
object.dataVersion = currentDataVersion;

Although the update process is done via command in this example, it could eventually be migrated to an automated process, for instance via some sort of onStartup JS event that runs on server startup (although there's no way to run such an event as of right now).

Any thoughts?
-= Ho Eyo He Hum =-
Regnak
UOX3 Newbie
Posts: 1
Joined: Wed Jul 21, 2021 11:11 am
Has thanked: 0
Been thanked: 1 time

Post by Regnak »

Hiya there !

First thoughts : that's a great idea !

I'm quite new here, so I don't know what has been done about this before, and since April 2021, but as I come from ServUO, was a JustUO dev and I've been there since RunUO 1.0, I thought I could share some ideas about what we used to do on RunUO based projects. Remember those are written in C# mostly, but I guess for my answer the most important aspect is the analysis of what is done about versioning on RunUO and forked projects.

So we used to call that Serialization/Deserialization. Basically, it's a versioning system. I think the update command is a very neat idea and isn't part of ServUO. It's just done automatically when you save your world (Serialization) and then load your world (Deserialization).

So each object has those two functions, and they're both based on a global persistence system. Saves are done in a subfolder 'Saves' and there you can find all sorts of bin/idx/tdb files, some XML's.
Welp, coming from Legends of Aria community admin program, I can share this also: they use 'db' files for all the saves and it seems a strong option. On UO emulator, and the serialization/deserialization system, you kinda work blind and if anything goes wrong, it's a world wipe. So there are improvement to make to what I'm going to explain.
Btw, LoA is written in Lua for scripts and mods, XML's for templates and Unity3D for any client modifications (so C#)

Anyways, let's focus on Ser/Deser function in ServUO.

Let's take an example, it's easier :)
e.g. PlayerMobile, which is the script for any player 'object', or in other words any 'Mobile' in game that is a player. The base script Mobile is split into two: PlayerMobile and BaseCreature.

You have inside that script two functions that can be useful to illustrate my thoughts : Serialize and Deserialize.
Serialize is triggered on world saves, wether they are automatic or manual.
Deserialize is trigger when the world is loading from saves, basically at every restart.

Just that, I really like your idea of a command, as it's something you can't do on ServUO. Over the years, when mixing with Ser/Deser, I had to do workarounds to simluate a command. Example : I wanted to convert old saves to a newer version. So I had to play with an old version of a Deserialize function, a new version of a Serialize version. Launch the server that way. Save manually to generate new world saves. Then switch to the new Deserialize function and run the server again.
Now imagine, how it was done, you have to do that for every 'object' in game that was modified with a newer version.
;)

But it's a very good system that would require a bit of improvement.

If we look deeper in the Serialize fuction (I'll skip any code here, as I guess it's just food for thoughts ^^):
- First step, you call the base function Serialize. It's the common part from the persistence scripts.
- Then you write a version number to your saves: e.g. Version 28. Remember, this is for the BaseCreature script, to any other objects can have different version numbers.
- Then you consider reading from version 28 to version 0. Going from the newest, to the oldest. We usually seperated the sections for each version with a comment: //version 28 .. //version 27 ... and so on.
- Basically, no IFs, no ELSEs, no CASEs, nothing. Just writing all the properties from the newest to the oldests in the world saves files.
- There is an exception though, when you reach version 0, or along the way, you may need to put an IF for later corrections. Delicate though.

And that's it for the serialization !

Any modification you want to do to a player, and you want it kept on worlds saves and restarts, it had to be added as a new version. For example, we want our players to become Bio Engineers, something very custom, well, you'll have to create version 29 (just by giving that variable the value of 29 when it was 28 before your modifications), add the comments (//version 29) and start writing all the new properties linked to a player there, on the top of the Serialize function.

Here's an example of C# Serialize function. Btw, you can find that on any GitHub related to RunUO and children. ;)

Code: Select all

		public override void Serialize(GenericWriter writer)
		{
			base.Serialize(writer);

			writer.Write(28); // version		
				           
			// Version 28
			writer.Write(m_PeacedUntil);
			// Version 27
			writer.Write(m_AnkhNextUse);
			// Version 26
			writer.Write(m_AutoStabled, true);
			// Version 25
			if (m_AcquiredRecipes == null)
			{
				writer.Write(0);
			}
			else
			{
				writer.Write(m_AcquiredRecipes.Count);

				foreach (var kvp in m_AcquiredRecipes)
				{
					writer.Write(kvp.Key);
					writer.Write(kvp.Value);
				}
			}
			
			...
			
Now Deserialize function. That one is easier to read and I like the way it was done.
It's just you read the version number. So if we added a number every time, not passing one, or not mixing too many exceptions, it should be clear how to read this. First you read the version number, then start a switch with cases to browse all the version from up to down.

Illustration (still C#) :

Code: Select all

		public override void Deserialize(GenericReader reader)
		{
			base.Deserialize(reader);

			int version = reader.ReadInt();

			switch (version)
			{
                case 28:
					{
						m_PeacedUntil = reader.ReadDateTime();

						goto case 27;
					}
				case 27:
					{
						m_AnkhNextUse = reader.ReadDateTime();

						goto case 26;
					}
				case 26:
					{
						m_AutoStabled = reader.ReadStrongMobileList();

						goto case 25;
					}
				case 25:
					{
						int recipeCount = reader.ReadInt();

						if (recipeCount > 0)
						{
							m_AcquiredRecipes = new Dictionary<int, bool>();

							for (int i = 0; i < recipeCount; i++)
							{
								int r = reader.ReadInt();
								if (reader.ReadBool()) //Don't add in recipies which we haven't gotten or have been removed
								{
									m_AcquiredRecipes.Add(r, true);
								}
							}
						}
						goto case 24;
					}
				...
                
So actually, imho, the order you add new version and insert them inside the scripts is really important. At least, unless you want a big spagetthi code lol :)

Hopefully this helps a little. Myself very very new here, less than a week, I'm discovering the amazing work you did there, Xuri, and all your mates of course, over the years. It's a really strong system with options that open so many possibilities.
I'll be there around working on my learning curve. Hopefully one day I can help more than this 'food for thoughts' answer :)

Take care ! Thanks a lot for UOX3 and keeping the project up over sooooo many years. Respect.

-Rek-
These users thanked the author Regnak for the post:
Xuri
User avatar
Xuri
Site Admin
Posts: 3704
Joined: Mon Jun 02, 2003 9:11 am
Location: Norway
Has thanked: 48 times
Been thanked: 8 times
Contact:

Post by Xuri »

Hey Regnak, and welcome!

Thanks for the insights! Although the way UOX3 handles saving and loading data is a bit different from RunUO/ServUO, this certainly provides food for thought on how to proceed with a data versioning system like this. For now it's on the backburner as I work on some other stuff, but I'll revisit it at some point in the future and have a proper think on how to best proceed.

I still want to implement a system like this or similar, if only to provide an "upgrade path" to make it easier to upgrade UOX3 when new versions are released, with new version of script files, etc. However it's done, though, it should be easy to understand and easy to use, and it should be impossible to mess up one's shard files by accident! :P

If you have any other thoughts to share about this - or anything else - feel free! You can also come hang out in our Discord server if that's your cup of tea.
-= Ho Eyo He Hum =-
Post Reply