ToolAPI Feature Requests

Maarc · Post by **Maarc** » Tue Feb 28, 2006 8:08 am

I figured I'd chuck this more in the coder's part, rather than the feature requests, because it's so intimately tied to the creation of new tools, and coding.

So ... anyone got any ideas/desires/requests for the ToolAPI? I can't guarantee I'll knock them off quickly, but if you can give a general sort of overview of something you think might be needed across more than a single app, or multiple times within an app, I might just be able to implement it.

* Add a WorldCollection class, that parses a directory of WorldFiles/WorldSections
* Add some validation capabilities (eg validating a ScriptSection) - Quite possibly through the creation of some validation schemas?
* Add a class to encapsulate the resource.bin files
* Investigate the possibility of validating JScript files
* Drum up some example msgboard files so that I can encapsulate those properly
* Add a way of iterating over a DefinitionCollection that hides the internal implementaiton (Hashtable), so that you can iterate over all the scripts if you wanted to.

Note, this API isn't aimed for client stuff (so, not MUL handling, except in so far as it effects the server). Basically, the API is a tool that can be used to help deal with UOX3 server data, essentially.

Note that:

Dictionaries can be encapsulated through a Script/ScriptSection already
UOX.INI can be encapsulated through a Script/ScriptSection already

stranf · Post by **stranf** » Tue Feb 28, 2006 8:37 am

Does it do .dfns?

I think a program, similar I guess to the spawn program (although I've never run that one), that shows a map of britannia, with all the weather regions.

Click on a region, and it opens a menu that lets you simply edit the weather chances for that region. I think we could get a pretty quick community weather.dfn file released with such a tool that would take advantage of the Community's new rewrite of the data.

Maarc · Post by **Maarc** » Tue Feb 28, 2006 8:39 am

Well, the API's more a building block for other tools to be built. Rather than people having to reinvent the wheel each time, they can reuse pieces of code to help them build tools quicker. eg Each person who writes a tool doesn't have to come up with the code to read/parse a DFN, for instance.

Post by **Xuri** » Tue Feb 28, 2006 10:13 am

How exactly does the ToolAPI work? Could you give an example? =)

Maarc · Post by **Maarc** » Tue Feb 28, 2006 12:55 pm

Essentially, it's a library that you can use in a .NET program (ultimately, I'd like to backport this to other languages, like C++, and maybe even VB). Giwo committed the character viewer to the CVS, so that's one example. But take an example (C#), loading the definitions.

Code: Select all

UOXData.Script.DefinitionTree dfnTree	= new UOXData.Script.DefinitionTree( dfnPath );
UOXData.Script.ScriptSection mSect		= dfnTree.FindEntry( "RACE 1", UOXData.Script.DFN_Categories.race );
if( mSect != null )
{
	foreach( UOXData.Script.TagDataPair t in mSect.TagDataPairs )
	{
		System.Windows.Forms.MessageBox.Show( "Tag: " + t.Tag + "\nData: " + t.Data.Value );
	}
}

That will load all the definitions that UOX has, then find a ScriptSection that is associated with RACE 1 in the RACE category. If it can find it, it will print out all the tag/data pairs in that section.

Code: Select all

UOXData.Script.WorldFile90 mFile = new UOXData.Script.WorldFile90( "C:\\app\\uox3test\\shared\\0.0.wsc" );

That will load one of the WorldFiles, and parse it.

Or say you want to create a new ScriptSection and store it in a file somewhere.

Code: Select all

UOXData.Script.ScriptSection newSection = new UOXData.Script.ScriptSection();
newSection.Add( "Xuri", "Rocks" );
newSection.Add( "Mumbo", "Jumbo" );
System.IO.StreamWriter mWrite = new StreamWriter( "C:\\app\\uox3test\\basic.txt" );
newSection.Save( mWrite );

That's a way to do it.

I've come up with a few other ideas, so I'll update the first post for that as well.

Maarc · Post by **Maarc** » Sat Mar 04, 2006 6:33 am

Okay, I've made a few updates in the last few days, don't know if anyone follows that particular branch of the CVS or not

So here's some update notes (minus test driver code that is in the changelog), and some details of what I'm currently working on.

Code: Select all

[color=yellow]3rd March, 2006 - Maarc[/color]
	Added ToInt32( byte[] toConvert, int offset ) and ToUInt32( byte[] toConvert, int offset ) to the Conversion class
	Added automatic parsing of a book's serial from the filename, if it's constructed via a filename
	Exposed the book's Serial, Title, Author, NumPages and Pages as properties
	Added basic classes to parse message boards.  This is done through MessageBoardScript and MessageBoardSection.
	Added basic constructor to Section() that accepts just a StreamReader
	Added functionality to be able to set the line of a page of a book through indexing
	Hid Sections from ClassicBookScript and MessageBoardScript
	Updated TrimCommentAndWhitespace to remove NULLs
	Added a TrimTrailingWhitespace command (in the event we don't want to actually trim a comment)

[color=yellow]3rd March, 2006 - Maarc[/color]
	Added Save() routines to ClassicBookScript and ClassicBookSection, and validated them against test books
	Adjusted some protection levels on some functions
	Added ToByteArray() functions that accept int/uint, short/ushort, or string, and chunk out a byte array of a particular length

[color=yellow]3rd March, 2006 - Maarc[/color]
	Added a ToString() function to Conversion.  It takes either a byte[] or char[] and returns a valid string
	Added InternalReset() functionality to the BaseScript and Section classes.  This allows pre-retrieval resetting of vars for constructed objects (child classes override as necessary)
	Updated and tested the ClassicBookScript and ClassicBookSection classes.  They now read a valid book/pages, but currently does *not* write out!  Saving of books is currently unimplemented
	Added indexing capabilities to ClassicBookScript and ClassicBookSection, as well as hiding some unnecessary functions
	The indexing ensures that a valid page/line is selected (if it doesn't, it returns NULL)

[color=yellow]2nd March, 2006 - Maarc[/color]
	Added a basic ClassicBookScript and ClassicBookSection class.  Note that this is *incomplete*, and requires testing against real data (which I don't have at the moment).
	Added a very basic DictionaryScript and UOXIni class.  They are a subclass of Script, with no additional information.  Essentially, they just *are* normal Scripts, but this way, we can conceptualise it a bit better.

[color=yellow]1st March, 2006 - Maarc[/color]
	Added a AccountScript and AccountSection class, for dealing with Accounts files
	Fixed a possible bug with script parsing that only became apparent with AccountScript (if the delimiter appeared more than once, we weren't adding it in properly for the value part)
	Added an AddSection routine to BaseScript, so one you create you can readily add in


[color=yellow]28th February, 2006 - Maarc[/color]
	Updated DefinitionTree so you can get a Collection based on the DFN Category you want (GetCollection() method)
	Added empty constructors for BaseScript(), WorldFile90() and Script(), so that we can create empty ones for populating later (not forced reads)

As for what I'm currently working on, it's DFN validation stuff. It won't be entirely transparent (ie it won't validate when it reads in), but it's an optional system by which people can validate DFNs. At the moment, it has the current goals

* Different schemas for different parts of the DFNs. A Race validates differently than an Item
* Flexible and easy to use, with an easy way to determine which schema you need to use
* Validates that a section only includes the TAGs that are actually valid for it
* Validates the value that accompanies a tag. If we're expecting a numeric, make sure it is. Expanded to include strings, uppercase strings, numerics, floats, double numerics (ie like TAG=10 30), and double floats (ie like TAG=10.5 11.5)
* Provides the capability to actually log errors that are found during validation
* Provide a way to sanitise a section. At the moment, I see this as stripping out invalid TAGs, as well as stripping out tag/value pairs, where the value is not in the right format (ie you specified a string rather than a numeric). This is also error logged.

Thoughts? Ideas? Suggestions?

As it stands, I have a basic schema in place for the general Race DFNs. I'll be going on to others soon.

Maarc · Post by **Maarc** » Sat Mar 04, 2006 9:03 am

Code: Select all

[color=yellow]4th March, 2006 - Maarc[/color]
	Added a validation scheme for trying to validate the format of a ScriptSection
	Please note that this does not validate that the data you put in is useful!  It only validates that it subscribes to the right syntax
	Essentially, it will allow you to find any bad/wrong tags, find any values that are the wrong type (eg string instead of number), and so on
	This does not stop, for example, of you using say, a GET tag that points to a non-existent section (it only checks to see that it is a valid syntax)
	Early schemas are in for:
		RANDOMCOLOR entries in colors dfn
		CARVE entries in carve dfns
		COMMANDLEVELS, COMMAND_OVERRIDE and actual command entries in command dfns
		CREATURE entries in creature dfns
		RACE entries in race dfns
		REGION, MIDILIST and INSTALOG entries in regions dfns
		REGIONSPAWN entries in spawn dfns

	Some basic test driver code to see how it works

	UOXData.Script.DefinitionTree dfnTree					= new UOXData.Script.DefinitionTree( dfnPath );
	UOXData.Script.ScriptSection mSect						= dfnTree.FindEntry( "CARVE 1", UOXData.Script.DFN_Categories.carve );
	UOXData.Script.Validation.Schemas.BaseSchema mSchema	= UOXData.Script.DefinitionTree.ValidationSchema( UOXData.Script.DFN_Categories.carve, mSect.SectionName );
	ArrayList errorLog										= new ArrayList();
	if( mSchema != null )
		mSchema.ValidateSection( mSect, errorLog );
	foreach( UOXData.Script.Validation.ValidateError ve in errorLog )
	{
		txtPackageDetails.Text += ve.ErrorMessage + " - " + ve.ExpectedType + " - " + ve.Tag + " - " + ve.Value + System.Environment.NewLine;
	}

Maarc · Post by **Maarc** » Sat Mar 04, 2006 1:59 pm

Code: Select all

[color=yellow]4th March, 2006 - Maarc[/color]
	Added schemas for
		HOUSE and HOUSEITEM entries in house dfns
		HTML entries in html dfns
		LOCATION entries in location dfns
		TILE and MAP entries in maps dfns
		BOOK, PAGE, GUMPTEXT, GUMPMENU, MOTD, TIPS and TIP entries in misc dfns
		ESCORTS and ESCORT entries in msgboard dfns
		BESTSKILL and DEFAULT entries in newbie dfns
		ORE_LIST, SKILL and ORE entries in skills dfns
		REGIONSPAWN entries in spawn dfns
		SPELL entries in spells dfns
		SKILL, PROWESS, FAME and MURDERER entries in titles dfns
		WEATHERAB entries in weather dfns

	The only schemas missing are for
		Advance
		Create
		Items
		Harditems
		NPCs
		Menus

Post by **Xuri** » Sat Mar 04, 2006 5:43 pm

Nice work, Maarc

Maarc · Post by **Maarc** » Thu Mar 09, 2006 7:44 am

Code: Select all

[color=yellow]9th March, 2006 - Maarc[/color]
	Fixed some subtle order of construction errors with Scripts and ScriptSections
	Added a GetFileEnumerator() method to DefinitionCollection so that you can now progressively access all the DFNs in the tree
	Added category tracking to ScriptSection, so that getting a schema is really easy
	Added an easier way to get at a validation schema
	Updated ScriptSection so that it also strips off comments and whitespace on the tag, not just data
	Updated UOXIni and Dictionary classes so that they have a default constructor
	Updated UOXIni so that it can deal with relative paths (using the Directory() method)
	Fixed a bug in the HTML Validation Schema (TYPE and UPDATE were wrong)

Basically, this allows us to do a quick validation based on the schemas that already exist. Essentially, this code following will do it. It's ripped out of a slight tinkering of the package installer (I'm using that as a quick play pen for testing some of this), and will dump out all the known errors (see later on).

Code: Select all

opnFileDialog.Filter = "INI files (*.ini)|*.ini|All files (*.*)|*.*" ;
if( opnFileDialog.ShowDialog() == DialogResult.OK )
{
	txtUOX3Ini.Text	= opnFileDialog.FileName;
	iniScript		= new UOXScript.UOXIni( txtUOX3Ini.Text );
	dfnPath			= iniScript.Directory( UOXScript.UOXIni.DirectoryPaths.Definitions );

	UOXScript.DefinitionTree dfnTree				= new UOXScript.DefinitionTree( dfnPath );
	ArrayList errorLog								= new ArrayList();

	this.progressReading.Minimum	= 0;
	this.progressReading.Maximum	= (int)UOXScript.DFN_Categories.NUM_DEFS;
	this.progressReading.Value		= 0;
	for( UOXScript.DFN_Categories m = UOXScript.DFN_Categories.FIRST_DEF; m < UOXScript.DFN_Categories.NUM_DEFS; ++m )
	{
		UOXScript.DefinitionCollection mCollect			= dfnTree.GetCollection( m );
		System.Collections.IDictionaryEnumerator myEnum = mCollect.GetFileEnumerator();
		while( myEnum.MoveNext() )
		{
			UOXScript.Script mScript = (UOXScript.Script)myEnum.Value;
			foreach( UOXScript.ScriptSection mSect in mScript.Sections )
			{
				UOXScript.Validation.Schemas.BaseSchema mSchema	= UOXScript.DefinitionTree.ValidationSchema( mSect );
				errorLog.Clear();
				if( mSchema != null )
					mSchema.ValidateSection( mSect, errorLog );
				if( errorLog.Count > 0 )
				{
					txtPackageDetails.Text	+= "Section " + mSect.SectionName + " has " + errorLog.Count + " errors " + System.Environment.NewLine + (string)myEnum.Key + System.Environment.NewLine;
					txtPackageDetails.Text	+= "============================== " + System.Environment.NewLine;
					foreach( UOXScript.Validation.ValidateError ve in errorLog )
					{
						txtPackageDetails.Text += ve.ErrorMessage + " - " + ve.ExpectedType + " - " + ve.Tag + " - " + ve.Value + System.Environment.NewLine;
					}
					txtPackageDetails.Text += System.Environment.NewLine;
				}
			}
		}
		progressReading.Value++;
	}
}

And the result of that, punched into a file, looks like this:

Code: Select all

Section LOCATION 33 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 236

Section LOCATION 51 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 235

Section LOCATION 81 has 2 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Tag does not exist in schema - Unknown - ( - 
Failed validation test - Numeric - Z - 254

Section LOCATION 122 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 253

Section LOCATION 135 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 253

Section LOCATION 141 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 253

Section LOCATION 163 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 230

Section LOCATION 171 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 255

Section LOCATION 243 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 236

Section LOCATION 254 has 1 errors C:/app/UOX3Test/dfndata/location\location.dfn
============================== 
Failed validation test - Numeric - Z - 251

Section SPELL 1 has 1 errors C:/app/UOX3Test/dfndata/spells\spells.dfn
============================== 
Tag does not exist in schema - Unknown - GINSING - 0

Section SPELL 2 has 1 errors C:/app/UOX3Test/dfndata/spells\spells.dfn
============================== 
Tag does not exist in schema - Unknown - GINSING - 1

Section SPELL 3 has 1 errors C:/app/UOX3Test/dfndata/spells\spells.dfn
============================== 
Tag does not exist in schema - Unknown - GINSING - 1

etc ... down to

Section SPELL 64 has 1 errors 
C:/app/UOX3Test/dfndata/spells\spells.dfn
============================== 
Tag does not exist in schema - Unknown - GINSING - 0

Section HOUSE 18 has 1 errors 
C:/app/UOX3Test/dfndata/house\house.dfn
============================== 
Tag does not exist in schema - Unknown - TYPE - 1

Section CREATURE 0x90 has 1 errors 
C:/app/UOX3Test/dfndata/creatures\creatures.dfn
============================== 
Tag does not exist in schema - Unknown - WATERCREATURE - 

Section CREATURE 0x91 has 1 errors 
C:/app/UOX3Test/dfndata/creatures\creatures.dfn
============================== 
Tag does not exist in schema - Unknown - WATERCREATURE - 

Section CREATURE 0x96 has 1 errors 
C:/app/UOX3Test/dfndata/creatures\creatures.dfn
============================== 
Tag does not exist in schema - Unknown - WATERCREATURE - 

Section CREATURE 0x97 has 1 errors 
C:/app/UOX3Test/dfndata/creatures\creatures.dfn
============================== 
Tag does not exist in schema - Unknown - WATERCREATURE -

The spell errors are easy fixes. We look for GINSENG, not GINSING

The location DFN ones are probably easy too (though whether it's a code or data fix is up for debate). Basically, z's are -127 to 128, and you wander over the top. The code will (probably) autowrap it properly, but I wouldn't assume it.

House and creatures, though, no idea. And remember, I haven't added in the validation schemas for items, npcs and so on. But it's a starting point

Maarc · Post by **Maarc** » Thu Mar 09, 2006 8:16 am

Code: Select all

[color=yellow]9th March, 2006 - Maarc[/color]
	Updated Validation Schema for MapTiles to reflect the new beta cycle

Grimson · Post by **Grimson** » Thu Mar 09, 2006 3:23 pm

Maarc wrote: House and creatures, though, no idea.

The "WATERCREATURES" tag doesn't exist anymore:

viewtopic.php?p=5256#5256

Post by **Xuri** » Thu Mar 09, 2006 6:05 pm

Fixed versions of spells.dfn and locations.dfn are up on CVS =)

It shouldn't surprise anyone to learn that both of these peculiarities (ok, the GINSING one is a BUG, the other doesn't really affect anything that I know of, since it seems that UOX3 or UO wrapped around to the negative side automatically) have been with us since the 0.68.x versions of UOX3 were released in 1998

0.68.025.020 (released by Tauriel in October 1998) to be exact where spells.dfn is concerned, while locations.dfn is a bit harder. It might originate from when location.scp, npc.scp, menus.scp and speech.scp were seperated from uox3.scp - i.e. version 0.68.009, released by Maytrix at an unknown date, though likely also in 1998 as this was the time where a new uox3 release was made every week by a different person, though historians argue that the abnormally high Z values could very well already have been present in the base uox3.scp before it was seperated into multiple files.

Maarc · Post by **Maarc** » Sun Mar 12, 2006 8:17 am

Doesn't surprise me in the slightest either, to be honest. Surprised no one complained that ginseng was never being used

Code: Select all

[color=yellow]12th March, 2006 - Maarc[/color]
	Added Validation Schemas for the NPC definitions

Ultima Offline eXperiment 3 forums

ToolAPI Feature Requests

ToolAPI Feature Requests