Data Loading

So my exporter *works* but how should I go about loading that data. I chose XML for a majority of my file formats and I want a robust solution for loading that into my game code.

In a previous version of the Hulkengine I created a system that would at runtime parse header files and generate essentially reflectance information. I then used this system to auto-magically load xml into a hierarchy of what were essentially data only classes. For the most part this worked well but it meant I had to include header files as part of the games assets bleck* and I had to either make a more robust parser or be careful of what I put in my headers. A better solution along these lines would be to generate C++ code from the headers and compile in special code for loading. This would load more optimally and remove the need for shipping with the headers.

I'm a big fan of parsers and code generation but I'm not sure of all the idiosyncrasies of this approach. I woul need to write an Antlr grammar. I'd also need to write a tool for code generation. In addition I'd need to create a build rule in visual studio and somehow get it to auto-add my auto-generated parsing code when necessary. I suppose I could add by hand but that doesn't have much sexiness does it? I actually think this is probably the best approach but well.... It seems kind-of a lot of work. Remember I'm doing this on my own time and dime so if I don't want to work on it I'm not going to. Well I suppose I could look online about existing code generators for XML schema files but mehh.

Instead I've decided for a much simpler and clunkier approach. Basically I'm going to force all my data classes to implement a serialization function. I'm a big fan of sexy coding and I realize this is not it. Gimme a sec though let me slap some paint on this jalopy.

Actually let me first point out all the horrible parts. All my data classes have to have a common base class. This means data files describing a db_vec4 in my math library will have to inherit from my data::base class in my data library. Wait it gets worse. This means my data only classes are all going to have a virtual table and procedures. I'll stop here and say I agonized over this all day but I couldn't come up with a better way. This is not to say a better way doesn't exist just I don't know it.

So here's what I came up with. First(well not really but for the sake of clarity) I created a data library. This has three core classes.

• base:
• factory:
• archive:

All of my data objects must inherit from data::base. I really am very proud of this naming convention. Basically the derived class must implement a getid() function for save and serialize() function for save/load. As a general rule although not enforced by the compiler I give each one a const string for their name and a static function to register with the factory for creation.

The factory exists to construct data-classes on load. When an array or a pointer is reach at load time the factory looks up the relevant id and returns the correct creation function. It's basically a map. Actually it is a map nough said.

Archives are my equivalent to a stream class. from the syntax it looks like you are adding entries to the archives when in reality it's more of a tool for navigating my save/load files. Rather than try to fully explain in English let me give an example and we'll go from there.

class db_vector4 : public data::base
{
public:
DECLARE_ARCHIVE(db_vector4);
float x;
float y;
float z;
float w;
};

DEFINE_ARCHIVE(db_vector4);
void db_vector4::serialize(data::archive& s)
{
s.begin("db_vector4")
.attribute("x", x)
.attribute("y", y)
.attribute("z", z)
.attribute("w", w)
.end();
}

Here's an example of a data class and its serialization function. Notice there are two macros there. One declares a few functions and the other defines them. Normally I'm anti-macro but I find it helpful in this case. The only thing interesting hidden by the macros is that I pass an allocator to the static creation function.

So there are a few benefits to this approach that I like. For one save/load is all in the same function. Another is that through the use of tabs I can make the serialization function look almost like the xml file it's going to save. Not really important but I find it clever. You don't actually need to use the classes with the archive. You can if you want to just write the values however you want. Also the the output format is an implementation of the archive so you could easily save csv or xml or binary if you wanted.

So let me give a few more examples.

void db_entity::serialize(data::archive& s)
{
s.begin("db_entity")
.attribute("name", name)
.array("components", m_components)
.end();
}

Here I'm save/loading an entity data class. basically an entity is a name along with a whole bunch of components. Notice the array function. This will on load iterate over children and factory construct the appropriate object for serialization.

void saveload(data::archive& b, options& in_options)
{
b.begin("profile")
.attribute("exportmesh", in_options.m_exportmesh)
.attribute("exportentities", in_options.m_exportentities)
.attribute("exportstage", in_options.m_exportstage)
.attribute("exportcameras", in_options.m_exportcameras)
.attribute("exportmodels", in_options.m_exportmodels)
.attribute("exportlights", in_options.m_exportlights)
.attribute("exportallmodels", in_options.m_exportallmodels)
.begin("mesh")
.attribute("channels", in_options.m_mesh.channels)
.attribute("influences", in_options.m_mesh.influences)
.attribute("exportpositions", in_options.m_mesh.bexportpositions)
.attribute("exportcolors", in_options.m_mesh.bexportcolors)
.attribute("exportnormals", in_options.m_mesh.bexportnormals)
.attribute("exportbinormals", in_options.m_mesh.bexportbinormals)
.attribute("exporttangents", in_options.m_mesh.bexporttangents)
.attribute("exporttexcoords", in_options.m_mesh.bexporttexcoords)
.attribute("exportboneweights", in_options.m_mesh.bexportboneweights)
.end()
.end();
}

This is the profile for my exporter notice how options is just a bunch of data it doesn't implement data::base.

Anyway the archive is the only remotely complicated class. Basically I create a different one for load and save and a different one for whatever format I'm serializing to. So for example I have and xmlloader and xmlsaver implemtation of archive. The loader has a reference to a factory.

Whenever I figure out how to post files up here I'll put up the source and then you can really start tearing it apart.

Well I've made my bed and now I will have to sleep in it. I suspect for the Hulkengine version 4 I'll revisit with some love for the auto-generation version but for now hacky serialization is what I'm going to stick with.

Loading data is fun;)