The Project

One thing I've learned over the course of developing my engine is that how you setup your project is important. If done correctly you can limit the number of crazy project settings you have and generally simplify and speedup development. Ok I'll admit alot of this is just to satisfy my personal coder OCD.

I don't have coder OCD at work but in my personal projects I'm anal to the extreme. Admittedly this is a problem that kills productivity but it does lead to what I consider to be nice code/projects. I'll address my code OCD in a future blog. For now I'll visit what I consider to be good rules for project settings. I suspect that a real build engineer will laugh at some of what I say but I think most of this reasonable.

The first thing to know is what you are trying to accomplish. My goal in regards to my engine is always simplicity. I want it to be as easy to work with and understand as possible. I want to reduce the number of project setting. This makes it easier to add new projects or to copy to a new machine etc. I want to be able to support multiple configurations win32/x64/xbox easily. This way if my target platform changes I can switch with it easily.

I've found there are a couple of categories of components that make up my project. I believe each to be essential but you may find you only have a few. That said here is how my project is broken up.

• Libs


• 3rdParty


• Apps


• UnitTests


• Plugins


• Wrappers


• Resources

Libs this is all the code libraries that make up my engine. I have graphics libraries, xml libraries etc etc. This is the engine really or atleast the portion that I code. I group all my libs under one folder. So it looks like "MyProject/Libs/graphics". There are two great benefits from putting all your cody libs in one folder. You can map one additional include path in your projects and then all your include paths look the same. So if I add "MyProject/Libs" to my include path then in any project that I wish to use my graphics library I can access it's header files by saying #include . This is great because you know where your header is coming from so if you have two myheader.h files it's clear which you are using. This same principle holds true for linking. You set your lib files to export to (soldir)/(Platform)/(debug)/libname. Then you only map one link dir. The use of visual studio macros makes everything work so nicely. Note as a general rule I try to do all my project editing on all configurations.

3rdParty directory is for all external libraries that are going to in general be unique to the game engine. So boost and directx would not go in here but some version of lua or antlr might. Really I think everything should go in here but I basically just put things in here that don't have installers. The reason to put external libs here is if you want to backup or do source control on your stuff it's easy to avoid grabbing this directory. It can also have benefits for include paths and linking similar to the libs directory. Not always though because there's no controlling how third party libs are layed out. Do not manually arrange third party libs. This should be obvious but basically once you change it you have to maintain your change.

Apps is the directory for all the applications you make using your engine. If you make one game or 100 test apps they all go here. The basic idea is you can setup most of the project settings once and then reuse when making a new app. Sometimes for one reason or another you may want to pull your app off into it's own solution. In that case put the .sln file for your new app at the same level as the .sln for the main project. Again this is so project settings stay the same.

UnitTest is the directory for unittesting individual libraries. I highly recommend unittesting libraries but that is another topic. Basically the advantage here is just to group all these together and similar to the apps directory to setup identical sets of project settings.

Plugins directory is just another grouping for items dependant on the libs directory. I try to keep these things seperate because I think it looks nice.

Wrappers is the directory for libs that have been wrapped for another language. If you want to expose a library for use in C# or python then you would put the code in here.

Resources is the directory for sharing common art assets across multiple projects. The idea is it's easy to setup common paths to art resources for different projects. In practice this is likely to change.

A long time ago I learned to setup project for use with make. Then I learned all the good stuff that is visual studio. Now I'm in the process of heading the other direction. I'm going to continue with visual studio for now but I tend to think I might use something different in the future.

I really had a lot more to say about project setup but I'm tired of writing about it. Which in my mind means 99% of people probably didn't read this far. If you take that number out of the 0 people that visit this blog then you see why I'm stoppoing.

Gareth

Data Loading

So my exporter *works* but how should I go about loading that data. I chose XML for a majority of my file formats and I want a robust solution for loading that into my game code.

In a previous version of the Hulkengine I created a system that would at runtime parse header files and generate essentially reflectance information. I then used this system to auto-magically load xml into a hierarchy of what were essentially data only classes. For the most part this worked well but it meant I had to include header files as part of the games assets bleck* and I had to either make a more robust parser or be careful of what I put in my headers. A better solution along these lines would be to generate C++ code from the headers and compile in special code for loading. This would load more optimally and remove the need for shipping with the headers.

I'm a big fan of parsers and code generation but I'm not sure of all the idiosyncrasies of this approach. I woul need to write an Antlr grammar. I'd also need to write a tool for code generation. In addition I'd need to create a build rule in visual studio and somehow get it to auto-add my auto-generated parsing code when necessary. I suppose I could add by hand but that doesn't have much sexiness does it? I actually think this is probably the best approach but well.... It seems kind-of a lot of work. Remember I'm doing this on my own time and dime so if I don't want to work on it I'm not going to. Well I suppose I could look online about existing code generators for XML schema files but mehh.

Instead I've decided for a much simpler and clunkier approach. Basically I'm going to force all my data classes to implement a serialization function. I'm a big fan of sexy coding and I realize this is not it. Gimme a sec though let me slap some paint on this jalopy.

Actually let me first point out all the horrible parts. All my data classes have to have a common base class. This means data files describing a db_vec4 in my math library will have to inherit from my data::base class in my data library. Wait it gets worse. This means my data only classes are all going to have a virtual table and procedures. I'll stop here and say I agonized over this all day but I couldn't come up with a better way. This is not to say a better way doesn't exist just I don't know it.

So here's what I came up with. First(well not really but for the sake of clarity) I created a data library. This has three core classes.

• base:
• factory:
• archive:

All of my data objects must inherit from data::base. I really am very proud of this naming convention. Basically the derived class must implement a getid() function for save and serialize() function for save/load. As a general rule although not enforced by the compiler I give each one a const string for their name and a static function to register with the factory for creation.

The factory exists to construct data-classes on load. When an array or a pointer is reach at load time the factory looks up the relevant id and returns the correct creation function. It's basically a map. Actually it is a map nough said.

Archives are my equivalent to a stream class. from the syntax it looks like you are adding entries to the archives when in reality it's more of a tool for navigating my save/load files. Rather than try to fully explain in English let me give an example and we'll go from there.

class db_vector4 : public data::base
{
public:
DECLARE_ARCHIVE(db_vector4);
float x;
float y;
float z;
float w;
};

DEFINE_ARCHIVE(db_vector4);
void db_vector4::serialize(data::archive& s)
{
s.begin("db_vector4")
.attribute("x", x)
.attribute("y", y)
.attribute("z", z)
.attribute("w", w)
.end();
}

Here's an example of a data class and its serialization function. Notice there are two macros there. One declares a few functions and the other defines them. Normally I'm anti-macro but I find it helpful in this case. The only thing interesting hidden by the macros is that I pass an allocator to the static creation function.

So there are a few benefits to this approach that I like. For one save/load is all in the same function. Another is that through the use of tabs I can make the serialization function look almost like the xml file it's going to save. Not really important but I find it clever. You don't actually need to use the classes with the archive. You can if you want to just write the values however you want. Also the the output format is an implementation of the archive so you could easily save csv or xml or binary if you wanted.

So let me give a few more examples.

void db_entity::serialize(data::archive& s)
{
s.begin("db_entity")
.attribute("name", name)
.array("components", m_components)
.end();
}

Here I'm save/loading an entity data class. basically an entity is a name along with a whole bunch of components. Notice the array function. This will on load iterate over children and factory construct the appropriate object for serialization.

void saveload(data::archive& b, options& in_options)
{
b.begin("profile")
.attribute("exportmesh", in_options.m_exportmesh)
.attribute("exportentities", in_options.m_exportentities)
.attribute("exportstage", in_options.m_exportstage)
.attribute("exportcameras", in_options.m_exportcameras)
.attribute("exportmodels", in_options.m_exportmodels)
.attribute("exportlights", in_options.m_exportlights)
.attribute("exportallmodels", in_options.m_exportallmodels)
.begin("mesh")
.attribute("channels", in_options.m_mesh.channels)
.attribute("influences", in_options.m_mesh.influences)
.attribute("exportpositions", in_options.m_mesh.bexportpositions)
.attribute("exportcolors", in_options.m_mesh.bexportcolors)
.attribute("exportnormals", in_options.m_mesh.bexportnormals)
.attribute("exportbinormals", in_options.m_mesh.bexportbinormals)
.attribute("exporttangents", in_options.m_mesh.bexporttangents)
.attribute("exporttexcoords", in_options.m_mesh.bexporttexcoords)
.attribute("exportboneweights", in_options.m_mesh.bexportboneweights)
.end()
.end();
}

This is the profile for my exporter notice how options is just a bunch of data it doesn't implement data::base.

Anyway the archive is the only remotely complicated class. Basically I create a different one for load and save and a different one for whatever format I'm serializing to. So for example I have and xmlloader and xmlsaver implemtation of archive. The loader has a reference to a factory.

Whenever I figure out how to post files up here I'll put up the source and then you can really start tearing it apart.

Well I've made my bed and now I will have to sleep in it. I suspect for the Hulkengine version 4 I'll revisit with some love for the auto-generation version but for now hacky serialization is what I'm going to stick with.

Loading data is fun;)

The Exporter

Exporting content from Max is a pain. Invariably I will be writing and rewriting this exporter until the day I die. Why you ask? Most game studios crank out and exporter in a week or so and then perform quick touch ups only as necessary. Well I can't argue with the success other studios have but I'll try to explain my thinking.

For one the exporter is the central piece of my content pipeline. It needs to handle exporting all kinds of content and it needs to do it easily. If the exporter is a pain to use for one reason or another that will slow down dev time. If it's buggy then it's useless. If it's not expandable then I'll constantly be using win32 gui designer to relay out my widgets. All that said the point is that my exporter needs to be rock solid. It needs to support umpteen bajillion options and it needs to be fast to use.

Now the first time I created a Max exporter I used the fast and loose approach. I thought who cares about the exporter it's the game engine that I really care about. Well I was wrong and I ran into big headaches. My first shortcut was to use classes from my game engine. This leads down the rabbit hole of all the problems with early version of my game engine but I'll just say I ended up having to link my exporter to directX, deal with auto-registration errors(a new peeve), and fight other non-exporter related bugs. All that said I did get a working exporter before. It just wasn't easy to use, wasn't robust, and took forever to add or remove features.

This time I decided to create an uber exporter. I really wanted to use C# for the interface but I didn't see any easy way of doing it. I suspect many exporters are made using MFC but I read somewhere in the docs that I wasn't supposed to use MFC and I've never used it before and didn't care to learn it. So I was forced to do some win32 gui programming. Win32 gui programming is about as much fun as punching yourself in the face.

To meet my first design requirement of flexibility I decided to use a tabbed view. This will allow me to simply add new tabs as more options or categories of options become available. The drawback here is usability. Artists and game designers will now have to click through multiple tabs to get the options they want for export.

To improve the usability of my exporter I created a profiles system. On the main tab of the exporter is a combo box for selecting a given profile or saving a new one. The profiles are just a list of export options. In theory content developers will only need to create a profile once and then they can rapidly export models using that same profile over and over again.

I created the exporter as a first class app. By that I mean I created the code for it as robustly as I know how. I didn't cram multiple classes into one file or include a bunch of extraneous libraries. I'm using the xml library from my game engine and a few third party libs like boost and stdlib but for the most part I'm keeping the dependencies down. I created some rough widget helper classes to make binding data easier. All in all I've spent considerable time up front in hope that it will pay off later on.

So what exactly am I currently exporting. Well when a user clicks export he will if all options are selected generate the following files and folders

• meshes
• models
• profiles
• scenes

Ideally there will be other folders in here for other game assets like sounds and scripts and what not but currently this is what max exports.

Meshes: Is the folder containing all the meshes the game can load. This is not a flat folder but contains subfolders for specifying a particular model that a mesh belongs to. Meshes are essentially binary data. They contain a mesh header describing the data and then a long list of face and vertex data. The data itself is setup so that it can be read as one big chunk when it comes time to load a mesh.

Models: This folder contains a bunch of xml files describing a game model. So if an asset is a hierarchy of meshes with textures and shaders this xml file is the description of that. When a scene exports each top level mesh node is treated as it's own model.

Profiles: This contains xml files for the options used for exporting. So an artist may have several sets for exporting different types of models and a designer may have several more for exporting his content.

Scenes: This folder contains a list of xml files describing a particular scene. Basically it lists all game entities and their components along with position in the game world. It also lists all assets used by the game world independantly.

I'm certain more features will be added to this damn thing but that's what it does now. There's lots more to say about exporting so I'm sure I'll revist with more posts in the future.

Later.

The Content Pipeline

There's the saying all roads lead to Rome. Well for the Hulkengine all roads lead to 3dsMax. Now I can already hear people grumbling. Believe me when I say I understand your concerns and worries but let me see if I can convince you I have valid reasons.

1. My artist works in Max

OK so I only have one reason. It's a powerful reason none the less. Anyway I haven't explained what I mean yet.

Max is where all my game content gets linked into the game. Max is my game editor. Now I'm not the first person or even game studio to try this. There are several drawbacks but the biggest being cost doesn't really apply to me. Basically because I'm planning on making small games even if I had a small team that would only be a few Max licenses. For big studios it's probably cheaper to create a game editor then have everyone use Max.

There are other problems beyond cost for using Max. The api is clunky and it's hard to create gui's for. You can use Maxscript for lots of things and I do but I'm not Maxscript guru and that means another language to support. I like the idea of using the right language for the problem but only when that language interests me and Maxscript definitely does not.

So how will this all work? What will I use max for exactly? Well obviously art assets. Artist will be able to create models/animations in Max click export and then run the game and view their newly created content. More interesting is the ability to attach game content in max. So once an artist has laid out the art for a level a designer will load that level up and begin to attach game content.

Game content will be attached using Maxscript widgets. Basically the designer will run the Hulkengine Maxscript and see a list of game specific components that he can attach to a Max Node. Then the designer can click export and all the art with game components will export and then he can see/play the game with his content.

The same is true for programmer or sound designer content. All content will go through Max and then into the game.

Here's hoping this works well. Note I had a terrible time placing these images so that's my first blogger negative.

Introduction

Well here goes... I've been thinking about putting my game engine online for years. I don't actually know anything about web design or hosting so I thought I would start generating content using something a bit more structured. This is the opposite of the way I normally do things so I suspect it will work well. Whether it does or not time will tell. Anyway welcome to Hulkengine Development all 2(maybe?) of you.

What I'm hoping to do is to document the design and development of my game engine the hulkengine.

The hulkengine is designed for fast prototyping of simple 3d games. My focus is on having a simple scalable content pipeline. I'm willing to make tradeoffs of performance for simplicity or flexibility. Ultimately I want to be able to create games like SuperMario or SmashTV with modern graphics.

I don't think there is any hope of creating the next Halflife2 or Doom3 game engine on my own. Nor do I pretend I'm going to surpass Ogre or any other existing engine in terms of features or performance. Instead I'm going to create the easiest to use 3d engine for programmers or non-programmers.

Ok well that's my first post.

Cheers...