Monday, June 22, 2009

Time

The biggest hindrance in making a game engine is finding time. I essentially have none. I have no time for this blog or for fixing bugs or adding features to my game engine. I'm not sure what the solution is.

I can say with some certainty that the approach I've taken of making the uber engine from start to finish is probably not the right one. I'm going on 9 months of this current engine iteration and although I think it's got good points it's not functional. Physics breaks and I'm redoing the camera system for the millionth time sigh...

Probably it's smarter to work with an existing engine and build in the points that you want to focus on. I think most people focus on gameplay. I suspect I will never get that far. Maybe one day.

If anyone was following this blog because they wanted to learn about engine development they should probably be discouraged reading this. Good thing nobody follows it.

Tuesday, May 19, 2009

Ping

Hello world. It's been several months since my last post but it's time to try again. I keep meaning to update this but it's hard to find time.

This is a link to my current engine development. I think it's pretty cool but it obviously has a long way to go. Hopefully people can start to see how a game can be made with it.

http://www.wuala.com/GarethCox/hulkengine/hulkengine%20May%2019%202009.avi

Cheers

Monday, January 12, 2009

Engine Design

Ultimately I want to use the Hulkengine to very rapidly make games like the recent remake of bionic commando or galaga etc. Too that end I've focused on simplicity as the driving motivation. Performance and flexibility are secondary issues. This is not to say that I don't consider performance or flexibility because I do. Too much as a matter of fact. It is just that when push comes to shove simplicity comes first. In the battle between flexibility and performance again performance comes in second. My thinking is that so long as the game engine is of reasonable quality I will be hard pressed to get enough content to truly push the performace envelope. Plus if I try to push the performance envelope I will never actually create anything. You can only push the performance/technology envelope if you focus on one particular area. As the sole author of the entire hulkengine that is not feasible.

So how do I keep my engine simple? Well the basic answer is to write lots of little libraries that do one thing. Then take those little libraries and put them under unit test so that I have some moderate assurance of their functionality. I'll admit I use unit tests almost as documentation on how to use my libraries more so than for verification. This is mostly because I'm new to the unit test world and because I'm lazy. Whenever I discover a bug or think there might be a bug I usually write a unit test to look for it. Writing unit tests has changed the way I code and I believe mostly for the better. Point is you should use unit testing.

The libraries themselves are as self contained as possible. Some like the math and data libary I use in other libs but generally I prefer to duplicate code rather than add dependancies. That's right. I'll say it again. I prefer to duplicate code rather than add dependancy. This is a recent fad with me but I think it makes sense. I want each of my libraries to be self contained packets of functionality. If library A depends on B and B on C and so on then any time I write an application I have to link to every single thing. I suppose this isn't the end of the world but it's complicated and over time increasing dependancy between the two libs will make them uttery inseperable. Alternatively you can interface everything and have complexity and waste where none is needed. I think the phobia against code duplication is well founded but my desire for libs with zero or very limited dependancy is stronger.

Wrap each library in a namespace. This makes it very simple to know where code is coming from. Along with header paths and namespaces code becomes significantly simpler and easier to trace. Unfortunately namespaces tend to temp you to name things the same across projects. So I might have graphics::allocator interface and scene::allocator interface. Unfortunately this doesn't play nice with visual assist so despite the fact that with the namespace it's reduntant I would recomend still prefixing your classes. I don't but then I pay the price when visual assist fails. It's annoying and often I wish I had bit the bullet and prefixed classes. On a quick note get visual assist if you use visual studio. It's a time saver.

Consider what the API is for your library. You don't want to expose any more than is absolutely necessary. This is a great way to simplify your code. You can have some fantastical scene graph lib but if you create a clean API to use that library then you've really accomplished something. Try to export handles to objects rather than objects themselves when possible. So if you have a graphics library that loads textures don't expose the texture pointer external to the library. Expose a handle for tracking that texture and manage the texture internal to the lib. That said do not not use singletons or globals in your library. This will be a non-starter on many projects so make sure that you allow users to create and maintain your manager structures.

Ok well I'll continue this post later. This is tiring.

Gareth

Tuesday, December 30, 2008

The Project

One thing I've learned over the course of developing my engine is that how you setup your project is important. If done correctly you can limit the number of crazy project settings you have and generally simplify and speedup development. Ok I'll admit alot of this is just to satisfy my personal coder OCD.

I don't have coder OCD at work but in my personal projects I'm anal to the extreme. Admittedly this is a problem that kills productivity but it does lead to what I consider to be nice code/projects. I'll address my code OCD in a future blog. For now I'll visit what I consider to be good rules for project settings. I suspect that a real build engineer will laugh at some of what I say but I think most of this reasonable.

The first thing to know is what you are trying to accomplish. My goal in regards to my engine is always simplicity. I want it to be as easy to work with and understand as possible. I want to reduce the number of project setting. This makes it easier to add new projects or to copy to a new machine etc. I want to be able to support multiple configurations win32/x64/xbox easily. This way if my target platform changes I can switch with it easily.

I've found there are a couple of categories of components that make up my project. I believe each to be essential but you may find you only have a few. That said here is how my project is broken up.
  • Libs

  • 3rdParty

  • Apps

  • UnitTests

  • Plugins

  • Wrappers

  • Resources
Libs this is all the code libraries that make up my engine. I have graphics libraries, xml libraries etc etc. This is the engine really or atleast the portion that I code. I group all my libs under one folder. So it looks like "MyProject/Libs/graphics". There are two great benefits from putting all your cody libs in one folder. You can map one additional include path in your projects and then all your include paths look the same. So if I add "MyProject/Libs" to my include path then in any project that I wish to use my graphics library I can access it's header files by saying #include . This is great because you know where your header is coming from so if you have two myheader.h files it's clear which you are using. This same principle holds true for linking. You set your lib files to export to (soldir)/(Platform)/(debug)/libname. Then you only map one link dir. The use of visual studio macros makes everything work so nicely. Note as a general rule I try to do all my project editing on all configurations.

3rdParty directory is for all external libraries that are going to in general be unique to the game engine. So boost and directx would not go in here but some version of lua or antlr might. Really I think everything should go in here but I basically just put things in here that don't have installers. The reason to put external libs here is if you want to backup or do source control on your stuff it's easy to avoid grabbing this directory. It can also have benefits for include paths and linking similar to the libs directory. Not always though because there's no controlling how third party libs are layed out. Do not manually arrange third party libs. This should be obvious but basically once you change it you have to maintain your change.

Apps is the directory for all the applications you make using your engine. If you make one game or 100 test apps they all go here. The basic idea is you can setup most of the project settings once and then reuse when making a new app. Sometimes for one reason or another you may want to pull your app off into it's own solution. In that case put the .sln file for your new app at the same level as the .sln for the main project. Again this is so project settings stay the same.

UnitTest is the directory for unittesting individual libraries. I highly recommend unittesting libraries but that is another topic. Basically the advantage here is just to group all these together and similar to the apps directory to setup identical sets of project settings.

Plugins directory is just another grouping for items dependant on the libs directory. I try to keep these things seperate because I think it looks nice.

Wrappers is the directory for libs that have been wrapped for another language. If you want to expose a library for use in C# or python then you would put the code in here.

Resources is the directory for sharing common art assets across multiple projects. The idea is it's easy to setup common paths to art resources for different projects. In practice this is likely to change.

A long time ago I learned to setup project for use with make. Then I learned all the good stuff that is visual studio. Now I'm in the process of heading the other direction. I'm going to continue with visual studio for now but I tend to think I might use something different in the future.

I really had a lot more to say about project setup but I'm tired of writing about it. Which in my mind means 99% of people probably didn't read this far. If you take that number out of the 0 people that visit this blog then you see why I'm stoppoing.

Gareth

Saturday, December 27, 2008

Data Loading

So my exporter *works* but how should I go about loading that data. I chose XML for a majority of my file formats and I want a robust solution for loading that into my game code.

In a previous version of the Hulkengine I created a system that would at runtime parse header files and generate essentially reflectance information. I then used this system to auto-magically load xml into a hierarchy of what were essentially data only classes. For the most part this worked well but it meant I had to include header files as part of the games assets bleck* and I had to either make a more robust parser or be careful of what I put in my headers. A better solution along these lines would be to generate C++ code from the headers and compile in special code for loading. This would load more optimally and remove the need for shipping with the headers.

I'm a big fan of parsers and code generation but I'm not sure of all the idiosyncrasies of this approach. I woul need to write an Antlr grammar. I'd also need to write a tool for code generation. In addition I'd need to create a build rule in visual studio and somehow get it to auto-add my auto-generated parsing code when necessary. I suppose I could add by hand but that doesn't have much sexiness does it? I actually think this is probably the best approach but well.... It seems kind-of a lot of work. Remember I'm doing this on my own time and dime so if I don't want to work on it I'm not going to. Well I suppose I could look online about existing code generators for XML schema files but mehh.

Instead I've decided for a much simpler and clunkier approach. Basically I'm going to force all my data classes to implement a serialization function. I'm a big fan of sexy coding and I realize this is not it. Gimme a sec though let me slap some paint on this jalopy.

Actually let me first point out all the horrible parts. All my data classes have to have a common base class. This means data files describing a db_vec4 in my math library will have to inherit from my data::base class in my data library. Wait it gets worse. This means my data only classes are all going to have a virtual table and procedures. I'll stop here and say I agonized over this all day but I couldn't come up with a better way. This is not to say a better way doesn't exist just I don't know it.

So here's what I came up with. First(well not really but for the sake of clarity) I created a data library. This has three core classes.
  • base:
  • factory:
  • archive:

All of my data objects must inherit from data::base. I really am very proud of this naming convention. Basically the derived class must implement a getid() function for save and serialize() function for save/load. As a general rule although not enforced by the compiler I give each one a const string for their name and a static function to register with the factory for creation.

The factory exists to construct data-classes on load. When an array or a pointer is reach at load time the factory looks up the relevant id and returns the correct creation function. It's basically a map. Actually it is a map nough said.

Archives are my equivalent to a stream class. from the syntax it looks like you are adding entries to the archives when in reality it's more of a tool for navigating my save/load files. Rather than try to fully explain in English let me give an example and we'll go from there.

class db_vector4 : public data::base
{
public:
DECLARE_ARCHIVE(db_vector4);
float x;
float y;
float z;
float w;
};

DEFINE_ARCHIVE(db_vector4);
void db_vector4::serialize(data::archive& s)
{
s.begin("db_vector4")
.attribute("x", x)
.attribute("y", y)
.attribute("z", z)
.attribute("w", w)
.end();
}

Here's an example of a data class and its serialization function. Notice there are two macros there. One declares a few functions and the other defines them. Normally I'm anti-macro but I find it helpful in this case. The only thing interesting hidden by the macros is that I pass an allocator to the static creation function.

So there are a few benefits to this approach that I like. For one save/load is all in the same function. Another is that through the use of tabs I can make the serialization function look almost like the xml file it's going to save. Not really important but I find it clever. You don't actually need to use the classes with the archive. You can if you want to just write the values however you want. Also the the output format is an implementation of the archive so you could easily save csv or xml or binary if you wanted.

So let me give a few more examples.


void db_entity::serialize(data::archive& s)
{
s.begin("db_entity")
.attribute("name", name)
.array("components", m_components)
.end();
}

Here I'm save/loading an entity data class. basically an entity is a name along with a whole bunch of components. Notice the array function. This will on load iterate over children and factory construct the appropriate object for serialization.


void saveload(data::archive& b, options& in_options)
{
b.begin("profile")
.attribute("exportmesh", in_options.m_exportmesh)
.attribute("exportentities", in_options.m_exportentities)
.attribute("exportstage", in_options.m_exportstage)
.attribute("exportcameras", in_options.m_exportcameras)
.attribute("exportmodels", in_options.m_exportmodels)
.attribute("exportlights", in_options.m_exportlights)
.attribute("exportallmodels", in_options.m_exportallmodels)
.begin("mesh")
.attribute("channels", in_options.m_mesh.channels)
.attribute("influences", in_options.m_mesh.influences)
.attribute("exportpositions", in_options.m_mesh.bexportpositions)
.attribute("exportcolors", in_options.m_mesh.bexportcolors)
.attribute("exportnormals", in_options.m_mesh.bexportnormals)
.attribute("exportbinormals", in_options.m_mesh.bexportbinormals)
.attribute("exporttangents", in_options.m_mesh.bexporttangents)
.attribute("exporttexcoords", in_options.m_mesh.bexporttexcoords)
.attribute("exportboneweights", in_options.m_mesh.bexportboneweights)
.end()
.end();
}

This is the profile for my exporter notice how options is just a bunch of data it doesn't implement data::base.

Anyway the archive is the only remotely complicated class. Basically I create a different one for load and save and a different one for whatever format I'm serializing to. So for example I have and xmlloader and xmlsaver implemtation of archive. The loader has a reference to a factory.

Whenever I figure out how to post files up here I'll put up the source and then you can really start tearing it apart.

Well I've made my bed and now I will have to sleep in it. I suspect for the Hulkengine version 4 I'll revisit with some love for the auto-generation version but for now hacky serialization is what I'm going to stick with.

Loading data is fun;)

Friday, December 26, 2008

The Exporter


Exporting content from Max is a pain. Invariably I will be writing and rewriting this exporter until the day I die. Why you ask? Most game studios crank out and exporter in a week or so and then perform quick touch ups only as necessary. Well I can't argue with the success other studios have but I'll try to explain my thinking.
For one the exporter is the central piece of my content pipeline. It needs to handle exporting all kinds of content and it needs to do it easily. If the exporter is a pain to use for one reason or another that will slow down dev time. If it's buggy then it's useless. If it's not expandable then I'll constantly be using win32 gui designer to relay out my widgets. All that said the point is that my exporter needs to be rock solid. It needs to support umpteen bajillion options and it needs to be fast to use.

Now the first time I created a Max exporter I used the fast and loose approach. I thought who cares about the exporter it's the game engine that I really care about. Well I was wrong and I ran into big headaches. My first shortcut was to use classes from my game engine. This leads down the rabbit hole of all the problems with early version of my game engine but I'll just say I ended up having to link my exporter to directX, deal with auto-registration errors(a new peeve), and fight other non-exporter related bugs. All that said I did get a working exporter before. It just wasn't easy to use, wasn't robust, and took forever to add or remove features.
This time I decided to create an uber exporter. I really wanted to use C# for the interface but I didn't see any easy way of doing it. I suspect many exporters are made using MFC but I read somewhere in the docs that I wasn't supposed to use MFC and I've never used it before and didn't care to learn it. So I was forced to do some win32 gui programming. Win32 gui programming is about as much fun as punching yourself in the face.

To meet my first design requirement of flexibility I decided to use a tabbed view. This will allow me to simply add new tabs as more options or categories of options become available. The drawback here is usability. Artists and game designers will now have to click through multiple tabs to get the options they want for export.
To improve the usability of my exporter I created a profiles system. On the main tab of the exporter is a combo box for selecting a given profile or saving a new one. The profiles are just a list of export options. In theory content developers will only need to create a profile once and then they can rapidly export models using that same profile over and over again.
I created the exporter as a first class app. By that I mean I created the code for it as robustly as I know how. I didn't cram multiple classes into one file or include a bunch of extraneous libraries. I'm using the xml library from my game engine and a few third party libs like boost and stdlib but for the most part I'm keeping the dependencies down. I created some rough widget helper classes to make binding data easier. All in all I've spent considerable time up front in hope that it will pay off later on.
So what exactly am I currently exporting. Well when a user clicks export he will if all options are selected generate the following files and folders
  • meshes
  • models
  • profiles
  • scenes
Ideally there will be other folders in here for other game assets like sounds and scripts and what not but currently this is what max exports.
Meshes: Is the folder containing all the meshes the game can load. This is not a flat folder but contains subfolders for specifying a particular model that a mesh belongs to. Meshes are essentially binary data. They contain a mesh header describing the data and then a long list of face and vertex data. The data itself is setup so that it can be read as one big chunk when it comes time to load a mesh.
Models: This folder contains a bunch of xml files describing a game model. So if an asset is a hierarchy of meshes with textures and shaders this xml file is the description of that. When a scene exports each top level mesh node is treated as it's own model.
Profiles: This contains xml files for the options used for exporting. So an artist may have several sets for exporting different types of models and a designer may have several more for exporting his content.
Scenes: This folder contains a list of xml files describing a particular scene. Basically it lists all game entities and their components along with position in the game world. It also lists all assets used by the game world independantly.
I'm certain more features will be added to this damn thing but that's what it does now. There's lots more to say about exporting so I'm sure I'll revist with more posts in the future.
Later.

The Content Pipeline

There's the saying all roads lead to Rome. Well for the Hulkengine all roads lead to 3dsMax. Now I can already hear people grumbling. Believe me when I say I understand your concerns and worries but let me see if I can convince you I have valid reasons.
  1. My artist works in Max
OK so I only have one reason. It's a powerful reason none the less. Anyway I haven't explained what I mean yet.

Max is where all my game content gets linked into the game. Max is my game editor. Now I'm not the first person or even game studio to try this. There are several drawbacks but the biggest being cost doesn't really apply to me. Basically because I'm planning on making small games even if I had a small team that would only be a few Max licenses. For big studios it's probably cheaper to create a game editor then have everyone use Max.

There are other problems beyond cost for using Max. The api is clunky and it's hard to create gui's for. You can use Maxscript for lots of things and I do but I'm not Maxscript guru and that means another language to support. I like the idea of using the right language for the problem but only when that language interests me and Maxscript definitely does not.

So how will this all work? What will I use max for exactly? Well obviously art assets. Artist will be able to create models/animations in Max click export and then run the game and view their newly created content. More interesting is the ability to attach game content in max. So once an artist has laid out the art for a level a designer will load that level up and begin to attach game content.

Game content will be attached using Maxscript widgets. Basically the designer will run the Hulkengine Maxscript and see a list of game specific components that he can attach to a Max Node. Then the designer can click export and all the art with game components will export and then he can see/play the game with his content.

The same is true for programmer or sound designer content. All content will go through Max and then into the game.

Here's hoping this works well. Note I had a terrible time placing these images so that's my first blogger negative.