Game Programming 101

Investing

richard@randomly.com — Mon, 05 Jan 2009 05:09:32 GMT

Okay, I'm now officially betting on myself finishing this one. I've invested a whopping $50 registering a domain name for my new game; you can drop by and see a screenshot of the Windows client's menu, since that's about the only thing that's working so far. But it's a start--and now if I don't finish, I'm out the $50 for no reason.

www.praetoronline.com

Incentive, here I come!

After Scripting

richard@randomly.com — Wed, 31 Dec 2008 05:45:49 GMT

Six weeks ago I said I wouldn't post again until I'd finished off the scripting engine. Did that: it's wrapped up, streaming and all. A fun sub-project complete with a compact command-line tool for running scripts and a nice robust runtime interface for linking the interpreter into other applications.

For the curious, the interpreter is downloadable here, and I've put the main runtime interface headers behind an lj-cut just to give you the flavor of the thing.

To be truthful, I finished all that over a week ago. Since then I've been working on the next step: a map editor for my next game, the game that will also use that scripting engine. I'm using the working title Praetor, so you'll probably catch me referring to that. I'll post more about that thing later, but the quick summary is that it's a collectible card game played out using armies deployed on a hexagonal playing board. Cells on the playing board have various terrains, and as you progress through the game you'll face opponents on different boards--so a map editor to lay out that terrain was essential. Here's a sample of what it lets you create:

The editor has a small amount left to do, but it's not a big project and is almost wrapped up already. Importantly, a lot of its code is content that will be shared with the game itself: one third is raw game-engine code, one third is common Banshee-based map-rendering code, and one third is specific to the editor itself. Should help me get the game itself running a little faster than if I had to rewrite all that from scratch.

namespace dualscript
{
   class interpreter
   {
      friend class runtime;
      friend class object;

      public:
         interpreter (linker &link);
         ~interpreter (void);

         void setImportStream (importstream *imp);
         void setExportStream (exportstream *exp);

         void run (void);
         void runtimeError (const char *psz, ...);

         runtime * newVoid (const string &name);
         runtime * newInt (const string &name, int val);
         runtime * newFloat (const string &name, float val);
         runtime * newString (const string &name, const string &text);
         runtime * newObject (const string &name, runtime *classtype, runtime *args);
         runtime * newArray (const string &name);
         runtime * newNamespace (string name);
         runtime * newClass (const string &name, runtime *nspace, runtime::classconstructor pfn);
         runtime * newGlobalMethod (const string &name, runtime *nspace, runtime::globalmethod pfn);
         runtime * newMemberMethod (const string &name, runtime *classtype, runtime::membermethod pfn);

         runtime * find (const string &name);
   };

   class runtime
   {
      friend class interpreter;
      friend class object;

      typedef runtime* (*globalmethod)(runtime *args);
      typedef runtime* (*membermethod)(runtime *obj, runtime *args);
      typedef void (*classconstructor)(runtime *obj, runtime *args);

      public:
         ~runtime (void);
         interpreter & getInterpreter (void);

         // Queries this thing's value as an int, float or string
         int getInt();
         float getFloat();
         string getString();

         // Changes this thing's value to be an int, float or string.
         void setInt (int val);
         void setFloat (float val);
         void setString (const string &text);

         // Manipulates this thing as an object.  You can assign an
         // arbitrary cookie to ride shotgun on an object--a good place
         // to stash a native object if you like.
         void* getCookie();
         void setCookie (void *cookie);

         // Manipulates this object if it's an array type.  Pushing just
         // adds a reference to the runtime's underlying value to the
         // array, so you still have to delete your runtime object to
         // free your own reference to it.  Likewise, when you pop an
         // element from the array, or even just use getIndex() or
         // front() or back() just to peek, you are granted a new runtime
         // object that you must delete eventually.
         void push_front (runtime *val);
         void push_back (runtime *val);
         size_t size (void);
         runtime * require (const string &name);
         runtime * getIndex (size_t ii);
         runtime * front (void);
         runtime * back (void);
         runtime * pop_front (void);
         runtime * pop_back (void);

         // Finds a child.  If this is a namespace, can be used to find
         // a variable, class, global method or child namespace; if this
         // is an object, can be used to find a member variable or method.
         // The path specified can be separated by "."'s.
         runtime * find (string name);

         // If this is a method, calls the method with the specified args
         // (which should be NULL or an array type).  The input args array
         // is deleted for you.  All methods return something; remember to
         // delete the result when you're done with it.
         runtime * call (runtime *args);

         // Insert the specified variable as a global or member var of the
         // current object or namespace.  You can still delete your reference
         // to this runtime afterwards; this process adds a new reference to
         // the entity.
         void insert (runtime *child);
   };
};

Interpreter Progress

richard@randomly.com — Sun, 09 Nov 2008 00:57:10 GMT

Things are going well on the scripting front: the parser has been enriched to handle array definitions properly, the compiler and generate about a dozen different bytecodes, and the interpreter is actually implemented fully enough to run scripts using those codes.

The result is that you can actually run simple scripts now: global variables and local variables all work, mathematical expressions are properly evaluated, and even things like += and -- are compiled and interpreted properly. Yay!

There's still a lot to go: in particular classes, methods, namespaces and array object references are still pending. Oh, and things that involve jumping around at runtime: if/else, for loops, trinary operator, logical-or/-and etc. But things are looking good.

For the die-hard curious out there, I'm posting my current .

-------------------------------------------------------------------------------
BYTECODE FILE FORMAT
-------------------------------------------------------------------------------

The output of the compiler takes the form of a dualscript bytecode blob,
whether saved as a .DSB file or simply passed in raw form directly to the
interpreter.  The content is identical either way.

The DSB file begins with a variable-length header, which is followed by an
array of variable-length records: the header contains data about the
overall script and the names used within it, while each record corresponds
with a class constructor or a class- or global-scope method.  The first record
contains global-scope commands.

All structures are stored little-endian; text is UTF-8 encoded.  The first
name record is always zero-length, and represents unnamed (global) space.

   struct DsbFileHeader {
      uint32_t signature;     // 0xB1DEC0DE
      uint32_t nameCount;     // number of names stored in header
      uint32_t methodCount;   // number of method records
      struct {
         uint8_t nameLength;        // length does not include terminator
         char nameText[nameLength]; // text is not null-terminated
      } names[nameCount];
   };

   struct DsbMethodRecord {
      uint32_t signature;     // 0xFEEDC0DE
      uint32_t nameIndex;     // References the method name (0 for global)
      uint32_t byteCount;     // Bytes in bytecode data
      uint8_t bytecode[byteCount];
   };



-------------------------------------------------------------------------------
OPCODE CONVENTIONS
-------------------------------------------------------------------------------

As stored in the bytecode file, each script starts at instruction address
zero.  These instructions are rebased to lay end-to-end in memory when the
scripts are loaded by the interpreter.

TOS     - top-of-stack.  The most recently pushed item; first to be popped.

NOS     - next-on-stack.  The item just below TOS on the stack.

opcode  - the first byte of an instruction.  Tells the interpreter what to do.
          Also implicitly indicates the number of params; for example,
          the "pop" opcode has no params, while "push" takes one value.

params  - content that follows the opcode, providing details of exactly
          should happen.

instr   - a full instruction, composed of opcode and params.  Instructions
          are and params are unpadded; everything is byte-aligned.

addr    - a reference to the start of a bytecode instruction.  This value is
          always a dword; in each compiled script addr=0 refers to the
          beginning of the file's content, while in memory addr=0 refers to
          the beginning of the first file's content--all others are rebased
          to concatenate from there.

type    - a reference to a type.  This value is again a dword, as described
          in the Types section above.

value   - a pointer to a value object; for example, TOS and NOS refer to
          values.  A value is always 4 bytes.  In the bytecode script, the
          value TODO


-------------------------------------------------------------------------------
DUALSCRIPT OPCODES
-------------------------------------------------------------------------------

assign {mathcode}

      Pops top two values off the stack, copies *(NOS) = TOS, pushes NOS.
      Runtime faults if the assignment involves bogus types of values
      (e.g., a namespace reference, a function pointer, or a const-lhs).
      The mathcode argument can indicate whether this is a simple
      assignment, or whether the LHS value is being adjusted in some
      manner first.

            x = 5;

               lookup-cg {nameid:x}
               constant-int {int:5}
               assign
               pop

            x <<= 5;

               lookup-cg {nameid:x}
               constant-int {int:5}
               assign {leftshift}
               pop


clone

      Pops TOS, creates an unnamed equivalent and pushes that, then re-pushes
      TOS.  This expression is used for post-increment and post-decrement
      expressions, which are otherwise problematic since Dualscript is
      reference-based (so it's hard to leave a value on the stack while the
      variable itself changes content afterwards).  In the example below,
      note that the pre-increment and post-increment are encoded identically
      except for the addition of a clone and pop expression.

            // x starts as 3
            y = ++x;    // x=4, then y=4
            z = x++;    // z=4, but x=5

               lookup-cg {nameid:y}       // stack has &y
               lookup {nameid:x}          // stack has &y, &x
               constant-int {int:1}       // stack has &y, &x, 1
               assign {+}                 // stack has &y, &x  (x now == 4)
               assign                     // stack has &y      (y now == 4 too)
               pop

               lookup-cg {nameid:z}       // stack has &z
               lookup {nameid:x}          // stack has &z, &x
             ->clone                      // stack has &z, 4, &x
               constant-int {int:1}       // stack has &z, 4, &x, 1
               assign {+}                 // stack has &z, 4, &x  (x now == 5)
             ->pop                        // stack has &z, 4
               assign                     // stack has &z         (z now == 4)
               pop

      Note also that this has one side-effect: a post-modified variable leaves
      an *unnamed* equivalent on the stack, while the pre-modified variable
      leaves the properly-named variable on the stack.  The only time this
      matters is if you try to use the thing as a function parameter:

            method foo requires x { ... }
            x = 5;
            foo (++x);  // works fine; passes x==6
            foo (x++);  // runtime error: foo requires "x", you passed unnamed


constant-int {int}
constant-float {float}
constant-string {string}

      Pushes a dsvalue representing a constant onto the stack.  The new value
      is unnamed, attached to the current scope but not added to the current
      namespace.

            x = 5;

               lookup-cg {nameid:x}
               constant-int {int:5}
               assign
               pop


invert

      Reverses the position of TOS and NOS.

            x = ~y;

               lookup-cg {nameid:x}    // stack has &x
               constant-int {int:5}    // stack has &x, &y
               constant-int {int:0}    // stack has &x, &y, 0
               invert                  // stack has &x, 0, &y
               math {~}                // stack has &x, ~y
               assign
               pop


lookup-cg {nameid}
lookup-cl {nameid}
lookup {nameid}

      Looks up the specified name and pushes the corresponding dsvalue onto
      the stack.  The difference between the ops reflects what should happen
      when the lookup fails: -cg creates a new global variable, -cl creates
      a new local variable, and the bland variant just runtime-faults.

            lhs = rhs;

               lookup-cg {nameid:lhs}
               lookup {nameid:rhs}
               assign
               pop


math {mathcode}

      Pops top two values off the stack, performs *(NOS) (op) TOS, pushes
      unnamed result.  Runtime faults if the assignment involves bogus types
      of values (e.g., a namespace reference, a function pointer, or a
      const-lhs).  The mathcode argument indicates what kind of operation
      is being done.  The difference between math and assign is that assign
      modifies the LHS variable and pushes it; math does not modify either
      existing value, but rather pushes an unnamed result.

            x = 3 + 5;

               lookup-cg {nameid:x}
               constant-int {int:3}
               constant-int {int:5}
               math {+}
               assign
               pop

            x <<= 5 | 7;

               lookup-cg {nameid:x}
               constant-int {int:5}
               constant-int {int:7}
               math {|}
               assign {leftshift}
               pop


member

      Resolves a name within an already-on-stack namespace.  At first this
      might seem stupid: if the caller asks for "x = mypkg.string;" then
      why not encode the whole RHS as a compound identifier and look it
      up at once?  The rationale is that the would-be namespace portions
      might be function calls or object members, who can't be resolved
      properly until runtime.

            namespace one.two;
            x = abc.mypkg.globalmethod().datamember.submember;

               // This first command is encoded as looking up "one.two.x"
               // since the nameid we record maps to that string.  In
               // practice the interpreter doesn't lookup "one.two.x"
               // starting at the namespace root; rather, it looks up "x"
               // starting in the "one.two" namespace and working upwards.

               lookup-cg {nameid:one.two.x}     // stack: &x
               lookup {nameid:one.two.abc}      // stack: &x, &abc
               lookup {nameid:mypkg}            // stack: &x, &abc, &mypkg
               member                           // stack: &x, &abc.mypkg
               lookup {nameid:globalmethod}     // stack: &x, &abc, &mypkg, &globalmethod
               member                           // stack: &x, &abc.mypkg.globalmethod
               constant-int {int:0}             // stack: &x, &abc.mypkg.globalmethod, 0 (no parameters)
               call-function                    // stack: &x, &result
               lookup {nameid:datamember}       // stack: &x, &result, &datamember
               member                           // stack: &x, &result.datamember
               lookup {nameid:submember}        // stack: &x, &result.datamember, &submember
               member                           // stack: &x, &result.datamember.submember
               assign                           // stack: &x
               pop


pop

      Pops the top value from the stack and throws it away.

            ;

               pop


return

      Pops the top address off the call stack and continues execution there.
      If the call stack is empty, then execution halts and returns to
      your program.  This method does not affect the exprStack, so anything
      you want to on the exprStack for a return value had better be there
      already.  This opcode implies a scope-end opcode as well.

            method foobar {
               return 5;
            }

               constant-int {int:5}
               // Note: no "pop" instruction: want to leave 5 on the stack!
               return


scope-begin
scope-inner
scope-end

      Controls visibility and release of local variables.  Scope-begin hides
      all existing local variables, creating instead a new local-variable
      scope; this behavior is common with function invocation, and in fact is
      implied by the 'call-function' opcode.  Scope-inner provides an
      interior scope that does not hide existing locals, but allows new locals
      to be destroyed on the next scope-end; this behavior is coded explicitly
      when a statement is written as a series of statements wrapped in braces.
      Scope-end closes a local variable scope; this behavior is implied by
      the 'return' opcode, and is explicitly encoded for other circumstances.

            x = 5;
            {
               y = 3;
            }
            x = y;   // runtime exception: "y" not found

               lookup-cg {nameid:x}
               constant-int {int:5}
               assign
               pop
               scope-begin
               lookup-cg {nameid:y}
               constant-int {int:3}
               assign
               pop
               scope-end
               lookup-cg {nameid:x}
               lookup {nameid:y}    // runtime exception: "y" not found
               assign
               pop


source

      Specifies where the next set of commands came from.  Useless at runtime
      except to report where we are when errors occur.

            foo();
            bar();

               source {nameid:file.ds} {1}   // line 1 of file.ds
               lookup {nameid:foo}
               call-function
               pop
               source {nameid:file.ds} {2}   // line 2 of file.ds
               lookup {nameid:bar}
               call-function
               pop

I'm filling in the docs as I go, so this describes accurately what the script does today--but it lacks several bytecodes that will be necessary to finish this thing off.

Rough Interpreter Design

richard@randomly.com — Thu, 16 Oct 2008 23:34:33 GMT

Been doing too much posting and not enough coding. I'm off to make some more progress on the compiler, which means I need to lay out the bytecode design, which in turn means I needed to summarize how the interpreter was going to run. So here are the notes on the interpreter; back when it's starting to work.

-------------------------------------------------------------------------------
INTERPRETER
-------------------------------------------------------------------------------

So the compiler takes the original script--just a text file--and pre-digests
it into a bytecode form.  The interpreter reads that bytecode and executes
it, performing the steps of the script.

An application can naturally include both portions in its runtime.  In the
app I'm working on at the moment, I only need the interpreter linked into
the final program; the compiling will happen early, and the compiled scripts
will be provided as add-on content for enriching the app.  Lots of ways to
put all this together.

The compiler is, ultimately, just a means to an end: its output is this nice
pretty bytecode, that the interpreter uses as input.  When the interpreter
runs, it keeps track of variables, namespaces, objects and so on; it follows
the bytecode in order to evaluate expressions and call methods.  Importantly,
your application can provide additional methods that the script can invoke
just as it could invoke script-supplied methods.  That's actually essential,
since those application-supplied methods are presumably very fast (being
native code, after all) while the script itself will be slow.  Thus the
script provides the logic, but your application provides the power.


-------------------------------------------------------------------------------
STACKS AND PILES
-------------------------------------------------------------------------------

The difference between a stack and a pile is mostly one of interpretation.  For
a stack, think of a spring-loaded plate holder at a buffet restaurant: put a
new plate on the top, and the whole thing sinks a bit; pull a plate off the
top and it raises a bit to keep that top plate at the same spot.  The position
of an object in the stack is always measured relative to the top of the stack:
this plate is top-of-stack (TOS), the one below that is next-on-stack (NOS),
and although there's some other plate down there ten plates down, we don't 
are about it because we can't reach it.  When a new object is added to a
stack, everything's position in the stack changes.

A pile is mechanically just the same, but without the spring.  In a pile we
measure from the fixed bottom of the pile upwards: that plate is on the
bottom and so has position zero, the next plate is at position one right above
that one, and so on.  Adding more stuff to the pile doesn't change anything
else's location within the pile, and since the pile isn't hidden inside a
spring-loaded chamber we can actually reach any pile element we want at any
time--unlike a stack where we really only care about what's at or near the top.


-------------------------------------------------------------------------------
RUNTIME STACK THEORY
-------------------------------------------------------------------------------

The Dualscript interpreter uses a stack-based expression evaluator.  That kind
of interpreter uses an execution stack where a modern computer would use
registers instead.  It's slow but much simpler to write than a modern compiler
and CPU emulator.

For example, let's say a compiler hits this expression:

      A = (5 + 3) * 2;

The compiler emits instructions like this to match:

      push address-of-A
      push 5
      push 3
      math add       // pops 3 then 5, adds them, pushes 8
      push 2
      math multiply  // pops 2 then 8, multiplies them, pushes 16
      assign         // pops 16 then address-of-A, assigns A=16, pushes 16
      pop            // expression complete; throw away the 16

How about a harder example?

      A = B = 3 * function ( 5 + 8 );

Just for the sake of example, let's say "function" returns 99.  I'm also
going to assume that the caller tells the function how many args are on the
stack for it, and the called function is responsbile for popping those args
and pushing the function result.  So, the compiler would generate this:

      push address-of-A       // stack has A
      push address-of-B       // stack has B, A
      push 3                  // stack has 3, B, A
      push 5                  // stack has 5, 3, B, A
      push 8                  // stack has 8, 5, 3, B, A
      math add                // stack has 13, 3, B, A
      push 1                  // stack has 1, 13, 3, B, A  (1=number of args)
      call function           // stack has 99, 3, B, A
      math multiply           // stack has 297, B, A
      assign                  // stack has 297, A       (B is now 297)
      assign                  // stack has 297          (A is now 297 too)
      pop                     // stack is empty

As you can see, this usage pattern is clearly stack-like and not pile-like:
at any given moment we're only messing with whatever's at the top of the
stack, and we don't really care how deep the stack is.  We'll eventually work
our way back to whatever's futher down as we consolidate the values on the top
into simplified expressions.

The actual operations up there are the bytecode, and there's a separate doc
to talk about the bytecode operations that Dualscript uses.  As I said,
evaluating that bytecode is actually pretty straight-forward; if the compiler
did its job right, then these instructions are each pretty trivial.  It's the
object and memory management that's difficult to get right.


-------------------------------------------------------------------------------
DUALSCRIPT VALUES
-------------------------------------------------------------------------------

Every constant and every variable in a script--whether a primitive like 15 or
"hello" or an object created by the script from a user-defined type--is
represented by a value.  To the interpreter a value is a pointer to a fixed-
size structure more or less like this (and yes, I know you can't put a
C++ std::string object inside a union):

      struct dsvalue {
         string name;
         uint32_t type;
         union {
            dsobject *obj;
            int64 inum;
            double dnum;
            string str;
         } data;
      };

Notice that every value has a name--in practice that's the name of a variable,
like "format" in the earlier example.  Those names are useful when printing
error messages, but importantly they're also used during function invocation
since function parameter order is flexible in Dualscript.

The type field is little more than a discriminator for the data area: is
this value an integer, a floating point number, a string, or a user-defined
type (or maybe simply undefined)?  If it's a user-defined type, which one?

On the other end of "data.obj" is a dynamically allocated structure that
includes a refcount, and that holds dsvalue structures for data members.
More about all that later.


-------------------------------------------------------------------------------
NAMESPACES
-------------------------------------------------------------------------------

A script can declare a namespace for its content:

      namespace movies.fiction;
      int svar;   // script variable
      method smethod { ... } // script method
      realgenius {
         global gvar = 12;  // global variable
         ivar = 5;   // instance variable
         method gmethod { ... }
         method imethod requires this { ... }     // instance method
      }

A namespace is just a list of names that map to stuff: classes, methods,
variables, or importantly, other namespaces.  In this example, the
namespace hierarchy looks like this:

      (root) contains {
         "movies" = child namespace
      }
      "movies" contains {
         "fiction" = child namespace
      }
      "movies.fiction" contains {
         "svar" = variable
         "smethod" = subroutine
         "realgenius" = namespace
      }
      "movies.fiction.realgenius" contains {
         (instance) = implicit child
         "gvar" = variable
         "gmethod" = subroutine
      }
      "movies.fiction.realgenius.(instance)" contains {
         "ivar" = variable
         "imethod" = subroutine
      }

The interpreter manages namespaces.  When trying to find a name, the
interpreter checks the current namespace first; if it can't find the name,
it walks up one level towards the root and tries again.  The "namespace"
command remains in effect until overridden with another command; just
"namespace;" returns to the root namespace.



-------------------------------------------------------------------------------
THE INTERPRETER PILE
-------------------------------------------------------------------------------

The interpreter's pile is just a place to store variables.  The pile grows
when new variables are defined, and the pile shrinks when the script leaves
the scope where some variables were defined.  Globals go on the pile too just
like local variables do--but, since we never leave global scope, they never
go away.  (Each class definition likewise gets one global variable to contain
its class-global data members.)  Every element in the pile is a full dsvalue
structure, not a pointer to one.

      {
         format = "Hello, World";
      }

This script is going to eventaully need two values on the pile: one for the
"Hello, World" constant and another one for the local variable "format".

Since these are the first ones mentioned in the script, the pile is empty
before this line is encountered.  When the compiler first hits this line,
it reads the word "format" and realizes it's defining a new variable.
(Since the compiler hadn't heard of the variable named "hello" before, if
it wasn't showing up as the left-hand-side of an assignment expression
then the compiler would issue a compile-time error.)

So the compiler hits that word "format" and emits two things into the bytecode
stream: first it emits a command to reserve space on  TODO

variable as a left-hand-side, the compiler emits the address of this value
onto the stack (so, it emits 0--this is on the bottom of the pile) and
proceeds.

The line causes the
interpreter to add a value for hello to the pile, and then a value for the
constant.  (I'll draw the first object on the bottom, like a pile should be.)
First, we reserve space for hello:

      0: { name="hello", type=undefined, data=null }

We haven't done any work with the variable yet; it's uninitialized so far, and
therefore has no content associated with it.

Now the compiler hits the = operator, so it recursively evaluates the rest of
the expression and them emits an "assign" opcode.  (There's a different doc
about how the recursive-descent parser works.)  Evaluating that right-hand-
side, though, encounters the constant string "Hello, World".  We need a value
to represent that, and since it's a primitive we know just how to set it up:

      1: { name=none, type=string|readonly, data=0x8E239FB0 }
      0: { name="hello", type=undefined, data=null }

The data=0x8E239FB0 pointer refers to a dynamically allocated, refcounted
structure like this:

      struct stringdata {
         size_t refcount = 1;
         string ss = "Hello, World";
      }

The reference count indicates that there's exactly one variable in the pile
who points to this thing, which is true right now.  The compiler has ended
up emitting a sequence like this:

      addpile null   // tells interpreter to add the hello value to the pile
      push 0         // pushes address of hello onto runtime stack
      addpile string "Hello, World"
      push 1         // now stack has &"HelloWorld" on top, &hello just below
      assign         // pops both values from stack, assigns, pushes one back
      pop            // semicolon pops that final value
      removepile 2   // removes top two values from stack

The assignment leaves the following on the pile:

      1: { name=none, type=string, data=0x8E239FB0 }
      0: { name="hello", type=string, data=0x8E239FB0 }
         ...with 0x8E239FB0's refcount==2

The removepile instruction causes the interpreter to get rid of the top two
items on the stack.  Removing each one decrements a reference from the target
stringdata structure; when its refcount drops to zero, the interpreter
immediately deletes the structure itself.  (If the structure were for a user-
supplied type, it would invoke the destroy member first.)


-------------------------------------------------------------------------------
DUALSCRIPT TYPES
-------------------------------------------------------------------------------

Starting with something trivial: type=0 means unknown; type=1 means number;
type=2 means string.  Anything higher than that means a user-constructed type.

The compiler works on a source-file-by-source-file basis: no prototypes or
externs required.  So any given text script is compiled into something self-
contained: it doesn't know what types another script might have exposed.

So, let's say a script defines type Foo.  When the compiler hits it, it could
pick identifier 10 to represent type Foo.  Two days later the compiler loads
a different file and compiles it separately, and it again picks identifier 10.
It doesn't know any better; every script is self-contained.

So, the interpreter has to resolve ties.  When the interpreter loads a
compiled script, it has to look through the script's data types and remap
each one to something not currently in use in memory.  So, if the interpreter
has already loaded a dozen scripts and is using types 0..50, then when it
loads a new script referring to a new type name, it will remap that name to
type 51 instead of zero.

The name of the type is relevant here.  If two otherwise-unrelated scripts
use the same type name, then it's a cinch they were made to work together
and are talking about the same type.  So the interpreter will remap them to
the same constant.

The interpreter also has to keep in mind a lot of other stuff about each
type.  Type constructs are intended to be additive: if script A defines
members a1 and a2 in a type, and script B defines members b1 and b2 in
that same type, then the interpreter will consider the type to have 4 members.
It's not secure, but who cares?

Recursive Descent Parsing

richard@randomly.com — Wed, 15 Oct 2008 23:48:12 GMT

If you look over the formal grammar, you'll see that it's defined as a hierarchy. For example, type A is defined as type B followed by a semicolon or an optional series of type C followed by a type D:

   type_a ::=   type_b ";"   |   [type_c]* type_d

A simplistic parser that recognizes such a definition would look roughly like this:

   bool parse_type_a (sourcedata &in)
   {
      pos = in.tellPosition();

      // See if we have a type_b followed by a semicolon
      if (parse_type_b (in)) {
         if (parse_token (in, SEMICOLON)) {
            return true;
         }
         in.setPosition (pos);  // didn't pan out; rewind try the other form
      }

      // Didn't have that; see if we have a series of type_c's followed by a type_d
      while (parse_type_c (in)) {
         ;
      }
      if (parse_type_d (in)) {
         return true;
      }

      in.setPosition (pos);
      return false;
   }

Notice that, on failure, we didn't scream and complain: if parse_type_b doesn't see a type_b syntax coming up next on the stream, then it's not a big deal. All it does is return false, indicating that we didn't see a type_b coming up--and importantly, we didn't advance the stream at all. (Notice that we might've successfully pulled a type_c or two off the stream, then failed to find a type_d following on. Gotta put those type_c's back.)

With an adequately complex grammar (like the one we're using for parsing C-like expressions), you can imagine how quickly this turns into a deep recursive stack. In practice, this is exactly how the recursive descent parser works: having defined the grammar very precisely, we write methods that exactly parse each basic type, and allow them to recurse pretty deeply. It's quite simple to implement, but can be unpleasant to debug; helps to add a tracing facility to show where you are in the parsing stack.

Starting the Compiler

richard@randomly.com — Sun, 12 Oct 2008 00:17:13 GMT

Once you've nailed down the grammar--and yes, I made several changes since my last post--it's time to start the compiler itself. The compiler takes two stages: parsing and generation.

In the parsing stage, we'll just write enough code to properly walk the script all the way through--doing nothing at all, just scanning through. By the time the parsing stage is done, the compiler will be able to detect and report all manner of syntax errors while silently accepting all properly-formatted scripts. The compiler will not, however, do anything else useful at all. This is the last stage we can really accomplish without giving some serious thought to how the interpreter is going to work.

The generation stage will be much harder: that's where we insert code into the existing parser to make the compiler actually generate some kind of machine-usable bytecode output. And that's when things get really fun.

I've just about finished my parser; it handles everything except flow control (for/do/while/if/else etc) and array initialization. When I've got that stabilized a little bit, I'll post in more detail about how that parser works, then move on to the interpreter and bytecode generator. Good geek stuff coming up!

Defining the Grammar

richard@randomly.com — Thu, 09 Oct 2008 04:49:47 GMT

Having designed the scripting-language-to-be, and having already broken an inbound script into tokens, the next step is to compile the thing. To do that, we'll need a formal grammar for the language: something more precise than the human-readable examples whipped up so far.

The traditional way to lay this out is through BNF format, and to make use of YACC ("yet another compiler compiler"). Again I'm going to skip over that thing and just roll the code myself, but writing out some BNF-like form for the language is essential--especially when you plan to use a recursive descent compiler, since the methods you write will almost exactly follow the BNF assignment statements.

A first cut at the grammar for Dualscript is below. I've smoothed out a few things in the language since I posted earlier: added a keyword here or removed one there, eliminated class-scope methods and data variables, and so forth. The result is a pretty simplistic scripting language that has a certain degree of agreeable symmetry. There are probably some significant errors below--in particular I've omitted the array definition syntax and I'm not happy with postfix_expr through postfix_operator, so I may need to rework those. But it's time for bed now, so this is going to have to suffice.

-------------------------------------------------------------------------------
LANGUAGE GRAMMAR
-------------------------------------------------------------------------------

Dualscript has a particularly simplistic grammar--at least, as simplistic as
anything can be that understands mathematical expression evaluation.  Since
the compiler requires a formal definition of the structure of the language,
though, I'm laying it out below.

Personal note: I don't care for or about any particular variant of BNF, so
this is more or less a mixture of various syntactical descriptions.  But I bet
you can follow it anyway.

Some conventions I've followed here:

   abc*         means zero or more instances of abc
   [abc]        means abc can optionally appear
   "abc"        refers to the keyword abc
   abc | def    matches either abc or def
   abc def      refers to abc then def in sequence


-------------------------------------------------------------------------------
GLOBAL SCOPE
-------------------------------------------------------------------------------

A script is, at heart, a series of global_statement expressions.  Each
global_statement is just something that defines a class, a method or a
namespace, or maybe just does some arbitrary thing like "X=5".


      script ::= global_statement*

      global_statement ::= class_definition |
                           method_definition |
                           namespace_definition |
                           enum_definition |
                           statement


-------------------------------------------------------------------------------
CLASSES AND METHODS
-------------------------------------------------------------------------------

These two--defining a class and defining a method--are intentionally nearly
identical syntactically.  (Later on we'll see that they're nearly identical
in implementation and invocation too, so that makes sense.)  Both start with
a short header, then have a brace-encapsulated array of statements.  Note that,
while classes can only be defined at global scope (e.g., no classdef statments
are allowed within other classdefs or within methoddefs), methods can be
defined both at global scope and at class scope (but still not inside another
method).

Remember the ellipses we allow on the input parameters for a method?  That's
Dualscript's equivalent for stdarg; all unrecognized input params get
thrown into that array.


      class_definition ::= "class" identifier "{" class_statement* "}"

      class_statement ::= statement | method_definition

      method_definition ::= "method" identifier method_params "{" statement* "}"

      method_params ::= "requires" identifier ["," identifier]* ["..."]


-------------------------------------------------------------------------------
ENUMERATIONS
-------------------------------------------------------------------------------

Enumerations are really trivial.  Note that I don't let you assign particular
values to enum entries; you just get whatever the compiler chooses.  Keeps you
from cheating.


      enum_definition ::= "enum" identifier "{" identifier ["," identifier]* "}"


-------------------------------------------------------------------------------
NAMESPACES AND IDENTIFIERS
-------------------------------------------------------------------------------

A quick word on these.  Although we haven't described them yet in the syntax,
there are two kinds of identifiers: the identifier syntax type is just for a
single C-compliant name (e.g., letters and underscores, with numbers allowed
too except in the first character), while a compound_identifier syntax type
allows one or more "identifier." prefixes first.  The namespace command is
the perfect example for introducing the compound identifier, so we'll do that
next.


      namespace_definition ::= "namespace" [compound] ;

      compound ::= { identifier "." }* identifier


-------------------------------------------------------------------------------
FLOW CONTROL
-------------------------------------------------------------------------------

C allows a lot of ways to make loops happen, and Dualscript supports all of
them except goto.  (Goto sucks, right?)  So we need to describe those things
too.  This is also where we define a "statement"--and, like C, notice that a
single "statement" thing can also be recursively defined as a bunch of
statement things wrapped in braces.


      statement ::= for_loop |
                    foreach_loop |
                    do_loop |
                    while_loop |
                    if_clause |
                    switch_clause |
                    "break" ";" |
                    "continue" ";" |
                    "return" [expr] ";" |
                    [expr] ";" |
                    "{" [statement]* "}"

      for_loop ::= "for" "(" [expr] ";" [expr] ";" [expr] ")" statement

      foreach_loop ::= "foreach" compound "in" expr statement

      do_loop ::= "do" statement "while" "(" expr ")" ";"

      while_loop ::= ""while" "(" expr ")" statement

      if_clause ::= "if" "(" expr ")" statement ["else" statement]

      switch_clause ::= "switch" "(" expr ")" "{" case_statement [case_statement]* "}"

      case_statement ::= "case" expr ":" statement |
                         "default" ":" statement


-------------------------------------------------------------------------------
EXPR
-------------------------------------------------------------------------------

Believe it or not, that's almost all there is to the overall language.  At this
point, we've defined the entire language of a script except for that pesky
"statement" type.  It's the hardest one we have to deal with, but only because
it allows expressions like this:

      x <<= 3 * (2 + function(17, 3 ^ myobject.data));

Anyway, we'll break it down into pieces.  First a few categories:


      assignment_operator ::= "=" | "*=" | "/=" | "%=" | "+=" | "-=" |
                              "&=" | "|=" | "^=" | "<<=" | ">>=" | ":="

      equality_operator ::= "==" | "!=" 

      relational_operator ::= "<" | ">" | "<=" | ">="

      shift_operator ::= "<<" | ">>"

      additive_operator ::= "+" | "-"

      multiplicative_operator ::= "*" | "/" | "%"

      prefix_operator ::= "++" | "--" | "+" | "-" | "~" | "!"

      constant ::= integer | float | character | string


Okay, now let's do expr itself.  The first step is pretty easy; you see this
rule applied a lot in for loops.  From there things get harder, with a deep
cascade supplying precedence for important operations.


      expr ::= one_expr [ "," one_expr ]*

      one_expr ::= [ unary_expr assignment_operator ]* conditional_expr

      conditional_expr ::= logical_or_expr [ "?" expr ":" conditional_expr ]

      logical_or_expr ::= logical_and_expr [ "||" logical_and_expr ]*

      logical_and_expr ::= bitwise_or_expr [ "&&" bitwise_or_expr ]*

      bitwise_or_expr ::= bitwise_caret_expr [ "|" bitwise_caret_expr ]*

      bitwise_caret_expr ::= bitwise_and_expr [ "^" bitwise_and_expr ]*

      bitwise_and_expr ::= equality_expr [ "&" equality_expr ]*

      equality_expr ::= relational_expr [ equality_operator relational_expr ]*

      relational_expr ::= shift_expr [ relational_operator shift_expr ]*

      shift_expr ::= additive_expr [ shift_operator additive_expr ]*

      additive_expr ::= multiplicative_expr [ additive_operator multiplicative_expr ]*

      multiplicative_expr ::= unary_expr [ multiplicative_operator unary_expr ]*


Whew.  That was pretty ugly, but it brings us all the way down to unary
expressions, which is really where the last bit of magic happens.


      unary_expr ::= [prefix_operator] postfix_expr

      postfix_expr ::= constant | value_expr [affix_expr]

      value_expr ::= identifier | "(" expr ")"

      postfix_operator ::= "++" |
                           "--" |
                           "." value_expr |
                           "(" [expr] ")" |
                           "[" expr "]"

Tokenizing

richard@randomly.com — Mon, 06 Oct 2008 23:56:13 GMT

A short article for an easy step. The first part of compiling or interpreting a script is breaking the script's text up into tokens. If you're using lex (which you probably should do), that's pretty easy--but since "easy" is no fun, I'll be doing it the old-fashioned hand-coded way.

A clever tokenizer won't run too far ahead of the actual compiler: that's called a streaming tokenizer, and it's marginally more complex than a tokenize-everything-and-remember-it-all-at-once tokenizer. I'm using the latter, since I've used up all my "easy is no fun"-ness in the first paragraph.

The basic tokenization loop looks more or less like this:

   as long as there's content left in the input script {

      skip whitespace (including EOLs)
      if we hit "//", skip to the end of the line and restart this loop
      if we hit "/*", keep reading until we hit "*/" then restart this loop
      if we ran into the end of the script, quit now

      if the next char is a digit, look for number sequences
         don't forget to look for hex and octal radixes ("0x5E13", "0777")
         don't forget to look for decimals and exponents ("15.37", "27e+5")
         remember to look for special cases ("0", "0.3" which look octal-ish)

      see if the next 3 characters match a 3-character token (like ">>=");
         if so, record that token into our output and restart this loop

      likewise, see if the next 2 characters match a 2-character token ("+=", "<<")

      likewise, see if the next character matches a 1-character token (":", ";" etc)
 
      if the next character is an apostrophe, crack character sequences like 'x'
         remember to handle encodings like '\t' for tab, '\n' for newline etc
         and of course '\x7f', '\127', '\035' should be supported too

      if the next character is a quotation mark, try to pull a whole string
         this is pretty easy--just skip the ", then keep reading until hit another one
         again, look for \ prefixes, and don't be fooled by \"

      okay, the next word must be plain text--either a keyword or something like a variable.
      scan forward until we run out of legal characters for either, and accumulate the text.
      then match against known keywords ("for", "return" etc)
   }

And that's it. No magic involved--just some simple text cracking. The result is that we can stop worrying about the text file that the user supplied; instead, we have a much more programmatically accessible array of tokens. The compiler will start pawing through those tokens to get its work done--in the next post.

Compiling and Interpreting Scripts

richard@randomly.com — Fri, 03 Oct 2008 22:29:32 GMT

Is this Game Programming 101 or not? Haven't done anything but whine about cell phones and talk about TKD for a year now. Time to get on with the programming!

"In the beginning, the Matrix was designed as a utopia for humans, but some say we lack the programming code to create a perfect world." -Agent Smith, Matrix

Yes, it's time to design a scripting language, complete with an interpreter. I'm doing this as part of another project--will get to that later--but it's a fun little self-contained project all on its own.

The first step in the project is to be sure you really need it. Adding a scripting language to a system can make it extremely flexible and powerful, but even the blunt approach I'm going to cover here is a good amount of work. If you can get by without it, by all means use a less general-purpose solution.

Okay, you passed that hurdle: you want a script after all--and by the same process, you've decided that downloading and incorporating a public Javascript engine or calling out to a Perl interpreter isn't doing it for you either. You need something new--some tricky new syntactical feature, integration with your home-rolled application or something you can sell without licensing problems. So a new scripting language and interpreter it will be.

Next step is to figure out what kind of syntax your language will provide. I've just finished a write-up of the major features of my next project, Dualscript. And I'll even put it behind a cut, because it's about 500 lines. Next up: posts about the bytecode, tokenizing, the recursive-descent parser and putting together a stack-based interpreter.

-------------------------------------------------------------------------------
SUMMARY
-------------------------------------------------------------------------------

Dualscript uses a recursive-descent compiler and a crappy little stack-based
interpreter, because they're both easy and performance isn't a big issue here.
The only consumer so far precompiles all scripts into bytecode before the app
is even released, and even the interpreter runs for only moments at a time.
It's an important feature, but not a time-critical one.

As with any other stack-based runtime, every expression involves pushing
content onto the stack and manipulating that stack, while a trailing semicolon
in the text means it's time to pop the last value off the stack in order to
end the expression.  For example, just "15;" would push 15 onto the stack
then pop it right back off--dumb but legal.  (And yes, the compiler has to
look for superfluous semicolons like "x=5;;" and skip them, or the runtime
stack will underflow when it tries to pop something that doesn't exist.)

The "dual" in dualscript refers to its double nature as both master and slave
scripts.  When you invoke a script as a master, it can export its decisions
to an external stream; when you invoke the same script as a slave, it reads
those decisions instead of making its own choices.  This ability is built into
the interpreter and supported syntactically within the language.

A game like Praetor, just as a convenient example, relies on this duality as
part of its intrinsic nature.  When user A casts a spell, the game runs the
card's "cast" method. That method runs as a master script, showing prompts to
the user ("sacrifice which creature?") and creating new in-game objects,
and exporting all its important decisions (like the UUID of a new piece) into
an import/export stream.  When user A's turn is over, that whole stream gets
sent to user B, so that when user B's computer replays user A's turn and tries
to invoke the same card's "cast" method, user B's computer imports all the
choices made the first time around--so user B sacrifices the same creature and
gets the same new UUID for a newly created object.


-------------------------------------------------------------------------------
PROGRAM STRUCTURE
-------------------------------------------------------------------------------

A script file is just a series of expressions, which are happily executed at
global scope; none of this everything-inside-a-class crap like Java demands.
This is a perfectly valid script:

      console.print (format = "Hello world!\n");

Like any other C-like language, this invokes a method called "print", in this
case taken from the "console" class.  As a parameter, it constructs a local
variable called "format", initializes its value to the string "Hello world"
and passes it to the method.

In this example, "console" is a class name.  The class and method were actually
supplied by the interpreter, but in theory a script could've implemented them
like this:

      console
      {
         method print requires format, args...
         {
            // code to lay out and print contents goes here
         }
      }

As you can see, the bareword "console" is defining the start of a class named
"console", and "method print" is defining a method within that class.


-------------------------------------------------------------------------------
PRIMITIVE DATA TYPES
-------------------------------------------------------------------------------

You've noticed already that Dualscript doesn't make you specify a lot of type
names.  A type is implicitly part of every value, so typenames are largely
unnecessary.

      method foobar requires param
      {
         if (param == "string") {
            return "here's your answer";
         } else if (param == 10) {
            return 15;
         }
      }

The only primitives are string-types and number-types; internally numbers are
64-bit floating point and strings are utf-8, but in practice you won't care.
There are no "int", "bool" or "string" keywords (though "null", "false" and
"error" are aliased to int 0 while "true" and "success" are aliased to int 1).


-------------------------------------------------------------------------------
AGGREGATE DATA TYPES
-------------------------------------------------------------------------------

Scripts can define aggregate types analogous to C++ or Java classes.  Like most
other object-oriented languages, Dualscript classes can have both methods and
data members, and those can be either class-scope or instance-scope.

      myclass
      {
         // Class data member
         global param1 = ... ;

         // Per-instance data member
         param2 = ... ;

         // Class method
         method function { ... }

         // Instance method
         method function requires this { ... }
      }

Expressions inside the class that start with "global" are executed when the
script is compiled and typically define variables whose scope is internal to
the class; all other expressions are executed when an object of that type is
constructed, and any variables are scoped to this particular instance.

A class can take input parameters to be used during instance construction,
using the same syntax as a method would use:

      myclass requires width, height
      {
         global style = "metric";
         mywidth = width;
         myheight = height;

         method print requires this
         {
            return "size is "+this.mywidth+"x"+this.myheight+", "+style;
         }
      }

The "requires this" and references to "this." indicate that a method is an
instance method rather than a class method.  They're also entirely optional;
the compiler will implicitly add "requires this" if anything in the method
uses a reference to "this", and the compiler will further recognize
references to instance members as implicitly needing "this."  So the
previous example can be simplified a bit:

      myclass requires width, height
      {
         mywidth = width;
         myheight = height;

         method print
         {
            return "size is "+mywidth+"x"+myheight+", "+style;
         }
      }

Any method that doesn't use "this" can be invoked as a class method or as an
instance method.  From outside the class, methods and members must be invoked
using classname.member notation, as in the console.print(...) example earlier.

Classes cannot nest, nor can methods contain class or method definitions.


-------------------------------------------------------------------------------
MEMORY MANAGEMENT
-------------------------------------------------------------------------------

Scripts can create objects easily:

      instance = new myclass (width = 5, height = 12);

Memory is automatically garbage collected immediately upon loss of the last
reference to an object.  If the object contains a "destroy" method, then that
method is invoked as soon as all interior object references are likewise
dropped.  For example:

      class1
      {
         console.print (format="create class1\n");
         method destroy {
            console.print (format="destroy class1\n");
         }
      }

      class2
      {
         console.print (format="start creating class2\n");
         member = new class1;
         console.print (format="finish creating class2\n");
         method destroy {
            console.print (format="destroy class2\n");
         }
      }

      new class2;
      console.print (format="done\n");

This script immediately prints the following:

      start creating class2
      create class1
      finish creating class2
      destroy class1
      destroy class2
      done

When the "new class2;" line hits that last semicolon, the interpreter throws
away the only reference to the new object (which was, until now, sitting on
the stack waiting to be assigned to something).  That immediately starts the
release process for the class2 object.  That release process iterates through
the data members of class2 and drops a reference for each of them, which in
this case causes the last reference to class1 to disappear as well.  When
class1 goes away it invokes its destroy method, and only afterwards is class2
able to finish being destroyed.


-------------------------------------------------------------------------------
ENUMERATIONS
-------------------------------------------------------------------------------

Just using what's defined above, one could implement an enum easily enough:

      myenum
      {
         global ONE_OPTION = 0;
         global OTHER_OPTION = 1;
         global FINAL_OPTION = 2;
         global size = 3;  // count of elements
      }

But that's a lot of repetition.  A shorthand variant is supplied:

      enum myenum
      {
         ONE_OPTION,
         OTHER_OPTION,
         FINAL_OPTION
      }

Either technique allows callers to use the values in the enum:

      value = myenum.ONE_OPTION;
      count = myenum.size;


-------------------------------------------------------------------------------
ARRAYS
-------------------------------------------------------------------------------

Dualscript supplies only a single-dimensional, automatically-sized array type.
The type acts more like a doubly linked list than anything, but does have
acceleration for random access.

      myarray = ( value1, value2, value3 );

      myarray[1] = 5;
      myarray.pushFirst (value0);
      myarray.eraseIndex (1);
      previoustail = myarray.popLast();

One can iterate through elements in an array using the foreach operator:

      sum = 0;
      foreach value in myarray {
         sum += value;
      }

      sum = 0;
      foreach value in ( 1, 12, 37, 99 ) sum += value;

One can also use the more C-like "for" operator, and of course "while" and
"do ... while" are likewise available.

      sum = 0;
      for (index = 0; index < myarray.size; ++index) {
         sum += myarray[index];
      }


-------------------------------------------------------------------------------
VALUES AND REFERENCES
-------------------------------------------------------------------------------

Internally, consider a constant as a pointer to a short structure like this:

      5
         5 has { name=undef, type=number, value=0x00000005 }

Similarly, a variable is a pointer to a structure like this:

      var = new myclass;
         var has { name="var", type="myclass", value=0x8C3412C9 }

When you perform an assignment, the interpreter replicates whatever is stored
inside those braces (but keeps the original name). So:

      x = 5;
         x has { name="x", type=number, value=0x00000005 }
         5 has { name=undef, type=number, value=0x00000005 }

      y = x;
         y has { name="y", type=number, value=0x00000005 }
         x has { name="x", type=number, value=0x00000005 }

      x = 10;
         y has { name="y", type=number, value=0x00000005 }
         x has { name="x", type=number, value=0x0000000A }

Consider the impact of using a derived type instead of a primitive--and note
that there's a second layer of indirection involved since "value" actually
is a pointer to dynamically allocated storage that holds the object's
real contents.

      x = new myclass(text="hi");
         // x has { name="x", type="myclass", value=0x8EF20557 ("hi") }

      y = x;
         // y has { name="y", type="myclass", value=0x8EF20557 ("hi") }
         // x has { name="x", type="myclass", value=0x8EF20557 ("hi") }

      x.changeText (newtext="bye");
         // y has { name="y", type="myclass", value=0x8EF20557 ("bye") }
         // x has { name="x", type="myclass", value=0x8EF20557 ("bye") }

In this case, then, it feels like X and Y are acting more like references
and pointers than values.  Dualscript likewise uses the Java style of
pass-reference-by-value for function parameters.  For example:

      method foo requires val
      {
            // val here is a new local, at address 0x8834CE90.  Its value
            // was copied from the input param's value just as above.
            // *0x8834CE90 = { name="val", type=string, value=0x8A983098 ("bar") }

         val = "foo";
            // The "val" variable is still at the same spot, but its value
            // has now been changed to point to this other string.
            // *0x8834CE90 = { name="val", type=string, value=0x8C440324 ("foo") }
      }

      val = "bar";
         // val is at address 0x84593278
         // *0x84593278 = { name="val", type=string, value=0x8A983098 ("bar") }

      foo(val);
         // val is still at address 0x84593278, whose contents are unchanged
         // *0x84593278 = { name="val", type=string, value=0x8A983098 ("bar") }

This is all exactly as Java does things, which makes me feel pretty bad since
I really don't like Java all that much.  It might feel a little weird at first,
but will become perfectly comfortable once your standards start slipping.


-------------------------------------------------------------------------------
POINTERS
-------------------------------------------------------------------------------

As you probably figured out from the above, Dualscript doesn't offer & or *
operators.  A variable that points to a derived object is by definition a
pointer, so you can construct linked lists and such just as you would in Java:

      classname {
         payload = "default payload";
         next = null;
      }

      o1 = new classname;
      o2 = new classname;
      o1.next = o2;


-------------------------------------------------------------------------------
FUNCTION PARAMETERS
-------------------------------------------------------------------------------

The weirdest thing in dualscript is probably its use of variable names during
function calls.  In Perl, there's a common parameter-passing convention:

      sub myfunction
      {
         my @args = @_;
         printf ("got %u, %s\n", $args{foo}, $args{bar});
      }

      myfunction (foo => 15, bar => "hello");

The recipient gets his parameters as the ordered list { foo, 15, bar, hello }
and will typically construct a hash from that list, then dig parameters out of
the hash by name ($args{foo}).  Dualscript uses something like this, but it's
both required and easier to use.

      method myfunction requires foo, bar
      {
         print (format = "got %u, %s\n", foo, bar);
      }

      myfunction (foo = 15, bar = "hello");

"But wait!" you say, "isn't the caller defining local variables foo and bar,
right there inside the parentheses when it calls the function?"  Yes, he is.
The compiler constructs local variables foo and bar within the context of the
caller, then passes their addresses into myfunction in whatever order.  The
compiler and interpreter take care of matching up input parameters with
arguments regardless of their order.  Variables constructed inside the
parentheses are scoped by the parentheses, just like a C99 "for" loop.

You'll also see this:

      foo = 5;
      myfunction (foo, bar = "hello");

In this case, there was already a local variable named "foo", and we chose to
pass it right along.  In this case, since the function wanted something called
"foo" anyway there was no need to construct a second local inside the parens;
the existing local worked fine.  If the name of the local wasn't "foo", though,
we would have to create a "foo" to pass in.

      notfoo = 5;
      myfunction (foo = notfoo, bar = "hello");

This conforms with myfunction saying "requires foo, bar" --if you don't pass
something named "foo", it can't run.

One extra note: functions can accept variable arguments by specifying exactly
one input argument with ellipses.  That variable is initialized as an array,
with one element for each otherwise-unexpected input argument.

      myfunction sum requires elements...
      {
         result = 0;
         foreach value in elements result += value;
         return result;
      }

      total = sum (5, 3, 12);
      // now total is 20

      total = sum ("hello", 5, 3, 12);
      // amusingly, now total is "hello5312", since string+anything == string

To return to the first example in this document, then:

      console
      {
         method print requires format, args...
         {
            // code to lay out and print contents goes here
         }
      }

One technically can call this method in any order, because the recognized
"format" argument is pulled out and everything remaining is inserted in order
into the catch-all "args" array.

      console.print (5, format="what is %u+%u?", 3);
      // prints "what is 5+3?"


-------------------------------------------------------------------------------
IMPORT/EXPORT
-------------------------------------------------------------------------------

Streams are what give dualscript its value; otherwise it's just a sick take-off
of javascript.  Exporting a value into the stream is easy:

      variable = 15;
      variable.export();

Importing is just as easy, and if the import returns false then there's
probably no input stream available:

      variable.import();

There's a very common sequence, actually, that looks like this:

      if (!variable.import()) {
         variable = ... ;
         variable.export();
      }

Since it's so common, there's a shorthand version:

      variable := ... ;

The idea is, if there's an import stream, this reads the value from the stream;
if there's not, this evaluates expression, copies it into value, and tries to
export it into any stream (if one has been provided).  This expression is so
common that it's often the only hint of streams that you'll see at all.

Keep in mind that, if you export a variable twice as a master script, you had
better import it twice as a slave script.  That's easy; if you simply stick
with using the ":=" expression then you literally can't go wrong.

A New Platform!

richard@randomly.com — Mon, 10 Mar 2008 04:35:52 GMT

I've been a big fan of the Danger Hiptop cell phone, and have made a decent wage writing games for it. But writing in Java hurts my pride, and when my third Hiptop in a row died a hardware death I decided it was time to rethink my choice of platform.

See, the Hiptop just doesn't seem to be going anywhere. After about four years now, their hardware has changed slightly: a two-directional wheel has replaced the one-directional one, and the screen resolution is better. The camera is maybe 1.6 megapixels instead of 1.0. Wow. But it still doesn't have anything more than rudimentary bluetooth, doesn't have any kind of GPS, its CPU-power-to-display-pixels ratio is *decreasing*, and its radio... god help me, but GPRS connectivity sucks even when it actually works. Four years, and they haven't done anything about some fundamental problems.

So I've swapped brands, and am now one of the herd playing with an iPhone. It's a new world, baby. Looking forward to picking up some Objective-C, learning some xCode, and capitalizing on the latest fad.

December? Already??

richard@randomly.com — Sat, 01 Dec 2007 19:17:11 GMT

Nathaniel earned a second yellow stripe this morning; I got to watch the whole thing, this time from up front as one of the black belts helping out with the ceremonies. Fun! He was remarkably mature: no squirming, no silliness, good effort. He took his time with his pattern and did it strongly, even though the other children were rushing their way through. I predict he's going to be phenomenal at this.

Isaac and Mom are off at their own testing right now; they should be back soon in fact. Ike's aiming to get the first bit of black on his belt as he pre-tests for 1st degree; Tara's working on her red belt. All fun. :)

In other news, NaNoWriMo is finally over. Since I can't write my way out of a wet paper bag, I don't plan to return to the jokingly-referred-to-as-a-manuscript for edits or anything, so I figured I'd just post it here. Have fun, and if you laugh yourself sick at my lack of writing prowess, don't say I didn't warn you.

Dust - Richard's NaNoWriMo book from 2007

VNC Shot Down

richard@randomly.com — Fri, 22 Jun 2007 14:53:30 GMT

Well, so much for that one.

I actually finished the VNC client; it turned into a nicely polished application, and works surprisingly well. I packed it with little things to help clean up the remote-manipulation experience while working on a high-latency link: it manages your cursor locally so there's no visible lag to cursor movement, batches up cursor- and keystroke- transmission to minimize bandwidth consumption, has a clever popup/overlay typing editor that shows you your keystrokes so that you can easily edit text without waiting 2 seconds for a screen refresh to find out what you actually typed, and so on. No question, it's the best VNC app I've ever seen for a cell phone. :)

Sadly, Danger shot it down:

Dear Random Software,

After careful evaluation, we do not believe this application to be a candidate for Catalog inclusion. If you have any questions, please feel free to contact me.

...and that's that. No justification, no explanation, no apologies--just "you wasted your time and we won't tell you why." Unprofessional.

I've since had a fairly lengthy conversation with a different person at Danger, who described his understanding for the limitation: a combination of high bandwidth and low demand. Big surprise there.

Anyway, time to move on to the next product. Mmmmm... maybe I should dust off Chess Club and resubmit it. Remember this one?

It was shot down for third-party server interaction. BUT, I've heard (from Danger themselves) that this wall is crumbling; there's already one app in the catalog that talks to third-party servers and has a recurring charge (to cover the ongoing network usage). The app itself is pretty nice, and it's possible that this time around it will get approved.

Problem is, ICC has changed their "guest" policy since I wrote that thing--you can no longer login as a guest indefinitely, which is going to be a ship-stopper for my client. I hate to say it, but if I want to ship this thing I'm going to have to port it to work with FICS (the Free Internet Chess Server) as well. That's a lot of work, but not impossible.

VNC Client

richard@randomly.com — Thu, 10 May 2007 05:07:52 GMT

From time to time, I work out of my house; I've got a suite of software packages that help me punch through the office firewall, connect to shared resources and even view and use my office computer as if I were sitting right in front of it.

That last feat is pretty fun, and is the basis of a new program I just finished for my cell phone: a VNC client. The idea is, with two clicks on my cell phone I can now connect to my desktop machine and see its display right there on the phone; I can type and the remote computer sees the keystrokes, and I can roll the trackball and watch the mouse cursor move. Basically, if you don't mind the brain-numbing latency (1-5 seconds for a full screen refresh), you can take over a remote machine very nicely.

Here's a few obligatory pics:

I've just submitted the application to the makers of my cell phone, as the first step in getting their business development and QA departments to look the application over. The quality of the application is very high--I'm pretty proud of it, actually--but I'm guessing there's only about a 50% chance that the app will get approved. There are two likely hangups: it doesn't work on older versions of the phone (requires a trackball, and old phones don't have one), and it uses a lot of network bandwidth (something carriers don't like--and if carriers don't like it, well, it's dead in the water).

On the other hand, having a good VNC client brings the Sidekick closer to feature-parity with other data-targetted PDAs out there. Who knows?

Black Belt Test

richard@randomly.com — Sat, 28 Apr 2007 22:27:51 GMT

Yet another exciting day: I tested for black belt this morning. The whole process took just under three hours--though half of it I was just standing by watching the junior members test, and helping out only by holding boards for breaking or playing attacker for self-defense tests.

The test went quite well, I think. The patterns went smoothly--didn't get called to repeat any, and several instructors complimented me on them later. Self-defense likewise was straight-forward, though we took them a little gently in deference to the exceptionally hard floors (the test was held in a local high school gym). I had only two rounds of sparring, one attacker each--drew strong opponents both times, and did much better than I'd expected. Breaking was great: nailed all my breaks on the first try, including the four-board one. I know the terminology cold, so I wasn't worried about that part.

Basically a much less painful test than I'd feared. Sa Bum Nim runs a very vigorous class, when the frequent refrain of "compared to this, testing will be easy." I don't know if it was really easy, but I certainly wasn't as tired after the 3-hour test as I am after a normal 1-hour class.

We'll find out the results of the test in two weeks, and if I passed (I'm pretty confident that I did), there's a ceremony in late May where new belts are awarded. Here's hoping I get to move on to learning some new patterns!

Update: the lists of test results were posted today, and indeed I passed. The ceremony where I'll receive my actual black belt will be held on the 19th--just a week from now.

New car!

richard@randomly.com — Fri, 27 Apr 2007 20:27:45 GMT

I recently traded in my '02 New Beetle. Kind of strange to remember how excited I was to get that car originally; over the last five years, it spent a great deal of time in the shop and accumulated thousands of dollars of repair bills. When I traded it in, it had 75,000 miles--was on its third windshield, had a driver-side window that still wouldn't roll up without yanking on it, a new clutch (replaced at 60,000), assorted weird squeaks and groans, a brake pedal that would audibly and physically pop when pushed, and so on.

So, here's hoping for better luck this time:

(large image 1, large image 2)

This tasty little thing has about twice the horsepower of my old car, weighs in at 100 pounds lighter, and has so much torque that you can bark the tires at 60mph if you slam the accelerator. Its handling is phenomenal--the comparison to steering a go-kart is well-deserved. It practically begs you to drive like an asshole.

In fact, I think I'll go for a drive now. :)

Yay Isaac

richard@randomly.com — Mon, 20 Nov 2006 13:07:47 GMT

Isaac and I attended our first TKD tournament yesterday. The crush of people--particularly at the beginning--was amazing. Since Tara was still at church when it started, for a while there I had all three boys--at least until Isaac went down to the main floor to compete.

Isaac did an amazing job. First off, he handled that huge crowd so well! The tournament was at the "main school"--a building we've been to only once before, and which is full of weird twisty passages and crammed with spectators and participants. But he paid attention, found out where he needed to be, and did his thing in the tournament without any help or instructions from us at all.

Tara arrived from church about 20 minutes before the childrens' competition began. We got a few photos of Isaac doing his pattern: he scored 9's and 9.5's, and ended up with 3rd place in his division. It was hilarious to see his jaw drop when they called his name for a trophy.

When it came time for Isaac's division to spar, Nathaniel and I wandered around the building and arrived outside a door right next to where they were fighting. Got a few video clips and some stills, but sadly his back was to us and there were occasional spectators in the way. Isaac lost his first match 2pts-3pts, then won his second 3pts-1pt. He ended up with a 3rd-place finish there as well, so he now has two trophies on the mantle. (Need to put up a shelf in his room.)

When Isaac was finished, Tara took the boys back to the house and I stuck around for the adult competition. The pre-black-belt division was too large so they split us up; I ended up in a different division from the others against whom I'd expected to compete. I took first in patterns and third in sparring--the former doesn't mean much as the others in my division had a lot of stage fright, but I struggle with sparring so I'm pretty proud of the latter.

Since Michael had a birthday Saturday and Isaac won trophies Sunday, Nathaniel got to pick where we went to dinner to celebrate. :)

More progress on Hero

richard@randomly.com — Fri, 10 Nov 2006 04:25:40 GMT

The first bit of coding for Hero is coming up! I've written yet more specs--which I won't bother you with here--and am ready to actually start producing something usable.

I'm targetting the server-side binary first, and the initial step there is to come up with a card compiler. That is, here on my machine I'm going to be producing a series of text files (and their associated images) that describe individual cards in the game. Here's a sample for what the cards will look like during the actual game play (click for larger version):

This particular card talks about a Medusa, which is just a convenient sample. (My son is reading a book about greek mythology, and he suggested it.) The relevant stats for this thing (which don't match the picture--sorry) are:

- It gets 1 AP per turn, and can have 3 maximum.
- It's slow: it can only move two squares per AP expenditure.
- It can't attack often: it has to use up 2 APs to attack.
- It can attack you up even if you're standing a square or two away.
- If it does attack you, you're screwed: you can't defend.
- Anyone hit by a Medusa gains the Rooted (can't move) and Harmless (can't attack) attributes.

Here's the full text file describing this sample card. The upcoming piece of code is the tool that reads files like these, cracks them into something machine-usable, and spits out compressed binary blobs that I'll be able to keep up on the server. [Again, the premise here is that I can buy deck A and you can buy deck B, and if you play a deck-B card against me, I need to know what it does. So I download information about that card from the server, by asking for one of these pre-prepared blobs.]

      short-name: Medusa
      long-name: Summon Medusa
      description: Unstoppable  Wise creatures avoid approaching the Medusa, whose gaze can turn any living creature to stone.

      cost: 3
      type: creature
      properties: unstoppable

      ap-init: 0
      ap-turn: 1
      ap-max: 3
      hp-init: 5
      hp-max: 5
      move: 1=2
      attack: 2=5 range 2
      defend: 1=5
      vision: 5
      hand: 0
      ap-tax: 0

      abilities: none

      effect: scope=creature, timing=attack, termination=continuous, action=grant rooted
      effect: scope=creature, timing=attack, termination=continuous, action=grant harmless

More Design Specs

richard@randomly.com — Fri, 27 Oct 2006 01:04:36 GMT

I've recently written another three or four low-level specs for some of the more easily separable pieces of the Hero game architecture. Still haven't tackled the big ones (the Game Rules Engine and its attendant data structures), but it's progress anyway.

I've also written a specification for the dedicated server process that will be running full-time on the web site to manage this. I've thrown out peer-to-peer networking along with all its complications, and am going with a full client/server approach instead. This was primarily motivated by the complications of trying to play a correspondence game when there's no guarantee that you and your opponent will ever both be online simultaneously, but that's kind of a cop out: truth is client/server is simply easier to deal with overall.

Anyway, for those who have incurable insomnia, here's the better part of the web server design.

Hero Web Server

Hero requires a dedicated process listening at a well-known address that can provide several features. This binary will run on the web server and will speak a simple binary protocol. Games that don’t involve network play don’t require interacting with the server at all, and so none of this stuff really applies in that scenario.

It’s important that the player is not forced to involve the internet in his enjoyment of the game: the single-player game (“straight out of the box”) should be adequately complete to provide a rich experience just playing against the computer. But it is probable that the most rewarding way to play the game will be against other humans—including most likely ones you don’t know personally.

All network communication in Hero is done from client applications to this central server process; attempts at peer-to-peer networking were jettisoned from the design because they cause great headaches for correspondence games.

Client Versions

A client version is composed of an array of integers:

• The major version number—odd numbers represent a mobile device and even numbers represent a desktop device, and higher numbers represent more recent releases.

• An array of integers. Each integer has a low word indicating a deck number and a high word indicating a patch level. Patch level 0 indicates this particular deck has not been purchased or is otherwise not installed; patch level 1 indicates the client has only the shipped-with-a-client collection of cards, and higher patch levels indicate that the server has provided updates to this card deck (in the form of modified or additional cards).

Connections

A client application will maintain a connection to the Hero server for as long as the client knows of at least one networked game in which the local player is involved—either as an active player, an observer, a challenger or a recently accepted player for a game that hasn’t yet started.

Upon connecting, the client will typically transmit a login request; the server will respond with a brief summary of what games the player is involved in and potentially updates to the local card deck. Thereafter, so long as the client remains connected, either the client or server may initiate messages to indicate changes in game-state.

The client will typically disconnect from the server within N minutes of ending the last networked game in which the local player participated.

A client may also attempt to login periodically—once at startup, then again every hour—even if the client knows of no ongoing network games involving this player. Connecting like this at startup protects against the local client having forgotten his actual list of games; connecting periodically thereafter enables the server to deliver messages.

Player Accounts

To play online requires that you have a Player Account. An account consists, essentially, of an alias and password; optionally a player may choose to associate an email address with the account (to which lost-password reminders can be sent). Player accounts are managed by the Server process, which maintains a vault for them and forces players to choose unique aliases.

A vault record for a player contains the following fields:

• String playerName
• String password
• String emailAddress
• Message[] messages
• GameID[] games

The games array includes all games that mention this player in any capacity—each may be an in-progress game, a game that this player has proposed and is still awaiting opponents, a game for which this player has been challenged but not yet accepted or declined, or a game for which this player has accepted a challenge but other players are still needed.

If a game in the array has a state indicating the game is over (e.g., a challenge was declined, the game request was withdrawn or someone has won), then the game ID will be removed from the list after being reported once during login. Likewise, a game ID is removed from a player’s list if the server is able to immediately report the fact of that game’s termination to the already logged-in user.

The messages array includes any messages waiting for this player that have not been delivered. These queued messages are delivered as soon as the player logs in; if new messages arrive while the player is still logged in, they’re delivered immediately. The player’s client software displays game-agnostic messages immediately; game-specific messages are delivered when the player next switches to that game.

Game Timing

There are two different ways to play this game:

• Live – No one really looks away from the display, but instead actively watches the whole time to watch what’s happening. No one wants to wait long for his opponents to move, so there’s a per-move time limit of probably 1-5 minutes. A player who is idle too long on his turn forfeits that turn, and three forfeits in a row cost him the game.

• Correspondence – You just don’t have time to play the whole game out at once, but you still want to play against other humans. So you find someone else with the same problem, and start a correspondence game: make a move when it’s your turn, then go away and do something else for a few hours or even the rest of the day. You’re expected to make some progress, though, so there’s a time limit of probably 24-72 hours. (A time-limit on the order of 1-2 hours becomes problematic: you can’t finish the game before it’s time to sleep, so whoever goes to sleep first loses the game. Even 12 hours could be a problem; if you stop checking your game at dinner time, you might have lost by breakfast.) In this scenario, a single turn-forfeit costs the whole game; it’s unfair to expect someone to time out for a week before giving the game up.

What these modes have in common is the idea of a per-move time limit, coupled with a forfeit counter. The game client will expose four basic settings for timing a game:

• Blitz – Each player gets two minutes per move; three timeouts forfeit the game
• Standard – Each player gets five minutes and up to two timeouts before a loss
• Correspondence – Players get 24 hours per turn, but lose after one timeout
• Extended – Good for the long game: you get 72 hours per turn and one timeout

When pairing players, all opponents must have requested the same time control in order to get paired together.

Starting a Game

When a player wants to start a game, the player specifies all of the following:

• For each of the N possible player colors, whether this player is:
o a local player (one of the slots must be this; more than one can be); or
o a computer player (selected from the local array); or
o a specific remote player (specified by name); or
o any remote player who chooses to participate; or
o no one

One chooses a local player by specifying which locally configured deck should be played; each deck has its own name, which in turn specifies that player’s Hero’s name during the game. Similarly, one chooses a computer player by selecting from among the automatically provided list, where each named computer player has an associated deck (or choice of decks) and playing strategy.

• The size of playing board: small, medium, large or huge

• The timing for the game: blitz (2min), standard (5min), correspondence (24hr) or extended (72hr)

Once these parameters are selected, the client considers whether there are any remote opponents listed; if not, the game starts and is played locally and involves no interaction with the server at all.

If the game description does include remote players, it gets sent to the server process. The server first considers whether there are any specific remote players named; if so, the server creates an ID for this new game and issues challenges to the named players. The game remains in this state until all challenges have been accepted; if any are declined, the game is cancelled. Once all challenges have been accepted, the server opens the game for other players to join in any still-open spots (see below) or commences the game immediately.

If the game contained remote players but none were specifically named, then the server first looks through all existing open games—that is, games with unassigned players—to see if this player’s challenge is consistent with them; if so, this player is added as a participant in the existing open game, filling a spot. Otherwise, the server assigns an ID for this new game, and returns that ID to the caller. The client displays something about how the server is looking for matching opponents, and the player is free to try starting more games or resume playing existing games.

An outstanding game request is automatically withdrawn if the player making the request disconnects before the game can be started. Such a withdrawal deletes the game itself and notifies all already-accepted players. (The withdrawing player does not get, or need, such a notice.)

Per-Game Data

The server maintains a lot of information about each ongoing networked game; this data is retained for N days after the game is concluded, then destroyed. (This delay allows the losing players a chance to discover what happened on the last turn, even during a correspondence game.)

Every networked game is assigned its own separate vault, keyed by a GameID—a unique integer assigned by the server. The records in the vault fall into three categories:

• GameDescription
Summarizes who attempted to start the game, and what parameters were specified. This record also includes an up-to-date summary of who is going to play: as players accept the challenge or are auto-joined as they attempt to start games, the game description is filled in. At first, this is the only record in the vault.

• InitialBoard
Constructed once the game begins and the first player’s client has generated a game board. The record contains a GameState that summarizes the entire state of the game before the first turn begins.

• PlayerMove
Appended as players make moves during their turns. A player’s turn consists of one or more PlayerMove records; a special PlayerMove record is used to indicate the player has chosen to end his turn, and it is now the next player’s turn. Game progress is measured by the number of moves that have elapsed since the beginning of the game.

Locking

The server is multi-threaded, and is expected to manage anywhere from dozens to thousands of simultaneous connections. Most of these connections will be idle at any given moment, but some race conditions are possible:

• More than one player could attempt to start a game at the same time; here we have to guarantee uniqueness for the newly created game IDs, as well as ensuring that if two players’ game-creation attempts are compatible we can collapse them into a single mutual game.

• One player may connect and request status at the same moment that another player in the same game is submitting a move.

• The housekeeping process on the server could decide that a player has been idle for too long and has forfeited the game, at the same time that that player attempts to submit a move.

…and so on. Race conditions of this sort require that the server provide a sufficiently robust locking model to guarantee serialized access to shared data structures; the model must be sufficiently fine-grained to allow multiple games to operate independently, and a strict lock hierarchy must be established to prevent deadlocks.

1. There is one vault used for all player records; this vault is guarded by a single reader/writer lock, and sits at the top of the locking hierarchy. Holding a reader lock is sufficient to allow any record in the database to be examined safely.

2. There is a global mutex that must be held when the list of games is potentially changing. The housekeeping process additionally holds this mutex while it iterates across all games in the database. Note that this mutex does not guard access to any particular game vault—just the collection thereof.

3. There is one vault per game, and each such vault is also guarded by a single reader/writer lock. There is no defined order for manipulating multiple game objects simultaneously, because such an operation is disallowed; transactions must release one game before attempting to lock another.

RPCs

Messages prefixed with CS_ are sent from client to server; messages with SC_ are sent from server to client. All messages are asynchronous and therefore involve only input parameters—fire-and-forget is the basic RPC policy here.

• CS_REGISTER
String playerName
String password
String emailAddress

Requests the creation of a new account. The emailAddress and password fields can be empty. The server eventually sends a SC_REGISTER_ACK message in response.

• SC_REGISTER_ACK
Bool success
if (!success)
String errorText

Indicates that the specified player account was accepted and that the client should attempt to login (by sending CS_LOGIN next). If the response indicates failure, an English description of the error (probably an illegal player name) is returned.

• CS_SEND_REMINDER
String nameOrEmail

Requests that the server look up the specified player account, either by player name or email address (whichever matches first), and send the account password to the registered email account for that player. The server eventually sends a SC_REMINDER_ACK message in response.

• SC_REMINDER_ACK
Bool success
if (!success)
String errorText
else
String emailAddress

Returns either a report of what email address has been provided with a reminder mail, or an English description of what condition (probably a missing player name) prevented the reminder from being sent.

• CS_LOGIN
String playerName
String password
ClientVersion clientVersionData

When the client first connects to the server, this is probably the first message it sends (it’s either going to be this, CS_REGISTER or CS_SEND_REMINDER). The client includes not only the login and password, but also an indication of what version of client this is and what patch level has been attained. The server eventually sends a SC_LOGIN_ACK message in response; on success, that response may include the data necessary to update the client to a newer patch level.

• SC_LOGIN_ACK
Bool success
if (!success) {
String errorText
} else {
ClientVersion patches
CardUpdates[] updates
GameSummary[] games
Message[] messages
}

Brings a client more or less up-to-date with what’s happening on the server. The newPatchLevel field reflects what the client should report as its patch level next time, assuming that the updates field is applied to the local card store. The updates sent by the server represent card decks owned by the client but not adequately patched yet, as reported by the clientVersionData field during login. The messages field contains delayed messages (see the Player Accounts section), and each entry in the games array summaries simply how many moves have been made, whose turn it is, and what each player’s state is (challenged, accepted, playing, forfeited, etc). If the client’s stored state doesn’t match any of this, the client will likely immediately ask for more information.

• CS_REQUEST_STATE
int GameID
bool needPlayerData
bool needInitialBoard
int knownMoves
int csumCurrentBoard

At any time, a client can use this message to find out the current state of a game; in practice, the client will likely issue these shortly after getting an SC_LOGIN_ACK message and will likely not need to use the call thereafter. The client provides a short summary of what it knows about the game. If knownMoves==0, or if the csumCurrentBoard does not match the server-stored checksum of the game state after knownMoves moves, then the server sends the full history of the game (including its initial board if requested or after a checksum mismatch); otherwise, the server sends only an incremental description containing moves since the client’s last known position.

• SC_GAME_UPDATE
int GameID
bool playerDataSent
if (playerDataSent)
PlayerData players[]

bool initialBoardSent
if (initialBoardSent)
InitialGameState initialState

int firstMoveSent
int movesSent
if (movesSent > 0)
MoveDetails[] moves

Contains the data necessary for a client to update its local cache of the game state to the most current level.

• CS_SEND_MESSAGE
String playerTo
int gameID
String text

Asks the server to deliver a player-to-player message. The method of delivery depends on the first two parameters:
o if (playerTo != “” && gameID != 0) then the message is delivered to one player, in the context of the specified game—that is, the recipient will get the message when the game next becomes active.
o if (playerTo != “” && gameID == 0) then the message is delivered to one player, immediately.
o if (playerTo == “” && gameID != 0) then the message is delivered to all of this player’s opponents in the context of the specified game.

• SC_MESSAGE
Message message {
String playerFrom
int gameID
bool isBroadcast
String text
}

Describes one message being delivered to this client. The client is responsible for choosing whether to display the message immediately or to wait until gameID next becomes the active game. Messages delivered in this fashion become the client application’s responsibility to display; if the client chooses not to deliver them immediately, then it should store them persistently until later since the server will not re-send the message upon next login.

• CS_SEND_MOVE
int gameID
MoveDetails move

Describes one action made by a local human or computer player in the specified game. The move may indicate that the player has chosen to end his turn; that’s a move like any other. The server is responsible for updating its canonical game state and forwarding the move to all other online players in the game via immediate SC_GAME_UPDATE messages.

• CS_START_GAME
TimeControl timeControl
BoardSize boardSize
PlayerRequest[] players

Allows the client to request a new game to begin. The client will eventually get an unprovoked SC_GAME_UPDATE message, telling the client of a new game that’s ongoing that the player should be observing.

• CS_CHANGE_PLAY
int gameID
enum {
ACCEPT_CHALLENGE,
DECLINE_CHALLENGE,
RESIGN_GAME
}

Allows a challenged player to accept or decline a challenge; also allows a player to withdraw from a game at any time. (If the game has not yet started, the player is simply removed from the list of players; if there are no human players left, the game is immediately deleted.)

TODO: any APIs for database administration?

Server-Side Operational Flow

The server’s operation begins when the housekeeping timer goes off or when a particular request is received over the network. Each such starting point is detailed below along with a rough outline of its operation thereafter.

Client Connected

The server must be able to maintain hundreds to thousands of simultaneous connections; therefore, having dedicated inbound and outbound threads for each connected socket is inappropriate. Instead, the server maintains one input and one output thread for each group of 64 connections, using a banshee netstreamgroup. (At least, it will as soon as I’ve written it.) The process of simply connecting does nothing more than bookkeeping for the netstreamgroup.

Each connection is associated with a player identity—the connection is initially anonymous, and the player identity is assigned after a successful CS_LOGIN attempt. These identities are also reverse-mapped: given an identity, one can determine the connection that player is using. When player A logs in, if that identity is already connected elsewhere that other connection is killed.

Client Disconnected

When a logged-in client disconnects, he is immediately withdrawn from any not-yet-started games in which he was otherwise a participant; this process may cause some new games to be deleted entirely if there are no human players remaining.

Other than this action, disconnecting has no other effects on the server.

Housekeeping Timer

Once every hour, the housekeeping timer kicks off a quick process to clean up the server’s databases. This process performs the following:

1. Obtain a write lock on the player database
2. Obtain the gamelist mutex
3. Iterate through all game vaults; for each:
• Obtain a write lock on the game’s vault
• If (the game has not changed within a month), or
if (the game has not changed within a week and it is not in progress) then:
o For each listed players, remove this game from the player.games array
o Delete the vault
• Drop the write lock on the game’s vault

In the future, the housekeeping timer will probably serve more functions, but this is a good one to start with.

CS_REGISTER Message Received

1. Validate incoming parameters or nack
2. Obtain write lock on the player database
3. Look up the supplied player name; if exists, drop lock and nack
4. Create new player account, drop lock, ack

CS_SEND_REMINDER Message Received

1. Obtain read lock on the player database
2. Look up the supplied player name / email address; if failure, drop lock and nack
3. Send a pretty email; drop lock, ack

CS_LOGIN Message Received

1. Validate incoming parameters or nack
2. Obtain write lock on the player database (going to remove messages etc soon)
3. Look up the supplied player name and check password; drop lock and nack on failure
4. Prepare a response message for the client:
• Move messages from player record to outbound list
• Check version info and prepare summary of patches
• For each game, read-lock game vault and find game summary then drop lock
5. Drop player database lock and ack

CS_REQUEST_STATE Message Received

1. Validate incoming parameters or nack
2. Obtain read lock on game vault
3. Summarize player list for outbound packet
4. Summarize initial game board for outbound packet
5. Summarize move list for outbound packet
6. Drop vault lock and ack

CS_SEND_MESSAGE Message Received

1. Obtain write lock on the player database (might be adding messages etc soon)
2. If player name specified, look up player record; drop lock and nack on failure
3. If game id specified, obtain read lock on game vault; drop lock and nack on failure
4. If player name specified, sendMessage to specified player; else
Iterate through players in game vault and sendMessage to each player
5. Drop locks

CS_SEND_MOVE Message Received

1. Obtain write lock on the game vault or nack
2. Update game state in vault to reflect this move
3. For each player listed in game vault header:
…if player connected, and player isn’t this caller, send SC_GAME_UPDATE
4. Drop write lock

CS_START_GAME Message Received

(flow chart omitted from journal because I'm lazy)

CS_CHANGE_PLAY Message Received

1. Obtain write lock on player database
2. Obtain write lock on the game vault or nack
3. Update player state in game and player databases
4. Send SC_GAME_UPDATE to all players
5. Drop locks

Omega Featured

richard@randomly.com — Sat, 14 Oct 2006 14:21:16 GMT

Omega is now the featured app on hiptop.com's main site: http://www.hiptop.com/#catalog

They made up the text here, cleverly taking the relevant game description from my own summary and inventing a midly amusing little story around it. :)


	Dear Child, You may as well take the last Omega tank out for a ride like you've been begging for all week since you got your license. Notice how the armor and shields are self-repairing. This will be useful since this tank is almost the only weaponry in the global military industrial complex not destroyed by the invading aliens. You might find some weapons and defenses lying about. You should pick these up because no one likes a cluttered battlefield. Also, you never know which ones will be most effective when dealing with the dozen different kinds of aliens out to kill you. No shooting laser cannons, launching missles or lobbing nuclear shells while in the garage please. And, oh yeah, if you wouldn't mind, please wipe out the aliens who have taken over the earth. Sincerely, Parental Unit P.S. The Omega has a full tank of gas and I expect it to be filled up when you bring it back. And no bleeding on the upholstery. P.P.S. Omega 3D is compatible with hiptop2/Sidekick II/hiptop3/Sidekick 3 devices.

Final Test

richard@randomly.com — Sat, 07 Oct 2006 19:04:43 GMT

Much thanks to my wife, who kindly stayed home with all three kids so that I could take part in today's Tae Kwon Do test. This was a comparably easy test: no patterns or breaking; just self-defense, sparring and terminology.

This was the final pre-test before black, so I don't have to test again until March '07 when I'll be trying for 1st-degree black. :)

Development Slowing Down

richard@randomly.com — Thu, 05 Oct 2006 00:57:53 GMT

I've got about 5 projects that I want to work on, but it's probably going to be a while before I make any progress on them: babies take up a lot of what used to be free time. But they're awfully cute...

richard@randomly.com — Thu, 21 Sep 2006 14:33:55 GMT

This morning, Omega finally made it to the "catalog"--the online list of games available for sale on my cell phone:

The game is available through all the major Hiptop resellers: T-Mobile, SunCom and Fido. :)

3 Boys

richard@randomly.com — Fri, 15 Sep 2006 15:07:16 GMT

We're heading out in about half an hour to finally meet our newly adopted son Michael, who has just arrived in San Francisco and will be arriving in Philadelphia late tonight. Baby pictures will follow as soon as they're available. :)

-- Update --

Off Topic (Sorry!)

richard@randomly.com — Sat, 19 Aug 2006 03:40:13 GMT

My wife is sick of hearing about these, but I can't stop laughing. I stumbled across this site earlier, and have selected some of my favorites for your reading enjoyment:

The Great Wall of China was originally created to keep Chuck Norris out. It failed miserably.
Most people have 23 pairs of chromosomes. Chuck Norris has 72... and they're all poisonous.
Chuck Norris can win a game of Connect Four in only three moves.
Chuck Norris once shot down a German fighter plane with his finger, by yelling, "Bang!"
Chuck Norris once bet NASA he could survive re-entry without a spacesuit. On July 19th, 1999, a naked Chuck Norris re-entered the earth's atmosphere, streaking over 14 states and reaching a temperature of 3000 degrees. An embarrassed NASA publically claimed it was a meteor, and still owes him a beer.
Chuck Norris can hit you so hard that he can actually alter your DNA. Decades from now your descendants will occasionally clutch their heads and yell "What the hell was that?"
Time waits for no man. Unless that man is Chuck Norris.
When Chuck Norris falls in water, Chuck Norris doesn't get wet. Water gets Chuck Norris.
Chuck Norris can believe it's not butter.
Chuck Norris can divide by zero.
While urinating, Chuck Norris is easily capable of welding titanium.
Chuck Norris knows the EXACT location of Carmen SanDiego.
Chuck Norris ordered a Big Mac at Burger King, and got one.
It takes Chuck Norris 20 minutes to watch 60 Minutes.
Chuck Norris doesn't believe in Germany.
Chuck Norris played Russian Roulette with a fully loaded gun and won.
Chuck Norris can sneeze with his eyes open.
Chuck Norris let the dogs out.
Chuck Norris once bench-pressed the entire state of Ohio.
Chuck Norris can cook minute rice in 30 seconds.
When Chuck Norris cuts an onion, the onion cries.
Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.
Chuck Norris qualified with a top speed of 324 mph at the Daytona 500 without a car.
Chuck Norris throws out the gum and chews the tin foil.
Chuck Norris does not style his hair; it lays perfectly in place out of sheer terror.

Major Components

richard@randomly.com — Sat, 22 Jul 2006 22:56:07 GMT

I've finished a rough breakdown of how the game will be put together. This is really the last step in the high-level design phase; from here the designs get much more detailed and cryptic.

This is the infamous Block Diagram that kicks off most software projects:

Each of these modules gets summarized below, though each will require a detailed low-level specification.

Application Resources

Contains a selection of pre-designed playing boards and the descriptions of all the possible cards for this application’s included ‘decks’. This content will be designed separately, compressed and attached to the application either as an external banshee database or as java app resources.

Card Database

Responsible for informing the Game Rules Engine all about a particular modifier card. The GRE doesn’t read the application resources directly, because the card may not exist there.

One of the goals of this game is to be able to interoperate with other versions of the application. Picture the Danger Catalog with three different games in it, each separately purchased for $4.99 (maybe with a discount if you’ve already bought one of them). You buy game A (with a certain list of cards encoded in it), and your friend buys game B (with a different list of cards). Yet you want to be able to play against each other.

When asked to resolve the meaning of a card, the database first consults its cache of card information; if that turns up the data it needs, then its job is done.

If the card isn’t in-cache, the database consults the application resources to see if it has some data about the card locally. If so, it decodes the resource and shoves the result into its cache, then returns the result.

If that fails, then the card must’ve been introduced by a peer. The database turns to the network to ask about the card, and updates its cache when it gets a response. (When the GRE asks for data about a card, it also specifies which peer probably triggered that query—usually the player whose turn just ended.)

The card database is also used by the local display, which needs to describe cards to the local player; in this capacity it caches text and icons. These also are served from cache.

Finally, the current game state also interacts with the card database, in order to advise it that particular cards are currently active and should not be thrown out of the cache.

Persistent Card Storage

When the card database receives data about a new card from a peer, it records that data persistently as well as in its local cache.

This persistent storage is required when loading a previously networked game—and is particularly essential on the Hiptop, where even switching away from the game requires that all in-memory caches be discarded to save memory.

Network Connections

The networking model here is driven by what the Danger Hiptop supports—that is, peer-to-peer networking is preferred over client/server architectures.

Typically the rules engine receives updates over the network from other players who have just completed a turn. There are many complications here in the realms of picking players and setting up a game, handling disconnections and so on.

It is anticipated that the desktop game—which has lower-latency networking—will probably be primarily a real-time turn-based game. The cell phone game, however, is likely going to be used often as a correspondence game—make your move, then close your phone and go do something else, expecting to get notified when it’s your turn again. The game must support both models.

The actual network protocol should be generic enough to support cell phone users playing against desktop users. Certainly the other content—cards, rules, game board dimensions, etc—will be compatible.

Game Rules Engine

The most complex part of the game: this is the logic engine that takes a description of a player’s turn and applies it to a Current Game State in order to determine the result. The GRE is responsible for accumulating actions performed by the user locally and eventually transmitting them to all peers when this player’s turn is completed.

Note that the GRE does not typically transmit the entire resulting game state—just the actions performed by the local user. The assumption is that the remote player’s receiving GRE will enact those same actions and end up with an identical game state. To verify this, the local GRE will checksum its initial and final game states and transmit those checksums as well; if the receiving player’s GRE ends up with a different checksum (or if it fails to perform the turn at all, perhaps because it lost the local save game), then the GRE will indeed transmit the entire game state to the recipient.

Current Game State

A static structure that encompasses the entire present state of a game, including board terrains, player creatures, active modifiers and so on.

The Game State can be initialized by the GRE for a new game or can be loaded from a serialized state; thereafter it is modified by the GRE as turns proceed. The Game State informs the local display as it changes, and makes different actions available to the local user.

Local User Input

Manages the input devices for the local player. These are quite different between the desktop and cell phone variants, and so each flavor requires a separate specification.

Serialization

Encodes a game state as a relatively compact chunk of data, which can be saved persistently for later. The serialized game state does not encode the cache of the card database; the database is responsible for maintaining any persistent storage that it needs.

Local Display

Works very closely with the local user input module, presenting the game on the local monitor or cell phone. As with the input module, the display’s form factor is clearly radically different between the cell phone and desktop device, and these will therefore require separate specifications.

I've also got short summary of how the game sequencing should work. This is less fancy stuff--but it's something that I typically slap on at the last minute and therefore botch horribly. "Sequencing" means, at least in this sense, all the messy menuing stuff that lets the user start a game, pause a game, load and save games, switch away, advance levels and generally just do everything else in the application except simply play the game. Because Hero has a lot of potential to be played as a correspondence-style game, I wanted to make sure I got this part right.

Game Sequencing

The application maintains a persistent store of ongoing games; any one of which can be active at once. The active game is the one that is currently being displayed. This active game, if there is one, can be either paused or running.

The application has a main menu which—among other things—lists all ongoing games. For each such ongoing game, the application indicates whether it is the local player’s turn or whether the game is still waiting for a remote opponent to move.

When the game is first run, there is no active game and the user is presented with this main menu. Thereafter, the user can return to that main menu at any time—doing so again means there is no longer an active game. The game that was previously active will appear in the list of games as one among several, with the current selection focus on that game—such that triggering it will return the user to play.

While playing a game, it’s possible that the user will switch away to another process. On the cell phone this is a particularly common activity, but even on the desktop the action must be streamlined to account for a play-by-correspondence feel. Switching away from the application does not clear the active game; it simply switches the active game from the running state to the paused state.

As long as the application is running, it is possible that it will receive input from a peer indicating that the state of the game has changed. If this change affects the active game, and if the active game is running rather than paused, then the change is made immediately to the active game and the turn is played out visually on the local display. In all other cases, the new turn data is cashiered away along with the target game—so that when the user next resumes the updated game (or un-pauses the active game), the opponent’s turn can be visually seen.

While switched away, the desktop application uses a bubble notice near the taskbar to indicate that a game state has changed; the cell phone application uses a notification over the clock area for the same purpose.

The splash screen for the game indicates a brief summary of whether there are any games in progress for which it is the local player’s turn.

Lost Connections

Cell phone connections are particularly vulnerable to being dropped, and desktop connections will be dropped if either player closes the game. The active game is not impacted by these lost connections; instead, if it is the local player’s turn he can continue without even being notified of the problem.

When the local player completes his turn, the application attempts to send the summary of his actions to all peers. If any peer fails to acknowledge that the turn was received within a reasonable period of time, the local application attempts to re-send it—and will continue trying, until it either succeeds or receives a game update from another peer.

Remember that more than two players may be involved: if player A completes his turn, that data will be transmitted to B and C. Assuming B receives the data and it is B’s turn to play, that player will probably make his own turn and proceed to transmit that update to A and C—even if C never got A’s transmission. In this case, A will receive B’s transmission and update his own game state, discarding the data that it failed to send to C.

This process is safe because C, when it finally discovers that it is out of date, will simply get a transmission of the full game-state rather than an incremental update—and the game continues normally.

So long as the application is running, it periodically (~N seconds) looks through all the ongoing games—even the active one—searching for games in which it is not the local player’s turn. For those games, the application makes a quick call to all its peers asking for the current game’s status; the peers reply simply with a checksum of the current game state. In this way, if one peer temporarily loses its connection and misses some content, it will catch up quickly once it reattaches to its peers.