Mobs and the Things They Do

It's been a while since my last post, which is kind of the default state of things on most blogs, so I don't know why I even bother saying it. I've been doing some other blogging on Destructoid lately, which is pretty fun. But enough of that. I've also been working on a few changes related to mobs and how they act. These changes are the baby steps at the beginning of implementing combat, which looks like it will be the first major "game" feature I do.

Looking forward to combat, I can see it as being a kind of special case of a "mobprog," that is, a mob behavior that responds to some stimulus. You hit me, I hit you back, I dodge, cast a spell, etc.. This means that mobs will be performing series of actions in an automated way.

Currently, there are a few ways to make a mob to do something. If you can type things, you can issue commands to a mob via the usual interpreter -> interaction mode -> mob path. Or you can order a mob to do something, again using roughly the same process. Finally, you can directly invoke methods on the mob object, for instance, doLook or doLook_Direct.

The last of these ways in particular will probably be used for implementing mobprogs, so I took a long, hard look at it to see if it does everything it should. Hint: it does not, which is why I'm writing this blog right now.

It turns out the way things were implemented, direct calls on a mob sidestep a lot of the necessary mechanics that bring it to life. In particular, it circumvents delays. Delays were inserted by the interaction mode when a user types in a command. They ensure that you can't make your mob do more actions more quickly than it "should be able to."

If you think about it, the right place for delays to be inserted and handled is in the mob itself, not in the interaction mode, which is just part of the interface between the player and the mob. The amount of delay from any particular action has nothing to do with the player and everything to do with the mob. So I moved it.

Let me just interject here that the next several paragraphs are an outline of code that I rejected after realizing what a bad idea it was. If you want, you can jump straight to my epiphany, correction, and what I did implement.

Now each mob has its own invocation queue, like interaction modes do. When a user types a command, if it's an action to be carried out by the mob (an ICCommand / ICInvocation), the interaction mode just queues it up to the mob; if it's an out of character command, the interaction mode takes care of executing it, since it has nothing to do with the mob itself.

Some restrictions apply

As I've described so far, all is not well. Where there used to be a single queue of invocations, now there are two, and moreover, there are multiple ways for commands to get into the two queues:

• A user can type an IC command, which is generally redirected from the interaction mode straight to the mob's queue.
• A user can type an OOC command, which stays in the interaction mode and is not seen by the character or the character's queue at all.
• Another mob can send an order to this mob, which is routed from the mob to the interaction mode and then back to the mob.
• Something programmatic like a mobprog can directly invoke something on the mob, which is never seen by the interaction mode but goes straight into the mob's queue.

And with all these different routes there are a few invariants we want always to hold:

• Commands should be executed "in order," i.e., commands shouldn't be able to "cut in line."
• Commands from any one source shouldn't be able to starve other sources.
• Commands should neither short-circuit their required delay nor incur excessive delays.

Let's see how the code enforces each of these conditions.

Maintaining ordering

Firstly, that commands should execute in order. Say that the following commands arrive, and are assigned between these two queues (the mob's queue and the intraction mode queue).

1. Mob queue: IC Command #1
2. Interaction mode queue: OOC Command #2
3. Mob queue: IC Command #3
4. Mob queue: IC Command #4
5. Interaction mode queue: OOC Command #5
6. Interaction mode queue: OOC Command #6

After they arrive, they'd be arranged in the queues in this order:

Mob queue	Interaction mode queue
IC Command #1	OOC Command #2
IC Command #3	OOC Command #5
IC Command #4	OOC Command #6

But of course, if we want them to execute in the right order, they'd need to be staggered a bit, like this:

Mob queue	Interaction mode queue
IC Command #1
	OOC Command #2
IC Command #3
IC Command #4
	OOC Command #5
	OOC Command #6

In order to enforce this ordering, we have to control when commands get executed. One way this is done is by having the interaction mode queue give precedence to the mob's queue. See this code:

# in MainInteractionMode
elsif mobIsIdle?()
    if invocation = @invoQueue.dequeue()
        # ...

As I described, the interaction mode doesn't permit new commands to be executed if the mob is busy. Of course, this isn't enough. Given our incoming commands above, this would produce the following (incorrect) ordering, because the mob is busy until it's queue is drained, and only then can the interaction mode do the OOC commands.

Mob queue	Interaction mode queue
IC Command #1
IC Command #3
IC Command #4
	OOC Command #2
	OOC Command #5
	OOC Command #6

In order to correct this, we need to have a mechanism for the mob to wait until the interaction mode is done, too. Recall that all commands except direct method calls on mobs pass through the interaction mode in some way. Usually, for IC commands, they pass straight through to the mob with little examination.

# in MainInteractionMode
if invocation.kind_of?(ICInvocation)
    if isIdle?()
        addInvocationForMob(invocation, true)
    else
        addInvocation(DeferredMobIntermodeInvocation.new(invocation))
    end
else

This code says that if the mob is currently idle, we pass commands straight through to the mob, but if it's busy, we actually put a placeholder command in the interaction mode queue, not the mob queue that, when it is executed, will then send the command to the mob. So the interaction mode controls the overall ordering here. Given this technique, the commands would be arranged this way in the queues:

Mob queue	Interaction mode queue
IC Command #1
	OOC Command #2
	Deferred(IC Command #3)
	Deferred(IC Command #4)
	OOC Command #5
	OOC Command #6

The first IC command goes to the mob because it's idle at that time. The rest are managed by the interaction mode queue.

...

What am I doing?

Hang on. Hang on a second. What the hell am I doing, anyway? I'm trying to convince you it was a good idea to take a single queue and split it up into three separate queues (I hadn't even gotten to describing the third one yet), then add in a bunch of extra (complicated, fragile) logic to enforce orderings that make them simulate a single queue. How on earth did I think this was a good idea?

Blogging to the rescue again. As I wrote this, I realized that I hadn't covered some particular interleaving, so I went to add in the code to fix it up. In the course of doing so, I found that one way to solve it was to make the mob add commands back into the interaction mode queue in certain cases. And then I realized that if I did this, I might as well go back to a single queue. So I did. I yanked out basically 1-2 weeks' worth of work splitting the queues as I consolidated them again.

But all is not lost; this post was about a couple things, only the first of which was the splitting of the queues. After that I wanted to talk about new capabilities for calling methods on mobs.

Changes to calling methods on mobs

Early in this post I mentioned that functions executed directly on a mob basically bypass any kind of queuing, delays, etc.. I've now fixed at least some of those problems.

Firstly, it's no longer permitted to call an "action" method directly on a mob, like this:

mob.doLookAt(otherGuy) # not cool

The proper way to do it is (slightly clunky, unfortunately):

mob.doAction(:doLookAt, otherGuy) # A-OK

There's one solid reason for changing this: to funnel all "actions" through a common code path where I can do various types of checking. Mobchecks, verifying delays, and deferring are all things that are done in doAction. Let's take a look at them.

Mobchecks

Mobchecks are a simple way of checking for validity of various conditions when doing a command. It turns out this is a pretty common thing to need to do. Check out this typical function in which I've bolded all of the error handling code that spits out a message when the target is missing or something.

def doWhisper(targetString, whisperedText)
    textError = nil

    if targetString
        if whisperedText
            bagSearchSpace = self.room.bagMobsInRoom()
            doWhisper_Direct(bagSearchSpace.searchByTargetString(targetString, self), whisperedText)
        else
            textError = 'Whisper what to him?'
        end
    else
        textError = 'Whisper to whom?'
    end

    if textError
        seeText(textError)
    end
end # function doWhisper

def doWhisper_Direct(target, whisperedText)
    if target
        evt = WhisperEvento.new(self, target, whisperedText)
        sendEvento(self.room, evt)
    else
        seeText('You don\'t see him here.')
    end
end # function doWhisper_Direct

This type of code has to go into every single function, and moreover, a lot of it is kind of repetitive--checking for valid arguments, targets, etc.. It would be nice to make these kinds of checks easier to perform, since there will be many of them...

def doWhisper(targetString, whisperedText)
    mcheck_arg(targetString, 'Whisper to whom?')
    mcheck_arg(whisperedText, 'Whisper what to them?')

    bagSearchSpace = self.room.bagMobsInRoom()
    doWhisper_Direct(bagSearchSpace.searchByTargetString(targetString, self), whisperedText)
end # function doWhisper

def doWhisper_Direct(target, whisperedText)
    mcheck_targetPerson(target)

    evt = WhisperEvento.new(self, target, whisperedText)
    sendEvento(self.room, evt)
end # function doWhisper_Direct

These "mobchecks" are just a convenient syntax for checking those same conditions. They're pretty simple, and use the throw/catch syntax in Ruby (not to be confused with raise/rescue) to early-out from the function at the point a check fails and using a supplied message.

Here's the machinery that makes it work:

module MobChecks
    # ...

    def mc_errOut(msg)
        throw(MC_ErrOut, msg)
    end # function mc_errOut

    def mcheck_arg(target, msg)
        if target == nil || target == ''
            mc_errOut(msg)
        end
    end # function mcheck_arg

    # ...
end

class Mob
    # ...

    def doAction_Real(symAction, *args)
        if (text = catch (MobChecks::MC_ErrOut) {
                      DebugOutput.debugOut(LComment) { "#{self} doing action: [#{symAction}, #{args.map() { |x| x.to_s() } }]" }

                      send(symAction, *args)
                  }).kind_of?(String)

            seeText(text)
        end
    end # function doAction_Real

    # ...
end

We simply catch the string associated with the error if any, and print out the error message.

Adding Delays

Delays for actions are important. It's how we prevent someone from doing too many things too quickly. It used to be that the delay for an action was embedded in the ICCommand object, but now it's take care of within the mob itself. This is better, because it allows actions regardless of their origins to have delays, and actions that have no associated command at all to have delays.

def doSlay(target)
    mcheck_amPhysical()
    mcheck_targetPerson(target)
    mcheck_personNearby(target)

    mcheck_physicalPresence(target)

    addWaitPulses(10)

    sendEvento(self.room, SlayEvento.new(self, target))
    target.die()
end # function doSlay

Clearly, slaying is something that should give one pause, so that's exactly what we do--10 pulses worth, in fact. For the time being there's also code in doAction_Real that checks to make sure any command has added some delay.

Deferring actions

When the mob is busy in the delay after an action, obviously any new action arriving at doAction shouldn't execute right away; it has to wait for the delay to run out first. We enforce this by deferring the action:

def doAction(symAction, *args)
    if isIdle?()
        doAction_Real(symAction, *args)
    else
        invoQueueInteractionMode().addInvocation(DeferredMobActionInvocation.new(symAction, *args, true))
    end
end # function doAction

As might be obvious, the DeferredMobActionInvocation is simply a wrapper around the action and its arguments. It goes to the end of the queue and executes in its appropriate order.

The new, simpler code also enforces at least two of the three invariants I mentioned earlier. The starvation one isn't completely taken care of, but it's addressed well enough for the most important case, that someone else shouldn't be able to order you nonstop so that you can't react.

So here endeth another lesson where blogging saved my MUD from bad, overcomplicated, what-the-heck-was-I-thinking design. Onwards to combat!

vim73

Looks like vim 7.3 was released about a month ago (old news, I know). The most exciting improvements I've noticed so far are better support for files with long lines that wrap (like big XML files) and better experience editing files on network shares. I found out at one point that for every keystroke you make when editing a file, vim would call a synchronous function that queries some property of the file. so if you're on a slow network, it could have a delay after every keystroke. Well, like I said, it seems like they fixed it, so that's cool.

JS Immediate Window: The Making Of

Today at work, I needed to try out some Javascript code in a test wrapper around Internet Explorer. IE proper has a Javascript debugger built in, which has a very handy "immediate window" in which you can type commands and have them take effect. IE8's tools are okay at best, but the IE9 Platform Preview Builds show a good deal of progress, and are quite usable. But I digress...

Anyway, I didn't have access to the usual set of Javascript tools that help with discovering functionality and trying out code before writing it up in an HTML page. I thought to myself, "wouldn't it be simple just to create a little Javascript tool that lets me type lines of code and show me the output?" And then I thought to myself, "Yeah, that ought to be pretty easy." So I set out to do just that.

My thinking was basically that the interpreter portion of the code would use eval to interpret the line and then I'd format the result as a string and display it. So I started out with something like this:

function handleLine(line)
{
    return (eval(line)).toString();
}

It's intuitive and straightforward. I tried it with this input:

>> var x = 1;
    undefined

>> x
    ReferenceError: 'x' is undefined

The first line is okay (var doesn't return the value of the assignment, for some reason), but the second one? That's not what I expected at all... and hopefully not what you'd hope for. Perhaps you've guessed the problem already, that eval injects code in the context in which it's called. Here, that context is the local variable scope of the function handleLine. Once that function returns, all the local variables are, of course, blown away.

So in order to mimic the kind of persistent state from line to line that would be really useful, I need some way of preserving the mutated context in which the eval executes.

I'm really lucky that I had taken a really rad programming languages class in the past. Having gone through some of the utterly mind bending homework assignments, I was at least somewhat equipped to reason through this problem and ultimately arrive at a working solution.

The solution revolves around the technique of using a closure to capture the current context and propagate it from line to line. It would be really nice just to be able to package up the current state of all variables, but you can't quite do that in Javascript, so we have to play some tricks.

The way to create a closure is simply to declare a new function. This function, whenever it is called, will be able to use all of the local variables available at the time it was defined. So after calling eval, we will create a new function to capture the new state of the environment after whatever side effects the eval had, and return it. This function will be used later to execute the next statement, using that new environment.

Enough talking! Let's code!

function evaler(str)
{
   var ret = null;

   try
   {
       ret = eval(str);
   }
   catch (ex)
   {
       ret = ex;
   }

   eval(evaler.toString());

   return {
       ret: ret,
       context: evaler
   };
}

var g_context = evaler;
function handleLine(line)
{
    var o = g_context(line);
    g_context = o.context;
    return o.ret;
}

There are only a couple parts here that are fairly tricky (the bolded part). Firstly, within the body of evaler sometime after the eval call, it defines a new closure (also called evaler, just for convenience). Then we return it. It has everything needed to preserve the current context and execute another line (and generate a new evaler to do the same again, ad infinitum).

Some less interesting features include the try/catch to make sure we don't let exceptions escape, and that we also return the actual output (in ret).

Now it works like we'd expect!

>> var x = 1;
    undefined

>> x
    1

I can even define functions and execute them. Kewl.

>> function whoa() { echo("sup"); }
    undefined

>> whoa();
sup
    undefined

Try it out if you'd like. I hope this was informative. I know I felt kind of clever afterwards.

JS Immediate Window

JS code:

You can press the Up and Down keys to scroll through history of previous commands typed. You can also read about how I created this, if you like.

How to use reg add to write hexadecimal REG_DWORDs

I can't find any documentation on the internet for how to do this, and since I just discovered it, I thought I'd give something back.

If you want to have reg.exe write hexadecimal values into the registry, you can do so like this:

reg.exe add "key" /v "value" /t REG_DWORD /v 0x555

For decimal, just pass the number like usual; that's the default format.

zoo times

this weekend I was at the san diego zoo. the place is massive, and it took us nearly two full days to see it all. it was fun, but also kind of depressing, as zoos usually are. it's not anything about animals in captivity, really. I now know enough about zoos to know that in general, animals are really pretty well cared for and can be quite happy in zoos. the san diego zoo in particular seems to go to a great deal of trouble to ensure this of their animals. plus, they have all these carefully managed breeding programs, which also seem carefully managed.

the depressing part is the mentality of people who go to the zoo and who kind of expect the animals to be there for their entertainment. this behavior can range from something as subtle as just wishing aloud for them to display themselves to more obnoxious acts like banging on the glass, calling out loudly, and whistling.

by and large, the animals are not there for our amusement. they're there because that's where they're living. it seems kind of distasteful to adopt the mindset that we should be able to demand some kind of favorable behavior from them. I'm guilty of it a little bit too, but at least I notice it.

nevertheless, it's great to be able to see these animals and learn about them, so I'll keep going to zoos.

Intro CS: an experiment

Intro to the intro

At one time in college I briefly considered becoming a computer science high school teacher. I found myself helping people around me quite a lot with CS homework and things, and it was pretty rewarding when it worked out. At another time, I tried to tutor a friend in CS, but I don't think I was patient enough, and didn't really have the right materials. I think I'll try again. This is an experiment.

This is not meant to be a tutorial in learning how to program. There are plenty of sites on the internet that will do that for you, with varying degrees of success. This is supposed to be an introduction to computer science. CS encompasses a few things other than simply programming. It also includes algorithmic thinking and understanding how computers work. Rest assured, learning how to program is an integral part of it. In fact, that will be our starting point with this...lesson. I guess that's the best word for it.

I should mention that I am using some CSE142 courses from the University of Washington for some of my reference and probably for programming problems.

The primary language I want to use in these lessons is Javascript. Javascript (or JScript as I may call it from time to time), is--as the link points out--a misunderstood language. It has a lot of expressive power, but also has a simple syntax with the added benefit of looking quite a bit like C/C++ and Java. In that respect, you could call it a gateway language. Har!

What is a program?

The core thing of interest in CS is a program. A program is just a series of instructions for the computer to do. You are telling the computer what to do, and in what order to do it, and it does it. Computers are hard workers, but fairly stupid, so it's up to you as a programmer to be very explicit about what steps you want it to take. That's what programming is about.

Programs are written in text files. You can use any text editor, like Notepad, to create or edit a text file, and then use that file as a program. But not all text files are programs. For that to happen, the contents of the file has to conform to a language. A language describes how the computer should interpret the contents of your text file as a program. Enough talking; let's see what a program looks like!

WScript.Echo("Hello World");

The conventional first program to write in any language to test it out is one that prints out the text, "Hello World", for some historical reason. If you create a new text file with just that single line in it, save it in c:\hello.js, and run it (on a command prompt) by using the command cscript c:\hello.js, you should see this output:

Hello World

You might be able to kind of sort of read that program as though it were English: echo "Hello World". For "echo", think, "say back to me." Now to clarify the various things that are going on with this program.

I said earlier that a program is a series of instructions. A better term for "instruction" is statement. A program is a series of statements. In this example, the hello world program contains just one statement. A statement is an instruction telling the computer to do something. When you run a Javascript program, the computer goes one line at a time from top to bottom through your file and does each line in order.

Programs also contain expressions. An expression is something that results in a value. Statements in a program can contain expressions that indicate more about what the computer should do, or how to do it.

Before I get into this next sample program, I need to point out one thing. If you see //, it is known as a comment. The computer ignores everything on the line following it. It's used to write additional clarifying information alongside code. Note that this can be used on its own line (the line begins with //, and so the entire line is ignored) or on the end of a line (the line has some code that is executed, and then //, so only things after the // are ignored).

// statement
WScript.Echo("Hello World");

// expression
WScript.Echo("Hello World");

As you see above, the text between the parentheses is an expression. The program is instructing the computer to Echo (that is, print out or output for the user to see) some text. The text to print out is an expression, which has the value "Hello World".

Other miscellanea about this particular example. WScript is not too important right now; just know that it's necessary and will be explained later.

Blank lines do nothing in a Javascript program. You can insert as many blank lines as you want in between any two lines of your program and receive the same result. For example, the following two programs do exactly the same thing.

WScript.Echo("Hi there");
WScript.Echo("Hi again");

WScript.Echo("Hi there");






WScript.Echo("Hi again");

Generally speaking, most statements in Javascript end in a semicolon. This rule and others comprise the syntax of the language. The syntax of a language is the set of rules that you must follow when writing your program so that the computer can interpret what you wrote. In English, if you neglected all punctuation, used the wrong punctuation, or put punctuation in strange places, someone else may have a hard time understanding what you mean. Likewise, if you do not adhere to the syntax of a language, the computer cannot run your program, because it can't understand you. Unlike humans, the computer is fairly unforgiving about syntax errors, and will usually just display an error message when you try to run your program, rather than try to guess your intent.

Other things from our example that are part of the syntax include the fact that pieces of text need to be enclosed in double quotes (""). The fact that blank lines are ignored and that comments (using //) cause the rest of the line to be ignored are also part of the syntax.

Functions, your best friend

Echo above is what is known as a function. A function is a group of statements that are executed together. That group of statements is called the function's body. When you use a function, the computer jumps to the first line of the function's body, does each line in turn until the end of the body, then jumps back to the next line where you were before.

I said, "use a function" in the previous paragraph. The proper term is calling, or invoking. These terms describe instructing the computer to jump to the function's body, run all of the lines in it, and jump back to the next line after the invocation.

In the example program above, you called a function named Echo. The body of that function happens to be in a different file, so you can't see it. Later, you will see functions whose body is visible to you. You should imagine that on the line that says WScript.Echo("Hello World");, the computer jumped to the function body of Echo (not visible to you), did several lines of program there, then jumped back.

One of the most important features of functions is that they can act on different data whenever called. In the following example (a new program), you are calling the same function, but supplying different data.

WScript.Echo("Hello");
WScript.Echo("Goodbye");

On both lines, the function being called is the same: Echo, but on the first line the data supplied is "Hello", whereas on the second line, the data supplied is "Goodbye". The same type of activity will take place on each line--displaying some text--but the specific text being displayed will be different.

I mentioned "supplying data to a function" a few times in the previous paragraphs. The proper term for the data being supplied is argument, or parameter. The more conventional term for "supply" is pass. So instead of "supplying data to a function", we say pass an argument to a function.

Briefly going back to syntax, function call arguments are always enclosed in parentheses, and immediately follow the function name.

Creating your own functions

One of the most important parts of programming is that you can define your own functions. The way you do this is by using the keyword function followed by the name you want to give to your function, the parameters that someone is allowed to pass to the function, and the function body. The body is enclosed in { } (curly braces), which is part of the syntax of the language and tells the computer where the body starts and ends.

Just to quickly review, remember that the parameters are the pieces of data that someone can supply when calling a function. And remember that the function body is the set of statements that are run, in order, when something calls the function.

Here is an example in which I have defined a function named Greet, then called it. You can write or paste this code in a new text file and save it as c:\greet.js. Then you can run it on the command prompt like you did before with hello.js, by using the command cscript c:\greet.js.

function Greet(person)
{
    WScript.Echo("Hello, " + person);
}

Greet("Owen"); // output is: "Hello, Owen"

There are a lot of interesting things about this example. On the very first line, I defined the function, which consists of what I said in the previous paragraph--giving the function's name, parameters, and body.

After the end of the function body, I call the function and pass a parameter to it. The parameter's value is "Owen". At this point, the computer jumps into the function body, to the line that says WScript.Echo("Hello, " + person);. The computer runs this line, which causes "Hello, Owen" to be displayed. Then the computer goes to the next line, but that is the end of the function body, so it jumps back to where it was before--the line after Greet("Owen");. Since this is the last line in the file, the program is done.

There's one crucially important detail here that I need to point out. This is one of the most difficult, fundamental concepts to grasp, so I will try to explain it thoroughly. When you pass an argument to a function, it can be used within the function body. In the example above, the argument "Owen" that you pass to Greet can be referred to within the function body as person. As we will see later, various different things can be passed as arguments to the function. No matter what, the code within the function body can always refer to the value that was passed in using the name person. One simplistic way to think about it for now is that everywhere you see person in the function body, the computer automatically substitutes the data that you passed as the argument.

By the way, You may be able to surmise from the example and code that + here appends one piece of text to another.

As you will discover later, using functions is one of the primary ways of breaking down a large problem into smaller pieces.

More on function definitions

I'd like you to try something with your greet.js example. I'd like to delete the last line, save the file, and run the program again. What do you see? You should see nothing at all, but why is that?

Remember that there were two steps that I explained in the example: defining the function and calling it. By deleting the last line, you removed the function call but not the definition. This should tell you that defining functions has no visible side effect. Just because a function with a certain name, parameters, and body exists doesn't mean that anything really happens. Only when it is called does the contents of its body run. You could define a million different functions; so long as you don't call any of them, your program will basically do nothing.

Variables, a constantly useful construct

Sometimes you need the computer to remember a piece of information. This is the purpose of a variable. A variable is a holding place for a piece of information.

We can use an analogy that a computer is like a building full of storage units. Each storage unit is empty and unused initially, but could be reserved by someone and used to store something. If the computer is the building, a variable is a little bit like a storage unit in the building, and your program is the customer who can reserve rooms.

When you declare a variable, you reserve a place in the computer that can be used to store data and give a name to that place, so that you can refer to it later. In the analogy, declaring a variable is like reserving a storage unit and picking a unique name or number for the room. Now you have an empty space that could be used to store something, and you know the room number so that you can get back to it later.

When you assign to a variable, you store an actual value in it. In the analogy, this is like putting some of your stuff in the storage unit. You use the room number to find the unit, then put something in there, then carry on with your life, knowing that your thing will stay in there until you take it out.

Finally, you can read a variable, which retrieves the value that you stored in it previously. In the analogy, this would be like going to the storage unit (again, using its room number to find it), and looking at what's inside. The analogy breaks down a little, because you don't actually take anything out of the room at this point.

Here is a new program that you can save and run. In it, we declare, assign to, and read a variable. Let's save it in c:\variable.js and run it.

// declare a variable
var someName;

// assign to a variable
someName = "Owen";

// read the variable by using it
WScript.Echo("Nice to see you, " + someName);
// output is: "Nice to see you, Owen"

There are a lot of things going on here that are important to know. The way to declare a variable, according to the syntax of the language, is to use the var keyword, followed by the name you want to use for the variable.

The way to assign to a variable (store something in it) is to use the variable name, followed by =, followed by the thing you want to store in it. In this program, I've stored a piece of text in this variable for later use. After the line reading someName = "Owen"; the value "Owen" will be stored in the variable named someName.

Finally, in the Echo line, we are reading the variable, simply by using its name in an expression. The expression it is used in is "Nice to see you, " + someName. As part of computing the value of that expression, the computer sees the variable named someName and reads its value (what you stored in it before) to substitute into the final piece of text.

More on function arguments

I'm going to combine parts of a greet.js and variable.js into a new program, as follows. Let's save it as c:\greet2.js and run it.

function Greet(person)
{
    WScript.Echo("Hello, " + person);
}

var someName;
someName = "Owen";

Greet(someName); // output is: "Hello, Owen"

There's one very important point that you need to understand here. Notice how when we define Greet, we write person as the name of the parameter, and refer to that data by the name person in the function body. And then notice that we wrote the line Greet(someName);. Well, someName doesn't "match" person... but it doesn't have to!

On the line where we call Greet, the computer determines the value of the parameter before that value is passed to the function body. So by the time the computer gets into the function body (the Echo line), it doesn't matter what we wrote outside of the function. The function body only knows that it was passed an argument that it knows as person.

A simplistic way to think of it for now is that Greet(someName) is automatically transformed by the computer into Greet("Owen") when the program runs.

That's one of the great powers of functions. No matter what expresion you pass to them, the data is available within the function body by referring to the same parameter name within. Similarly, you have freedom to pass any expression to the function as an argument, and the function will be able to handle it, so, for instance, you're not restricted in what you name your variables.

Exercises and solutions

To see whether you've understood any of this, here are a series of sample programs that make use of some of these constructs. If you've understood well, then in the comments where I ask what the expected output is, you should be able to guess exactly what the output should be. Each code block below is a separate, complete program that you could save in a text file and run with cscript, but try to guess the output before resorting to that.

// problem 1

function Func1(text1)
{
    WScript.Echo("Func1 " + text1);
}

Func1("abc"); // what is the output?

Problem 1 solution - click to show or hide

Func1 abc

// problem 2

function Func1(text1)
{
    WScript.Echo("Func1 " + text1);
}

var text = "def";
Func1(text); // what is the output?

Problem 2 solution - click to show or hide

Func1 def

// problem 3

function Func2(text1, text2)
{
    WScript.Echo("Func2 " + text1 + ", " + text2);
}

Func2("aaa", "ddd"); // what is the output?

Problem 3 solution - click to show or hide

Func2 aaa, ddd

// problem 4

function Func2(text1, text2)
{
    WScript.Echo("Func2 " + text1 + ", " + text2);
}

var text1 = "abc";
var text2 = "def";
Func2(text2, text1); // b) what is the output?

Problem 4 solution - click to show or hide

Func2 abc, def

// problem 5
function Func1(text1)
{
    WScript.Echo("Func1 " + text1);
}

function Func3(text1)
{
    Func1(text1);
}

Func3("def"); // what is the output?

Problem 5 solution - click to show or hide

Func1 def

// problem 6

function Func2(text1, text2)
{
    WScript.Echo("Func2 " + text1 + ", " + text2);
}

function Func4(text1)
{
    Func2(text1, text1);
}

var text2 = "rrr";
Func4(text2); // what is the output?

Problem 6 solution - click to show or hide

Func2 rrr, rrr

// problem 7
function Func1(text1)
{
    WScript.Echo("Func1 " + text1);
}

function Func5(text1, text2)
{
    var text3 = text1 + " " + text2;
    Func1(text1);
    Func1(text3);
}

Func5("dep", "ped"); // what is the output?

Problem 7 solution - click to show or hide

Func1 dep
Func1 dep ped

// problem 8
function Func1(text1)
{
    WScript.Echo("Func1 " + text1);
}

Func1(""); // what is the output?

Problem 8 solution - click to show or hide

Func1

// problem 9

function Func1(text1)
{
    WScript.Echo("Func1 " + text1);
}

var text1 = "efg";
text1 = text1 + text1;
Func1(text1); // what is the output?

Problem 9 solution - click to show or hide

Func1 efgefg

wow autohotkey wow

so I got a new netbook recently, because my old one (acer aspire one) up and died all of a sudden. the new one is a hp mini 210 hd. as far as netbooks go, it’s pretty rad. it has a high resolution screen, which is great until I realized I couldn’t read anything. so I set the computer to high DPI. How ludicrous is that? a netbook running with high dpi.

anyway, the main thing I find wrong with it is some puzzling choices with the keyboard. there are no dedicated page up, page down, home, and end buttons. Instead, to do that, you have to hit the dreaded Fn key plus up, down, left, or right. that would be fine, I guess, if there were an Fn key on both the left and the right. as it is, it’s only on the left, so to do a page down, I have to use both hands. annoying.

Another weird choice is making the key for “printscreen” the same as the key for “insert”, but making it so that the primary function is printscreen and the secondary (Fn) function is insert. I guess I can kind of understand that. I am probably one of the minority who sometimes uses ctrl+insert and shift+insert for copy and paste, respectively.

I found a program called sharpkeys that allows for some limited remapping of keys, apparently using some built-in windows feature having to do with scancodes. I say limited because a big limitation is the inability to map a combination of keys to a single other keystroke. in particular, I am interested in making right-control + up/down do my page up/down for me.

After some searching, I found a program called autohotkey, which does all this and way more. I’m using just the basics right now, but it can do all kinds of stuff, like run arbitrary programs upon receiving a keystroke, scan windows that are open on the system, append to files, and tons of other things.

here’s my script that I’m using for it right now. it’s pretty simple and slick. I no longer need to use sharpkeys either.

AppsKey & Down:: Send {WheelDown}
AppsKey & Up:: Send {WheelUp}

RCtrl & Down:: Send {PgDn}
RCtrl & Up:: Send {PgUp}
RCtrl & Left:: Send {Home}
RCtrl & Right:: Send {End}

PrintScreen::Ins
Ins::PrintScreen

cuttin on my finger

I managed to cut my left index finger last night while cutting through some english muffins to make breakfast sandwiches. I held the muffin in my left hand and cut with a gigantic bread knife down towards my hand, like an idiot. it cut a little too smoothly by accident, and I ended up slicing the tip of my left index finger.

now I have the opportunity to observe what daily activities I absolutely need that part of my body for. the good news seems to be that I don’t really use it for much that I couldn’t use my left middle finger for instead.

• pulling the left trigger on my xbox 360 controller. I don’t know how well I’d adjust to doing this with my middle finger. I think that would throw off my grip a lot
• wiping water out of my eyes. not a critical dependency

hm, nothing else comes to mind right now. I may update later if I think of something.

richmond science museum

this morning I bumped up against the wall of the shower while having my shower. it was cold, and reminded me of my childhood. when I was in maybe first grade, our class took an overnight field trip to the richmond science museum. the place was utterly magical for me, since it was my first experience at a science museum like that. I remember two specific experiences there.

the first, which is what reminded me of the museum this morning, was an exhibit where a series of metal bars were placed in a row. every other bar was either pretty hot or very cold. when you placed your hand on it, so that it touched several of the bars, you experienced a sensation quite unlike any other, probably led primarily by your body’s confusion at having such conflicting feelings so close to one another.

the other thing I remember was a kind of workshop that we as a class attended. in it, the instructor created something she called simply “goop”. it was a little bit slimy, and oozed like a very thick liquid when poured, but when you squeezed it hard enough it resisted, like a dense piece of clay or something. the instructor insisted that it could be created in our own homes with ingredients as simple as just corn starch and water, but I was totally disbelieving at the time.

mass effect 2

I started playing mass effect 2 today. it seems to be better in pretty much every way compared to the first one, which I also liked a great deal. I managed to make my main character just a little weird looking, by accident. so yeah, that’s cool I guess.

some people may be annoyed that they removed such fundamental RPG concepts as armor and weapon management, but actually, I’m not really missing it. the loadout system gives me some choice about it while not making me constantly manage my inventory. and face it, if you played through ME1 you are undoubtedly familiar with how much time was spent in inventory management. ugh.

the one other thing that is just downright bad is that the cool “special” armors like the cerberus armor and blood dragon armor have helmets that, while they look cool (I guess), totally obscure your face. the facial animation has gotten even better since ME1. characters raise their eyebrows, avert their eyes, and make faces with their mouths, and I really kind of want to see shepherd doing those things during conversations. furthermore, conversations are a pretty big part of mass effect. apparently there is a thread on the bioware forums clamoring for an option to remove your helmet. I (and 15 pages worth of people in the forum) hope they address that in an update.

currently a level 7 sentinel. exciting! the scene where you first see the new normandy gave me goosebumps. it was really well executed, cinematically and musically.

3d platformer

I was reading this (pretty interesting) article on game innovation today. They had an interesting bit in it where they said that Nintendo invented the 3d platformer. is that true? was mario 64 really the first 3d platformer? it was really, really good, but I don't know my trivia well enough to know if it represented the invention of the genre.

dirty harry

i finished watching the movie, dirty harry last night. it was okay, i guess. the pacing was too slow and some of the camera work seemed pretty awkward. there was also way more nudity than i expected to see in a film.

i'll say this, though, the music was fantastic. the drum beats and guitar lines were super catchy. i'd listen to that soundtrack.

oh, and now i get the reference to the line, "you gotta ask yourself, 'do i feel lucky?' Well? do ya, punk?" Actually, the best part about that line is there's lingering doubt about whether harry himself knows if his gun is out of ammo. if he's not sure, then he's totally bluffing with his confident demeanor. i like to think that every time he's asking that question, he's also asking it to himself.

podcasts

lately i've been listening to some podcasts at work. it started when i tried out the penny arcade podcast, which i liked so much that i listened to every episode basically straight in a row. this is still my favorite podcast of all the ones i've checked out. it's basically just gabe and tycho talking about whatever for 1-2 hours, and somewhere in the middle they come up with the idea for the comic for the next day. i pretty much think they're some of the funniest people alive, so hearing them ramble on forever is just awesome. not to mention... the gratuitously flowery way that tycho writes his posts? yeah, he talks like that too. awesome.

Once i ran out of those (*sob*), I eventually looked around for some other sources. I tried the Major Nelson podcast, which was fine. I mean, it had content. in fact, it had content i wouldn't find in some of the podcasts i went to later, for example, an interview with one of the mass effect 2 developers. that's kind of cool. of course, this one is an official microsoft podcast of sorts, so you won't hear major nelson, e, etc. talking about the kinds of crazy things gabe and tycho do. not only that, the tone is way more businesslike (in comparison, i guess). each of the speakers goes around taking turns talking about various things with some discussion. they have "segments". you certainly won't hear them all bust up laughing and shouting for a minute straight over some absurd scenario. oh well.

don't get me wrong; i like major nelson fine. i think he's a big ol teddy bear.

Then I took a break for a while. Last week someone at work linked to the destructoid podcast, or "podtoid", as they call it. I initially listened specifically for a particular section, but ended up listening through the rest of it and liking it. Since then i've started listening from way back, and i'm continuing to like it. i like that it's way more informal than some other podcasts. i'm not sure what i think about here being 4-6 people on the air. sometimes it gets a bit out of hand. certainly if you can't tell them apart some of the jokes are lacking context (because they spend about a quarter of the time making fun of each other).

what I do like is that all of the speakers seem pretty intelligent. a lot of the show is one person sparking a discussion, then all the members going around discussing it for another half hour or something. i've heard several really good discussions on there. they also all have somewhat conflicting views and tastes, so it's not all of them just agreeing, which I tended to find in other places.

they have three sections of each podcast, mainly: game of the week; perhaps some topic; and listener questions. game of the week is each person talking about what they've been playing that week. usually this ends up getting into what they like and dislike about it, and what other members like and dislike about it. sometimes then they talk about some particular agreed-upon topic for a while, which can be good. then they answer reader questions, which again sparks more discussion.

they're usually insulting each other and laughing and hollering a lot. it seems to be a pretty fun time. when the sports dude starts talking about sports, i totally zone out, though. also, all i can imagine is kutner from house (warning: spoilers) whenever he's talking.

destructoid is part of the gamercast network, so i browsed there a little. i was curious what girls would talk about on these, so i tried the sarcastic gamer pink podcast. i listened for about 10 minutes before turning it off. i don't know if it would have gotten better after that, but I could not stand the incessant loopy giggling that permeated it.

i read game informer, so i had this wild and crazy idea that maybe they have a podast. indeed, they do. I checked that out too. I gotta say, it felt quite a bit like the major nelson podcast. is that how most podcasts are? again, it had kind of high quality content associated with it, which probably comes from being a very well established piece of gaming journalism. they were able to talk about halo reach and their various impressions, from visiting the bungie studios. good content, not as interesting tone. i rate the discussion aspects somewhere between MN and dtoid.

yeah, these are all gaming podcasts. i like games. what can i say?

on a tangential note, i really really hate it when i see a sign for "the best gaming around!" but oh wait, they're talking about gambling. sheesh.

efficiency: scripts, or: up front investment

whenever i have to do something annoying at work more than once, i write a program or a script to do it for me next time. i'm a big believer that an up-front investment of a little while is worth it in the long run in terms of time and effort saved, even if it sets me back a bit temporarily. it's kind of the "work smarter, not harder" credo.

efficiency: laptop battery life

i think about weird stuff all the time. I guess most people do, but think their own weird stuff is weirder than others' weird stuff. one category of weird stuff i think about is efficiency. kind of like the dad from cheaper by the dozen (great book, by the way). he's a motion efficiency expert, which seems like a pretty rad job to me.

when i turn off my laptop, i always think about when i will next use it. if it's going to be within the next 15 minutes, i'll put it into sleep mode. if it's going to be longer than that, i'll put it into hibernate. the idea being that the additional power needed to shut down and start up the laptop "costs" a certain amount of time in sleep mode. I haven't scientifically figured out what that threshold is, so i've set it mentally based on convenience.

what if i'm not sure? well, my laptop is set to go into hibernate automatically after 20 minutes, so if i'm not sure, i'll put it in sleep mode, knowing that if i don't turn it back on, it will shut itself off after a while. pretty sweet power saving capabilities in windows makes this possible.

bayonetta

I recently beat Bayonetta... three times. I started out planning just to play through it once, but then penny arcade challenged me with a contest to beat it on hard mode. I don't generally end up beating games multiple times, usually because I play through the first time slowly enough that I don't feel the need to. bayonetta, though, was gripping. And I wanted a t-shirt.

it's easily one of the best games i've played in ages. the fighting mechanics are ridiculously well polished. when you get the hang of it, you can blaze through enemies feeling totally unstoppable.

a lot of people didn't like the story much. they complained either about how it was too complicated or too cliched (light vs. dark, in broad terms). Maybe that's true. Even if the particulars of the story didn't overly interest me, the game world did. I especially liked the lore about the angels and heavenly figures themselves. apparently there are something like 9 ranks of angels in increasing order of power or status. Each rank has various types of footsoldiers and a kind of captain. then, aside from these "regular" angels are several "auditiones", or overseers, I guess.

The character and enemy designs were great. The enemies (angels) in particular were really unique designs and had a lot of great details, too. The human(oid?) characters like bayonetta, jeanne, luca, rodin, and cereza were also very well done, though that is more dependent on taste. At the very least, the animation was fantastic. Even the facial animation was pretty good, something that few games seem to do really well.

I didn't mind the half-animated cutscenes either. I can understand how it could be seen as a trick to save budget, but it also lent a unique style.

I had absolutely no problems with any of the voice acting either, which is much more than I can say about most games.

yeah, so after i beat it on normal, i had to beat it on hard for the PA contest. then i was having enough fun that I beat it on "nonstop infinity climax" mode too. For accomplishing this, you get a really sweet lightsaber. I still want to go back and get some of the final unlockables, like beating the game in under X hours and getting enough money for the platinum ticket to fight rodin.

Hotboot and SxS

How many times do I need to fix hotboot? I fixed it once before, but after recompiling Ruby 1.9 at some point, it stopped working again. Now with my performance sweep out of the way, I debugged this yet again. The way I fixed this before relied heaviliy on calling _get_osfhandle and _open_osfhandle in the right places to convert between CRT filehandles and Windows HANDLEs.

I used the Win32API class to do this:

GET_OSFHANDLE = Win32API.new("msvcrt", "_get_osfhandle", ["I"], "I")
OPEN_OSFHANDLE = Win32API.new("msvcrt", "_open_osfhandle", ["I", "I"], "I")

# and to call them:
def Hotboot.handleFromFileDesc(fd)
    return GET_OSFHANDLE.Call(fd)
end # end function handleFromFileDesc

def Hotboot.fdFromHandle(handle, flags=0)
    return OPEN_OSFHANDLE.Call(handle, flags)
end # end function fdFromHandle

What I found was that after recompiling Ruby 1.9, it no longer links against msvcrt.dll, but instead against msvcr90.dll. This is a big problem, because those two functions modify private DLL state about filehandles. If my code is using one dll's version, and Ruby is using another's, then it won't see each other's effects as they should. Of course, to fix this I need to make my code use msvcr90.dll.

The naive approach, changing "msvcrt" to "msvcr90" doesn't work, because msvcr90.dll is a SxS module. Just making that change leads to the infamous R6034 error. I spent a bunch of time following red herrings about adding manifest files, copying the CRT dll locally, and turning on loader snaps, all the while getting ever more frustrated.

It was while I was writing up a question to ask on stackoverflow that I got back into thinking up other ideas. I thought: well, if LoadLibrary is having trouble because of SxS configuration, maybe I shouldn't try to load it, but instead reference the instance that's already loaded. After all, by the time this code is called, ruby has been running for a while, and certainly will have msvcr90.dll already loaded.

But even using GetModuleHandle wouldn't work, so I started thinking it had something to do with where I was loading the library from. Previously I wasn't specifying a path to the library, just giving the name. Now I tried giving the full path to the dll in the sxs folder, and for whatever reason, it worked!

So I wrote a little program to call EnumProcessModules on my ruby instance, find the full path of the dll I need, and print it out. I parsed this output and passed this full path to Win32API.new. Life is good again.

DL

As I was searching around I happened to see someone mention that Win32API was being deprecated after Ruby 1.9. That's not cool... I use that! It turns out that Win32API is just a wrapper around a Ruby library called DL. And somehow that link is the only reasonable reference/guide for it that I've been able to find. How crazy is that?

DL is cool. You already saw how I was loading the functions before. Now see how it's done after, with DL.

module CRT
    extend DL::Importer

    # ...

    modpath = Utilities.findLoadedModuleFullPath("msvcr90.dll")
    dlload(modpath)

    extern("int _get_osfhandle(int)")
    extern("int _open_osfhandle(int, int)")

    #...
end

# and to call them:
def Hotboot.handleFromFileDesc(fd)
    return CRT._get_osfhandle(fd)
end # end function handleFromFileDesc

def Hotboot.fdFromHandle(handle, flags=0)
    return CRT._open_osfhandle(handle, flags)
end # end function fdFromHandle

I actually kind of like the new way using DL, especially the part where you just write out the function signature instead of what looks like packed array templates.

The Great Logging Worker Thread Adventure

In my previous entry, I talked about some ways I tried to increase performance or responsiveness of the MUD. One of the ideas I talked about was putting my logging functionality into a separate thread. I went through a few iterations of it before I finally arrived at a solution that is both fast and preserves the functionality I want. One of the intermediate solutions also had a really cool idea (if I may say so myself), and I want to share that/document it.

It's funny: I was going to call this post "the great logging worker thread fiasco"... but then I managed to make it work, so I had to change the title. I guess that's a good thing.

The justification for doing this was to increase responsiveness. If the logging code is synchronous/blocking, then whenever I log something, the user has to wait for that logging to happen. The user doesn't care! They just want to play. In fact, they don't even know it's happening; all they know is my MUD sucks because it's dog slow.

So I wanted to use threading to handle this in the background, i.e. at a lower priority. It could happen whenever it was convenient. More so, I thought of using a worker thread. Obviously, that link is for C++ and Windows, but hopefully I'm not abusing the term for the idea I'm thinking of. The key point is that the logging worker thread doesn't exit, but instead hangs around waiting for new work to arrive. This avoids the cost of creating and destroying a thread.

Actually, Ruby threads are supposed to be "green threads", not kernel threads, so maybe this particular aspect doesn't really matter. I could take some profiles to verify that. In fact, the fact that these aren't hardware threads could potentially make this entire idea a non-starter. I'd have to do some simulations to verify that too.

Let's first consider the naive implementation.

def DebugOutput.debugOut(msg, cats = LComment)
    trace = caller(1)
    enqueueAction() {
        debugOut_internal(msg, trace, cats)
    }
end # end function debugOut

def DebugOutput.workerThreadProc()
    isRunning = true

    while isRunning
        toLog = nil

        @@mutexEventDoLogging.synchronize() {
            isRunning = @@isRunning
        }

        @@mutexEventDoLogging.synchronize {
            @@condDoLogging.wait()
            toLog, @@toLog = @@toLog, []
        }

        toLog.each() { |prc|
            prc.call()
        }
    end
end # function workerThreadProc

I've skipped some of the innards here that perhaps you can imagine. debugOut is the producer, if you will, and workerThreadProc is the consumer--the thread that sits around (the wait call) and does the actual printing (the prc.call() call). What's wrong with this code? Well, one major and a few secondary problems.

To see the major problem, we need to look first at what the logging output actually looks like.

connec@20   accept              Waiting to accept a connection.

There are four parts here: filename ("connection.rb", truncated), line number, function name (accept), and the actual message being logged. The first three of these are generated automatically in the logging code. It's useful to have, to quickly see in the log file where a message was logged from to go look at the code in that file. Herein lies the problem, though.

Look back up to the code and notice the caller(1) call. It turns out this is really expensive (like 80% of the time spent in the logging code). You might also be able to tell that this work isn't deferred in the worker thread; it's on the main thread before the enqueueAction call. The unfortunate fact is that this is necessary. The caller function grabs a full stack trace at the point where it's called. I use that information to parse out the file, line, and function name for the logs. One can perhaps imagine how that might be expensive. And, since it grabs the stack at the point where it's called, it must be done on the main thread; on the worker thread, the stack will just look like the worker thread's stack.

Try Continuations

So I got to thinking, how can I continue to make the correct caller call, but do it on the worker thread, so it doesn't get in the way? In my recent programming languages course, I learned about something very cool that Ruby 1.9 got: continuations. The "continuation sandwich" description in the wikipedia article is pretty cool. Let me summarize the idea for those too lazy to click on the link. A continuation is basically taking a snapshot of the execution state at some time and storing it in a variable. Later, you can jump right back to that point and resume execution from the next line. But it's not really time travel; everything you've done in the meantime is still persisted.

To apply continuations to this problem, roughly the following steps would take place. I'm not altogether clear on whether this is an example of a coroutine.

1. On the main thread, create a continuation at the point where the caller call should happen, without actually calling it.
2. Enqueue the logging call as usual, also passing in the continuation object.
3. In the worker thread, create another continuation to mark the current place.
4. Call the first continuation, also passing in the second continuation.
5. Now we're back in the code main thread (but hopefully still running in the logging thread). It's set up (see #1) so that right now we call caller.
6. Having procured the caller information, now the main code invokes the second continuation, passing the caller information too.
7. This moves us back to the worker thread proper and the normal flow of execution.

So yeah... that was confusing. Maybe I should just post the code and annotate it with numbers? OK.

def DebugOutput.debugOut(msg, cats = LComment)
    cont = callcc { |cc| # 1
        enqueueAction() {
            debugOut_internal(msg, cc, cats) # 2
        }
    }

    if cont.kind_of?(Continuation) # 5
        trace = caller(1) # also 5
        cont.call(trace) # 6
    end
end # end function debugOut

def DebugOutput.debugOut_internal(msg, cc, cats = LComment)
    trace = callcc { |cont| # 3
        cc.call(cont) # 4
    }

    # 7
    # write logs using trace
end # function debugOut

The callcc function creates a continuation. It takes a block, in which you receive (via the block's parameter) the continuation object that was just created. The return value from the callcc call is not the continuation object that was just created! The return value is whatever value is passed to continuation.call later. For example, the return value from callcc at point #3 is the variable trace that was passed to cont.call at point #6. The syntax is maddening, but it works.

So if you can puzzle all this out in your head (it took me a while), it's pretty rad. Too bad it just flat out doesn't work. Oh, it works on paper, and even works in the single-threaded case, but if you try to call a continuation from a thread other than the one on which it was created, Ruby just throws an exception saying you're not allowed to do that.

The plane was full of Miracle Whip, but it never left the hangar.

Try __callee__

I was looking through methods and noticed that Kernel exposes a method called __callee__ that tells the current method being executed. So if every call to debugOut included the value of this method at that point, it would at least get me the function name that a logging call was executed in. The problem is, I really don't want to update every single call site with an extra parameter that is always the same code.

I was pretty sure this wouldn't work, but I gave it a try anyway: I put __callee__ as the default value for another parameter to debugOut, hoping it would get executed in the context of the caller, not the callee (debugOut).

def DebugOutput.debugOut(msg, cats = LComment, callee = __callee__())

Nope, this always gets the value debugOut. Oh well.

Think about macros

I started thinking a bit about C++, and the way I'd do this there. We actually have an exception library at work that does have this feature of printing out the site that threw the exception. To make it work, we used the __FILE__ and __LINE__ macros.

There are macro processors for Ruby out there. For instance, rewrite looks interesting. But it doesn't seem to be under active development, and there is some question about its speed. It also has some external dependencies, and after last time, I don't feel like putting myself through that pain just yet.

Try Proc.source_location

At some point along the way, something I'd seen tickled my mind. I remember printing out a Proc object and noticing that it showed the file and line number where it originated. Surely I can programmatically get a hold of this information. Indeed, I can! It's source_location.

But to get at this, I'd need a proc that was created at the right place. We've already established that I can't create an empty one in an optional parameter to debugOut. And the prettiness of the code matters, so I'm not going to create that by hand everywhere.

Instead I decided to turn the log message into a proc itself. Check this out:

# Before:
def DebugOutput.debugOut(msg, cats = LComment)
    writeLogs(msg)
end

# After:
def DebugOutput.debugOut(cats = LComment, &procMsg)
    writeLogs(procMsg.call())
end


# Before:
DebugOutput.debugOut("called this crazy function", LComment)

# After:
DebugOutput.debugOut(LComment){"called this crazy function"}

See, this is syntax I can live with. I don't mind curly braces too much, and this has the added benefit of deferring the evaluation of the string until the proc is called... I think. Best of all, this works. I had to change every single call to my logging to fit the new syntax, but it's great. I'm down to 5 sec in startup time, and the majority of that is spent actually interpreting the source code, not in actually executing MUD code.

Minor problem: synchronization

Way back above I alluded to minor problems with the naive implementation. It turns out that all the synchronization and waiting and signaling I'm doing here is slow. When I eventually did get this threading approach working, I found that it sometimes accounted for over 50% of the time. So I removed it. It's just logging, after all; if I lose a couple log messages I highly doubt I'll miss them.

After removing it, the code for the worker thread and the code to control the worker thread is really simple. You don't want to see what it was before, when I had to avoid pesky deadlocks and stuff.

def DebugOutput.startLoggingThread()
    DebugOutput.ASSERT(!@@thDebugLogging, "trying to start logging thread twice")
    @@isRunning = true

    @@thDebugLogging = Thread.new() {
        DebugOutput.workerThreadProc()
    }
end # function startLoggingThread

def DebugOutput.stopLoggingThread()
    @@isRunning = false
    @@thDebugLogging.join()
    @@thDebugLogging = nil
end # function stopLoggingThread

def DebugOutput.enqueueAction(&procLog)
    @@toLog.push(procLog)
end # function enqueueAction

def DebugOutput.workerThreadProc()
    while @@isRunning
        sleep(0.5)

        toLog, @@toLog = @@toLog, []

        toLog.each() { |procLog|
            procLog.call()
        }
    end
end # function workerThreadProc

One big change is now that instead of being notified that there is work to do, the thread wakes up periodically to check for work (the sleep call in the loop).

I guess the moral of the story is that sometimes you have to try tons and tons of crazy stuff until you finally hit on something that actually works.

First Performance Sweep

As usual, it's been quite a while since I've updated with anything on-topic. And as usual, I haven't gone forward with implementing any real new features. I had classes and things that took priority for a while, and progress in this last bit of work was very slow for very dumb reasons. The focus for my last "sprint" was performance. It's only fitting that since I now work on performance testing of my product at work that I look at performance of my MUD!

Since I try to avoid premature optimization whenever possible, I tried to use profiling techniques to figure out what was making my startup performance so slow. I had a suspicion already, but I wanted proof.

Profiling in Ruby (on Windows)

Let me just say: profiling in Ruby kind of sucks. Or maybe it's just on Windows that it sucks. I mean, Windows is already a second class citizen in the Ruby world, but still...

The state of the art in profiling in Ruby as far as free stuff goes seems to be something called ruby-prof, but don't let that fool you; you don't want to try to get ruby-prof to work on Windows. Instead, you want to get jeremy-ruby-prof, which is some dude's attempt at fixing bugs in ruby-prof and getting it workable on Windows. Don't get me wrong. It is workable, but I have repressed memories of having to bludgeon the build process to get it to finally compile. That is actually a theme I'll return to later.

There are actually other tools out there. I tried to get perftools.rb to work (unsuccessfully). I could never get it to build properly. It looks like it might have been updated since I tried, though, so maybe eventually I'll give it another chance.

You could compare this with profiling C++. It's night and day. C++ has a lot of great tools like Valgrind, VTune and XPerf. I suppose the usual response to this is either a) Ruby is still in its infancy, or b) because Ruby is in its infancy, first it needs to work before we should worry about it working fast.

At work we have some really good instrumented profiling tools, along with visualizations for the profiles that allow things like comparing two profiles against each other and finding where they diverge. If anyone knows about any such tools available widely, I'd love to know about them. ruby-prof has extensibility for writing out a profile into any output format. If any good UI for profiles exist, I can try writing a printer for that profile format.

Bottlenecks

So I was finally able to get the Ruby profiler working, and I took some traces. I found two pretty bad bottlenecks, one surprising and one totally unsurprising. The unsurprising one was my XML loading implementation. I already knew that I made too many passes over too many objects. The somewhat surprising one was my logging infrastructure. I found over a quarter of my time spent in just logging. For something purely aside from the game proper, that's pretty unacceptable.

Logging

A while ago I revamped all of my logging code so that I could assign categories to each logging call and allowed for adding filters that would redirect different categories to different log files. It turns out I introduced some components in this that slowed things down a lot.

Chief among them was a bitfield class that implemented a bitfield of arbitrary size. The way I used this is that when I made a logging call, I would pass an array of categories that the log message belongs to. Then the system enumerated a list of registered loggers and asks each one if they would like to log the message. Any that answer, "yes" get called again to do the actual logging.

The first part of this that I noticed was really slow was the implementation of the bitfield. Well, I'm sure the implementation itself was just fine, but I realized I don't need all the flexibility of an arbitrary size bitfield. I decided instead just to limit myself to 32 different categories, and store my category mask in a single number. By replacing all of the array handling from the bitfield with simple binary operators, it sped things up by about 30%.

The second part of this that I noticed was slow was a call to Kernel#caller. I use this to extract the file name, function name, and line number of the site where the logging call was made. I haven't yet found a good way to get around this cost.

I also noticed that I had set IO#sync = true, which causes each write to have to be flushed to disk. Turning this off helped reduce the time spent in the write call, since it's instead done less often and probably in the background by the OS.

Once again, as I write this entry, I think of things to try. Talking about how something as ancillary as logging affected responsiveness led me to think of other ways to gain responsiveness. I spent some time investigating doing my logging in a worker thread, and I'll discuss this spectacular failure in another post.

XML Loading

Man, I could write an entire entry just about this, but instead I'll cram it all in here. There are often many ways to tackle a performance problem. One way is to use a more efficient algorithm. That's not what I did here with my XML loading. I still make too many passes over the data. Oh well. What I did do is swap out the XML library I used.

I had been using REXML previously. It comes with all Ruby installations, so that's cool, but it's also pretty much the slowest XML library available for Ruby, at least according to various independennt sources and some articles. I decided to try out libxml-ruby, which is a set of bindings for libxml.

As with all Linux/Unix originating libraries, there are numerous dependencies that you have to resolve. I don't even remember now how many weird other libraries I had to find or build in order to get this to work on Windows. I guess on Linux this is all a cakewalk, but I'm "pampered" by now.

The speed is the real deal. After translating all the XML calls in my code from REXML to libxml, I got a 50% speedup, and this is on top of the logging work I did before. My total MUD startup time went from about 20 seconds to about 10. Loading a character carrying a bunch of nested objects also sped up very noticably.

libxml trickiness

But all was not totally simple with libxml. There is a peculiar phenomenon that manifested with it that I had to handle with care. If you look at the documentation for Document#find, it mentions that after you do any xpath query, you have to manually nil out the xpath result object you received and manually run the GC, or you risk crashing.

To explain it a little more, when you call find, which performs an xpath query, it takes a reference to the document from which it came. This is necessary to prevent the GC from freeing the document while the xpath result is still in use. Then, if you are done with the document and the xpath result at about the same time and release all references to them both, the GC will collect both of them probably in the same pass. The order in which it collects them is indeterminate. If it collects the xpath result first, then it's fine; but if it collects the document first, then the xpath result, which has a reference to the document, will access freed memory and crash.

I didn't much like the idea of running the GC manually all the time, so I came up with what I think is a pretty neat trick. It is based on the assumption that XML documents and xpath objects on my MUD are short-lived. That is, they're created, accessed for a little while, and then thrown away. I call my trick "document tracking."

In the first part, I make sure that any document creation is done through a single, central function. In it, I make sure to track the document object I return.

def XmlGeneral.createNewDocument(str = nil, isString = false)
:
    doc = LibXML::XML::Document.new()
:
    trackDocument(doc)
    purgeUnusedDocuments()
    return doc
end # function createNewDocument

def XmlGeneral.trackDocument(doc)
    @@docs.push(doc)
end # function trackDocument

Tracking is pretty simple, as you can see: I just keep track of the documents that have been created. Obviously, I don't want to keep them forever. The purging is the interesting part.

def XmlGeneral.purgeUnusedDocuments()
    ids = @@docs.map() { |d|
        d.object_id
    }

    # should free most or all of the objects dependent on documents
    GC.start()

    @@docs = []

    ##########
    # notice how this proc here is empty. we have to do this to get the
    # object space to update itself from what's actually present. otherwise
    # the _id2ref calls below will not work
    ##########
    ObjectSpace.each_object(LibXML::XML::Document) { |o| }

    ids.each() { |i|
        begin
            @@docs.push(ObjectSpace._id2ref(i))
        rescue RangeError
        end
    }

    DebugOutput.debugOut("doctracking: #{ids.length} objects before, #{@@docs.length} after\n", LComment | LLoading | LSaving)
end # function purgeUnusedDocuments

Let me walk through this. The strategy is to grab all of the object ids for the documents being tracked, then start the GC. This will free any xpath results that are not being used. These are the problematic ones I talked about above. By the end of this, the only xpath results left should be ones legitimately in use.

Then I throw away all of my references to the documents and try to get them back again, using _id2ref, which takes one of my previously saved object ids and tries to fetch the object for it. If the document is not being used anymore by anyone, then I won't be able to get the instance again (RangeError). If it's still in use, I'll get its instance back, still being tracked for next time.

Essentially, I'm doing the same thing as naturally happens--freeing documents and the xpath objects that refer to them--but I've split up this process into two steps to eliminate the nondeterminism.

In practice, this works pretty well. Every several documents I go through purges a few. I didn't see it ballooning upwards or anything. From my code snippets you may notice that I'm purging on every document create. I could very easily tune this to purge every n creations instead, which is probably what I'll do eventually.

At some point I will go through and actually make the XML loading algorithm faster, probably by building some kind of dependency graph in one pass and making one more traversal in such a way that I don't encounter any unfulfilled dependencies. That's the cause for multiple passes in the loading code currently.