Netmon for Debugging

On the project I'm working on instead of working on my MUD (since I haven't felt inspired about it lately) I'm in need of doing some protocol analysis. Yeah, network protocol analysis. I bet lots of people swear by Wireshark, but my tool of choice is Netmon, since I've used so much of it at work. Netmon has great parsers that let me drill in deeply to all the protocols involved in my project. I'm developing a server, and want to capture network traffic to see if my implementation is correct.

One problem is that Netmon doesn't work well when capturing loopback traffic. I do a lot of my development on the bus, with limited or no Internet access, so I tend to start both the server and client on my laptop. How, then, can I get my traffic into Netmon to use its cool interface to analyze it?

The method is pretty simple, really. There are three steps: have my code dump traffic to a file; consolidate those files into a Netmon capture file; and add to Netmon's parsers to recognize my new format.

Logging my traffic to files is pretty straightforward. In my case, I want to separate out the "messages", so I log only one per file. I order the files with a unique number, plus an indicator about whether it is a read from the network or a write to the network."

Then I wrote a program to read in each file and write it into the Netmon capture format instead. Netmon provides a really great API set, which has a reference included in the free download of Netmon.

This program was pretty straightforward. I created a capture file with NmCreateCaptureFile, read each file in turn into a buffer, and added it using NmBuildRawFrameFromBuffer and NmAddFrame.

I added a little "special sauce", if you will. Or if you won't. Whatever. In Netmon, the "frame parser" is the top-level parser invoked on every frame. It multiplexes the contents of the frame to different sub-parsers (Ethernet, Wifi, etc.) based on the NDIS_MEDIUM MediaType value stored in the frame (which is a parameter to NmBuildRawFrameFromBuffer). I don't want my frames to be parsed as Ethernet or Wifi or whatever, so I supplied my own value for MediaType.

We're now getting to the part about making Netmon recognize my format. The Frame parser has a big switch statement specifying how to parse the body of the Frame according to MediaType. I just need to insert a clause for my custom media type.

// in frame.npl
switch(MediaType)
{
    case 0:
    case 1:
        ETHERNET Ethernet;
    case 6:
        WiFi WiFi;
    case 0xFFFC:
        PayloadHeader PayloadHeader;
    case 2:
        TOKENRING TokenRing;
...
    case 0xFFFB:
        NetworkInfoEx NetworkInfoEx;

    case 0x2323:
        MungeTLS MungeTLS;
}

This works, and multiplexes to my custom parser, which I will show momentarily. But first, note that this method requires modifying or overriding the built-in frame parser in frame.npl. That's not very nice. The good people on the Netmon team must have had the exact same thought, because they added support into the parser language to insert case clauses into the parent parser's code to redirect it to a subparser--exactly what I did here, but in a non-invasive way!

[ RegisterAfter(Frame.NetworkInfoEx, MungeTLS, 0x2323) ]
Protocol MungeTLS
{
...

The first parameter to RegisterAfter is the clause in the switch statement after which the clause for handling MungeTLS should be inserted. The second parameter is the name of the field/variable to be inserted into the parent structure (the data type is implied by the protocol definition that immediately follows). The last parameter is the value to be used in the case clause to trigger redirecting to this sub-parser.

The model is a little specific, but there is a prescribed way to write these parsers, and it works very well.

Finally we get to the description of my actual "protocol", which I have lovingly called MungeTLS, and which I will expound on more in a later blog post. The main feature is that it wraps a proper TLS message and attaches a little metadata to the front. So far, the only metadata it has is the "directionality" of the frame (which of course is indicated by the "r" and "w" suffixes on the filenames I mentioned early in this post).

When my program generates the Netmon capture from the set of files, it writes the "direction" of the traffic into an extra first byte in the frame, which is parsed by the MungeTLS Netmon parser and used to indicate it in the UI.

const MediumMungeTLS = 0x2323;
const Flags_Receive = 0x0;
const Flags_Send = 0x1;
const Flags_Mask_SendReceive = 0x1;

[ RegisterAfter(Frame.NetworkInfoEx, MungeTLS, MediumMungeTLS) ]
Protocol MungeTLS
{
    [
      Property.Source = MTLS_Source(Flags & Flags_Mask_SendReceive),
      Property.Destination = MTLS_Destination(Flags & Flags_Mask_SendReceive)
    ]
    UINT8 Flags
    {
        UINT8 Unused:7;
        UINT8 SendReceive:1;
    }

    TLS InnerTLS;
}

Table MTLS_Source
{
    switch (value)
    {
        case Flags_Receive : "Client";
        case Flags_Send : "Server";
    }
}

Table MTLS_Destination
{
    switch (value)
    {
        case Flags_Receive : "Server";
        case Flags_Send : "Client";
    }
}

The Source and Destination properties are shown in a special place in the frame summary window in Netmon, usually occupied by things like the source/dest IP or MAC address. In this case, the strings "Client" and "Server" are shown instead, which is good enough for me!

More on Template Partial Specializations and Inheritance

Last time we talked about template partial specialization and some ways that it gets hung up. I posted a sort of solution which kind of sort of works, but in fact did not solve the problem I was having. I should have probably tried applying it to my code before posting it, huh? Such is the haste of impatient programmers.

The way I used the template last time was too simplistic. What I didn't do was include any class names in the template parameters, only values. It turns out this causes some interesting, unintuitive issues. Let's take a look at a code sample.

I apologize for the contrived, abstract example; I was having trouble coming up with anything good. We have a class called TimesN with a template parameter specifying what N is, which is baked into the class. It has a DoTimes function that takes the other number and does the multiplication. That other number can either be produced by some object or is simply a number itself, depending on the template parameter T.

#include <windows.h>
#include <stdio.h>

// Typically, T is a helper class that implements GetNum()
template <typename T, int N>
class TimesN
{
    public:
    virtual void DoTimes(T t);
};

class NumberWrapper
{
    public:
    NumberWrapper(int t) : t(t) { }
    NumberWrapper() : t(0) { } // aside: MSVC11 doesn't support peer ctors

    int GetNum() const { return t; }

    private:
    int t;
};

template <typename T, int N>
void TimesN<T, N>::DoTimes(T t)
{
    // just do the multiplication
    printf("t=%d, N=%d, t*N=%d\n", t.GetNum(), N, t.GetNum() * N);
}



// subclass and specialize for T=int, as shown in last blog post
template <int N>
class IntTimesN : public TimesN<int, N>
{
    public:
    void DoTimes(int t);
};

template <int N>
void IntTimesN<N>::DoTimes(int t)
{
    printf("t=%d, N=%d, t*N=%d\n", t, N, t * N);
}

int __cdecl wmain(int argc, wchar_t* argv[])
{
    TimesN<NumberWrapper, 3> x;
    x.DoTimes(NumberWrapper(6));

    IntTimesN<4> y;
    y.DoTimes(5);
    return 0;
}

This is exactly as prescribed in my last blog post, and that post worked fine, so this one should too, right? WRONG! When you compile this, you're greeted by:

main.cpp(28) : error C2228: left of '.GetNum' must have class/struct/union
        type is 'int'
        main.cpp(26) : while compiling class template member function 'void TimesN<T,N>::DoTimes(T)'
        with
        [
            T=int,
            N=4
        ]
        main.cpp(36) : see reference to class template instantiation 'TimesN<T,N>' being compiled
        with
        [
            T=int,
            N=4
        ]
        main.cpp(52) : see reference to class template instantiation 'IntTimesN<N>' being compiled
        with
        [
            N=4
        ]

If you read the error message carefully, you'll see it is trying to work out the implementation of IntTimesN::DoTimes. I suppose it makes sense, due to the inheritance we've put in place, that this would require instantiating the template for the superclass, TimesN<int>. And now the class TimesN<int> has to be compiled in full, irrespective of its use as a superclass, the same as if I'd simply declared an object of type TimesN<int>. And this requires compiling the DoTimes<int> method, which of course fails.

WHEW. Well, I was confused/annoyed by this before, but now it makes perfect sense to me. Writing helps solve problems, you guys!

Oh right, the solution? Yes yes, I worked around this. Thinking critically about what's going on, the problem is trying to compile the DoTimes<int> method, so the most direct route is to make sure this method does not have an implementation. That's right, we're going to switch up the inheritance. TimesN will become an abstract base class, and we'll have two child classes, ObjectTimesN<T, N> (which subsumes the former implementation from TimesN) and IntTimesN<N>, which stays the same.

The code, for your viewing pleasure:

#include <windows.h>
#include <stdio.h>

template <typename T, int N>
class TimesN
{
    public:
    virtual void DoTimes(T t) = 0;
};

template <typename T, int N>
class ObjectTimesN : public TimesN<T, N>
{
    public:
    void DoTimes(T t);
};

class NumberWrapper
{
    public:
    NumberWrapper(int t) : t(t) { }
    NumberWrapper() : t(0) { }

    int GetNum() const { return t; }

    private:
    int t;
};

template <typename T, int N>
void ObjectTimesN<T, N>::DoTimes(T t)
{
    printf("t=%d, N=%d, t*N=%d\n", t.GetNum(), N, t.GetNum() * N);
}



template <int N>
class IntTimesN : public TimesN<int, N>
{
    public:
    void DoTimes(int t);
};

template <int N>
void IntTimesN<N>::DoTimes(int t)
{
    printf("t=%d, N=%d, t*N=%d\n", t, N, t * N);
}

int __cdecl wmain(int argc, wchar_t* argv[])
{
    ObjectTimesN<NumberWrapper, 3> x;
    x.DoTimes(NumberWrapper(6));

    IntTimesN<4> y;
    y.DoTimes(5);
    return 0;
}

I've taken the liberty of highlighting the differences from the first code sample. I've actually incorporated this solution into my project and it is working just fine now. A lot of hair tearing, I tell ya what, but now me and template partial specializations are on good terms again.