Electronic, Program, and Algorithmic Trading

I'm baaack

2010-06-30T09:56:00.001-04:00

After nearly a 2-year hiatus from this blog, I'm back. I've been working hard on the trading system, things are going well.

Going to be posting some results soon.

IP4 Exhaustion

2009-04-06T13:31:00.001-04:00

Market fragmentation

2008-06-13T10:10:00.006-04:00

It's fascinating how the US equity markets have evolved as electronic trading has come into its own. One of the areas of particular interest right now is market fragmentation. First, a little bit of introduction on the topic for the uninitiated.

In the late 90's (before decimalization), there was a scandal that had to do with market makers not protecting their customer orders. In other words, a market maker would be making a market and then a customer order arrived that would better their quote. The market makers at times would fail to put the customer order in front of their own quote. The problem went even deeper than that, with market makers actually only quoting in even-eighths (i.e., they kept spreads artificially wide by having a gentleman's agreement not to quote in odd-eighths). Needless to say, when the SEC discovered this they were not amused.

I'm greatly simplifying things here, but what happened was that the SEC came down with various order-handling rules (informally called the "Manning Rule": http://en.wikipedia.org/wiki/Manning_rule). Limit order protection, limit order handling, and limit order display being chief among the rules, their intent was to ensure that all orders are handled fairly when in the hands of a market maker.

So how did the market makers respond? Well, they started dumping all of their "nuisance" orders (these are the customer orders that they now had to protect) into ECNs. ECNs, being designed to electronically handle massive amounts of orders, were ideally suited to carry out the market makers' obligations and make sure that the order was protected. What happened, though, was that the ECNs grew to the point of being larger than many of the market makers. A lot of this had to do with day-trading... With all of the resident liquidity from the market makers, all an ECNs needed was liquidity takers (day traders) and they had a nice bit of crossing volume. We all know the story behind Island, Archipelago, etc. Keep in mind that the ECNs did not participate directly in price discovery... this was still the province of the market maker (or exchange specialist). More on this in a moment.

So when decimalization came in, the scene changed drastically... Volumes skyrocketed, trade sizes decreased, and in general the market truly started to fragment. One one side of the fence, you had old-school institutions and brokers who both lamented the decrease in size (and spread!). They suddenly had to work for a living and actually add value on their respective trading desks. You can still bump into these guys, who fervently cling to the idea that decimalization was a Bad Idea. They'll even try to explain how it ultimately hurts the retail investor because mutual funds have to pass the higher trading costs on to them. These guys are the same guys who are mad that algorithmic trading is replacing them on the trading desk for certain functions. 'Nuff said.

So I just mentioned algorithmic trading; this is certainly a big buzzword these days. In a nutshell, algorithmic trading was developed to cope with the fast pace of the post-decimalized world. It started off with smart routing, evolved into basic algorithms (VWAP, Implementation Shortfall), and continues to evolve to this day. It was a way to block trade in decimalized markets that had no more blocks. But there was another solution to this problem: Dark-pools. A dark-pool allowed traders to put large blocks of liquidity into non-publicly displayed markets, and then put additional restrictions on them (minimum execution size, etc). This was a way of un-fragmenting the liquidity in the market; the other side of the same coin as algorithms.

But what is happening now should cause everyone to sit up and take notice. The dark-pools (there are more than 40 now, I've lost count) have grown to the point where they (collectively) have significant liquidity. And NONE of this liquidity is protected by Reg NMS, nor does it participate in price discovery on exchanges. Doesn't this sound like a step backwards?

Adding fuel to the fire are the industry participants who want to perpetuate (and further) this lopsided market structure.
Take this recent article, for example: here.

Seth Merrin, undeniably an industry pioneer, comes up with an incredibly wrong argument to make his point that Exchanges should route flow to ATSs. First, he draws an analogy between the GAP buying jeans from a guy on the corner. We all know that this is not how things work, because it would be very inefficient. But he uses this analogy to justify the argument that exchanges should route their retail flow to ATSs (oh, and he just happens to run an ATS by the way). First off, Seth, you need to go pick up an economics textbook and realize that in the tangible goods markets, there are costs that do not exist in electronically traded markets: Storage costs, shipping costs, and decay costs, for example. These and other reasons are why the GAP doesn't buy from a guy on the corner. With electronically traded equity shares, those costs do not exist, or they are quite minimal because of the efficiencies of electronic marketplaces.

So, what is the answer? I firmly believe in free-market forces. We need to abandon the notion that retail and institutional liquidity should be treated differently. We need to stop removing large amounts of liquidity from the price discovery process (like LiquidtNet and other darkpools do). That being said, I am VERY much in favor of competition, and so I am not advocating a Central Limit Order Book (CLOB). I do like Reg NMS, and all that needs to change is for ALL pools of liquidity to protect quotes ala Reg NMS. Also, all venues should be forced to display a quote (and thus participate in price discovery). Finally, note that these restrictions do NOT preclude venues from creating alternative trading models. For example, mid-point matching is still a very viable model, as is call-market auctioning (ala POSIT).

BTW you might be wondering why I am a fan of POSIT and not other parasitically-priced dark pools; it's simple: POSIT doesn't let you cancel or replace your order for a certain amount of time before the match, and they also start the match at a random time somewhat close to the published time. This greatly reduces gaming. Also, POSIT lets anyone in; buy-side, sell-side, and even retail (if they are big enough). LiquidNet, on the other hand, only allows buy-side shops in.

Ah, I'm done. Drop me a line and let me know what you think.

The probable improbable

2008-03-02T18:56:00.006-05:00

OK, as promised to my 1 riveted reader(s), (and no, that was not generated by a script... just my attempt to be funny), I am resuming the blog.

So, after a long trek down math lane, I've got three strategies that are operating now. And hell yeah, I was RIGHT to code my own infrastructure. First off, I am easily able to do things that (from inside knowledge) are not so simple with "professional" packages. And it didn't cost me anything (unless you count two years of my life). Assuming I make $0 an hour, that is a good bargain.

Don't think too much about that last statement. Please.

Anyhoo, I've learned several things (not all at once... this stuff takes months (that means 15-18, not 1 or 2) of live trading to learn):

1. Shit happens, so be ready for it. The part of your code that is commented with, "I'll code this later -- very improbable" will get run more often than you think.
a. You can get fills before ACKs and many other seemingly incorrect behaviors... be ready for them.
b. You can have orders "hanging" with no ACK, or REJ for quite some time (minutes if not longer!)
c. Be ready for odd lots. Plenty of them, mein froind. If you're trading less than round lots, I would (seriously) recommend doing at least 100 shares. Dealers LOVE you if you do odd-lots with them, and market structure rules don't protect you as much.
d. Don't assume that you get good fills from your broker. You often get fills waaaay outside of the market, and you have to phone them and have a FIX log pissing contest. Be a turd, because it's your money. 9 times out of 10 you are right.
e. Quotes frequently get stale. I could have a whole blog on data sources, scrubbing, filtering, and sampling.
f. Trades get printed late (as per the 90-second rule). Don't use trade volume or price as an "immediate" indicator.
f. Symbols change names. They get acquired. They delist. Other symbols take their place (oh, that was fun when that one happened).
g. Corporate actions. Learn it, live it, love it. "f" is really a corollary to this.

2. You model will break. Then it will SEEM to unbreak. Then it will break. Trust me, it was broken to begin with. Go back and see what was going on. Many times you have overfitted (i.e., biased your model to the data). To get rid of this, you can do out-of-sample tests. You may also have a model that is not explanatory. Add more indicators but only enough such that you don't overfit. I would recommend either the AIC (Akaike Information Criterion) of BIC (Bayesian Information Criterion) to make sure that the new indicators are worth it.

3. Don't count on all the academic papers/theories to be right. They're brilliant works to be sure, but they are very academic. It's good to use as a way of viewing a problem, not always as a way of solving a problem. Kind of like how all that trig you took is an interesting way of viewing triangles, but not a great way to build a pyramid

Here's some more tidbits.

First, trading is a LOT of psychology. Seriously. Listen to the guys like Jim Cramer with his rules. Another one I like is George Sleezak (futures trader, used to play with a guy who founded the band "Chicago". Very cool). This stuff is pretty sound advice. One of my favorites that a buddy (Chris H.) came up with:

"If you have a position, imagine yourself on the opposite side. If you feel comfortable, then get out NOW." This does have psychological underpinnings because we tend to see potential gains differently than potential loss in terms of risk.

Another one (very important, even for a guy like myself):

Reverse entry points are NOT exit points. Every trade should have a separate piece of expectational logic to decide when to ENTER and when to EXIT. Translation: If your model goes long a name, then generates a sell signal in the same name, you are going to get hurt if you use that ENTRY signal (in the reverse direction) as an exit point. Worse yet, as a reverse point.

Each trade is it's own journey. You exit the desert because you run out of water.... you don't exit Atlantis for the same reason. OK I was never that good at literature but hopefully you get my analogy.

OK now for the geeky part of things:

1. Trading is about (duh) maximizing gains while minimizing risk. What the heck does that MEAN though? Well, financial engineers will tell you it's about maximizing the Sharpe (or for some, Sortino) ratios. Simply not true unless you are trading for a mutual fund. More on this in a moment. Remember, these are marketing terms and mutual funds use them to raise money because they make a ton of dough on MAINTENANCE fees.

2. Why is it that GAINS and LOSSES and so easily quantified in terms of dollars, whereas RISK is such a slippery pig? BARRA will have one definition, Northfield will have another, there are many models out there. One thing is true though: If you completely eliminate risk from your portfolio, then you should (theoretically) have returns like a T-Bill but with more transaction costs and tax implications. Instead of using other people's models, how about this notion: What does risk mean to YOU? Think upon that and come up with an answer before you hit the button.

3. Diversification. Wow. Markowitz was a very smart dude, but there are a few key assumptions that we should ALL take a closer look at (disclaimer: he has a Nobel Prize and I don't. But I'm hopeful.) :

a. Most of modern economics assumes rational investors. This is why they (when talking about Paraeto efficiency) speak of purchasing a baskets of GOODS. Who would buy a basket of BADS? In truth, some investors are NOT rational.

b. Also realize that people either "can" or "should" do rational things. If you "should" do something but for some reason "can't", well then you have an inefficiency.

More on the next blog, I hope you enjoyed my humor and content (I think I'd prefer it, in fact, if you enjoyed them in that order).

No time like the present

2008-02-21T23:47:00.003-05:00

I've not blogged in a long while. I shall resume presently. Stay tuned. Many new updates.... just need to get them all down on e-paper.

Timer chains

2007-06-03T10:21:00.003-04:00

My last post dealt with how I added deep subscriptions and I hinted at timers. Well, I've just about gotten the timers working, and it was a pretty challenging task.

For starters, I am creating a distributed system (i.e., timers firing over a network). This means I have to create a thread to listen for timer requests. When a timer request comes in, I have to notify the timer thread (this is the one that blocks) of the new timer.

The tricky part is the timer thread itself. I used pthread_cond_timedwait() as the main blocking function; it basically lets me wait on either a signal from another thread OR a timer. Keep in mind that since that thread can only block on one timer at a timer, each new timer added to the system must "chain" itself in an in-memory structure. For example, if I have a timer that is set to expire in 5 seconds and then another thread requests a timer that will go off in 2 seconds, I have to block for 2 seconds (the new timer) and then block for 3 seconds (the old timer, minus the 2 seconds that have already passed). This isn't too difficult to code. The main thing to watch out for is timers that expire at the same time.

One interesting thing I found on multiprocessor systems with pthread_cond_timedwait() is that you can have spurious signals and timeouts (actually, one doc said you can get only spurious signals, not spurious timeouts, but I assumed that either could be spurious). Therefore, you can't assume that everything's cool when the call unblocks.

Lastly, timers are not exact. You can have a signal that expires JUST before a timeout, but because it's possible that slightly more time expired than the timer was set for, you also have a timeout that you have to simulate. Fun.

Timers and deep subscriptions

2007-05-27T15:56:00.003-04:00

OK, the next Big Project is to add timers to the DB. Why do I need this? Well, for starters, the order placement strategy needs them. I need to be able to create an order that will change it's characteristics (limit price) or cancel itself after a certain amount of time.

The trick about timers is that there may be more than one timer per stock. I won't get into the details as this is part of my strategy, but it creates an interesting architectural problem. To see why, let's review how my tables and subscriptions work.

A table has 2 descriptive characteristics:
TableNum: This is a unique integer that is the "name" of each table
NumFields: Each table has a nonzero integer number of fields.

In SQL databases, you would also have a type for each field (e.g., string, float, etc) but my DB only supports "double" as the type for a field.

In DB, when you read or subscribe to a table, you get ALL rows (i.e., there is no filtering like with the WHERE clause in SQL). This has worked well so far, because each stock has it's own tables for bids, asks, and trades. This means that for 100 stocks, I have 300 tables for quotes (each stock has 3 tables: bids, asks, trades)

This makes things easy from a programming point of view, but to add more stocks to the system, I have to change the DB schema, which is a pain. The right way to do things is to create what I will call "deep subscriptions". This means that every subscription will have a MASK that specifies which rows are interesting to the caller.

For example, suppose that I have a "trades" table that has 3 fields:
Time
Price
Size

Remember, the number of the table "encodes" which stock I'm looking at.

OK, so if I create a mask that looks like this:

*, *, 10000

then I am interested in all trades that have an exact size of 10000 shares. The asterisk is a wildcard an allows all values to pass through.

Well, being that I have limited the DB to only doubles, how can I specify a wildcard and yet not use up any values that I might want to filter on? Clearly I cannot simply use 0 or -1, as these are values that I might want to filter on. Well, there's a simple answer: Nan (for the uninitiated, a NaN is a special value in floating-point land that indicates that the number is Not-A-Number. This is different from inifinity. You would get NaN if you take the square root of a negative number. You would get inf when you divide a real number by zero). For information on this somewhat arcane subject, here ya go:

http://en.wikipedia.org/wiki/NaN

In the standard math.h file, there is a function called "nan()" that provides me with a "quiet-nan". By "quiet", it means that the FPU hardware will not generate an exception when this number is operated upon (e.g., a compare operation). There is also a macro "isnan()" that I can use to see if a number is a NaN.

Now, by simply altering the TableSubscribe() function to support a mask, I have "deep subscriptions" that are effectively like a WHERE clause in SQL. Note that I am limited in my comparisons.... In SQL you can say:

SELECT * FROM table1 WHERE field1 > 5

Here, the "greater-than" operator is used. I can only support the equality operator, like this:

SELECT * FROM table1 WHERE field1 = 5

But that's fine for now..... Baby steps.

So getting back to timers, I need to create a TIMER_REQUESTS table and a TIMER_ACTIVITY table which allow me to implement timers in tables yet selectively subscribe to only the timers that I care about.

With the subscription masks, this is now a piece of cake. The deep subscription feature also lets me compress my DB table structure to something much more maintainable and scalable.

I'll leave it to the reader to figure out exactly how I plan to implement timers. The trick here is that they need to provide the correct firings regardless of whether I'm running in realtime or when backtesting. I'll give you a hint: Check out the dynticks feature in the new Linux kernel. Same idea, sort of.

I'll have the prime rib, sir

2007-05-12T21:50:00.003-04:00

Hooray, ring one up for the good guys! Finally got all the signals working and I'm getting some great results. Doing a few thousand trades per day (!) and the alpha signal is consistently positive. The first day was 1.625%. The worst day was still positive.

What amazes me is that my risk manager (which uses several heuristic rules to keep me from getting into bad shape) is not having to do much. I've got all sorts of constraints in there, some of which are:

- maximum position per name
- maximum number of trades per name
- maximum amount of capital in use
- dollar neutrality

...and several other things which are part of the alpha signal, so forgive me if I don't mention them here ;-)

So far, not running out of capital during the day, which is another amazing thing. The holding period is so short, and the signal is by nature keeping things pretty capital un-intensive, so things are working out even better than expected. I'm now having to consider scalability -- which is a high-quality problem to have at this growth stage!

The next steps are some signal-tweaking, which is going to be all the easier now that I've tied myself into a nice graphing package. Truth be told, I can backtest until the cows come home, but the most effective results thus far have come from an intelligent combination of statistics, calculus, and -- you guessed it -- human intuition. A good idea is something you have to have to begin with... the math is only there to express it or evaluate it. I am not looking to create a strategy that is right 50.000001% of the time and then to apply a billion dollars to it. I'm looking for something that is right a much higher percentage of the time and which does not take up much capital for the same risk-adjusted return.

I'll get some more results out there when I have a larger sample.

Also, FYI, things ran perfectly smoothly even on Thurday's NASDAQ quote problems. You gotta love automation.

Backtesting

2007-04-08T23:39:00.003-04:00

Backtesting is something that you've got to do to get any trading strategy running correctly and profitably. Some basics:

1. Don't overfit. Make sure to have a good amount of data, but only tune your model coefficients against half of it. Then, after you've done your tuning, simulate against the other half. If you still get good results, that's a good thing. If you get crappy results, go change your model, because it's not predictive (or there was alpha decay, in which case you need to change your model anyway).

2. Backtesting is a HUGE performance hog. On a MacBook Pro (2 cpus) it takes me 7 minutes to get through 1 days tick data (100 symbols). This is pretty good, but I can speed it up -- there's a lot of disk I/O when you read the stored tick data into memory. I'm going to change it so that's a one-time hit. One other thing I architected was the ability to split tasks up across processes (using TCP/IP). This is great when you've got a ton of CPUs available, but networking overhead will kill you. I only use this is a production environment when I need to have stability (i.e., one process cannot crash another). For pure speed, though, getting rid of the TCP/IP overhead and getting everything into 1 process to maximize CPU usage is killer. With my architecture, no problemo. Then 1 CPU is used for the OS (GUI, etc) and the other CPU is solely dedicated to the backtesting.

Once I finish all my backtesting I'll post the results!

Getting GNUstep installed AND building a simple project

2007-04-08T23:21:00.003-04:00

Installing GNUstep -- OK, I read how you can just say "apt-get install gnustep" and that DOES work great! Good job guys. OK, now the gdomap daemon is running (which is a good sign that _something_ happened during the install... gdomap is the GNUstep Distributed Object handler, basically it supports NSInvocation). But how do I compile a simple Objective-C program?

TO BE CLEAR -- These instructions are for getting a "non-gui" program working (i.e., no AppKit).

Here are the steps:

a. (I assume you already ran apt-get install gnustep)
b. apt-get install gnustep-core-devel
c. apt-get install gcc
d. apt-get install make
e. apt-get install gobjc
f. In your .bash_profile, add "source /usr/share/GNUstep/Makefiles/GNUstep.sh"
g. Log out and log back in so that the environment variables get set
h. Now you've got to create a makefile for your program that you're going to make. Name it "GNUmakefile" so you know that it's not a regular makefile. Here are the contents for a basic one:

include ($GNUSTEP_MAKEFILES)/common.make
TOOL_NAME = main
main_OBJC_FILES = main.m
-include GNUmakefile.preamble
include $(GNUSTEP_MAKEFILES)/tool.make
-include GNUmakefile.postamble

Note that the preamble and postamble files are optional.... these do not need to be present, they are used in case you want to define your own variables but obviously you do not want to edit the master common.make or tool.make files.

Note that I used TOOL_NAME and tool.make -- You will find that this is how you get the FoundationKit stuff to link (i.e. libobjc).

One caveat to all of this: In my stumbling through everything, I also did apt-get on a few other things. I am not sure if these are needed, but I didn't know how to un-get them to check:

libobjc-lf2 (I have no idea what this is... Before using TOOL_NAME and tool.make, I was using objc.make and OBJC_PROGRAM_FILES, which gave me a linker error. I incorrectly thought this package would fix it. I don't think it's needed.)

gnustep-devel (I have no idea what the difference betwen gnustep-devel and gnustep-core-devel is)

Now, to test it out, create the main.m file, something easy:

#import "Foundation/Foundation.h"

@interface Cool:NSObject
int x;
@end

@implementation Cool
@end

int main(int argc, char *argv[])
{
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
Cool *c = [[Cool alloc] init];
return 0;
}

Yes I know this program is silly, but it tests to see if the whole mess works. It does (!!!!!!)

In summary, Objective-C and the FoundationKit are incredible IF YOU CAN GET THE DAMN THINGS INSTALLED. The documentation out there is incredibly variable, stale, and/or plain wrong. I'd love to donate my time to getthing this solved, but I am not sure my skills are up to it. Therefore, I wrote this blog entry. As far as I know, It's 100% correct with no hacks.

NOTE: On Ubuntu and Debian (upon which Ubuntu is based), "apt-get install GNUstep" works like a charm and avoids all of these hassles. Great work on that guys!

Linux vs OS X (performance)

2007-04-08T22:17:00.003-04:00

Holy crap, what started out to be a "simple" experiment to run everything on the latest Linux kernel was nothing short of a nightmare. Here's what happened (this story does have a happy ending though):

1. Had to decide which Linux distro to use. Ubuntu seemed easy, especially since it had a Live-CD (which other distros DO have) and I've been hearing good things about it. I have to compliment the Ubuntu team -- the installation was absolutely easy and flawless. It was running a 2.4.x kernel, which was not the latest though. Also, my CPU was always at 50% usage, even when the system was clearly idle. WTF?

2. I figured that maybe the right thing to do would be to upgrade to the newer kernel. This is not a hard thing to do (once you've searched the 'net for a decent newbie kernel-bulding guide). 3 or 4 tries later (based on different kernel building instructions from different people..... argh!) I still wasn't there. Oh well.

3. Guess what? GNUstep (which is the FoundationKit and ApplicationKit stuff you need with Objective-C) is not really available through the Synaptic package manager. Well, I think it's available with the unstable or testing versions of the distro.... which I didn't have. Crap. This doesn't affect most people but I need this package.

4. OK, I found out that Debian (upon which Ubuntu is based) does support GNUstep via the apt-get command. Hooray.... I went and downloaded the Live-CD for Sarge. Worked perfectly, the only issue was that the graphics card was in some low-resolution mode (VESA support only) and my wireless networking card was not supported out of the box. Yes, I did try to build ipw2200. What a joke.... Needless to say I did not succeed. Not because of the ipw2200 people (their driver definitely works because I was using it under Ubuntu). Let's just say that I couldn't figure out how to get the build process to see the kernel headers. Yes, I'm a newbie, but that's not the point -- it should be much easier if Linux is ever going to get to the mainstream.

5. Hey, wouldn't ya know it, Debian worked f-ing great. I was really impressed. The CPU utilitization issue was gone (nice) and other than having no wireless networking, very low-resolution video, and no sound, everything was great. In case you are wondering, I've got a Sony Vaio laptop, and it's about 2 years old. Pentium M 1.73GHz, 1Gig ram, GeForce 6200 card, Intel wireless. Sony model PCG-7A2L (which is not on their website for some reason).

6. I wanted to try to fix the video thing, so I tried going to the NVidia website and downloading their driver. Again, could not build it because I don't know how to get it to see the kernel headers (I did download the headers, in case you are wondering).

7. Oh well, I was happy with it -- I am using it as a number cruncher, not a graphing device. But hey, wait a minute, Etch (Debian 4.0) just came out! Must....have.....new....kernel. Geek lust.

8. OK, I downloaded the LiveCD, installed it with no problems....and low and behold, the dang CPU is at 50% again. Higher resultion graphics though..... But still, crap!

9. Reinstalled Sarge.

10. Inspiration while installing.... I bet the graphics driver is what's sucking up all the CPU. Must.....have....new....kernel. Back to the 4.0 LiveCD, but I installed this time without the Desktop option.

11. CPU issue is gone (hooray) but still no sound and no wireless. Ah well, I can live without those.

So what about the OS X versus Linux speed comparison? Well, there is no comparison. for CPU-intensive tasks, I have no idea why, but Linux is about 3 times faster than OS X. I was running the same app, same input data, etc. The OS X hardware was different, which makes a detailed comparison useless (different disk drive, CPU, etc). But I was running a single-threaded app, so the CPU thing was not too much of an issue. The disk, on the other hand, probably skewed the results quote a bit. Anyway, on a MacBook Pro (Core 2 duo @ 2GHz) the MacBook took 3 times as much time to run the same benchmark when I made it less CPU/FPU intensive. When I turned on all the internal CPU/FPU stuff in the code, the MacBook Pro skunked the (single CPU) Sony. I can't be sure, but I think the Linux kernel is faster on equivalent hardware by about a factor of 3.

Good job Linux guys, great stuff!

Simulating Reality

2007-04-07T06:56:00.003-04:00

So.... all's well here on the algo trading front. One thing that I've been working on adding lately is a full-fledged exchange simulator. Why is this needed? Well, for simple order placement strategies, you don't really need one. Every entry order is a limit order slightly through the market (to protect you from getting screwed by the specialist or market maker). Therefore, any exchange simulator you write would pretty much just execute the order at the inside. This assumes that the order is fairly small, of course. It also assumes that you "reach" the quote before any other participants.

When you add more sophisticated order types to the picture (limit orders, stop orders, trailing stop orders, etc) then you have to hunker down and write a full-fledged exchange simulator.

So the first thing I did was create a model to represent the "real" market -- quotes coming in from either a saved file or from the live network. This wasn't too bad, just a two-tiered linked list with some hashing to make it O(1). Next, I created a separate structure to store limit orders. The structure for storing orders is almost identical to the one that stores quotes. The only real difference is that the quote structure stores the orderID and senderID (in case you have multiple strategies running, you want to make sure the fills go back to the correct strategy). The quote structure has to store the market maker ID (or exchange ID).

Voila! After a little codiing, I'm all set. Just for grins, I benchmarket the object at around 6.8MM quotes per second (on my laptop). Plenty good.... Now I'm going to get back to some serious backtesting, but now with cool limit and stop orders!

Up and Running

2007-03-02T01:08:00.003-05:00

It's been quite a while since I posted to this blog, mainly because I've been busy with the real part of the trading system -- the math. I have quite a few experiences that I can share, so here goes.

First, you have to have an idea about what in the heck your model is going to be about. Are you doing some simple "technicals"-style strategy? For example, is the strategy based on moving averages or Elliot Wave theory or Relative Strength Indicators? (If so, I submit that you are following a herd mentality and you might want to rethink your choice to become a trader). Sometimes this indicators are useful because other people think they are useful (i.e., a self-fulfilling prophecy) but in general they are more explanatory than predictive. Just watch CNBC and you'll see what I mean -- any self-respecting technical analyst can tell you exactly why the market did what it did. And they have lots of impressive vocabulary (head and shoulders patterns, breakouts, etc). But few if any of them will tell you which pattern is going to be useful to predict tomorrow's movement. Or worse yet, they will hedge their bets by telling you that "one of these three patterns will emerge tomorrow". And each of those three patterns describes something different. Therefore, no matter what happens, one of those three patterns will match what occusr in the market. Talk about having all of your bases covered.... (steps down from soapbox).

So, if you are going to create a model, think long and hard about what it is you are trying to do. What data are you going to look at? How are you going to scrub it? How are you (if at all) going to sample it? At what intervals?

Then you have to get around to creating the actual model. A good model is trying to express some fundamental truth about the data it describes. A good model is not a curve-fitting exercise -- anyone can do that. Which brings me to the final point, which is that you must backtest your model and then test with out-of-sample data to see how you did. This should be obvious to all but the most casual statisticians.

Ahhhhh, so with all that in mind, this is exactly what was done. Months of it. From Mathematica to Excel to hand-coded stuff, I used every tool in the book. To visualize the data I even wrote my own graphing application (thank-you, Objective-C for making that easier than in would be in any other language, IMHO).

And the result is a working strategy, ready to be tested in the market. Tune in next week to see how it goes.....

Sounds good to me

2007-02-13T02:00:00.003-05:00

One of the main design pricinples behind the whole project is to get human beings out of the way in situations where they would only introduce errors. But it would be a bad idea to completely eliminate the human brain from the process entirely; it is, after all, quite a bit more powerful than any compter ever invented. The trick is to present information to the brain in a way that the brain can process it accurately and usefully.

There are two forms of input that the brain can process thousands of times more quickly and accurately than a computer: Visual and Auditory. I've already got the Visual stuff covered (apologies if I don't put details here, but it is a proprietary trading system....). Next for the audio.

To be honest, this was a piece of cake. All I had to do was find the correct sound funcitons in CoreAudio and Cocoa, specifically NSSound and all of the technologies in CA (like AudioUnits, effects, etc). Apple has done a very good job designing this and it shows.

Figuring out which sounds to play and when is the real trick. There are a lot of peculiar side-effects that the human brain brings into the picture. Desensitization to repeated sounds is probably the most obvious. Also, tone separation is key -- the human brain has a hard time separating tones that are harmonic, but it is fantastic when detecting atonality. To see this, go to a keyboard and play a C-major. Now play an inversion -- they will sound very close, except that one will have a higher dominant to the tonic (I'm not sure I'm using those terms correctly, but it means that even though they are the same chord, one will sound higher). Next, play a C-7. Ouch.... This is so very different than the two inversions of C-Major, and your brain can easily distinguish between the two.

I'll leave the rest to your imagination....

DB is done (version 1.0)!

2006-10-28T16:53:00.003-04:00

OK the database is officially at version 1.0, which means that I'm satisfied that it's ready to go into extended use for more trading. I haven't posted in a while, but needless to say it's been running (with some nudges from me....) for a while now.

Changes to finish off the DB include:

1. Multithreading the server, which was easy with pthreads
2. Read and BulkRead commands that respond with Write commands to the caller (asynchronously, of course)
3. Remote subscriptions (this was more of a pain than I thought)

But it's done!

I've already run some tests and I'm getting some good results. Here's what the system does:

1. Loader parses a line from a pre-recorded input file (human-readable text)
2. Loader serializes a "write" command and sends over TCP/IP to the DB process
3. DB inserts it into the proper table
4. DB serializes a "write" command and sends over TCP/IP to the Subscriber process
5. Subscriber parses the write command (and then does nothing)

Three processes communicating via TCP/IP and using a human-readable textfile as the input (needless to say, much of the system's time is wasted in a silly sscanf() routine and very little time is spent in the DB or Subscriber code).

Let's just say it's pretty friggin' fast, plenty for what I'm using it for, and there's also plenty of room for speed tweaks later down the road.

Objectifying the database, and reading the news (RSS)

2006-10-12T21:13:00.003-04:00

Ugh.... just spent several hours doing something that I originally didn't think I would have to do -- eliminating the C API and shifting completely to C++. The need to do this became apparent when using proxy objects to serialize messages on the wire. You want to re-use the code that deserializes the messages, (call this a "Deserializer" object. Well, since bytes came come willy-nilly over the network, this object is stateful (it may contain a partial message and have the need to resume parsing where it left off). Okay, that was not so bad. Actually the code looks a lot cleaner now that it's C++. I stand by my decision to initially start off with C though -- there were a lot of things that (at the time) would have been very hard to design correctly, mainly because I've had to readjust my assumptions as I've progressed. Any initial attempt to UML-design this thing would have been disastrous (perhaps I'm just not a skilled designer....). Anyway, it's fixed now!

As a side project, I now have the ability to read news. This was very, very easy -- a couple of lines of code that use:

NSXMLDocument
NSXMLElement
NSArray

and you're done. Voila, I can now read any news I want and scan for keywords (like "scandal" or "lawsuit".... hehehehe)

Benchmarks

2006-10-03T05:52:00.003-04:00

The more I use the Cocoa frameworks, the more I realize that event-based programming (which requires a run loop) is the way to go in terms of making an application robust. Right now, the database is event based and uses a select() statement as it's message handler.

This has several problems, not the least of which is the fact that select() statements can't look for other types of messages (such as application-based messages) without doing something silly like creating a semaphore or timer and using that as an intermediary. Yech.

So this got me wondering, I wonder how much speed does one give up by using Objective-C message passing (i.e., what is the overhead of ObjC_msgSend() ? To test this out, I downloaded this guy's code from the 'net (I don't have the shortcut anymore but I'll make sure to give him credit once I find it again) which gives a program 4 seconds to see how many function calls it can make. I also wrote a C-version (which obviously gives up the notion of encapsulation) and a C++ version (which uses vtables).

Using a cached selector (IMP .... stuff) for ObjC, here are my results for number of "function calls" per second:

ObjC = ~ 135MM / s
C++ = ~ 186MM / s
C = ~ 348MM / s

Well, this is pretty conclusive evidence that there's overhead to ObjC and C++. I'm now officially glad that I used C instead!

Objectively Bridging the Gap

2006-10-01T13:45:00.003-04:00

Wow. I think my brain just exploded. I have a database written in pure C, coupled with serializers/deserializers written in C++. These communicate over sockets code written in C. This arrives into the Cocoa app via some objective-C++ code, which gets the info into the GUI using and Objective-C message. And it all worked the first time around. Whoa.

As I refactor my code, I am slowly realizing that the simplicity of development that you get with C is soon outweighed by the code reuse and encapsulation of C++. This contradicts my previous decision to keep the database in pure C, but I do stand by my initial decision to prototype in C. The code was quick and easy to write, and I didn't get bogged down with all of the C++ crap (virtual destructors, new and delete, etc).

All in all, I'm very impressed with the way Apple has tied together Objective-C and C++ (they call this Objective-C++). If you know both languages, it makes perfect sense and is very intuitive. I get the best of both worlds -- with C++ I get speed and direct access to memory via pointers (yes, they are useful sometimes, Java people), and with ObjC I get true OO.

Sidenote: I do not consider C++ to be as pure of an OO system for reasons beyond the scope of this blog, but start with the fact that C++ does not have a runtime and so there are many things that you cannot do at runtime. Introspection is a kludge, and everything is strongly typed and bound at runtime unless you hack around with pointers.

Visualizing the problem

2006-09-30T20:12:00.003-04:00

Orignally, I wasn't going to have anything that graphs the data in the database. It's a black-box strategy, why do I need to look at something that operates so fast that I would not be able to keep up? I was wrong. More and more, I know that I need a GUI to visualize what the strategy is doing. So, based on the platform that I am on I started a new program, DBVisualizer.

Having used all of these in the past on various platforms (mostly Windows):

VB / ActiveX
C++ / MFC (what a joke)
Java (both AWT and Swing)
C#
Web (AJAX)

Here is my conclusion:

VB is out as I do not consider Basic a real language (sorry, folks.... it's very powerful now in 2006, bit it still has remnants of it's past that I can't ignore)

MFC - outdated (is it even still supported?)

Java - Again, I can't stand Java (as a developer.... as a user, I think it's wonderful)

Sadly, I do not know C# but I hear very good things about it.

Web - I don't know AJAX and I really have no need to put this program in a browser

....and the winner is.....

Cocoa, which is the API for GUI apps on Macs. For those who do not know, has an interesting history:
Cocoa Wikipedia entry

If you like pure object-orientedness (think Smalltalk) and rapid development on a rich set of frameworks and widgets, this is something you should check out. Only runs on OS X though. Well, this is not entirely true, check out GNUStep

=======

OK, so I've been working on this thing for a few weeks. I must say, although I knew a lot about Foundation Kit (the framework that provides basic classes like strings and dictionaries, etc), I had a lot to learn about AppKit (the framework that provides GUI elements).

Interface Builder is awesome. It gives you everything that a UI development tool should, and it's very intuitive and easy to use. NeXT had this technology back in 1993!!! Talk about being waaaay ahead of Microsoft.
Openstep, the cousin to NeXTSTEP

XCode is awesome, but there's one thing that I pray they fix in XCode 3: When you create a class that implements delegate methods of another class (perhaps designated via a formal protocol), XCode should have a dropdown that allows you to automatically insert (in both the .h and .m files) delegate methods. Right now I have to cut'n'paste out of the documentation. C'mon, this would be easy to do and would save a ton of typing! (If someone else knows how to do this, please drop me a line... perhaps there's some feature of XCode that I don't know about).

The one problem that I ran into had to do with raw sockets code.... I did not want to write BSD sockets code and then somehow try to shoehorn that into the run loop. And, BTW, I found out the hard way that you do not want to even go near the NSConnection/NSSocketPort stuff. Those are not intended for raw messages over TCP/IP. They are specifically designed for distributed objects. And believe me, I spent hours ignoring all the advice on the 'net to that effect. I read through all the documentation and kept trying because I was convinced that there was a way to make it work and take advantage of all the Cocoa goodness inherent in being part of the Foundation Kit. It won't work, I'm telling you.... And if you do go through the trouble of putting raw data into the NSPortMessage object, the result is just ugly. It's not bad engineering on Apple's part, they just didn't intend for the classes to be used that way. The answer, and I'm pleased to say the code this guy wrote works excellent (and is well-designed), is to use NetSockets (here's a link to the developer's website, please check it out as he looks quite talented in his coding skills so his other stuff might be good too):

Blackhole Media NetSockets page

I owe this person a big thank you.... After hours of digging through the distributed objects stuff, his stuff solved my problem after literally taking only about 5 minutes to read his RTF and write some code. Bravo, well done. I did run into some problems when using 2 NetSockets in the same run loop, not sure what was happening but I got a memory access error that occurred in his code. If I can find it (and if it's not a bug in my code) then I'll try to help him patch it, in the spirit of giving back!

Speed or Stability? Both!

2006-09-22T17:43:00.003-04:00

I finally got the code running that allows me to write a strategy, and then run it either it out-of-process (OOP) or in-process (IP). What this means is that I can bring up the database and then open/close the OOP strategy willy-nilly without fear of crashing the database. When the strategy is STABLE, then I move it in-process (IP) and backtest it. The speed you gain is unreal, since you are completely bypassing the TCP/IP stack.

Yes, I know, I could have used shared memory and had the same speed along with the process separation, but that would not allow me to distribute the processes across machines, now would it? Also, shared memory protocols require that you basically manage all the buffering (and access) yourself, unless you use a library (there are several on the Internet if you Google for it). TCP/IP is a wonderful IPC mechanism that is stable, high-performance, and allows you to distribute across a network. Perfect.

Oh, and by the way, the coolest thing is that the strategy code is completely unchanged, regardless of whether it's running IP or OOP. Again, no transcription error.

It's all about the math

2006-09-16T18:56:00.003-04:00

Now for the fun stuff... I've got my equations plugged into Mathamatica (a GREAT product if you've never used it!) and I've solved for my coefficients (these are what the backtesting will "tune"). I put it in CForm[] and voila! C code for my equations. I cut and paste into a file and put a function wrapper around it.

I am of course leaving out MANY details, but that's for 2 reasons:

1. My strategy is private silly! Drop me a line if you want to chat about trading strategies, I'll go into more detail in a private conversation.

2. I didn't just "put a function wrapper" around an equation... Ask me for details. This has to do with what type of function you've created and what you do with the outputs. For example, if it's a simple signal, then somewhere else you will decide _what_ to do with the signal (for example, compare it to a value and buy/sell based on that). This starts to get into backtesting, I'll cover more of that in a later blog.

NOTE: There were a few things that Mathematica didn't do right like for example using the Pow() function instead of Power(). This is easily fixed (remember, no cut and paste! use inline functions to that call the correct functions instead). I am really paranoid about doing any cut-and-paste to the Mathematica code at all.

Mind the (data) store!

2006-09-16T18:40:00.003-04:00

Next up, I'm going to need some sort of database to store all of my preciouussss numbersssesss (Gollum). The idea I have is to use a fast database (I'd love to have Kx, Times Ten, or Vhayu but I don't want to spend that kind of dough!).

Over the course of several days, I tried:

MySQL
BerkelyDB
SQLite
(and since I'm on a platform that supports it, Oracle -- the free version)

Keeping in mind that I will be using tick data, so the memory usage and latency/throughput are important.

=================

I am sorry to say that the only database that performed at all well was BerkeleyDB. Oracle performance was a joke. MySQL (which can be configured to use BDB underneath) is trully a wonderful product but it's too slow, even when using in-memory tables. SQLite is much, much faster than mySQL as you would expect (it also doesn't do as much), but it's still pretty slow for realtime usage.

BerkeleyDB is FAST but for reasons I won't go into here, it seemed like the wrong solution. Please write me if you are curious, and I'll happily share my findings with you. It's a FANTASTIC product, and I was 99% about to use it.

So what did I go with? Man, I'm gonna get flack for this, but I'm rolling my own. What the heck would I do that for?

- I don't need any SQL functionality at all. I don't need foreign keys. I don't need atomic transactions, rollbacks, and all of the other things that a robust SQL engine would provide. This does NOT mean that I'm throwing disaster recovery out the window, of course (that would be foolish indeed, when developing a trading system!).

All I need is something this is:

"A lightweight process that uses a TCP/IP-based API that supports realtime storage and retrieval of time-series data (no strings or other types of data), and which has facilities for TCP/IP-based hot failover"

And yes, I know about ACE and Boost. It's much more than I need, and it will run slower than what I'm going to build.
Also, God forbid I run into a bug in their code, I would have no idea how to fix it and I cannot be beholden to them to turn things around quickly. Great products, but I'm going to pass.

So here's where we are so far:

1. Custom C++ client that speaks the TWS API protocol -- done!
2. C-based "database" that is very quick and stable.... in progress

Incidentally, the database supports a C-based API (not CPP) because I don't want to waste time in vtable lookups and I hate dealing with constructors and destructors when all I really want is encapsulation. Also, I support an "in-process" API for code that is linked with the DB process, and also a "proxy" API that is exactly the same but goes over the network. Easy and simple!

The people vs. Java

2006-09-16T18:22:00.003-04:00

OK the Java samples that IB gives you work just fine, and they are easy to build. The class design is quite logical (EClientSocket, etc) and obviously you are all set when new versions come out -- just copy over the old versions and recompile your code.

So why can't I get the damn thing to work with my own code? Perhaps I am a dinosaur and I don't grok Java, but I am sick and tired of trying to figure out what the CLASSPATH should be and why the IB examples work and my own do not (unless I put them in the same directory as the IB code). I tried with hand-coded compiles and I also tried downloading an IDE and letting it figure things out (I used Eclipse). No dice. The JVM can find my files just fine, but it can't find the IB code. What is the point of the fucking CLASSPATH if it only works when you are executing from the same directory as your files? And BTW I went to the web and looked at many examples of how to set the CLASSPATH -- apparently I'm not the only one who thinks this is a bit confusing:

http://www.kevinboone.com/classpath.html

Check out his phrase "widespread lack of comprehension" before you call me an idiot, thanks. Java has many nice things going for it but ease of VM use is not one of them and I have run out of patience, spending hours on something that should take 5 minutes.

So I instead bullheadedly read through the Java code and use it to create my own C++ version. Snap in a little sockets code (not complex at all, you really only need to be able to send and receive NULL-terminated strings). Seriously, it was only about 30 minutes of coding, and that includes all of the client-negotiation stuff at the beginning of the session, and it also handles dropped connections (and tries to reconnect). Best of all, this process is TINY as it requires no VM and links to multithreaded shared system libraries. Perhaps I am a dummy but I think the Java VM sucks (the language is nice and the frameworks are rich, but I'm not sold on using something that was originally (Oak) designed as an embedded stack-based interpreter).

The jury rules against Java ;-)

...and the winner is...

2006-09-16T18:13:00.003-04:00

Interactive Brokers. The commissions are right, they have most markets (and asset classes), and they have a clean and robust API. If anyone out there knows of other brokers that have similar funcitonality and lower prices, let me know please, thanks!

Their TWS application is written in Java so the UI is a little slow , but overall it's a nice application that does all the basics (order management, position keeping, charting, news, etc).

Their API is interesting.... They give you a choice of Java, ActiveX, C++ (a DLL), and DDE. Since I'm not coding in Excel I see no reason to even consideer DDE. ActiveX is a bloated abomination, probably only there for the legions of VB "programmers" out there. C++ and Java are the only options for serious coding, IMHO.

Well, now for the shocker: Windows is no match for Unix in terms of stability . I'm not putting my trading platform on something that has a registry, DLL-hell, and many virii. This is a matter of choice, and it's a very polarizing topic, but I choose to use Unix (I'm going to leave out which particular distro because that's an even more polarizing argument -- write me if you are curious, I spent many months comparing just about everything, so if you are curious I'm happy to share my results).

So by using Unix, I can't use DLLs (I refuse to use WINE or any other such emulator) and therefore C++ is out. Java it is, then!

OMSes, brokers, and languages, Oh My!

2006-09-16T17:45:00.003-04:00

I have already been working on my trading engine for months before starting this blog, so here's an update on my current status and the decisions I have made on the way to getting here.

The first thing I had to decide long, long ago was exactly what I am trying to build. The best definition would probably be: "A trading engine that supports ultra high-frequency trading and which lets me create, backtest, and use any conceivable strategy".

Decision #1 (I can hear the groans from the reader already): I am not going to use any of the software that I found on the 'Net. Why am I reinventing the wheel? Well, take a look at what's out there. There is some extremely high-quality software (TradeStation, Interactive Brokers TWS, REDI, RT, and even the institutional stuff like FlexTrade, Triton, InfoReach, Portware, and TradeFactory). All of these handle the OMS functionality that would be needed and most of them have some sort of language that can be used to code strategies.

The first problem I have with (some of ) these platforms is something I call "transcription error". When I use Mathematica (or Maple, SAS, etc) to create a model for trading (and by this I mean only the equations, not the backtesting), I use automatic methods of code generation. For example, in Mathematica I use the function CForm[]. Then I cut'n'paste the results into a C file. Hence, no transcription error (unless I screw up the cut'n'paste, in which case I am a moron). Believe me, this is an important feature because some of the equations I deal with a quite, quite complex and can fill up a page. You do NOT want to retype that and risk a typo. You also do NOT want to search-and-replace function and/or variable names... You will end up with something that is not what you think it is. Bottom line: I do not want to code my strategy in one of these platforms' custom languages, no matter what. To much risk of transcription error. Use a standard language that Mathematica can generate.

Also, I know that most of these platforms have APIs that I can use to interface directly with their trading engine. I could interface with the platform using any language with the proper bindings. So the next thing I need to ask myself is, should I just pick one of these platforms and do that? Hmmm, in the sort term the answer is a definite YES. You simply get too much bang for the buck doing this, and I do not want to write a FIX engine and OMS when there are more important things to get done first.

My Trading Engine

2006-07-28T16:45:00.003-04:00

This blog is specifically meant to document my journey of creating a full-fledged electronic trading application for my own personal use. This includes the technology part as well as the formulation of strategies, and I'll even document my trading days. I started this blog *internally* many months ago, and I've just now dumped everything to the website, so many of the initial dates will be the same day.... Keep in mind that there were many days or weeks inbetween those posts.

If you are a trader and/or don't want to read about the technology, I would skip to the posts that say something about math or trading. Start with "It's all about the Math"

==========

For those of you out there who have built something similar, I'm sure you'll get a chuckle as you watch me make my mistakes and learn from them. Please feel free to write your comments and help me out whenever you see me about to trip!

One thing I'm also going to do is be very opinionated about the industry and it's practices. I'm a guy that's been on the Street for a while so I will definitely have a few things to say about what goes on. Again, all comments are welcome.