Option data captured into database

I managed to capture stock option data into database. I made two mistakes and found the tricks to get it working. The first one was to set expiration Friday instead of the technical expiration Saturday, e.g. stockOption.setExpiration(friday). The second one was to add ib.genericTickList=101,100 setting into conf-base.properties. Below is the IB documentation on Generic Tick Types:

http://www.interactivebrokers.com/php/apiUsersGuide/apiguide/api/generic_tick_types.htm

Integer ID Value	Tick Type	Resulting Tick Value (see list below)
100	Option Volume (currently for stocks)	29, 30
101	Option Open Interest (currently for stocks)	27, 28
104	Historical Volatility (currently for stocks)	23
106	Option Implied Volatility (currently for stocks)	24
162	Index Future Premium	31
165	Miscellaneous Stats	15, 16, 17, 18, 19, 20, 21
221	Mark Price (used in TWS P&L computations)	37
225	Auction values (volume, price and imbalance)	34, 35, 36
236	Shortable (see “Tick Values” for more information)	46

OHLC Bar

I think I have missed one important element in my EPL, the OHLC Bar. That will most probably useful to mimic candle stick chart with some intervals.

• https://code.google.com/p/algo-trader/wiki/AlgoTraderDocumentation#Market_Data
• https://groups.google.com/forum/?fromgroups#!searchin/algo-trader/ohlc
• https://groups.google.com/forum/?fromgroups=#!searchin/algo-trader/ohlc/algo-trader/vZ5UBeLMkYE/enccRWIrJ2IJ
• https://groups.google.com/forum/?fromgroups=#!searchin/algo-trader/ohlc/algo-trader/JwLFM69vdGU/2vSmbNiiFIAJ
• https://groups.google.com/forum/?fromgroups=#!searchin/algo-trader/ohlc/algo-trader/8NvhNQzh9x0/uuz-H6Eq5KsJ

select * from Tick.std:groupwin(security.id).custom:ohlcbar(dateTime, last); 

select
        first(open) as open,
        max(high) as high,
        min(low) as low,
        last(close) as close
from
        Bar.win:length_batch(2);

 select
        first(price) as open,
        max(price) as high,
        min(price) as low,
        last(price) as close
from
        OHLCTick.win:time_batch(1 min)
group by
        ticker;

insert into Bar
select first as open, last as last, min as low, max as high,
* from Tick.std:groupwin(security.id).custom:ohlcbar(dateTime, last);

Smooth RSI

I was able to get RSI values from below EPL.

insert into RSI
select talib("rsi", currentValueDouble, 14) as value
from Tick(security.symbol = "GOOG");

But the series of values were not smooth. I wanna try to get smooth RSI line chart. Perhaps I can take the SMA as input for RSI, like this:

insert into SMA
select talib("sma", currentValueDouble, 5) as value
from Tick(security.symbol = "GOOG");

insert into RSI
select talib("rsi", sma.value, 14) as value
from pattern [every timer:interval(2 minutes) -> sma=SMA];

Or the other way? Making a smooth SMA line chart from RSI values.

insert into RSI
select talib("rsi", currentValueDouble, 14) as value
from Tick(security.symbol = "GOOG");

insert into SMA
select talib("sma", rsi.value, 5) as value
from pattern [every timer:interval(2 minutes) -> rsi=RSI];

Hacking TA-Lib

TA-Lib source code is available from http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-msvc.zip

I opened the com.tictactec.ta.lib.Core class. The RSI method signature looks like this:

   public RetCode rsi( int startIdx,
      int endIdx,
      double inReal[],
      int optInTimePeriod,
      MInteger outBegIdx,
      MInteger outNBElement,
      double outReal[] )

I think I made a mistake in double inReal[]. I was supplying single latest stock quote. Instead, I think I'd better supply an array of current and past stock quote in specific interval.

But when I supplied the array as a parameter, AlgoTrader can't compile the EPL. I asked this in AlgoTrader forum https://groups.google.com/forum/?fromgroups=#!topic/algo-trader/PtaMF04ay6Q

Below is the reply from Andy:

You should only provide the scalar value (tick.currentValueDouble). The GenericTALibFunction will collect all values and provide them as an Array to the talib function. Have a look at the JavaDoc of GenericTALibFunction.

Open Source Trading Platforms List

http://www.ta-lib.org/hdr_dev.html
A few of the products using the open source TA-Lib. If there we charting features in these products, I could probably hack into the code to see how TA-Lib get called.

http://www.ta-lib.org/hdr_lnk.html

http://www.traderslaboratory.com/forums/tools-trade/11086-open-source-trading-platforms-master-list.html

http://algotradingindia.blogspot.in/2012/05/open-source-trading-platforms-list.html

TA-LIB sucks

Well, I started writing my first strategy in Esper EPL. Not so difficult to reach a "Hello World" happy flow.

But too little documentation for TA-LIB http://www.ta-lib.org . No idea how to customize the parameters of MACD, RSI, STOCH etc to represent 2-minutes candle stick chart.

I think I better source other ways to generate Technical Analysis indicators. Let me hack into AIOTrade's http://aiotrade.com/ code to find out if I can do full customization on those Technical Analysis indicators.

Oh ya, that also bring the hints that I am going to use AIOTrade as my front end for AlgoTrader. Cool?

Esper Complex Event Processing

I have skimmed through Esper CEP official documentation quickly http://esper.codehaus.org/esper-4.6.0/doc/reference/en-US/pdf/esper_reference.pdf .

The EPL syntax is quite similar to SQL, with plenty extension. I think I'm ready to use EPL. Now I can understand the existing EPL used in AlgoTrader.

To develop strategies using EPL, I need to know TA-LIB http://ta-lib.org/ . I couldn't find many documentation about TA-LIB. Below are some simple documents:

http://www.ta-lib.org/function.html
http://qtstalker.sourceforge.net/talib.html
http://tadoc.org/
http://www.heatonresearch.com/content/technical-analysis-library-ta-lib-tutorial-java

AlgoTrader, I'm ready

By following the guide http://code.google.com/p/algo-trader/wiki/AlgoTraderQuickStartGuide , I got AlgoTrader binary run successfully. Though in Linux environment, I got to touch up the shell script, which is duplicated from Windows batch script.

I downloaded the source code according to the svn link in http://code.google.com/p/algo-trader/source/checkout . I made use of git-svn, so I can have the history in my local machine. In Eclipse, I installed the E-Git and J-Git plugins.

With the official documentation http://code.google.com/p/algo-trader/wiki/AlgoTraderDocumentation , I got my development environment setup. I also got the AlgoTrader "launchers" run in Eclipse. I spent some time skimmed through all the source code. Beautiful code, impressive!!

When I wanted to collect data (every 5 seconds) through Interactive Broker API, I got "duplicate record" error. Initially I thought Esper statement was propagating datetime with wrong precision. Later after I searched the word "duplicate" in https://groups.google.com/group/algo-trader , I found the thread "Duplicate entry error when persisting tick to DB". Andy Flury's replied "tick.setDateTime(date) in handlePersistTick sets the rounded Time".

Finally, I collected data every 5 seconds by doing the following modifications in com.algoTrader.service.MarketDataServiceImpl class's handlePersistTick(Tick tick) method:

            // get the current Date rounded to MINUTES
            //Date date = DateUtils.round(DateUtil.getCurrentEPTime(), Calendar.MINUTE);
            //tick.setDateTime(date);
            tick.setDateTime(DateUtil.getCurrentEPTime());


What's next? I guessed I got to spend sometime reading Esper Complex Event Processing's documentation at http://esper.codehaus.org/esper/documentation/documentation.html . Esper CEP is the heart of AlgoTrader, the soul of strategies building.

Paying for real-time market data

At Interactive Brokers, I got to pay USD 10 per month for real-time market data (US Securities & Commodities Non-Professional Bundle). Therefore, I must make full use of the money I'm paying for.

I decided to build my own historical database using IB's feeds. According to here http://www.interactivebrokers.com/en/p.php?f=marketData

All customers initially receive 100 concurrent lines of real-time market data (which can be displayed in TWS or via the API) and always have a minimum of 100 lines of data.

I think I can make use of at least 80 concurrent lines of real-time market data to build my historical database via the API. Whereas I can leave the remaining 20 concurrent lines of real-time market data for TWS.

Get ready for Algo Trader

I got my Interactive Broker new account approved. Yet pending for funding.

I'm about to start exploring algo-trader. I have read through the official wiki pages from https://code.google.com/p/algo-trader/w/list . These documentations gave me good foundation to play with the code.

Right before I start to run my first algo-trader strategy, I got further hints from a first time user in AlgoTrader google groups. https://groups.google.com/forum/?fromgroups#!topic/algo-trader/DLNCbxSNznw

... A very minor issue, but MarketDataStarter.launch tries to use the database AlgoTraderLight, but the included sql file, algotrader.sql, sets up the database with the name AlgoTrader.

I then ran into a slightly more frustrating issue that caused RMI to hang for a minute or so and then give an error about an ObjID already in use. It turned out the problem was that my hostname wasn't set in /etc/hosts, which screwed up RMI. Maybe it would be worth mentioning in the Wiki that Linux users should try pinging their hostname and make sure that it's set to a local IP address.

I was then able to able to run the Periodic strategy and place trades on IB. However, I noticed that transactions and position updates were not being recorded in the database. I needed to add "ib-trades" to the MODULES field in the BASE record of the strategy table. Perhaps it should be set by default or mentioned in the Wiki....

Demo account @ Interactive Broker

I do not have admin right @ my Windows XP company laptop. So, I downloaded Interactive Broker's TWS for Unix. http://www.interactivebrokers.com/en/control/systemstandalone.php?os=unix&ib_entity=llc

I got the TWS started by modifying the startup command to: java -cp jts.jar;hsqldb.jar;jcommon-1.0.12.jar;jfreechart-1.0.9.jar;jhall.jar;other.jar;rss.jar -Xmx512M -XX:MaxPermSize=128M jclient.LoginFrame . 

Login as user [edemo] with password [demouser].

Login successful !!

Startup - Quantitative Options Trading

I am planning to start up a small independent private Quantitative Trading part-time business focusing on Options. How small is it? Probably with just US$ 3,000 initial capital for trading and additional US$ 7,000 to fulfill minimum account opening requirements. Let's see how long can I sustained with this mini amount of money.

As far as trading is concerned, I'm most inspired by:

1. Jeff Augen's series of small booklet focus on options trading
2. Ernie P Chan's book on Quantitative Trading: How to Build Your Own Algorithmic Trading Business
3. Cao Ren Cao's articles & books (HK financial columnist and independent investor)
4. Jesse Livermore's books
5. Alexander Elder's book, Trading for a Living
6. Van K. Tharp's book, Trade Your Way to Financial Freedom
7. Oliver L. Velez and Greg Capra's book, Tools and Tactics for the Master Day Trader: Battle-Tested Techniques for Day, Swing, and Position Traders
   

In order to start my business, I need to spend time learning two software tools:

1. Interactive Broker's trading software, Trader Workstation, nick named TWS
2. algo-trader, a GPL'ed open source Java Algorithmic Trading System based on Esper, InteractiveBrokers and FIX - http://code.google.com/p/algo-trader/
   

I cannot learn TWS until I open a brokerage account with Interactive Brokers. I choose IB due to low commission, international trading and API support for Algorithmic Trading. I guessed I will first spend more time dealing with algo-trader. algo-trader is available since August 2011, it is not a mature tool. So, I am prepared to dig into the source code and get something working. I expect I will contribute some code and patch to algo-trader when I'm developing and testing my trading strategies.

Running a private business, will I be lonely? To some extent, yes. Top traders/ investors make lone decision all the time. On the other hand, I may get interaction with active members in algo-trader. Perhaps as his reader, I could also interact with Ernie P Chan at his blog, http://epchan.blogspot.com/ .