1.1 OVERVIEW
THIS book is about data analysis in finance. What useful information could one extract from data? To allow us to go into great depth in this book, we focus on information about regime changes, which means changes in the collective behaviour of the traders in the market. Being able to recognize regime changes in the market is important for traders and regulators. This book starts by asking “what are the data telling us about the market”. Then it explains how the information extracted from the data could help us monitor the market, to see whether it has entered a different regime. Then, as a proof of concept, it explains how a trader could benefit from such information. We shall explain that both knowledge representation and machine learning (two important branches of Artificial Intelligence (AI)) play important roles in information extraction.
How one represents knowledge determines how one could reason about it. Instead of using time series to summarise price changes in the market, this book takes the Directional Change (DC) approach. In time series, one samples a transaction price at fixed intervals, for example, daily closing prices. Directional change is a data-driven approach. It lets data tell us when to sample a transaction price. This will be explained in details in Chapter 2. By looking at price changes from a different angle, we are able to extract new information from data. Such new information complements what we observe under time series. We shall show that being able to see with two eyes (time series and directional change) is better than seeing with one (time series alone).
We cannot observe the individual trader’s behaviour, but the price dynamics in the market reflects their collective behaviour. Statistical properties of the price dynamics is observable. With machine learning, we could use the price dynamics to estimate the hidden processes that drive the market into different regimes. We can also use past observations to statistically estimate whether the market is going to change from one regime to another. Such estimations give traders a chance to adjust their trading strategies. By nature of the directional change definition, the market can be monitored tick-by-tick. This makes the proposed method particularly suitable for high-frequency finance [20].
This book should be seen as a proof of concept. It shows that data does not have to be sampled at fixed intervals (as in time series). Data can be sampled based on events (directional changes). It shows that machine learning can be used to find hidden models in data. It shows that markets can be monitored to detect regime changes. It shows that regime information can help algorithmic trading. It demonstrates a new research framework, a framework in which more effective market monitoring methods, more effective machine learning methods, more effective algorithmic trading algorithms can be applied.
1.2 RESEARCH OBJECTIVES
The aim of this study is to identify and measure the underlying trend of regime change in the financial market, so as to create a practical and theoretical framework to monitor the financial markets. This study aims to answer the following four research questions:
1. Traditionally, most research on regime change starts with summarising data as time series. In this book, we wanted to see if regime change can also be detected under the framework of DC. Therefore, we proposed a new methodology to detect regime change with a data summary under DC in Chapter 3.
2. As a DC-based regime change detection method is proposed, a number of further questions are raised. For example, would regime change detected under DC be the same as regime change detected under time series? Or would they be different? We then focus on evaluating the effectiveness of these two approaches (DC and time series) on regime change detection, in our first research chapter, Chapter 3.
3. Once regime changes can be effectively detected under the data-driven approach of DC, our next aim is to see what classifications or taxonomy can then be applied to regime change. One possible option is to characterise “normal market regime” and “abnormal market regime” in financial markets. The aim is to cover different markets, different periods and different data types, to see whether they share anything in common, which can be used as the factors that determine the market into the two categories. This topic is discussed in our second research chapter, Chapter 4.
4. Leading on from research to establish the parameters for normal and abnormal regimes through the mechanism of DC, the next aim is to take an in-depth look at what leads to the shift from one market regime to another. The aim is to track the financial market to see whether it is entering into one regime from another. In particular, whether the market is shifting to an abnormally volatile regime from a normal, less volatile regime. This would allow us to monitor the status of the financial market in real time, under our specially developed DC analysis and machine learning techniques. And, as a further move, this study could lay the foundation for establishing a practical financial early warning system.
5. Being able to track regime changes allows us to know the current status of the financial market. The early warning signals of regime changes allow investors to better understand the market. However, we wonder if this information would be useful for practical trading. One way to find out is to develop trading algorithms based on the regime tracking signals. By comparing the performance of the designed trading algorithms, we should find the impact of regime tracking information on practical trading.
In summary, our fundamental research objective is therefore to establish a methodology of regime change recognition, and to be able to go on to classify different types of market regimes and dynamically track regime changes under DC, as an alternative way to understand the operation of the financial market and its characteristics. Lastly, we attempt to establish practical trading algorithms based on the regime tracking information.
1.3 BOOK STRUCTURE
This book is organised as follows:
In Chapter 2, we begin with reviewing the principal current research and literature on regime change. Besides, this chapter also provides a general overview of the concept of Directional Change and its application. Lastly, the relevant machine learning techniques, which are adopted in this book, are outlined.
In Chapter 3, we propose a new methodology to detect regime changes in financial markets based on Directional Change. The proposed method is then compared with the conventional approach in regime change detection.
In Chapter 4, we extend our analysis of regime changes detection to classify different market regimes. In particular, we attempt to characterise what is a “normal market regime” as well as an “abnormal market regime”. In the empirical study, we investigate regime changes in ten different financial markets and then classify the market regimes that occur.
In the next research chapter, Chapter 5, we examine the features of what makes up the bridge, or movement, between different market regimes, and indicate the possibility of constructing a programme to monitor the financial markets in real time, so as to be able to track what is the current regime in the financial markets.
In Chapter 6, we propose two simple trading algorithms, which make use of the regime tracking information that is generated by the method presented in Chapter 5. By comparing the performance of the designed trading algorithms, we demonstrate the usefulness of the tracking signals for practical trading.
Thus, the four research chapters are proposed to be a programme of new theoretical and empirical research on the topic of regime change, with, for the first time, the examination to be carried out using Directional Change, a data-driven approach, and examining and testing for the strengths and weaknesses of such an approach, and where such an approach can fit in with seeking to understand and monitor the workings of the financial market, a global market with many ramifications.
Chapter 7 summarises the findings of this book and highlights some take-home messages. It highlights the significance and limitations of the work reported and suggests promising future directions.