Detecting Regime Change in Computational Finance

Data Science, Machine Learning and Algorithmic Trading

Jun Chen, Edward P K Tsang

• Publisher: Chapman and Hall/CRC
• Year: 2020
• ISBN:
  9781000220360
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Mining
• Index: Computer Science

Based on interdisciplinary research into "Directional Change", a new data-driven approach to financial data analysis, Detecting Regime Change in Computational Finance: Data Science, Machine Learning and Algorithmic Trading applies machine learning to financial market monitoring and algorithmic trading. Directional Change is a new way of summarising price changes in the market. Instead of sampling prices at fixed intervals (such as daily closing in time series), it samples prices when the market changes direction ("zigzags"). By sampling data in a different way, this book lays out concepts which enable the extraction of information that other market participants may not be able to see. The book includes a Foreword by Richard Olsen and explores the following topics:

• Data science: as an alternative to time series, price movements in a market can be summarised as directional changes

• 
  
  

• Machine learning for regime change detection: historical regime changes in a market can be discovered by a Hidden Markov Model

• 
  
  

• Regime characterisation: normal and abnormal regimes in historical data can be characterised using indicators defined under Directional Change

• 
  
  

• Market Monitoring: by using historical characteristics of normal and abnormal regimes, one can monitor the market to detect whether the market regime has changed

• 
  
  

• Algorithmic trading: regime tracking information can help us to design trading algorithms

• 
  
  

It will be of great interest to researchers in computational finance, machine learning and data science.

About the Authors

Jun Chen received his PhD in computational finance from the Centre for Computational Finance and Economic Agents, University of Essex in 2019.

Edward P K Tsang is an Emeritus Professor at the University of Essex, where he co-founded the Centre for Computational Finance and Economic Agents in 2002.

CHAPTER 1

Introduction

1.1 OVERVIEW

THIS book is about data analysis in finance. What useful information could one extract from data? To allow us to go into great depth in this book, we focus on information about regime changes, which means changes in the collective behaviour of the traders in the market. Being able to recognize regime changes in the market is important for traders and regulators. This book starts by asking “what are the data telling us about the market”. Then it explains how the information extracted from the data could help us monitor the market, to see whether it has entered a different regime. Then, as a proof of concept, it explains how a trader could benefit from such information. We shall explain that both knowledge representation and machine learning (two important branches of Artificial Intelligence (AI)) play important roles in information extraction.

How one represents knowledge determines how one could reason about it. Instead of using time series to summarise price changes in the market, this book takes the Directional Change (DC) approach. In time series, one samples a transaction price at fixed intervals, for example, daily closing prices. Directional change is a data-driven approach. It lets data tell us when to sample a transaction price. This will be explained in details in Chapter 2. By looking at price changes from a different angle, we are able to extract new information from data. Such new information complements what we observe under time series. We shall show that being able to see with two eyes (time series and directional change) is better than seeing with one (time series alone).

We cannot observe the individual trader’s behaviour, but the price dynamics in the market reflects their collective behaviour. Statistical properties of the price dynamics is observable. With machine learning, we could use the price dynamics to estimate the hidden processes that drive the market into different regimes. We can also use past observations to statistically estimate whether the market is going to change from one regime to another. Such estimations give traders a chance to adjust their trading strategies. By nature of the directional change definition, the market can be monitored tick-by-tick. This makes the proposed method particularly suitable for high-frequency finance [20].

This book should be seen as a proof of concept. It shows that data does not have to be sampled at fixed intervals (as in time series). Data can be sampled based on events (directional changes). It shows that machine learning can be used to find hidden models in data. It shows that markets can be monitored to detect regime changes. It shows that regime information can help algorithmic trading. It demonstrates a new research framework, a framework in which more effective market monitoring methods, more effective machine learning methods, more effective algorithmic trading algorithms can be applied.

1.2 RESEARCH OBJECTIVES

The aim of this study is to identify and measure the underlying trend of regime change in the financial market, so as to create a practical and theoretical framework to monitor the financial markets. This study aims to answer the following four research questions:

1. Traditionally, most research on regime change starts with summarising data as time series. In this book, we wanted to see if regime change can also be detected under the framework of DC. Therefore, we proposed a new methodology to detect regime change with a data summary under DC in Chapter 3.

2. As a DC-based regime change detection method is proposed, a number of further questions are raised. For example, would regime change detected under DC be the same as regime change detected under time series? Or would they be different? We then focus on evaluating the effectiveness of these two approaches (DC and time series) on regime change detection, in our first research chapter, Chapter 3.

3. Once regime changes can be effectively detected under the data-driven approach of DC, our next aim is to see what classifications or taxonomy can then be applied to regime change. One possible option is to characterise “normal market regime” and “abnormal market regime” in financial markets. The aim is to cover different markets, different periods and different data types, to see whether they share anything in common, which can be used as the factors that determine the market into the two categories. This topic is discussed in our second research chapter, Chapter 4.

4. Leading on from research to establish the parameters for normal and abnormal regimes through the mechanism of DC, the next aim is to take an in-depth look at what leads to the shift from one market regime to another. The aim is to track the financial market to see whether it is entering into one regime from another. In particular, whether the market is shifting to an abnormally volatile regime from a normal, less volatile regime. This would allow us to monitor the status of the financial market in real time, under our specially developed DC analysis and machine learning techniques. And, as a further move, this study could lay the foundation for establishing a practical financial early warning system.

5. Being able to track regime changes allows us to know the current status of the financial market. The early warning signals of regime changes allow investors to better understand the market. However, we wonder if this information would be useful for practical trading. One way to find out is to develop trading algorithms based on the regime tracking signals. By comparing the performance of the designed trading algorithms, we should find the impact of regime tracking information on practical trading.

In summary, our fundamental research objective is therefore to establish a methodology of regime change recognition, and to be able to go on to classify different types of market regimes and dynamically track regime changes under DC, as an alternative way to understand the operation of the financial market and its characteristics. Lastly, we attempt to establish practical trading algorithms based on the regime tracking information.

1.3 BOOK STRUCTURE

This book is organised as follows:

In Chapter 2, we begin with reviewing the principal current research and literature on regime change. Besides, this chapter also provides a general overview of the concept of Directional Change and its application. Lastly, the relevant machine learning techniques, which are adopted in this book, are outlined.

In Chapter 3, we propose a new methodology to detect regime changes in financial markets based on Directional Change. The proposed method is then compared with the conventional approach in regime change detection.

In Chapter 4, we extend our analysis of regime changes detection to classify different market regimes. In particular, we attempt to characterise what is a “normal market regime” as well as an “abnormal market regime”. In the empirical study, we investigate regime changes in ten different financial markets and then classify the market regimes that occur.

In the next research chapter, Chapter 5, we examine the features of what makes up the bridge, or movement, between different market regimes, and indicate the possibility of constructing a programme to monitor the financial markets in real time, so as to be able to track what is the current regime in the financial markets.

In Chapter 6, we propose two simple trading algorithms, which make use of the regime tracking information that is generated by the method presented in Chapter 5. By comparing the performance of the designed trading algorithms, we demonstrate the usefulness of the tracking signals for practical trading.

Thus, the four research chapters are proposed to be a programme of new theoretical and empirical research on the topic of regime change, with, for the first time, the examination to be carried out using Directional Change, a data-driven approach, and examining and testing for the strengths and weaknesses of such an approach, and where such an approach can fit in with seeking to understand and monitor the workings of the financial market, a global market with many ramifications.

Chapter 7 summarises the findings of this book and highlights some take-home messages. It highlights the significance and limitations of the work reported and suggests promising future directions.

CHAPTER 2

Background and Literature Survey

THIS chapter will discuss the research literature on the concept of regime change in the financial market, and on our research topic of the application of Directional Change (DC) to study regime change. In addition, the machine learning techniques that are used in this book are also outlined.

2.1 REGIME CHANGE

Regime Change (RC) is a term more often associated with politics and government rather than with financial markets. In this book, the term is being used to describe what is a significant change in price behaviour in financial markets. The question then arises as to what kind of change in price can be considered as a regime change, rather than simply the normal fluctuations of the financial market. The links between different categories of regime change is noted by [22]: “since the financial crash of 2008, the global financial markets have been subject to prolonged periods in which their behaviour has been dominated by a single, over-arching economic regime, often determined by the stance of monetary policy. When these regimes have changed, the behaviour of the main asset classes (equities, bonds, commodities and currencies) has been drastically affected, and individual asset prices within each class have also had to fit into the overall macro pattern.” It is therefore the observed price behaviour in financial markets that can have profound social and political implications.

In the field of macroeconomics, the term regime change is used to explain dramatic breaks in many economic cycles, often with dramatic and unpredictable political, governmental and economic crises causing financial ruptures [31].

Hamilton [32] compared the US unemployment rate since the Sec-ond World War and the subsequent periods of economic recession. He discovered that the US economy periodically entered into an episode when the unemployment rate rapidly rose. Such evidence indicated that the economy frequently oscillated between a steady period and a turbulent period, moving from one regime to another. Another example of the fluctuations of market regimes is indicated in the global financial crisis of 2008-2009, where the behaviour of asset ...