Big Data Science in Finance

Irene Aldridge, Marco Avellaneda

• Publisher: Wiley
• Year: 2021
• ISBN:
  9781119602972
• Edition: 1
• Topic: Informatica
• Subtopic: Informatica generale

Explains the mathematics, theory, and methods of Big Data as applied to finance and investing

Data science has fundamentally changed Wall Street—applied mathematics and software code are increasingly driving finance and investment-decision tools. Big Data Science in Finance examines the mathematics, theory, and practical use of the revolutionary techniques that are transforming the industry. Designed for mathematically-advanced students and discerning financial practitioners alike, this energizing book presents new, cutting-edge content based on world-class research taught in the leading Financial Mathematics and Engineering programs in the world. Marco Avellaneda, a leader in quantitative finance, and quantitative methodology author Irene Aldridge help readers harness the power of Big Data.

Comprehensive in scope, this book offers in-depth instruction on how to separate signal from noise, how to deal with missing data values, and how to utilize Big Data techniques in decision-making. Key topics include data clustering, data storage optimization, Big Data dynamics, Monte Carlo methods and their applications in Big Data analysis, and more. This valuable book:

• Provides a complete account of Big Data that includes proofs, step-by-step applications, and code samples
• Explains the difference between Principal Component Analysis (PCA) and Singular Value Decomposition (SVD)
• Covers vital topics in the field in a clear, straightforward manner
• Compares, contrasts, and discusses Big Data and Small Data
• Includes Cornell University-tested educational materials such as lesson plans, end-of-chapter questions, and downloadable lecture slides

Big Data Science in Finance: Mathematics and Applications is an important, up-to-date resource for students in economics, econometrics, finance, applied mathematics, industrial engineering, and business courses, and for investment managers, quantitative traders, risk and portfolio managers, and other financial practitioners.

Chapter 1
Why Big Data?

Introduction

It is the year 2032, and with a wave of your arm, your embedded chip authenticates you to log into your trading portal. For years, Swedes have already been placing chips above their thumb to activate their train tickets or to store their medical records.¹ Privacy, Big Brother, and health concerns aside, the sheer volume of data collected by IDs from everything from nail salons through subway stations is staggering, yet needs to be analyzed in real time to draw competitive inferences about impending market activity.

Do you think this is an unlikely scenario? During World War II, a passive ID technology was developed to leave messages for one's compatriots inside practically any object. The messages were written in tin foil, but were virtually unnoticeable by one's enemy. They could last forever since they didn't contain a battery or any other energy source, and they were undetectable as they did not emit heat or radiation. The messages were only accessible by the specific radio frequency for which they were written – a radio scanner set to a specific wavelength could pick up the message from a few feet away, without holding or touching the object.

Today, the technology behind these messages has made its way into Radio-Frequency Identification devices, RFIDs. They are embedded into pretty much every product you can buy in any store. They are activated at checkout and at the exit, where giant scanners examine you for any unpaid merchandise in your possession. Most importantly, RFIDs are used to collect data about your shopping preferences, habits, tastes, and lifestyle. They know whether you prefer red to green, if you buy baby products, and if you drink organic orange juice. And did you know that nine out of every ten purchases you make end up as data transmitted through the Internet to someone's giant private database that is a potential source of returns for a hedge fund?

Welcome to the world of Big Data Finance (BDF), a world where all data have the potential of ending up in a hedge fund database generating extra uncorrelated returns. Data like aggregate demand for toothpaste may predict the near-term and long-term returns of toothpaste manufacturers such as Procter & Gamble. A strong trend toward gluten-free merchandise may affect the way wheat futures are traded. And retail stores are not alone in recording consumer shopping habits: people's activity at gas stations, hair salons, and golf resorts is diligently tracked by credit card companies in data that may all end up in a hedge fund manager's toolkit for generating extra returns. Just like that, a spike in demand for gas may influence short-term oil prices.

Moving past consumer activity, we enter the world of business-to-business (B2B) transactions, also conducted over the Internet. How many bricks are ordered from specific suppliers this spring may be a leading indicator of new housing stock in the NorthEast. And are you interested in your competitor's supply and demand? Many years ago, one would charter a private plane to fly over a competitor's manufacturing facility to count the number of trucks coming and going as a crude estimate of activity. Today, one can buy much less expensive satellite imagery and count the number of trucks without leaving one's office. Oh, wait, you can also write a computer program to do just that instead.

Many corporations, including financial organizations, are also sitting on data they don't even realize can be used in very productive ways. The inability to identify useful internal data and harness them productively may separate tomorrow's winners from losers.

Whether you like it or not, Big Data is influencing finance, and we are just scratching the surface. While the techniques for dealing with data are numerous, they are still applied to only a limited set of the available information. The possibilities to generate returns and reduce costs in the process are close to limitless. It is an ocean of data and whoever has the better compass may reap the rewards.

And Big Data does not stop on the periphery of financial services. The amount of data generated internally by financial institutions are at a record-setting number. For instance, take exchange data. Twenty years ago, the exchange data that were stored and distributed by the financial institutions comprised Open, High, Low, Close, and Daily Volume for each stock and commodity futures contract. In addition, newspapers printed the yield and price for government bonds, and occasionally, noon or daily closing rates for foreign exchange rates. These data sets are now widely available free of charge from companies like Google and Yahoo.

Today's exchanges record and distribute every single infinitesimal occurrence on their systems. An arrival of a limit order, a limit order cancellation, a hidden order update – all of these instances are meticulously timestamped and documented in maximum detail for posterity and analysis. The data generated for one day by just one exchange can measure in terabytes and petabytes. And the number of exchanges is growing every year. At the time this book was written, there were 23 SEC-registered or “lit” equity exchanges in the U.S. alone,² in addition to 57 alternative equity trading venues, including dark pools and order internalizers.³ The latest exchange addition, the Silicon Valley-based Long Term Stock Exchange, was approved by the regulators on May 10, 2019.⁴

These data are huge and rich in observations, yet few portfolio managers today have the necessary skills to process so much information. To that extent, eFinancialCareers.com reported on April 6, 2017 that robots are taking over traditional portfolio management jobs, and as many as 90,000 of today's well-paid pension-fund, mutual-fund, and hedge-fund positions are bound to be lost over the next decade.⁵ On the upside, the same article reported that investment management firms are expected to spend as much as $7 billion on various data sources, creating Big Data jobs geared at acquiring, processing, and deploying data for useful purposes.

Entirely new types of Big Data Finance professionals are expected to populate investment management firms. The estimated number of these new roles is 80 per every $3 billion of capital under management, according to eFinancialCareers. The employees under consideration will comprise:

1. Data scouts or data managers, whose job already is and will continue to be to seek the new data sources capable of delivering uncorrelated sources of revenues for the portfolio managers.
2. Data scientists, whose job will expand into creating meaningful models capable of grabbing the data under consideration and converting them into portfolio management signals.
3. Specialists, who will possess a deep understanding of the data in hand, say, what the particular shade of the wheat fields displayed in the satellite imagery means for the crop production and respective futures prices, or what the market microstructure patterns indicate about the health of the market.

And this trend is not something written in the sky, but is already implemented by a host of successful companies. In March 2017, for example, BlackRock made news when they announced the intent to automate most of their portfolio management function. Two Sigma deploys $45 billion, employing over 1,100 workers, many of whom have data s...