Federated AI for Real-World Business Scenarios

Dinesh C. Verma

• Publisher: CRC Press
• Year: 2021
• eBook ISBN: 9781000462593
• Edition: 1
• Topic: Computer Science
• Subtopic: Computer Science General
• Index: Computer Science

This book provides an overview of Federated Learning and how it can be used to build real-world AI-enabled applications. Real-world AI applications frequently have training data distributed in many different locations, with data at different sites having different properties and different formats. In many cases, data movement is not permitted due to security concerns, bandwidth, cost or regulatory restriction. Under these conditions, techniques of federated learning can enable creation of practical applications. Creating practical applications requires implementation of the cycle of learning from data, inferring from data, and acting based on the inference. This book will be the first one to cover all stages of the Learn-Infer-Act cycle, and presents a set of patterns to apply federation to all stages. Another distinct feature of the book is the use of real-world applications with an approach that discusses all aspects that need to be considered in an operational system, including handling of data issues during federation, maintaining compliance with enterprise security policies, and simplifying the logistics of federated AI in enterprise contexts. The book considers federation from a manner agnostic to the actual AI models, allowing the concepts to be applied to all varieties of AI models. This book is probably the first one to cover the space of enterprise AI-based applications in a holistic manner.

Appendices

Appendix 1: Frameworks for Federated Learning

Any federated learning system needs to create a software to implement the federated learning algorithms. The framework provides a basic infrastructure around which federated learning solutions can be built. The framework needs to be consistent with the overall approach for training AI/ML within the enterprise. The framework software would also support an interface which would allow the software developers to invoke different existing functions for federated learning functions, communicate across nodes involved in federated learning, and provide some libraries of existing algorithms.

An example framework for federated learning is federated tensorflow [132] (available at url https://www.tensorflow.org/federated). This framework is open-source and offered by Google researchers. It assumes that a neural network system based on Google tensorflow and the default mode of training is being done on Android phones.

Another framework geared for enterprise computing is offered by IBM [133] (available at urlhttps://ibmfl.mybluemix.net). It is available as a community edition which is open-sourced and is an extension to IBM commercial products dealing with distributed data. The community framework provides a library of federated learning algorithms which include federated learning across neural networks, decision trees and several other algorithms. As a library of python functions, it is geared more towards machines running federated learning routines as opposed to support on mobile phones.

Baidu, the leading search engine in China, has also published its framework for federated learning as an extension to its broader framework for AI called Pad-dlePaddle (available at https://github.com/PaddlePaddle). The larger PaddlePaddle is an alternative to learning frameworks like tensorflow or PyTorch, and since middle of 2000 include the tools that support federated averaging support. PaddlePaddle is geared towards learning solutions in industrial environments.

In addition to the above frameworks published by large companies, the Flower framework [134] (available at https://flower.dev) has been published by researchers from European Universities. The flower framework is targeted at researchers creating new algorithms and extending federated learning to many new different environments.

Given the significance and importance of federated learning, more such frameworks are likely to emerge. Organizations implementing federated learning-based solutions may opt to choose one of the existing frameworks, or implement their own framework for federated learning. The choice of the approach would depend on factors such as the closeness to existing approaches for AI/ML within the organization, any licensing considerations, and ease of use. As an example, organizations relying on products from IBM to run their functions might find the IBM framework more suitable for their needs, while companies in China may prefer PaddlePaddle. Other organizations may find it easier to use the Google framework or to develop their own federated learning approaches on a completely new AI framework, e.g. a new framework on PySoft.

Appendix 2: Adversarial Federated Learning

The main focus of the book has been in the context of businesses that have data distributed among many different sites and need to learn an AI model without necessarily moving data to a single location. In Section 2.3.1 of Chapter 2, we had briefly introduced the concept of consumer federated learning. Within the context of consumer federated learning, adversarial learning is an important concern.

Consumer federated learning referred to the scenario where several mobile phones were maintaining data about their user which was mined and analyzed locally. Consumer federated learning has been the focus of intense academic research and has resulted in several publications [30, 31, 135]. Example applications of such consumer-focused federated learning include predicting keyboard type-ahead or predicting the set of user browser queries. The motivation is usually protection of user privacy in the sense that private data belonging to the user need not move outside the mobile phone.

Consumer federated learning has the advantage that all learning is done for a specific mobile application which can have a fixed format of input over which the models are being trained, thereby avoiding the complexities of the algorithms described in Chapters 4 and 7. The learning can be coordinated to happen in a synchronized manner, avoiding the issues and approaches discussed in Chapter 7. Thus, the systems implementing consumer federated learning can focus on creating the model and using it in the context of the specific mobile application.

The challenges associated with consumer federated learning have to deal with a significant large number of end points with data, which results in each mobile phone having data that is very specific to the user. Unfortunately, this also means that the data at each node is too small to make a meaningful model in most cases. In order to work around this lack of data, cohorts of users are used to combine data from several users into a larger group where sufficient data can be used to create a useful model [136], but this means that data has to be moved out of the mobile phone to a set of cohorts in order to create the model. This either requires complex security protocols to maintain privacy, like secure multi-party computation [137], that are very inefficient, or else violate the privacy requirement that was the original motivation for federated learning. Another challenge is the existence of malicious parties which can launch adverserial attacks on federated learning.

The current prevalent practice in the industry is to work around the privacy requirements of consumer data on the mobile phone using business arrangements which require users to agree to the use of their data in exchange for the benefit delivered by the mobile application. As a result, the primary motivation for consumer federated learning can be eliminated with simple business mechanisms, and the approaches are mostly of academic interest. While the high level of academic interest is shown in the large number of papers published on this subject, federated learning on mobile phones for consumers is likely to stay within the realm of academic research and unlikely to see wide adoption in the enterprise. Businesses are more likely to adopt a business mechanism to collect the data centrally and build AI models from them, which provides a more simplified and traditional implementation of AI-based systems.

Adversarial federated learning is an important challenge associated with consumer federated learning. When several consumer devices are generating data and that data is being used to train a common model, there is a high probability that someone may be feeding bad data or bad models into the system. The number of ways in which the common model being trained together can be attacked in a variety of ways, not just by one single participant but also by a set of maliciously coordinated cohort of clients.

A variety of attacks that can be targeted at different forms of federated learning can been described [138, 139, 140], which can create challenges for creating good models. In a massively distributed deployment of federate learning, it would be reasonable to expect that there would be pranksters and malicious entities, which is another reason for suspecting that widespread consumer federated learning is not likely to be used within enterprise applications anytime shortly. Such malicious entities can carefully select the weights they provide at different stages of the federated model training, leading to corruption of the model that is eventually created.

There have been approaches proposed which would mitigate the impact of such poisoning attacks, which include adjusting the averaging process of weights into it done in a hierarchical manner with a randomly selected groups of clients [141, 141]. This will reduce the impact of a malicious individual and randomize the malicious cohort members into different groups which should prevent the attacks if the malicious entities are a small fraction of the overall participants. Another safeguard can be used by keeping the individual parameters of the model being trained hidden from the other participants, who can only see the aggregated model. Since adverserial attacks need to be crafted carefully based on the type of models from other clients providing their models, putting in the privacy mechanisms can prevent targeted attacks against federated learning systems [142].

Other traditional mechanisms for detecting and preventing malicious parties, e.g. reputation-based schemes [67] and using mechanisms like differential privacy [104], secure multiparty computations [137] also provide some resilience against malicious entities.

We have not focused on this aspect within this book since this is a vulnerability which is typical for consumer federated learning but not likely to arise in the context of enterprise federated learning. In the context of enterprise federated learning, business arrangements provide a sufficient disincentive for anyone to participate willingly in a malicious adversarial attack on the federated learning process. While some inadvertent adverserial attacks may happen if software of one of the participants is compromised, the mechanisms for handling the variety of trust situations in Chapter 6 can provide sufficient safeguards for those situations.

References

1. S. Legg, M. Hutter, et al., “A collection of definitions of intelligence,” in Frontiers in Artificial Intelligence and Applications, vol. 157, p. 17, IOS Press, 2007.
2. C. Grosan and A. Abraham, “Rule-based expert systems,” in Intelligent Systems, pp. 149–185, 2011.
3. G. Cirincione and D. Verma, “Federated Machine Learning for Multi-Domain Operationsat the tactical edge,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006, p. 1100606, International Society for Optics and Photonics, 2019.
4. W. D. Nothwang, M. J. McCourt, R. M. Robinson, S. A. Burden, and J. W. Curtis, “The human should be part of the control loop?,” in 2016 Resilience Week (RWS), pp. 214–220, IEEE, 2016.
5. G. A. Seber and A. J. Lee, Linear Regression Analysis, vol. 329. John Wiley & Sons, 2012.
6. D. G. Kleinbaum, K. Dietz, M. Gail, M. Klein, and M. Klein, Logistic Regression. Springer, 2002.
7. L. Atzori, A. Iera, and G. Morabito, “The internet of things: A survey,” Computer Networks, vol. 54, no. 15, pp. 2787–2805, 2010.
8. R. Kohavi, “The power of decision tables,” in European Conference on Machine Learning, pp. 174–189, Springer, 1995.
9. L. Rokach and O. Maimon, “Decision trees,” in Data Mining and Knowledge Discovery Handbook, pp. 165–192, Springer, 2005.
10. C. Feng and D. Michie, “Machine learning of rules and trees,” Machine Learning, Neural and Statistical Classification, pp. 50–83, 1994.
11. M. A. Nielsen, Neural Networks and Deep Learning, vol. 2018. Determination Press San Francisco, CA, 2015.
12. C. C. Aggarwal, Neural Networks and Deep Learning. Springer, 2018.
13. H. Abdi and L. J. Williams, “Principal component analysis,” Wiley Interdisciplinary Reviews: Computational Statistics, vol. 2, no. 4, pp. 433–459, 2010.
14. H. Abdi and D. Valentin, “Multiple correspondence analysis,” Encyclopedia of Measurement and Statistics, vol. 2, no. 4, pp. 651–657, 2007.
15. S. Suthaharan, “Support vector machine,” in Machine Learning Models and Algorithms for Big Data Classification, pp. 207–235, Springer, 2016.
16. B. Steffen, F. Howar, and M. Merten, “Introduction to active automata learning from a practical perspective,” in International School on Formal Methods for the Design of Computer, Communication and Software Systems, pp. 256–296, Springer, 2011.
17. J. G. Kemeny and J. L. Snell, Markov Chains. Springer-Verlag, New York, 1976.
18. L. Rabiner and B. Juang, “An introduction to Hidden Markov Models,” IEEE ASSP Magazine, vol. 3, no. 1, pp. 4–16, 1986.
19. T. Murata, “Petri Nets: Properties, Analysis and Applications,” Proceedings of the IEEE, vol. 77, n...

Table of contents

• Cover
• Title Page
• Copyright Page
• Dedication
• Preface
• Table of Contents
• Introduction to Artificial Intelligence
• Scenarios for Federated AI
• Naive Federated Learning Approaches
• Addressing Data Mismatch Issues in Federated AI
• Addressing Data Skew Issues in Federated Learning
• Addressing Trust Issues in Federated Learning
• Addressing Synchronization Issues in Federated Learning
• Addressing Vertical Partitioning Issues in Federated Learning
• Use Cases
• Appendix 1: Frameworks for Federated Learning
• Appendix 2: Adversarial Federated Learning
• References
• Index