Federated Learning: A Comprehensive Guide to Decentralized Machine Learning
Artificial intelligence (AI) models thrive on vast amounts of data to achieve accurate predictions and insightful analysis. However, traditional machine learning (ML) approaches, which rely on centralized datasets, face challenges related to data storage, privacy concerns, and processing limitations. Federated learning has emerged as a groundbreaking solution, addressing these issues by enabling decentralized model training.
Federated learning represents a transformative shift in how AI models are trained, offering a decentralized approach that prioritizes data privacy and security. In this article, we will delve into the intricacies of federated learning, exploring its types, working mechanisms, real-world applications, challenges, and the frameworks that facilitate its implementation.
What is Federated Learning?
Federated learning is a decentralized approach to training machine learning models. It enables collaborative training of models on decentralized data, where the data is distributed across multiple devices or nodes, such as smartphones, IoT devices, and edge devices. This eliminates the need to access or transfer large datasets, as sensitive information remains on the node, preserving data privacy.
Instead of centralizing data in a single location for model training, federated learning brings the model to the data. Each device trains the model locally using its own data, and only the model updates are shared with a central server. The central server aggregates these updates to create an improved global model, which is then redistributed to the devices for further training. This iterative process allows the model to learn from diverse data sources without compromising data privacy.
How Federated Learning Works
Federated learning starts with initializing a global machine learning model on a central server. The central server distributes the global model to connected client nodes, which can be other servers or edge devices such as smartphones and Internet of Things (IoT) devices.
Read also: Federated Hermes Internship Program
Upon receiving the global model and all the necessary details, each client node proceeds with training. The model makes improvements from the new data that we got from the device. The model changes are summarized as an update and communicated to the cloud. This communication is encrypted.
The central server aggregates all the client node updates. There are different forms of aggregation, but a common method is federated averaging, which calculates the weighted average of all updates. On the cloud, there are many updates coming in from multiple users. These all updates are aggregated and the final model is built.
With every cycle, the models gather a varied amount of information and improve further without creating privacy breaches.
Types of Federated Learning
Federated learning can vary based on the structure of datasets or the nature of client nodes. There are various strategies that are used for Federated Learning.
Cross-device federated learning: This approach uses devices with volatile connectivity and limited computing resources, such as mobile phones and IoT devices.
Read also: Step-by-Step Federated Learning
Cross-silo federated learning: Unlike the cross-device federated learning approach, cross-silo entails a limited number of servers or data centers with stable connectivity and computational resources powerful enough to store and process huge volumes of data. Cross-silo federated learning can be valuable in industries such as finance and healthcare. For instance, a consortium of hospitals can train a shared model on their own patient data to enhance the diagnosis or prediction of certain diseases.
Horizontal federated learning: In horizontal federated learning, client node datasets share the same features or structure but have different samples.
Vertical federated learning: Conversely, vertical federated learning involves client node datasets that share the same samples but have a different structure or features.
Centralized Federated Learning: Here, a central server is used to perform different steps of the algorithm. The central system is subjected to selecting the nodes at the beginning of the training process and then it is also responsible for aggregating the model updates that we received from different nodes/devices. Here, all the selected nodes, send the updates to this central server and hence it is the bottleneck of the system. This method can cause bottleneck problems.
Decentralized Federated Learning: In Decentralized Federated Learning, the nodes themselves can coordinate to get the updated model. This approach can help in preventing the single server problems, that we can get from the centralized federated learning, as in this the model updates are shared between the interconnected nodes without the need of the central system. Here, the model's performance is totally dependent on what network topology we opt for.
Read also: Mitigating Class Imbalance in FL
Heterogeneous Federated Learning: This learning involves a large no of heterogenous clients e.g., mobile devices, and IoT devices. These devices can differ in software or hardware configurations. Recently, a Federated learning framework called HeteroFL has emerged, specifically designed to tackle the challenges posed by heterogeneous clients with varying computation and communication capabilities.
Federated Learning Algorithms
Let’s explore the commonly used strategies and algorithms used for federated learning.
Federated stochastic gradient descent (FedSGD): In traditional SGD, the gradients are computed on mini-batches, which are a fraction of data samples obtained from the total samples. In the federated setting, these mini-batches can be considered different client devices that comprise local data. In FedSGD, the central model is distributed to the clients, and each client computes the gradients using local data. These gradients are then passed to the central server, which aggregates the gradients in proportion to the number of samples present on each client to calculate the gradient descent step.
Federated averaging (FedAvg): Federated averaging is an extension of the FedSGD algorithm. Clients can perform more than one local gradient descent update. Instead of sharing the gradients with the central server, weights tuned on the local model are shared. Finally, the server aggregates the clients' weights (model parameters). Federated Averaging is a generalization of FedSGD-if all the clients begin from the same initialization, averaging the gradients is equal to averaging the weights. Therefore, Federated Averaging leaves room for tuning the local weights before sending them to the central server for averaging.
Federated learning with dynamic regularization (FedDyn): Regularization in traditional machine learning methods aims to add a penalty to the loss function to improve generalization. In federated learning, the global loss must be computed based on local losses generated from heterogeneous devices. Due to the heterogeneity of clients, minimizing global loss is different than minimizing local losses. Therefore, FedDyn method aims to generate the regularization term for local losses by adapting to the data statistics, such as the amount of data or communication cost. This modification of local losses through dynamic regularization enables local losses to converge to the global loss.
Benefits of Federated Learning
Federated learning offers several advantages over traditional centralized machine learning approaches:
Privacy: Federated learning eliminates the need to access or transfer large datasets. The privacy-preserving architecture of federated learning systems means that sensitive data never leaves a device. Only the encrypted model updates are shared with the central server, assuring data security. Additionally, secure aggregation techniques such as Secure Aggregation Principle allow the decryption of only aggregated results.
Data Security: Regular exchanges between client nodes and the central server can result in substantial bottlenecks. For better communication efficiency, consider strategies such as compressing model updates before transmission, quantization and sparsification to relay a subset of the updates or only essential updates.
Compliance: Because data is kept and processed locally, federated learning can help enterprises comply with data protection regulations.
Data Diversity: Federated learning’s decentralized design can bolster data diversity that can help mitigate bias. And thanks to greater data diversity, models can be made more generalizable. Financial institutions can work together to diversify data for credit risk assessment models, allowing better credit access for underserved groups.
Real-World Applications of Federated Learning
Federated learning holds the promise of helping solve real-world problems, with organizations joining forces even across borders and geographical regions.
Healthcare: Hospitals and research institutions can train shared deep learning models that aid in drug discovery for rare diseases. The sensitive nature of healthcare data and its restricted access due to privacy issues make it difficult to scale machine learning systems in this industry globally. With federated learning, models can be trained through secure access to data from patients and medical institutions while the data remains at its original premises. It can help individual institutions collaborate with others and makes it possible for the models to learn from more datasets securely. This is particularly important for AI document processing in healthcare, where patient records must stay within secure hospital networks. Federated learning lets hospitals collaborate on AI models without moving sensitive documents off-premises. Additionally, federated learning can allow clinicians to gain insights about patients or diseases from wider demographic areas beyond local institutions and grant smaller rural hospitals access to advanced AI technologies.
Finance: Financial institutions can work together to diversify data for credit risk assessment models, allowing better credit access for underserved groups. Fraud detection is an important federated learning use case for banking and insurance. This can be challenging without federated learning, considering data privacy protection laws such as the EU’s GRPR, China’s PIPL and the recent EU AI Act, which prohibits cross-border data sharing.
Retail: Retailers can use federated learning to track sales and inventory across multiple locations without revealing any customer data, allowing them to maximize stock levels and lessen waste.
Smart Cities: Smart cities can take advantage of federated learning to glean insights from the myriad devices and sensors scattered around urban areas while keeping resident data private.
Smartphones: Smartphones are one of the most common ways to witness federated learning in action. Word prediction, face recognition for logging, or voice recognition while using Siri or Google Assistant are all examples of federated-learning-based solutions. It helps personalize the user experience while maintaining privacy.
Transportation: Self-driving cars use computer vision and machine learning to analyze the surroundings and interpret the learning in real-time. In order to continuously adapt to the environment, models need to learn from diverse datasets to improve precision. Relying on a traditional cloud-based approach would slow down the systems. Using federated learning can speed up the learning and make the models more robust.
Manufacturing: With federated learning, the product recommendation systems can be improved based on the broader set of data obtained. AR/VR can be used to detect objects and assist with remote operations and virtual assembly. Federated learning can help improve detection systems to create optimal models. Another example is using federated learning in industrial environment monitoring. Federated learning makes it easier to perform a time-series analysis of the industrial environment factors obtained using multiple sensors and companies while maintaining confidential data's privacy.
Challenges and Limitations of Federated Learning
Federated learning signifies a transformative shift in training AI models, but it also comes with limitations.
Communication Efficiency: Regular exchanges between client nodes and the central server can result in substantial bottlenecks. For better communication efficiency, consider strategies such as compressing model updates before transmission, quantization and sparsification to relay a subset of the updates or only essential updates. Federated learning requires frequent communication between nodes during the learning process. Thus, it requires not only enough local computing power and memory, but also high bandwidth connections to be able to exchange parameters of the machine learning model. However, the technology also avoids data communication, which can require significant resources before starting centralized machine learning.
Data Heterogeneity: Federated learning’s decentralized design can bolster data diversity that can help mitigate bias. However, this also means that data is not identically distributed and can be imbalanced. Statistical heterogeneity This problem is posed by the multiple variations of data present across the client devices. For example, some devices may have high-resolution image data, while others can only store low-resolution pictures, or languages might vary based on geographical location. These instances denote that data is non-i.i.d in a federated learning setting, which is in contrast with the assumption of i.i.d data in normal algorithms. This might cause problems in the data structuring, modeling, and inferencing phases.
Systems Heterogeneity: Systems heterogeneity is also an issue, with devices having different computing capabilities. With the large number of devices playing a role in federated learning networks, accounting for differences in storage, communication, and computational capabilities is a huge challenge. Additionally, only a few of these devices participate at a given time, which may lead to biased training. Such heterogeneities can be handled by the techniques of asynchronous communication, active device sampling, and fault tolerance.
Privacy and Data Protection: Privacy and data security are some of the biggest concerns with federated learning. Although the local data stays on the user device, there’s a risk for the information to be revealed from the model updates shared in the network. Some of the common privacy-preserving techniques that can solve this problem include: Differential privacy-adding noisy data that makes it difficult to discern real information in case of data leaks Homomorphic encryption-performing computation on encrypted data Secure multiparty computation-spreading the sensitive data to different data owners so that they can collaboratively perform computation and reduce the risk of privacy breach
Lack of access to global training data: Lack of access to global training data makes it harder to identify unwanted biases entering the training e.g.
Federated Learning Frameworks
Implementing federated learning for real-world applications can be complex, but several frameworks exist to train models on decentralized data and streamline server and client workflows.
Flower: Flower is an open source framework for collaborative AI and data science. It can be used to craft federated AI systems with numerous connected clients. It has built-in training and evaluation workflows, privacy-preserving algorithms and learning algorithms for federated averaging and FedProx.
OpenFL: OpenFL is a Python-based open source federated learning framework originally created by Intel and now under The Linux® Foundation. OpenFL works with deep learning frameworks such as PyTorch and machine learning libraries including TensorFlow.
TensorFlow Federated (TFF): TensorFlow Federated (TFF) is an open source framework developed by Google for machine learning on decentralized data. TensorFlow is the go-to framework for Federated Learning tasks, providing a robust and flexible environment for this decentralized approach to Machine Learning. Federated Learning in TensorFlow allows for a central model to be trained on data distributed across multiple devices, a collaborative process that enhances privacy and data security.
FedCV: As the research in computer vision progresses with large-scale Convolutional Neural Networks and dense transformer models, the scarcity of tools and techniques to implement it in the federated setting becomes evident. The FedCV framework is built to bridge the gap between research and the real-world implementation of federated learning algorithms. FedCV is a unified library for federated learning to address computer vision applications of image segmentation, image classification, and object detection. It provides access to various datasets and models through easy-to-use APIs. The framework consists of three major modules: Computer Vision Applications layer High-level API Low-level API. The high-level API consists of models for computer vision tasks of image segmentation, image classification, and object detection. The users can use the existing data loaders and data partitioning schemes. Also, they can create their own non-i.i.d (identical and independent distribution) data, which can test the robustness of federated learning methods (as the real-world data is usually non-i.i.d). The high-level API also provides implementations of state-of-the-art federated learning algorithms such as FedAvg, FedNAS, and many more. The training can be completed in a reasonable time due to the available support of distributed multi-GPU training. Additionally, the algorithms can be trained using novel distributed computing strategies. The user-oriented API design enables easy implementation and flexible interactions between clients and workers. The low-level API consists of enhanced security and privacy primitive modules that allow secure and private communication between servers present at different locations.
NVFlare: NVFlare, an open-source federated learning framework that’s widely adopted across various applications, offers a diverse range of examples of machine learning and deep learning algorithms.
tags: #federated #learning #model #explained
