Federated Learning Fundamentals

Federated Learning Fundamentals

Federated learning (FL) is a rapidly growing paradigm in artificial intelligence that enables collaborative model training while preserving data privacy. This course covers the core concepts, architectures, and privacy‑preserving techniques that underpin modern FL systems. By the end of this module, learners will understand how FL differs from traditional centralized learning, the various types of FL, and the cryptographic tools that protect sensitive information.

Why Federated Learning?

Traditional machine‑learning pipelines often require raw data to be collected in a central repository. This approach raises several challenges:

• Privacy regulations such as GDPR and HIPAA restrict data movement.
• Bandwidth constraints make transmitting large datasets impractical.
• Data ownership concerns discourage organizations from sharing proprietary information.

Federated learning addresses these issues by keeping raw data on the client devices and only sharing model updates. This principle is the foundation of FL and is explored in detail below.

Core Principle: Model Updates, Not Raw Data

The most fundamental rule of federated learning is that only model updates such as gradients or weight deltas are transmitted to the server. The server never receives the raw data itself. This design enables:

• Compliance with privacy laws.
• Reduced network traffic because updates are typically much smaller than the original datasets.
• Retention of data sovereignty for each client.

In practice, each client performs local training on its own data, computes the gradient (or the difference between the new and old model), and sends this information to a central aggregator.

Types of Federated Learning

Federated learning can be categorized based on how participants share data dimensions. Two primary variants are Horizontal Federated Learning and Vertical Federated Learning.

Horizontal Federated Learning (HFL)

HFL assumes that all participants have the same feature space but own different samples (rows) of data. This scenario is common when multiple devices (e.g., smartphones) collect the same type of sensor data from different users.

• Example: Thousands of phones each record accelerometer readings for activity recognition.
• Key benefit: The global model can be trained on a much larger sample set without changing the feature representation.

Vertical Federated Learning (VFL)

VFL assumes that participants share the same sample space (i.e., the same set of entities) but have different feature spaces. In other words, each party holds different attributes about the same users.

• Example: A bank holds financial transaction data while a telecom company holds call‑detail records for the same customers.
• In VFL, the sample space is shared among participants, enabling joint model training without exposing each party’s proprietary features.

Privacy‑Preserving Techniques in Federated Learning

Even though raw data never leaves the client, model updates can still leak sensitive information. To mitigate this risk, several cryptographic and hardware‑based methods are employed.

Differential Privacy (DP)

DP adds carefully calibrated random noise to each client’s update before aggregation. This noise makes it mathematically difficult to infer any single data point from the aggregated model.

• Noise is typically drawn from a Gaussian or Laplace distribution.
• The privacy budget (ε) controls the trade‑off between model accuracy and privacy.

Secure Multi‑Party Computation (SMPC)

SMPC enables a group of clients to jointly compute a function (e.g., sum of gradients) while keeping each individual input private. The server only sees the aggregated result, not the individual contributions.

• SMPC guarantees that the server never learns any single client’s raw update.
• Common protocols include secret sharing and oblivious transfer.

Homomorphic Encryption (HE)

HE allows computations to be performed directly on encrypted data. Clients encrypt their model updates, send them to the server, and the server aggregates the ciphertexts without ever decrypting them.

• After aggregation, a trusted party (or the clients themselves) decrypts the final model.
• This technique is especially useful when the server is considered semi‑trusted.

Trusted Execution Environments (TEE)

TEE leverages hardware‑protected enclaves to create a secure area of memory where code can run isolated from the rest of the system. The key hardware feature is the hardware‑protected enclave, which ensures that even a compromised operating system cannot read the data inside.

• Examples include Intel SGX and ARM TrustZone.
• TEEs can be used to perform secure aggregation inside the enclave, guaranteeing that plaintext updates never leave the protected area.

Federated Learning Architectures

FL systems can be organized in different topologies. The most common is the centralized architecture, where a single server orchestrates training rounds.

• The central server selects a subset of clients, distributes the current global model, collects updates, and performs aggregation.
• Alternative architectures include peer‑to‑peer (decentralized) and blockchain‑based coordination, but these are less prevalent in production.

Putting It All Together: A Typical FL Workflow

1. Initialization: The central server creates a global model and broadcasts it to selected clients.
2. Local Training: Each client trains the model on its private data for a few epochs.
3. Privacy Enhancement: Clients optionally apply DP noise, encrypt updates with HE, or secret‑share them for SMPC.
4. Secure Aggregation: The server aggregates the protected updates (either in plaintext after decryption, via SMPC, or directly on ciphertext using HE).
5. Model Update: The aggregated result is used to update the global model, which is then redistributed for the next round.

This loop repeats until the model converges or a predefined performance threshold is reached.

Key Takeaways

• Federated learning keeps raw data on client devices; only model updates are transmitted.
• Horizontal FL shares the same feature space across clients, while vertical FL shares the same sample space.
• Privacy techniques such as differential privacy, SMPC, homomorphic encryption, and TEEs protect updates from leakage.
• In a centralized FL setup, a central server coordinates training rounds and aggregates updates.

Further Reading & Resources

To deepen your knowledge, explore the following resources:

• Federated Learning: Challenges, Methods, and Future Directions
• Google’s TensorFlow Federated Documentation
• OpenAI’s Research on Federated Learning
• Differential Privacy Overview