data-driven-docs

Living documentation for evolving technologies

View on GitHub

Concurrent Programming

Table of Contents

What’s Concurrent Programming

Concurrent programming is a programming paradigm that deals with the execution of multiple tasks or processes simultaneously. It enables programs to handle multiple operations at the same time, improving responsiveness, resource utilization, and performance.

Key Characteristics:

Unlike sequential programming where instructions execute one after another, concurrent programming allows multiple sequences of operations to make progress during overlapping time periods.

Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once. - Rob Pike

Back to top

Concurrency vs Parallelism

While often used interchangeably, concurrency and parallelism are distinct concepts:

Concurrency

Concurrency is about structure - the composition of independently executing tasks.

Time →
CPU: [Task A]--[Task B]--[Task A]--[Task C]--[Task B]
      ↑ Context switching between tasks

Parallelism

Parallelism is about execution - the simultaneous execution of multiple tasks.

Time →
CPU 1: [Task A--------------]
CPU 2: [Task B--------------]
CPU 3: [Task C--------------]
      ↑ True simultaneous execution

Relationship:

Back to top

Core Concepts

Processes and Threads

Process:

Thread:

┌─────────────────────────────────────┐
│          Process                     │
│  ┌──────────┐  ┌──────────┐         │
│  │ Thread 1 │  │ Thread 2 │         │
│  └──────────┘  └──────────┘         │
│                                      │
│  ┌────────────────────────┐         │
│  │   Shared Memory        │         │
│  └────────────────────────┘         │
└─────────────────────────────────────┘

Back to top

Thread Safety

A piece of code is thread-safe if it functions correctly when accessed by multiple threads simultaneously.

Thread-safe requirements:

Common thread-safety techniques:

Back to top

Synchronization

Synchronization is the coordination of concurrent tasks to ensure correct execution.

Synchronization Mechanisms:

  1. Mutex (Mutual Exclusion)
    • Ensures only one thread accesses a resource at a time
    • Lock before access, unlock after
  2. Semaphore
    • Controls access to a resource pool
    • Allows N threads to access simultaneously
  3. Monitor
    • Combines mutex with condition variables
    • Wait/notify mechanisms
  4. Barriers
    • Synchronization point where all threads must arrive before proceeding
  5. Read-Write Locks
    • Multiple readers or single writer
    • Optimizes for read-heavy workloads

Back to top

Race Conditions

A race condition occurs when the program’s behavior depends on the relative timing of events.

Example:

Thread A:              Thread B:
read balance (100)     read balance (100)
add 50                 add 30
write balance (150)    write balance (130)  ← Lost update!

Solution: Synchronize access to shared state.

Back to top

Deadlock

Deadlock occurs when two or more threads are blocked forever, each waiting for the other to release a resource.

Four necessary conditions (Coffman conditions):

  1. Mutual exclusion: Resources cannot be shared
  2. Hold and wait: Threads hold resources while waiting for others
  3. No preemption: Resources cannot be forcibly taken
  4. Circular wait: Circular chain of waiting threads

Example:

Thread A:              Thread B:
lock(Resource 1)       lock(Resource 2)
lock(Resource 2) ←─┐   lock(Resource 1) ←─┐
                   │                       │
                   └───── DEADLOCK ────────┘

Prevention strategies:

Back to top

Shared State vs Message Passing

Two fundamental approaches to concurrent programming:

Shared State (Shared Memory)

Threads communicate by reading/writing shared variables.

Advantages:

Disadvantages:

Example paradigms: Traditional multi-threading, OpenMP

Message Passing

Threads communicate by sending messages (no shared state).

Advantages:

Disadvantages:

Example paradigms: Actor model (Erlang, Akka), Go channels, MPI

Back to top

Concurrency Models

Thread-Based Concurrency

The traditional model where the operating system manages threads.

Characteristics:

Use cases:

Languages: Java, C++, C#, Python (with GIL limitations)

Back to top

Event-Driven Concurrency

Single-threaded event loop processing events from a queue.

Characteristics:

Use cases:

Languages: JavaScript/Node.js, Python (asyncio), Rust (tokio)

Back to top

Actor Model

Independent actors communicate via asynchronous messages.

Characteristics:

Principles:

Use cases:

Languages: Erlang, Elixir, Akka (Scala/Java), Pony

Back to top

Coroutines

Lightweight, cooperative concurrent tasks.

Characteristics:

Use cases:

Languages: Kotlin, Go (goroutines), Python (async/await), Lua

Back to top

Common Patterns

Producer-Consumer

One or more producers create work items; one or more consumers process them.

Components:

Benefits:

Back to top

Thread Pool

Reusable pool of worker threads that execute submitted tasks.

Benefits:

Components:

See also: Java Executors

Back to top

Future/Promise

Placeholder for a value that will be available in the future.

Characteristics:

Variants:

See also:

Back to top

Lock-Free Data Structures

Data structures that use atomic operations instead of locks.

Techniques:

Benefits:

Challenges:

Back to top

Benefits and Challenges

Benefits

  1. Improved Responsiveness
    • UI remains responsive during long operations
    • Better user experience
  2. Resource Utilization
    • Utilize multiple CPU cores
    • Overlap I/O with computation
    • Better throughput
  3. Performance
    • Parallel execution on multi-core systems
    • Reduced latency for I/O operations
  4. Scalability
    • Handle more concurrent users/requests
    • Efficient use of system resources
  5. Modularity
    • Natural decomposition of problems
    • Independent components

Back to top

Challenges

  1. Complexity
    • Harder to design and understand
    • Non-deterministic behavior
    • Difficult to test and debug
  2. Race Conditions
    • Subtle timing-dependent bugs
    • Hard to reproduce and fix
  3. Deadlocks
    • System hangs completely
    • Can be difficult to detect
  4. Synchronization Overhead
    • Lock contention reduces performance
    • Context switching costs
  5. Memory Consistency
    • Visibility of shared state
    • Memory model complexities
    • Cache coherence issues
  6. Testing Difficulties
    • Non-deterministic execution
    • Heisenbugs (disappear when debugging)
    • Need for stress testing

See also: Java Memory Model

Back to top

Examples

Thread Creation (Multiple Languages)

Java:

// Using Thread class
Thread thread = new Thread(() -> {
    System.out.println("Running in thread: " + Thread.currentThread().getName());
});
thread.start();

// Using ExecutorService (recommended)
ExecutorService executor = Executors.newFixedThreadPool(4);
executor.submit(() -> {
    System.out.println("Task executed by: " + Thread.currentThread().getName());
});
executor.shutdown();

Python:

import threading

def worker():
    print(f"Running in thread: {threading.current_thread().name}")

# Create and start thread
thread = threading.Thread(target=worker)
thread.start()
thread.join()  # Wait for completion

Go:

package main

import (
    "fmt"
    "sync"
)

func worker(id int, wg *sync.WaitGroup) {
    defer wg.Done()
    fmt.Printf("Worker %d executing\n", id)
}

func main() {
    var wg sync.WaitGroup

    for i := 1; i <= 5; i++ {
        wg.Add(1)
        go worker(i, &wg)  // Launch goroutine
    }

    wg.Wait()  // Wait for all goroutines
}

Rust:

use std::thread;

fn main() {
    let handle = thread::spawn(|| {
        println!("Running in thread");
    });

    handle.join().unwrap();  // Wait for thread
}

Back to top

Synchronization Example

Problem: Multiple threads incrementing a shared counter (unsafe).

Java (with synchronization):

public class Counter {
    private int count = 0;

    // Synchronized method ensures thread safety
    public synchronized void increment() {
        count++;
    }

    // Or using explicit lock
    private final Object lock = new Object();

    public void incrementWithLock() {
        synchronized (lock) {
            count++;
        }
    }

    // Modern approach: AtomicInteger
    private AtomicInteger atomicCount = new AtomicInteger(0);

    public void incrementAtomic() {
        atomicCount.incrementAndGet();  // Lock-free!
    }
}

Python (with lock):

import threading

class Counter:
    def __init__(self):
        self.count = 0
        self.lock = threading.Lock()

    def increment(self):
        with self.lock:
            self.count += 1

Go (with mutex):

type Counter struct {
    mu    sync.Mutex
    count int
}

func (c *Counter) Increment() {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.count++
}

Back to top

Message Passing Example

Go (channels):

func main() {
    messages := make(chan string)

    // Producer goroutine
    go func() {
        messages <- "ping"
    }()

    // Consumer (main goroutine)
    msg := <-messages
    fmt.Println(msg)  // "ping"
}

Erlang (actor model):

% Spawn a process that receives messages
receiver() ->
    receive
        {From, Message} ->
            io:format("Received: ~p~n", [Message]),
            From ! {self(), "acknowledged"},
            receiver()  % Tail recursion
    end.

% Send a message to the receiver
Pid = spawn(fun receiver/0),
Pid ! {self(), "Hello"}.

Rust (channels):

use std::sync::mpsc;
use std::thread;

fn main() {
    let (tx, rx) = mpsc::channel();

    thread::spawn(move || {
        tx.send("Hello from thread").unwrap();
    });

    let received = rx.recv().unwrap();
    println!("{}", received);
}

Back to top

Languages

Modern programming languages provide varying levels of support for concurrent programming:

Native Concurrency Support

Strong Standard Library Support

Library/Framework Support

Back to top

Ref.

Books:

Articles and Papers:

Online Resources:

Language-Specific:

Related Paradigms:

Get Started