How Computer I/O Works: Buses, DMA, GPUs, Multicore CPUs, and SoCs

In the previous chapters, we followed computation from a single transistor all the way to complete instruction execution. We saw how the CPU fetches and decodes machine code, performs operations, accesses memory, and writes results back into architectural state.

A useful computer still needs more than a CPU and RAM.

It must receive input, display output, store files, communicate over networks, move large blocks of data, and divide work among several processors. Those jobs require device controllers, drivers, interrupts, Direct Memory Access, system interconnects, multicore coordination, GPUs, and increasingly integrated system-on-chip designs.

This chapter explains how the CPU communicates with the rest of the computer and how modern systems combine several specialized processing units into one coordinated platform.

What You Will Learn

By the end of this chapter, you will understand:

• what input and output mean in computer architecture;
• how device controllers expose registers and buffers;
• how memory-mapped I/O differs from ordinary RAM;
• why some architectures also support port-mapped I/O;
• when polling is useful;
• how interrupts notify the CPU;
• how interrupt controllers route device events;
• what Direct Memory Access does;
• why DMA still requires CPU setup and memory-system coordination;
• how an IOMMU protects DMA-capable devices;
• why modern systems use interconnects instead of one simple shared bus;
• what PCI Express does;
• how device drivers connect software to hardware;
• how multicore processors execute work in parallel;
• why cache coherence and memory ordering matter;
• how GPUs differ from CPUs;
• what a System on a Chip contains;
• what unified memory does and does not mean;
• how heterogeneous computing combines CPUs, GPUs, and accelerators.

The CPU Is Part of a Larger System

A CPU can execute instructions and manipulate values, but it cannot provide a complete user experience by itself.

A working computer also needs hardware that can:

• receive keyboard, mouse, touch, camera, and sensor input;
• produce video and audio output;
• communicate through wired or wireless networks;
• store information after power is removed;
• control timers and clocks;
• communicate with external peripherals;
• move data between memory and devices;
• accelerate specialized workloads.

These components communicate through controllers and interconnects.

The operating system coordinates access so that multiple applications can use the hardware safely and efficiently.

What Is Input and Output?

Input/Output, usually shortened to I/O, describes communication between the processor-memory system and other devices.

Input brings information into the system.

Examples include:

• keyboard events;
• mouse movement;
• network packets;
• microphone samples;
• camera frames;
• storage reads;
• sensor measurements.

Output sends information from the system.

Examples include:

• display updates;
• audio samples;
• network transmissions;
• printer data;
• storage writes;
• commands sent to a motor or controller.

Many devices are bidirectional. A network adapter, storage controller, USB controller, and graphics processor can both receive and transmit information.

The CPU Usually Talks to a Device Controller

The CPU rarely controls the physical details of a device directly.

Instead, it communicates with a device controller.

A controller may contain:

• configuration registers;
• status registers;
• command registers;
• data buffers;
• queues;
• interrupt logic;
• DMA engines;
• protocol-specific hardware.

For example, a storage controller translates high-level read and write requests into operations understood by the storage device.

A network controller receives frames, verifies them, places their contents into memory buffers, and notifies the operating system.

A display controller reads image data and generates the timed electrical signals needed by a display interface.

This division allows the CPU to issue commands without implementing every low-level protocol in software.

Device Registers

Device controllers commonly expose registers that software can read or write.

Typical register types include:

Status Register

Reports conditions such as:

• ready;
• busy;
• error;
• data available;
• transfer complete.

Command Register

Starts or configures an operation.

For example:

START_TRANSFER = 1

Data Register

Provides a small amount of data directly.

This is common for simple or low-bandwidth devices.

Address Register

Contains the memory address of a buffer used for a transfer.

Length Register

Specifies how many bytes or descriptors should be processed.

Interrupt-Control Register

Enables, disables, acknowledges, or configures interrupts.

The exact register layout is defined by the device or platform specification.

Memory-Mapped I/O

In memory-mapped I/O, device registers are assigned addresses within a processor-visible address space.

The CPU uses load and store instructions to access those addresses.

Conceptually:

STORE device_command_address, command_value
LOAD  status_value, device_status_address

Although the instructions resemble normal memory operations, the target is a device controller rather than ordinary RAM.

This allows the processor to use the same general addressing and instruction mechanisms for both memory and devices.

Memory-Mapped I/O Is Not Ordinary Memory

A memory-mapped device region can behave differently from RAM.

Possible differences include:

• reads may have side effects;
• writes may start hardware operations;
• values may change independently of the CPU;
• accesses may need a specific width;
• access order may matter;
• caching may be disabled or restricted;
• speculative access may be unsafe;
• some addresses may not support every operation.

For example, reading a status register might clear an interrupt condition.

Writing a command register twice might start two operations.

A compiler and CPU must therefore follow the ordering and access rules required for device memory.

Operating systems use special primitives rather than treating device registers as ordinary variables.

Port-Mapped I/O

Memory-mapped I/O is common, but it is not the only model.

Some architectures support a separate I/O address space accessed through special instructions.

This is often called:

• port-mapped I/O;
• isolated I/O;
• programmed I/O ports.

Traditional x86 systems, for example, support special input and output instructions for an I/O-port space in addition to memory-mapped device access.

The distinction is:

• memory-mapped I/O uses normal memory-addressing instructions;
• port-mapped I/O uses a separate address space and specialized instructions.

Many modern devices still rely heavily on memory-mapped control structures.

A Simple Device-Control Example

Imagine a fictional device with these registers:

0xF0000000  COMMAND
0xF0000004  STATUS
0xF0000008  BUFFER_ADDRESS
0xF000000C  LENGTH

Software might perform:

1. Write the RAM buffer address to BUFFER_ADDRESS.
2. Write the byte count to LENGTH.
3. Write START to COMMAND.
4. Wait for completion.
5. Read STATUS to check for errors.

The waiting step can use polling, an interrupt, or another synchronization mechanism.

Polling

Polling means repeatedly reading a device’s status to determine whether an event has occurred.

Conceptually:

while device_status != READY:
    keep checking

Polling is simple and can provide low response latency when an event is expected almost immediately.

It is useful in situations such as:

• very short hardware waits;
• boot firmware;
• tiny embedded systems;
• simple devices;
• real-time loops;
• high-throughput networking under heavy load;
• situations where interrupt overhead would exceed the wait.

The Cost of Polling

Polling can waste processor time and energy when events are infrequent.

If a device completes one operation every second, checking its status millions of times between events is inefficient.

Polling also creates traffic on the system interconnect and may prevent a CPU core from entering a low-power state.

The key trade-off is:

• frequent polling can reduce response latency;
• interrupts can reduce unnecessary checking.

Polling is therefore not always bad. Its value depends on expected wait time, event rate, latency requirements, and processor availability.

Interrupts

An interrupt is a hardware or software event that requests processor attention.

A device can raise an interrupt when:

• data arrives;
• a transfer finishes;
• an error occurs;
• a timer expires;
• a buffer needs servicing.

The CPU can continue other work until the event occurs.

This makes interrupts useful when device events are unpredictable or relatively infrequent.

Interrupt Handling Step by Step

A simplified hardware-interrupt sequence is:

1. A device or controller raises an interrupt request.
2. An interrupt controller records and routes the request.
3. The processor reaches a point where the interrupt can be accepted.
4. Enough execution state is preserved.
5. Control transfers to an interrupt handler.
6. The handler identifies and services the event.
7. The interrupt condition is acknowledged or cleared.
8. The interrupted execution context is restored.
9. Normal execution resumes.

The exact sequence depends on the architecture and operating system.

Interrupt Controllers

A modern system can contain many interrupt sources.

An interrupt controller helps:

• collect interrupt requests;
• assign priorities;
• route events to particular CPU cores;
• mask selected interrupts;
• represent interrupt vectors;
• handle message-signaled interrupts.

Without such coordination, every device would need a direct dedicated signal path to every processor.

Interrupt Vectors

An interrupt vector identifies the type or source of an interrupt.

The processor or operating system uses that value to find the appropriate handler.

Different vectors may represent:

• timer interrupts;
• network events;
• storage completion;
• device errors;
• inter-processor interrupts;
• architecture-defined exceptions.

Interrupt-vector organization is platform-specific.

Interrupts vs Exceptions

Interrupts and exceptions both transfer control to special handlers, but the terms usually describe different event sources.

Interrupt

Generally associated with an asynchronous event outside the current instruction stream.

Examples include:

• timer expiration;
• network packet arrival;
• device completion.

Exception

Associated with instruction execution.

Examples include:

• page fault;
• invalid instruction;
• divide-by-zero;
• protection violation.

Architectures differ in terminology, but this distinction is useful.

Interrupt Overhead

Interrupts are not free.

Handling an interrupt may require:

• saving and restoring state;
• changing privilege level;
• disturbing caches and pipelines;
• running operating-system code;
• acknowledging the device;
• scheduling deferred processing.

A device producing extremely frequent interrupts can consume substantial CPU time.

This is sometimes called an interrupt storm.

Systems reduce this cost with techniques such as:

• interrupt coalescing;
• batching;
• deferred processing;
• hybrid polling;
• distributing interrupts across cores.

Hybrid Polling and Interrupts

High-performance systems often combine both techniques.

For example, a network interface may:

1. raise an interrupt when new traffic arrives;
2. temporarily switch to polling while many packets are queued;
3. process several packets as a batch;
4. re-enable interrupts when traffic becomes quiet.

This avoids an interrupt for every packet while also avoiding constant polling when no work exists.

The best strategy depends on workload conditions.

Programmed I/O

In programmed I/O, the CPU directly reads or writes device data through device registers.

For a small transfer, the processor might repeatedly copy values between a device data register and RAM.

This is straightforward, but it consumes CPU instructions for every transferred unit.

Programmed I/O is appropriate for small, simple, or infrequent transfers.

For large transfers, DMA is usually more efficient.

Direct Memory Access

Direct Memory Access, or DMA, allows a device or DMA engine to transfer data between a device and main memory without requiring the CPU to copy every byte itself.

The CPU still participates by:

• allocating or preparing buffers;
• configuring the device;
• providing memory addresses;
• setting transfer lengths;
• establishing permissions;
• handling completion and errors.

Once configured, the device can move the data through the system interconnect.

DMA Read and Write Directions

From the device’s point of view:

Device Reads Memory

The device obtains data from RAM.

Examples include:

• a network adapter reading a packet to transmit;
• a storage controller reading data to write to an SSD;
• a GPU reading vertex or texture data.

Device Writes Memory

The device places data into RAM.

Examples include:

• a network adapter storing a received packet;
• a storage controller delivering read data;
• a camera controller writing a video frame.

Terminology can vary, so documentation should clarify whether “read” is described from the CPU, memory, or device perspective.

DMA Transfer Step by Step

A simplified DMA receive operation might work as follows:

1. The operating system allocates a memory buffer.
2. The driver makes that buffer usable by the device.
3. The driver writes the buffer address into a descriptor.
4. The driver tells the controller that the descriptor is ready.
5. The device receives data.
6. The device transfers the data into RAM.
7. The device records completion status.
8. The device raises an interrupt or updates a completion queue.
9. The driver verifies the result.
10. Software consumes the data.

The CPU avoids copying each byte, but it still controls setup and completion.

Descriptor Rings and Queues

High-performance devices often use memory-resident descriptors.

A descriptor may contain:

• buffer address;
• buffer length;
• operation flags;
• ownership information;
• completion status.

Descriptors can be arranged in:

• rings;
• linked lists;
• submission queues;
• completion queues.

The CPU and device exchange ownership of these structures.

This reduces the number of direct device-register operations and allows many transfers to be queued at once.

DMA Does Not Bypass the Whole System

The phrase “the device transfers directly to RAM” is useful, but incomplete.

DMA traffic still travels through:

• device controllers;
• PCIe or another interconnect;
• bridges or fabrics;
• memory controllers;
• cache-coherence or consistency mechanisms;
• address-translation and protection structures.

DMA reduces CPU copying. It does not remove the rest of the computer architecture.

DMA and Cache Coherence

CPU caches may contain copies of memory that a device reads or writes.

The system must ensure that the CPU and device observe the correct data.

Depending on the platform:

• DMA may participate in hardware cache coherence;
• software may flush modified cache lines before a device reads them;
• software may invalidate cache lines after a device writes them;
• special uncached or coherent mappings may be used.

Device drivers rely on operating-system DMA APIs because the correct handling is architecture-specific.

The IOMMU

An Input-Output Memory Management Unit, or IOMMU, translates and protects memory accesses initiated by devices.

It is conceptually related to the CPU’s MMU.

An IOMMU can provide:

• DMA address translation;
• device isolation;
• access permissions;
• safer device assignment to virtual machines;
• controlled sharing of buffers;
• protection against accidental or malicious DMA access.

Without protection, a bus-mastering device might be able to access physical memory beyond the buffers intended for it.

Device Virtual Addresses

A device may receive an I/O virtual address rather than a raw physical RAM address.

The IOMMU translates that address into the physical pages backing the buffer.

This allows the operating system to present a contiguous device-visible region even when the physical pages are not contiguous.

The details vary by platform and device.

Buses and Interconnects

A bus is a communication path shared by components.

Traditional computer diagrams often separate:

• address bus;
• data bus;
• control bus.

This model explains the logical categories of information being transferred.

Modern systems frequently use point-to-point links, switches, packetized protocols, and on-chip networks rather than one electrically shared bus.

The broader term interconnect is often more accurate.

Bandwidth and Latency

Interconnect performance has at least two important dimensions.

Bandwidth

The amount of data that can be transferred per unit of time.

It is often measured in:

• bytes per second;
• transfers per second;
• bits per second.

Latency

The delay before a request or transfer produces a response.

A link can provide high bandwidth while still having noticeable latency for one small transaction.

High-bandwidth devices benefit from batching and multiple outstanding requests.

Bus Width Does Not Alone Determine Capacity

A simple teaching system may use an address bus with one wire per address bit.

Under that model, n address bits can distinguish up to 2^n address values.

However, real system limits also depend on:

• implemented address width;
• address decoding;
• virtual-memory rules;
• physical-memory-controller capability;
• chipset and board design;
• reserved device regions;
• operating-system support.

A 32-bit address space contains 2^32 byte addresses, but the usable RAM available to software may be smaller because parts of that space can be reserved for devices and system structures.

Shared Buses

On a shared bus, several devices use one communication path.

Only one or a limited number of participants can transfer at a time.

An arbiter decides who gains access.

Shared buses are conceptually simple but can become bottlenecks as the number and speed of devices increase.

Modern high-performance systems therefore use hierarchical and switched interconnects.

Point-to-Point Links

A point-to-point link connects two endpoints directly or through a switching fabric.

Benefits can include:

• greater aggregate bandwidth;
• simultaneous independent transfers;
• improved scalability;
• better electrical signaling at high speed;
• clearer fault isolation.

PCI Express is a prominent example.

PCI Express

PCI Express, or PCIe, is a high-speed serial interconnect used by devices such as:

• discrete GPUs;
• NVMe storage controllers;
• network adapters;
• accelerator cards;
• capture devices;
• expansion controllers.

PCIe uses point-to-point links rather than the parallel shared-bus design of older PCI systems.

A PCIe connection contains one or more lanes.

Each lane carries data in both directions.

Links are described with widths such as:

x1
x4
x8
x16

A wider link can provide more aggregate bandwidth, assuming the endpoints and generation support it.

PCIe Transactions

Software and devices do not simply place raw values on one global bus.

PCIe uses packetized transactions for operations such as:

• memory reads;
• memory writes;
• configuration access;
• messages;
• completion responses.

A PCIe hierarchy may contain:

• a root complex;
• switches;
• endpoints;
• bridges.

The physical topology and lane allocation vary between systems.

Message-Signaled Interrupts

Many PCIe devices use message-signaled interrupts.

Instead of asserting a dedicated interrupt wire, the device sends a special transaction representing the interrupt.

Benefits include:

• scalable interrupt counts;
• better routing;
• support for multiple queues;
• reduced dependence on shared physical lines.

Modern high-performance devices often use several interrupt vectors so different queues can be assigned to different CPU cores.

Memory Interconnects

The CPU communicates with DRAM through a memory controller and memory channels.

Modern processors commonly integrate the memory controller onto the processor package or die.

Memory access still involves:

• address translation;
• cache lookup;
• coherence;
• request queues;
• memory-controller scheduling;
• DRAM commands;
• data transfer.

A “memory bus” is therefore not one simple wire bundle from every core directly to every RAM cell.

On-Chip Interconnects

A multicore processor or SoC needs an internal fabric connecting:

• CPU cores;
• cache slices;
• memory controllers;
• GPUs;
• media engines;
• I/O controllers;
• accelerators.

Possible interconnect organizations include:

• rings;
• crossbars;
• meshes;
• networks-on-chip;
• hierarchical fabrics.

The choice affects:

• latency;
• bandwidth;
• power;
• scalability;
• cache coherence;
• physical layout.

As chips become more complex, moving data efficiently can be as important as the execution units themselves.

What Is a Peripheral?

A peripheral is a hardware component that provides input, output, storage, communication, sensing, or another service outside the CPU’s main execution core.

Examples include:

• keyboard controller;
• USB device;
• network adapter;
• SSD;
• sound device;
• display adapter;
• camera;
• printer;
• sensor;
• accelerator.

The exact boundary is contextual. A GPU integrated into an SoC may still behave as an I/O and compute device even though it is physically on the same chip.

Device Drivers

A device driver is software that knows how to control a particular device or device class.

A driver may:

• configure device registers;
• allocate DMA buffers;
• build command descriptors;
• handle interrupts;
• manage queues;
• report errors;
• expose a standard operating-system interface;
• enforce security and access rules;
• manage power states.

Applications usually interact with operating-system abstractions rather than directly manipulating hardware registers.

Examples include:

• files;
• sockets;
• graphics APIs;
• audio APIs;
• input-event interfaces;
• device-control requests.

Applications Do Not Always Avoid Hardware Completely

Most user applications cannot directly access arbitrary device registers because the operating system protects them.

However, some systems support controlled forms of direct or near-direct access, including:

• memory-mapped device buffers;
• user-space driver frameworks;
• GPU command submission;
• high-performance networking queues;
• device assignment to virtual machines.

These mechanisms still rely on permissions, drivers, kernels, runtimes, or IOMMUs to preserve isolation.

The statement “applications never talk to hardware directly” is therefore too absolute.

System Calls and Device Access

An application often requests I/O through a system call.

For a file read:

1. The application requests bytes through an operating-system API.
2. The kernel identifies the file and device.
3. The filesystem maps the request to storage blocks.
4. The storage driver prepares one or more device commands.
5. The device transfers data, often with DMA.
6. Completion is reported.
7. The kernel makes the data available to the application.

Caches may satisfy some reads without touching the physical storage device.

Multicore Processors

A multicore processor contains multiple CPU cores in one package or chip design.

Each core normally has its own:

• execution units;
• architectural registers;
• instruction-processing state;
• low-level control logic.

Cores may share:

• last-level cache;
• memory controllers;
• on-chip interconnect;
• power and thermal budget;
• I/O interfaces.

The exact sharing arrangement varies by processor.

Why Multicore CPUs Became Common

Increasing clock frequency can improve performance, but it also increases power and thermal pressure.

Other physical and architectural limits include:

• voltage scaling constraints;
• long wire delays;
• diminishing instruction-level parallelism;
• memory latency;
• power-density limits.

Adding cores allows several instruction streams to make progress at once.

This does not automatically accelerate every program. Software must expose useful parallel work.

Concurrency vs Parallelism

Concurrency means multiple tasks can make progress during overlapping periods.

They do not need to execute at the exact same instant.

A single core can support concurrency by switching between tasks.

Parallelism means multiple tasks or operations execute at the same time on different hardware resources.

A multicore processor enables true parallel execution of several software threads.

Concurrency is about organizing multiple activities.

Parallelism is about simultaneous execution.

Threads and Cores

An operating system schedules software threads onto logical processors.

A logical processor may correspond to:

• one physical core;
• one hardware thread within a core;
• another architecture-defined execution context.

If more runnable threads exist than logical processors, the operating system time-slices them.

Each context switch changes which thread’s architectural state is active.

Simultaneous Multithreading

Some CPU cores support more than one hardware thread.

This is called Simultaneous Multithreading, or SMT.

The hardware threads may have separate architectural register state while sharing many core resources, such as:

• execution units;
• caches;
• schedulers;
• branch-prediction structures;
• bandwidth.

SMT can improve utilization when one thread cannot use every core resource.

It does not duplicate the entire physical core.

Shared-Memory Multicore Systems

Threads on different cores can communicate through shared memory.

One core may write a value that another core later reads.

Correct communication requires attention to:

• cache coherence;
• memory ordering;
• atomicity;
• synchronization;
• data races.

The fact that all cores can address the same RAM does not mean every core instantly observes every write in a simple universal order.

Cache Coherence

Different cores may cache copies of the same memory line.

A cache-coherence protocol coordinates those copies so that writes do not leave other cores indefinitely using stale data.

The protocol may track states such as:

• shared;
• modified;
• invalid;
• exclusive;
• owned.

Specific state names and transitions depend on the protocol.

Coherence answers questions about the visibility of individual cache lines.

It does not by itself define every ordering rule for all memory operations.

Memory Ordering

Processors may reorder or overlap loads and stores internally.

The architecture defines a memory model describing which effects software may observe.

Some architectures provide a relatively strong ordering model.

Others permit more reordering.

Multithreaded code uses:

• atomic operations;
• locks;
• fences;
• acquire and release semantics;
• language-level synchronization;

to establish correct communication.

Without synchronization, two threads can have a data race even when the cache-coherence hardware is functioning correctly.

Race Conditions

A race condition occurs when program behaviour depends on the timing or ordering of concurrent operations.

Consider two threads incrementing a shared counter:

counter = counter + 1

This operation may involve:

1. reading the old value;
2. adding one;
3. writing the new value.

If two cores interleave these steps without synchronization, one increment can be lost.

Locks, atomic read-modify-write instructions, and other synchronization mechanisms prevent such errors.

False Sharing

Two independent variables can occupy the same cache line.

If different cores repeatedly modify those variables, the coherence system may move ownership of the line back and forth even though the threads are not logically sharing one variable.

This is called false sharing.

It can severely reduce performance.

Data layout and padding can help separate heavily written values onto different cache lines.

NUMA Systems

Large multicore or multiprocessor systems may use Non-Uniform Memory Access, or NUMA.

All processors can access the shared address space, but memory attached to the local processor or node is usually faster to reach.

Remote memory requires traversal through additional interconnects.

Operating systems attempt to place:

• threads;
• memory pages;
• I/O devices;

near one another when beneficial.

NUMA awareness matters for large servers and memory-intensive applications.

Inter-Processor Interrupts

CPU cores sometimes need to signal one another.

An inter-processor interrupt can be used to:

• request rescheduling;
• invalidate stale TLB entries;
• coordinate shutdown;
• deliver operating-system events;
• stop or wake another core.

These signals are part of the control mechanisms that allow a multicore operating system to manage shared state.

CPU vs GPU

A CPU is designed for low-latency execution of diverse instruction streams.

A GPU is designed for high throughput across large numbers of similar operations.

The distinction is not simply “few strong cores versus thousands of weak cores.” Modern CPUs and GPUs are both complex processors with caches, schedulers, execution units, and sophisticated control logic.

Their priorities differ.

CPU Design Priorities

A CPU commonly emphasizes:

• strong single-thread performance;
• sophisticated branch prediction;
• low-latency caches;
• out-of-order execution;
• fast context switching;
• flexible control flow;
• general-purpose operating-system support.

CPUs are well suited to:

• serial or lightly parallel code;
• irregular algorithms;
• branch-heavy workloads;
• operating systems;
• interactive application logic;
• task coordination.

GPU Design Priorities

A GPU commonly emphasizes:

• high arithmetic throughput;
• many execution lanes;
• large numbers of concurrent threads;
• latency hiding through scheduling;
• high memory bandwidth;
• specialized graphics and matrix hardware.

GPUs are well suited to workloads with abundant data parallelism, such as:

• graphics rendering;
• image processing;
• video operations;
• machine learning;
• physical simulation;
• numerical computing;
• matrix and vector calculations.

GPU Cores Are Not Directly Comparable to CPU Cores

A marketing count of GPU “cores” does not describe units equivalent to complete CPU cores.

A CPU core can independently execute a complex general-purpose instruction stream with substantial control and scheduling hardware.

A GPU execution lane is normally part of a wider grouped execution structure.

The programming and scheduling model differs.

Core counts across CPU and GPU architectures should therefore not be compared directly.

SIMD, SIMT, and Vector Execution

GPUs often execute groups of threads in a coordinated manner.

Depending on the vendor and programming model, this is described with terms such as:

• SIMD;
• SIMT;
• wavefront;
• warp;
• subgroup.

Several lanes perform the same instruction on different data elements.

This provides excellent throughput when the threads follow similar control flow.

Branch Divergence

If threads in one execution group take different branch paths, the GPU may need to execute the paths separately while masking inactive lanes.

This is called branch divergence.

Divergence reduces lane utilization and can lower performance.

GPUs therefore work best when nearby threads perform similar operations.

Hiding Memory Latency

GPUs often keep many groups of threads ready.

When one group waits for memory, the scheduler can run another group.

This hides latency through massive concurrency.

A CPU instead invests more hardware in reducing the latency of a smaller number of instruction streams through:

• large caches;
• branch prediction;
• speculative execution;
• out-of-order scheduling.

Both approaches aim to keep execution hardware busy.

CPU and GPU Cooperation

A common heterogeneous workflow is:

1. The CPU prepares commands and data.
2. The CPU submits work to the GPU.
3. The GPU processes many data elements in parallel.
4. The GPU signals completion.
5. The CPU consumes or presents the result.

Communication may use:

• shared virtual memory;
• unified physical memory;
• explicit buffer copies;
• command queues;
• synchronization objects.

The best model depends on the hardware and software platform.

Discrete vs Integrated GPUs

A discrete GPU is a separate device, often with dedicated graphics memory.

It commonly communicates with the CPU over PCIe.

An integrated GPU is built into the same processor package or SoC and often shares system memory.

Integrated does not automatically mean slow, and discrete does not automatically mean faster for every workload.

Performance depends on:

• architecture;
• memory bandwidth;
• power budget;
• cooling;
• software;
• workload;
• accelerator support.

What Is a System on a Chip?

A System on a Chip, or SoC, integrates many system functions into one chip or tightly integrated package.

An SoC may contain:

• CPU cores;
• GPU cores;
• memory controllers;
• cache;
• display engines;
• media encoders and decoders;
• neural-processing accelerators;
• security processors;
• audio hardware;
• camera-processing units;
• storage controllers;
• USB or PCIe controllers;
• wireless or cellular interfaces;
• power-management logic.

The exact contents depend on the product.

Smartphones, tablets, embedded devices, game consoles, and many modern personal computers use highly integrated SoCs.

SoC Does Not Necessarily Mean RAM Is on the Same Die

Some systems place memory chips in the same package as the processor.

Others place memory modules on the motherboard.

A marketing description such as “system on a chip” does not guarantee that DRAM cells are fabricated on the same silicon die as the CPU and GPU.

Integration can occur at several physical levels:

• same die;
• same package;
• closely connected package;
• same board.

The architectural benefit comes from tight coordination and high-bandwidth interconnects, not merely from one label.

Unified Memory

Unified memory can refer to several related designs in which processors or accelerators share a memory system or address space.

Possible benefits include:

• reduced need for explicit CPU-to-GPU copies;
• easier sharing of data structures;
• flexible memory allocation;
• lower software complexity;
• efficient zero-copy workflows.

However, the phrase does not mean every processing unit has identical access latency or bandwidth.

CPU cores, GPU cores, accelerators, and devices can still have:

• different caches;
• different access paths;
• different consistency rules;
• different preferred data layouts;
• different bandwidth demands.

Unified addressing and physically shared memory are related but not identical concepts.

Heterogeneous Computing

A modern SoC or computer can contain several types of processor.

Examples include:

• general-purpose CPU cores;
• GPU compute units;
• neural-processing units;
• digital signal processors;
• media engines;
• cryptographic accelerators;
• image signal processors.

This is called heterogeneous computing.

The system assigns each task to hardware suited to its structure.

For example:

• the CPU handles control-heavy application logic;
• the GPU renders graphics;
• a media engine decodes video;
• a neural accelerator runs matrix-heavy inference;
• a security processor manages protected keys.

Specialization can improve performance and energy efficiency.

Big and Small CPU Cores

Some processors combine high-performance and energy-efficient CPU cores.

The performance-oriented cores may provide:

• wider execution;
• larger caches;
• higher clock frequencies;
• stronger single-thread performance.

Efficiency-oriented cores may use:

• less chip area;
• less power;
• simpler structures;
• lower peak performance.

The operating system scheduler chooses where to run threads according to workload, priority, thermal conditions, and power policy.

The specific design and instruction compatibility vary by platform.

The Operating System as Resource Manager

The operating system coordinates the major parts of the system.

It manages:

• processes and threads;
• virtual memory;
• devices and drivers;
• interrupts;
• filesystems;
• security and permissions;
• CPU scheduling;
• power states;
• network communication;
• memory allocation.

Applications use abstractions such as:

• files;
• sockets;
• processes;
• windows;
• graphics contexts;
• audio streams;
• memory mappings.

These abstractions hide most hardware-specific details while preserving controlled access.

Data Movement Often Limits Performance

Arithmetic units can perform enormous numbers of operations, but they need data.

A system may be limited by:

• cache misses;
• memory bandwidth;
• PCIe transfers;
• storage latency;
• synchronization;
• device queue depth;
• NUMA placement;
• memory copies;
• serialization between processing units.

For many workloads, reducing unnecessary data movement is more valuable than increasing arithmetic capability.

This principle influences:

• cache design;
• SoC integration;
• unified memory;
• accelerator placement;
• near-memory computing;
• zero-copy I/O.

Zero-Copy I/O

Zero-copy describes techniques that reduce unnecessary copying between buffers.

A traditional data path may copy information several times:

Device -> Kernel buffer -> Application buffer -> Another device

A zero-copy design may allow components to share or remap buffers instead.

Examples include:

• memory-mapped files;
• shared DMA buffers;
• sendfile-style operating-system operations;
• shared CPU-GPU memory;
• network buffer reuse.

“Zero-copy” does not always mean no physical data movement occurs. It usually means avoiding extra CPU-mediated copies between software buffers.

Security and Device Access

I/O hardware has access to sensitive system resources.

Security mechanisms include:

• privileged driver execution;
• memory protection;
• IOMMU isolation;
• signed firmware;
• secure boot;
• restricted device access;
• encrypted storage;
• virtualization boundaries;
• permission-controlled APIs.

A faulty or malicious device driver can compromise system stability.

DMA-capable devices require particular care because they can initiate memory accesses independently of normal CPU instructions.

Virtual Machines and Device I/O

A hypervisor allows several virtual machines to share one physical system.

Device access may be provided through:

• emulated devices;
• paravirtualized devices;
• shared host drivers;
• direct device assignment;
• virtual functions.

An IOMMU helps isolate DMA so an assigned device can access only memory belonging to the correct virtual machine.

Virtualization adds another layer of address translation and control.

Common I/O and System Architecture Misconceptions

Every Device Looks Exactly Like Ordinary RAM

Memory-mapped registers can have side effects, ordering requirements, and non-cacheable behaviour.

Every Architecture Uses Only Memory-Mapped I/O

Some architectures, including x86, also support separate port-mapped I/O.

Polling Is Always Inefficient

Polling can be effective for short waits, high event rates, and latency-sensitive loops.

Interrupts Have No Cost

Interrupt handling consumes CPU time and can reduce performance when events are extremely frequent.

DMA Means the CPU Is Completely Uninvolved

The CPU and operating system still configure buffers, permissions, descriptors, completion handling, and errors.

DMA Bypasses Caches and Address Translation in Every System

Coherency rules and IOMMU translation vary by platform.

A Bus Is Always One Shared Set of Wires

Modern systems commonly use switched, point-to-point, and packetized interconnects.

A 32-Bit Address Bus Always Provides 4 GB of Usable RAM

Reserved address regions, implementation limits, and system design can reduce usable memory.

Every Application Is Completely Separated from Hardware

Most access is mediated, but controlled user-space device access and mapped queues are possible.

Each GPU Core Is Equivalent to a CPU Core

CPU and GPU execution units use different organizations and programming models.

More CPU Cores Automatically Make Every Program Faster

Software must contain parallel work and avoid synchronization bottlenecks.

Cache Coherence Eliminates the Need for Synchronization

Coherence maintains cache-line consistency. Software still needs atomics, locks, or memory-ordering rules.

Unified Memory Means Every Processor Has Identical Performance

Shared addressing or memory does not remove differences in cache, latency, bandwidth, and execution architecture.

SoC Means Every Component Is Fabricated on One Silicon Die

Integration can occur on the same die, within one package, or through another tightly connected design.

Frequently Asked Questions

How does a CPU communicate with an I/O device?

The CPU communicates with a device controller through registers, queues, memory buffers, and interconnect protocols. The operating system’s driver configures those mechanisms.

What is memory-mapped I/O?

Memory-mapped I/O assigns device registers addresses within a processor-visible address space so the CPU can access them with load and store instructions.

What is the difference between polling and interrupts?

Polling repeatedly checks device state. Interrupts allow a device to notify the processor when attention is required.

Is polling ever better than interrupts?

Yes. Polling can be better for very short waits, extremely high event rates, deterministic loops, or cases where interrupt overhead is too large.

What is DMA?

DMA allows a device or DMA controller to transfer data between the device and memory without requiring the CPU to copy every byte.

Does DMA use physical addresses?

A device may use physical addresses or I/O virtual addresses translated by an IOMMU, depending on the platform.

What is an IOMMU?

An IOMMU translates and protects device-initiated memory accesses, helping isolate DMA-capable devices and virtual machines.

What is PCI Express?

PCI Express is a high-speed packetized point-to-point interconnect used for GPUs, storage, networking, and other expansion devices.

What is the difference between a bus and an interconnect?

A bus traditionally refers to a shared communication path. Interconnect is a broader term that includes shared buses, point-to-point links, rings, meshes, and switched fabrics.

What is a device driver?

A device driver is operating-system software that configures hardware, manages data transfers, handles interrupts, and exposes a standard interface to applications.

What is the difference between concurrency and parallelism?

Concurrency allows multiple tasks to make progress during overlapping periods. Parallelism means tasks execute simultaneously on separate hardware resources.

What is cache coherence?

Cache coherence coordinates copies of shared cache lines so cores do not indefinitely use stale values after another core writes them.

Why are atomic operations needed if caches are coherent?

Coherence does not make a multi-step software operation indivisible. Atomic operations and synchronization establish correct shared-memory behaviour.

What is the main difference between a CPU and GPU?

A CPU prioritizes low-latency, flexible execution of diverse control-heavy workloads. A GPU prioritizes high throughput across many similar operations.

What is an SoC?

An SoC integrates CPU cores and many other controllers or accelerators into one chip or tightly integrated package.

What is unified memory?

Unified memory describes systems where processors or accelerators share memory resources or an address space. Exact behaviour varies by platform.

Why do modern systems use specialized accelerators?

Specialized hardware can perform a narrow class of work with higher performance or lower energy use than a general-purpose CPU.

Key Takeaways

A modern computer is a coordinated system of processors, memory, controllers, devices, and interconnects.

The central ideas are:

1. Devices are commonly controlled through controller registers and command queues.
2. Memory-mapped I/O uses normal load and store instructions but does not behave exactly like RAM.
3. Some architectures also support port-mapped I/O.
4. Polling trades CPU work and energy for predictable or low response latency.
5. Interrupts notify the processor but introduce handling overhead.
6. High-performance systems often combine polling and interrupts.
7. DMA lets devices move data without CPU copying every byte.
8. The CPU still configures DMA and handles completion and errors.
9. IOMMUs translate and protect device-initiated memory accesses.
10. Modern systems use point-to-point and packetized interconnects instead of only simple shared buses.
11. PCIe connects many high-performance peripheral devices.
12. Device drivers translate operating-system requests into hardware operations.
13. Multicore processors run several instruction streams in parallel.
14. Cache coherence, memory ordering, and synchronization are all required for correct shared-memory software.
15. GPUs prioritize throughput across data-parallel workloads.
16. CPU and GPU core counts cannot be compared directly.
17. SoCs integrate several processing and I/O components into one tightly connected design.
18. Unified memory simplifies sharing but does not make every access path identical.
19. Heterogeneous systems assign different tasks to specialized processors and accelerators.
20. Data movement, latency, and synchronization can matter as much as arithmetic speed.

We began this series with the smallest electronic switch and gradually built an entire computer system:

Transistors
    -> Logic gates
    -> Adders and storage circuits
    -> Arithmetic Logic Unit
    -> Control Unit
    -> Memory hierarchy
    -> Instruction execution
    -> I/O, multicore processors, GPUs, and SoCs

At every level, the same principles return:

• store state;
• move information;
• transform data;
• coordinate events.

A computer becomes powerful not because one component does everything, but because many specialized components exchange information through carefully designed interfaces. The CPU remains central to general-purpose execution, but the complete system comes to life through memory, devices, accelerators, operating-system software, and the interconnects that bind them together.