8.5. Asynchronous I/O in PostgreSQL

PostgreSQL 18 (2025) introduced the Asynchronous I/O (AIO) feature to improve read performance.

The buffer manager, therefore, has been refactored, especially for cases where the executor performs sequential scans, bitmap heap scans, or vacuum processes that read target pages sequentially. (AIO in PostgreSQL does not support asynchronous writing yet.)

This section explains the basic concept of AIO, describes its implementation, and details its operation, focusing on Sequential Scan.

Additionally, the Appendix provides sample io_uring programs for readers unfamiliar with the framework.

8.5.1. Asynchronous I/O

Figure 8.15 compares the execution flow of conventional synchronous reads and asynchronous reads. (The page cache is omitted to simplify the explanation.)

• Synchronous Read: Figure 8.15 [1] — an application issues a read() system call, waits until the operation finishes, and then issues the next request. Consequently, multiple read operations run sequentially.

• Asynchronous Read: Figure 8.15 [2] — an application can submit multiple read requests without waiting for each individual request to complete. The OS processes these requests in the background, and the application later collects the results.

8.5.1.1. Benefits of AIO

Asynchronous I/O provides performance benefits for the application and the storage.

• Application: Applications can execute other tasks while waiting for read operations to complete.

• Storage: The effect is significant for SSDs. SSDs can internally process multiple I/O requests in parallel, achieving high throughput when many read requests are issued simultaneously. This benefit stems from the multiple queues and high queue depths in NVMe SSDs.

Due to these benefits, PostgreSQL can issue many page read requests at once, allowing the OS and modern SSDs to exploit their internal parallelism. As a result, buffer pool loading and large table scans achieve significantly higher throughput.

8.5.2. AIO Implementation in PostgreSQL

PostgreSQL provides two implementations of asynchronous I/O: io_uring and io_worker background workers.

These can be chosen by the io_method configuration parameter. Since io_uring is a Linux-specific feature, other operating systems, such as macOS and BSD, must select io_worker.

8.5.2.1. io_uring

io_uring is an asynchronous I/O interface introduced in Linux kernel 5.1 (2019).

When using io_uring, PostgreSQL creates a pair of shared ring buffers called the Submission Queue (SQ) and Completion Queue (CQ). These queues are shared between user space and the kernel.

The flow of an asynchronous read operation using io_uring is shown below (Figure 8.16 [1]):

1. Prepare and Submit Requests: PostgreSQL prepares one or more read requests, as described below, and enqueues them into the SQ. PostgreSQL then submits these requests to the kernel.

2. Execute Request: The kernel retrieves requests from the SQ and performs the requested reads, utilizing storage-level parallelism.
   The retrieved data is copied directly into the specified buffer pool slot via DMA (Direct Memory Access). When a read operation completes, the kernel enqueues a completion event (CQE) into the CQ.

3. Wait Completions: PostgreSQL waits for the CQ to receive completed requests. Once a completion event is found, PostgreSQL can read the page that has been loaded into the target buffer pool slot.

Readers unfamiliar with io_uring programing can refer to the Appendix for simple examples.

Read Request

From the perspective of PostgreSQL, a read request is a pair consisting of a database-specific page identifier (BufferTag) and a buffer pool slot identifier (buffer_id).

When using io_uring, PostgreSQL translates these identifiers into standard OS parameters: the target file descriptor (fd), the physical byte offset, the read size (8 KB), and the destination memory address (a specific buffer pool slot).

Vector I/O (Scatter-Gather I/O)

Linux and Unix support vectored I/O (via readv() and writev()), which allows multiple contiguous blocks to be read from or written to a file using a single system call.

io_uring also supports vectored I/O, allowing PostgreSQL to read multiple contiguous relation blocks with a single read request.

As shown in Figure 8.17, when PostgreSQL reads contiguous blocks, it prepares a single system call (io_uring_prep_readv()) instead of executing a separate system call for each individual block:

1. Prepare and Submit: The backend prepares and submits a single vectored read request for relation blocks 0-3, specifying the destination buffer pool slots (buffer_ids 3, 1, 4, and 7).

2. Execute Request: The kernel reads the requested blocks into the specified buffer pool slots.

3. Wait Completions: PostgreSQL waits for a single completion event before processing the loaded pages.

During a Sequential Scan, PostgreSQL typically reads contiguous relation blocks. Vectored I/O combines these blocks into a single read request, reducing system call overhead and requiring only a single completion event.

Appendix 2 provides a simple io_uring program using Vector I/O.

8.5.2.2. io_worker

For operating systems that do not support io_uring (such as macOS, BSD, and Solaris), PostgreSQL provides an alternative implementation called io_worker (Figure 8.16 [2]). One or more io_worker background worker processes execute I/O requests.

In io_worker mode, PostgreSQL creates its own SQ and CQ structures in shared memory.

Backend processes prepare read requests into the SQ and submit them for execution. The io_worker background processes retrieve these requests, perform the reads, and enqueue completion events into the CQ.

Compared with io_uring:

• The io_worker background workers execute I/O requests using conventional synchronous read system calls. Although multiple workers may process requests concurrently, the achievable parallelism and scalability are generally lower than with io_uring.

• Request execution requires coordination between backend processes and io_worker processes through shared-memory queues and process wakeups. As a result, additional communication and scheduling overheads are incurred.

8.5.3. Basic Concept of Asynchronous Sequential Scan

The following subsections explain Sequential Scan using AIO.

Although this mechanism relies on simple functions, combining multiple components for performance optimization makes it complex. Therefore, this explanation is divided into two stages:

1. This subsection describes the basic behavior of an AIO-based Sequential Scan in a highly simplified setting. To focus on the core mechanism, this model assumes a cold-start scenario where the buffer pool caches no pages and storage contains all data to be read.
2. The next subsection builds upon this foundation by incorporating the omitted features to present the full implementation details.

While the pseudocode in Section 8.4 focused on low-level buffer descriptor processing, the following explanation operates at a higher level of abstraction:

• BufferAlloc() abstracts all buffer descriptor management excluding the physical read operation; lower-level functions are omitted.
• Details such as PIN processing are also omitted.
• do_exec(tuple) abstracts all tuple processing, such as WHERE-clause filter evaluation and incremental computation of aggregation functions.

8.5.3.1. ReadStream Mechanism and stream_buffers Abstraction

The basic strategy of an AIO-based Sequential Scan is straightforward:

• Process the pages of the target table sequentially from the beginning.
  • Read-ahead a certain number of blocks asynchronously in advance.

The buffer manager tracks and manages these read-ahead blocks using a dedicated data structure called the ReadStream structure.

typedef struct InProgressIO
{
    int16       buffer_index;
    ReadBuffersOperation op;
} InProgressIO;

/*
 * State for managing a stream of reads.
 */
struct ReadStream
{
    int16       max_ios;
    int16       io_combine_limit;
    int16       ios_in_progress;
    int16       queue_size;
    int16       max_pinned_buffers;
    int16       forwarded_buffers;
    int16       pinned_buffers;

    /*
     * Limit of how far, in blocks, to look-ahead for IO combining and for
     * read-ahead.
     *
     * The limits for read-ahead and combining are handled separately to allow
     * for IO combining even in cases where the I/O subsystem can keep up at a
     * low read-ahead distance, as doing larger IOs is more efficient.
     *
     * Set to 0 when the end of the stream is reached.
     */
    int16       combine_distance;
    int16       readahead_distance;
    uint16      distance_decay_holdoff;
    int16       initialized_buffers;
    int16       resume_readahead_distance;
    int16       resume_combine_distance;
    int         read_buffers_flags;
    bool        sync_mode;      /* using io_method=sync */
    bool        batch_mode;     /* READ_STREAM_USE_BATCHING */
    bool        advice_enabled;
    bool        temporary;

    /* scan stats counters */
    IOStats    *stats;

    /*
     * One-block buffer to support 'ungetting' a block number, to resolve flow
     * control problems when I/Os are split.
     */
    BlockNumber buffered_blocknum;

    /*
     * The callback that will tell us which block numbers to read, and an
     * opaque pointer that will be pass to it for its own purposes.
     */
    ReadStreamBlockNumberCB callback;
    void       *callback_private_data;

    /* Next expected block, for detecting sequential access. */
    BlockNumber seq_blocknum;
    BlockNumber seq_until_processed;

    /* The read operation we are currently preparing. */
    BlockNumber pending_read_blocknum;
    int16       pending_read_nblocks;

    /* Space for buffers and optional per-buffer private data. */
    size_t      per_buffer_data_size;
    void       *per_buffer_data;

    /* Read operations that have been started but not waited for yet. */
    InProgressIO *ios;
    int16       oldest_io_index;
    int16       next_io_index;

    bool        fast_path;

    /* Circular queue of buffers. */
    int16       oldest_buffer_index;    /* Next pinned buffer to return */
    int16       next_buffer_index;  /* Index of next buffer to pin */
    Buffer      buffers[FLEXIBLE_ARRAY_MEMBER];  /* typedef int Buffer;  */
};

ReadStream consists of an array named buffers[] and various internal metadata. This array acts as a circular FIFO queue with an active window size defined by combine_distance, as illustrated in Figure 8.18 [1]. An array index named oldest_buffer_index tracks the head of this queue, which represents the oldest unconsumed buffer.

Throughout this document, to abstract away the low-level pointer arithmetic of the ring buffer, this entire FIFO queue mechanism is represented simply as stream_buffers (Figure 8.18 [2]).

Enqueue Operation

Figure 8.19 illustrates the enqueue operation, comparing its low-level implementation with the high-level stream_buffers representation.

Consider loading the 2nd block (0-indexed) of a relation table into the 5th slot of the buffer pool.

In this case, the corresponding buffer descriptor is also allocated in the 5th slot of the buffer descriptor layer. At this moment, the buffer manager stores the integer value $5$ into the buffers[] array at the position currently pointed to by oldest_buffer_index.

In the stream_buffers representation, it stores the buffer descriptor itself (indicated by the block number $2$ in Figure 8.19 [2]).

Dequeue Operation

Figure 8.20 details the dequeue operation, showing how an element is removed from both the actual ring buffer and the abstract representation.

As shown in Figure 8.20 [1], when the head buffer identifier is dequeued, the oldest_buffer_index advances one slot to the right. The total number of remaining buffers tracked by pinned_buffers decreases accordingly.

In the stream_buffers representation (Figure 8.20 [2]), this dequeue operation is represented by deleting the leftmost element from the list.

8.5.3.2. Core AIO Operations

This subsection defines functions that encapsulate basic AIO operations.

Buffer Allocation and Enqueueing

read_ahead() function allocates buffer descriptors for the target relation’s blocks and enqueues them into stream_buffers.

stream_buffers   = []
combine_distance = N

/* Simple Version */
read_ahead(stream_buffers, combine_distance, blockNum)
    offset = 0
    while stream_buffers.pinned_buffers < combine_distance
        bufDesc = BufferAlloc(blockNum + offset)
        offset += 1
        bufDesc.state.IO_IN_PROGRESS = 1
        stream_buffers.enqueue(bufDesc)

Because this subsection assumes that the backend must read all pages from storage, the logic is simplified:

• Repeat until the number of pinned_buffers reaches combine_distance:
  1. Call BufferAlloc() to obtain a pinned buffer descriptor.
  2. Set the state of the buffer descriptor to IO_IN_PROGRESS and enqueue it into stream_buffers.

Preparing and Submitting Asynchronous Read Requests

prepare_and_submit() function prepares and submits asynchronous read requests for the buffer descriptors newly enqueued by read_ahead().

Because a Sequential Scan reads contiguous blocks, it utilizes Vector I/O (Scatter-Gather I/O) to combine multiple blocks into a single read request (see Appendix 2 for details).

/* Simple Version */
prepare_and_submit(stream_buffers)
    /* submits a Vector I/O read for the newly added buffers */
    submit(stream_buffers)

Waiting for Asynchronous Read Completion

WaitReadBuffers() function waits for the completion of a specific I/O operation and marks the corresponding buffers as valid.

/* Simple Version */
WaitReadBuffers(stream_buffers)
    wait_cqe(stream_buffers)       /* block until read completes */
    for each bufDesc in stream_buffers
        bufDesc.state.IO_IN_PROGRESS = 0
        bufDesc.state.VALID          = 1

8.5.3.3. Sequential Scan With AIO

Using the functions defined above, the pseudocode below shows the basic behavior of an AIO-based Sequential Scan.

ExecSeqScan()
    stream_buffers   = []
    combine_distance = 4
    blockNum         = 0
    /* Initialize stream_buffers and submit read request for the first round. */
(1) read_ahead(stream_buffers, combine_distance, blockNum)
    prepare_and_submit(stream_buffers)

(2) while blockNum <= last_block

        /* Dequeue the next buffer descriptor from stream_buffers */
(3)     current_buf = stream_buffers.dequeue()

(4)     if current_buf.state.IO_IN_PROGRESS == 1
            /* The buffer is still being read from storage. */
            WaitReadBuffers(stream_buffers)

        /* Read-ahead blocks and submit the read request if stream_buffers is empty. */
        if stream_buffers.pinned_buffers == 0
(5)         read_ahead(stream_buffers, combine_distance, blockNum)
            prepare_and_submit(stream_buffers)

        /* Process tuples of current_page */
(6)     for each tuple in current_buf
            do_exec(tuple)

        blockNum += 1

1. For the first round, read-ahead combine_distance (= 4) pages, enqueue them into stream_buffers, and submit an asynchronous read request for those pages.

2. Process the pages of the target table sequentially.

3. Dequeue the next buffer descriptor from stream_buffers.

4. If the current buffer descriptor’s block is still being read from storage, call WaitReadBuffers() and wait until the read completes.

5. When stream_buffers becomes empty, read-ahead the next batch of blocks and submit the read request.
   This timing yields the AIO benefit: the OS performs the next read asynchronously while the backend processes the tuples of the current page.

6. Process all tuples contained in current_buf.

Figure 8.21 illustrates this sequence assuming all pages are read from storage.

• Previous Round:

  • At the end of the round, stream_buffers becomes empty. Therefore, read_ahead() enqueues the next four blocks ($n$, $n+1$, $n+2$, $n+3$), and prepare_and_submit() submits the asynchronous read request.
  • Immediately after the submission, the backend processes the tuples of the last block of the previous round (blockNum = $n-1$). This tuple processing runs concurrently with the OS read operation for blocks $n$ through $n+3$.

• Current Round:

  1. Processing blockNum = $n$:

     • stream_buffers.dequeue() returns the buffer descriptor for block $n$, which is still IO_IN_PROGRESS.
     • WaitReadBuffers() waits until the read completes.
     • Upon completion, WaitReadBuffers() marks all four blocks ($n$, $n+1$, $n+2$, $n+3$) as VALID in one step.
     • The backend then processes the tuples of block $n$.

  2. Processing blockNum = $n+1$, $n+2$, $n+3$:

     • These buffers are already VALID (marked by the previous WaitReadBuffers() call), so processing continues without additional wait time.

  3. End of round - stream_buffers is empty:

     • read_ahead() enqueues blocks $n+4$ through $n+7$, and prepare_and_submit() submits the next asynchronous read request.

Comparison with Synchronous Sequential Scan (Version 16 or Earlier)

Prior to the introduction of AIO, PostgreSQL performed Sequential Scans in a strictly synchronous and serialized manner, as illustrated in Figure 8.22.

In this traditional model, the backend process repeats a serialized cycle for each block (referred to here as a round):

1. Buffer Allocation: The ReadBuffer() function invokes BufferAlloc(), which searches for or creates the corresponding buffer descriptor for the target block ($n, n+1, n+2, \dots$).

2. Synchronous Storage Read: The backend invokes a standard synchronous read() system call. At this point, the backend process is completely blocked.

3. I/O Termination & Tuple Processing: Once the I/O completes, the backend wakes up, unlocks the buffer via TerminateBufferIO(), and immediately processes the tuples (do_exec()).

8.5.4. Actual Implementation of Asynchronous Sequential Scan

While the fundamental pipeline remains as described in the previous simplified model, the buffer manager incorporates several real-world mechanisms to optimize efficiency under active workloads. Specifically, the implementation:

• Handles cache hits during read-ahead.
• Dynamically adjusts the read-ahead window based on real-time execution behavior.
• Bypasses stream overhead via a Fast Path mode when detecting a sequence of consecutive cache hits.

The following subsections detail each of these production mechanisms.

8.5.4.1. Cache Hit Processing in Read-Ahead

When read_ahead() encounters a cache hit, it stops scanning further blocks and prepares a read-request using only the contiguous cache-miss blocks accumulated up to that point.

Assume that the (n+2)-th block is already cached in the buffer pool. Figure 8.23 illustrates this behavior.

1. Since stream_buffers is empty, read_ahead() attempts to refill it with subsequent blocks.

   • read_ahead() starts from n-th block and enqueues blocks into stream_buffers.
   • When it encounters a cache hit (in this example, (n+2)-th block), it stops the read-ahead process.

2. prepare_and_submit() prepares and submits a vectored read-request for the two contiguous blocks (n-th and (n+1)-th blocks).

3. WaitReadBuffers() waits for the completion of the read-request.

4. The scan processes n-th, (n+1)-th, and (n+2)-th blocks. The first two blocks are obtained from the completed I/O request, while (n+2)-th blockis already available in the buffer pool.

Stopping the read-ahead process at the first cache hit ensures that all blocks accumulated in stream_buffers are physically contiguous in the relation file. Consequently, the buffer manager can submit them as a single Vector I/O request (see Appendix 2 for details), maximizing sequential read efficiency.

Figure 8.24 [1] illustrates the behavior when reading subsequent blocks from storage after a cache hit has been processed.

1. After processing the (n+2)-th block, read_ahead() refills stream_buffers up to the combine_distance width, starting from the (n+3)-th block.

2. prepare_and_submit() prepares a multi-block read-request from stream_buffers and submits it.

Conversely, Figure 8.24 [2] illustrates the behavior when subsequent blocks also result in consecutive cache hits. In this scenario, the cached buffer descriptors are enqueued into stream_buffers one by one, and a single-block round of tuple processing is executed repeatedly.

8.5.4.2. Adaptive Read-Ahead Mechanism

The length of combine_distance in stream_buffers changes adaptively.

At the start of a Sequential Scan, combine_distance initializes to 1. The incremental algorithm is as follows:

• Increment: The value of combine_distance doubles with each round (e.g., from 1 to 2, 2 to 4, 4 to 8, and so on).
  • The maximum value of combine_distance is the lesser of io_combine_limit/BLCKSZ and (queue_size - 1), calculated as:

Figure 8.25 shows the transition of the combine_distance size when io_combine_limit/BLCKSZ is set to 16.

While combine_distance increases rapidly to reach the specified maximum parallelism limit as quickly as possible, it decays very gradually.

• Decay: When cache hits continue for more than (queue_size - 1) blocks, the value of combine_distance decrements by 1.
  • Continued cache hits cause combine_distance to decrement by 1 each time until it reaches 1.
  • A cache miss immediately triggers the increment mode.

Figure 8.26 shows the fluctuation of the combine_distance value. This example assumes that the relation file is already cached in the buffer pool starting from the n-th block.

1. Initial cache hits: From the n-th block to the (n + (queue_size -1))-th block, the buffer manager performs the cache-hit processing described in the previous subsection. During this period, the length of combine_distance remains unchanged.

2. Gradual decay: When cache hits persist beyond the (queue_size - 1) threshold, the value of combine_distance decrements by 1 at each subsequent block.

3. Increment: When a cache miss occurs and a block must be read from storage, the mechanism immediately switches back to the increment mode. In this example, combine_distance doubles from 2 to 4.

8.5.4.3. Fast Path Mode: Bypassing stream_buffers

When combine_distance is 1 and cache hits continue, the buffer manager transitions to Fast Path mode, which reads buffer slots directly without using stream_buffers. This mode aims to accelerate processing by bypassing the entire management process of stream_buffers.

If a cache miss occurs during Fast Path mode, the buffer manager immediately transitions from Fast Path mode back to the normal read-ahead mode.

Figure 8.27 [1] shows an example of entering and leaving Fast Path mode.

When all blocks are already cached in the buffer pool (Figure 8.26 [2]), the buffer manager uses stream_buffers only to process the 0-th block. After that, it processes all subsequent blocks in Fast Path mode.

Appendix: io_uring Examples

Knowledge of io_uring implementation examples clarifies Asynchronous I/O (AIO) in PostgreSQL. Below are two simple examples.

These serve as a toy model of a buffer manager, asynchronously reading data from a file into slots in memory (Figure 8.28).

The programs asynchronously read a 32-byte file (rel.data) into the BufferPool array using four 8-byte read requests.

$ cat rel.data
A0000000B0000001C0000010D0000011

#define BUFFER_SIZE 8
#define PAGE_SIZE 8

char BufferPool[BUFFER_SIZE][PAGE_SIZE + 1];

In these examples, the four logical blocks of rel.data are placed into BufferPool slots 3, 7, 1, and 0.

/*
 * Mapping Table: Logical Block -> Physical Buffer Slot
 * Defines the destination slot in BufferPool for each sequential file block.
 * - Block 0 (Bytes 0-7)   -> BufferPool[3]
 * - Block 1 (Bytes 8-15)  -> BufferPool[7]
 * - Block 2 (Bytes 16-23) -> BufferPool[1]
 * - Block 3 (Bytes 24-31) -> BufferPool[0]
 */
#define QUEUE_SIZE 4

int dest[QUEUE_SIZE] = { 3, 7, 1, 0 };

A.1: The Multiple-Read Approach

The following example demonstrates a simple io_uring program.

The complete source code is shown below:

$ cat rel.data
A0000000B0000001C0000010D0000011

$ cat aio_read.c
#include <liburing.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

#define BUFFER_SIZE 8
#define PAGE_SIZE 8
#define QUEUE_SIZE 4
#define REL_FILE "rel.data"

int
main()
{
    int fd;
    struct io_uring ring;
    char BufferPool[BUFFER_SIZE][PAGE_SIZE+1];

    /*
     * Mapping Table: Logical Block -> Physical Buffer Slot
     * Defines the destination slot in BufferPool for each sequential file block.
     * - Block 0 (Bytes 0-7)   -> BufferPool[3]
     * - Block 1 (Bytes 8-15)  -> BufferPool[7]
     * - Block 2 (Bytes 16-23) -> BufferPool[1]
     * - Block 3 (Bytes 24-31) -> BufferPool[0]
     */
    int dest[QUEUE_SIZE] = { 3, 7, 1, 0 };

    /* Initialize io_uring environment and open target relation file */
    io_uring_queue_init(QUEUE_SIZE, &ring, 0);
    if ((fd = open(REL_FILE, O_RDONLY)) < 0)
        return 1;

    memset(BufferPool, 0, sizeof(BufferPool));

    /*
     * Prepare and submit read requests
     */
    for (int i = 0; i < QUEUE_SIZE; i++) {
        struct io_uring_sqe* sqe = io_uring_get_sqe(&ring);
        int buff_id = dest[i];
        io_uring_prep_read(sqe, fd, BufferPool[buff_id], PAGE_SIZE, i * PAGE_SIZE);
        sqe->user_data = i;
    }
    io_uring_submit(&ring);

    /*
     * Wait for completion events
     */
    for (int i = 0; i < QUEUE_SIZE; i++) {
        int blockNum;
        int buff_id;
        struct io_uring_cqe* cqe;

        /* Wait for one io_uring completion event */
        io_uring_wait_cqe(&ring, &cqe);

        /* Mark io_uring completion event as consumed */
        io_uring_cqe_seen(&ring, cqe);

        /* Output the results */
        blockNum = (int)cqe->user_data;
        buff_id = dest[blockNum];
        printf("CQ[%d]: BlockNum_%d -> BufferPool[%d] offset=%3d bytes=%2d: [%s]\n",
              i+1, blockNum, buff_id, blockNum * PAGE_SIZE, PAGE_SIZE, BufferPool[buff_id]);
    }

    close(fd);
    io_uring_queue_exit(&ring);
    return 0;
}

$ gcc -o aio_read aio_read.c -luring

$ ./aio_read
CQ[1]: BlockNum_0 -> BufferPool[3] offset=  0 bytes= 8: [A0000000]
CQ[2]: BlockNum_1 -> BufferPool[7] offset=  8 bytes= 8: [B0000001]
CQ[3]: BlockNum_2 -> BufferPool[1] offset= 16 bytes= 8: [C0000010]
CQ[4]: BlockNum_3 -> BufferPool[0] offset= 24 bytes= 8: [D0000011]

First, the program opens the file and initializes an io_uring instance.

#define REL_FILE "rel.data"

int fd;
struct io_uring ring;

if ((fd = open(REL_FILE, O_RDONLY)) < 0)
    return 1;

io_uring_queue_init(QUEUE_SIZE, &ring, 0);

Figure 8.29 illustrates the subsequent flow to process multiple read requests.

1. Prepare Read Requests: io_uring_get_sqe() provides one Submission Queue Entry (SQE) for each read request. io_uring_prep_read() then populates each SQE with a read request, specifying the destination buffer, read size, and file offset. The user_data field stores the block number to associate each completion event with its corresponding request later.

   for (int i = 0; i < QUEUE_SIZE; i++) {
       struct io_uring_sqe* sqe = io_uring_get_sqe(&ring);
       int buff_id = dest[i];
       io_uring_prep_read(sqe, fd, BufferPool[buff_id], PAGE_SIZE, i * PAGE_SIZE);
       sqe->user_data = i;
   }

2. Submit Read Requests: After all four read requests are ready, io_uring_submit() submits them together.

   io_uring_submit(&ring);

3. Read Blocks and Queue Completion Events: The kernel reads the requested blocks from the storage device. Once each read completes, the kernel enqueues a completion event into the Completion Queue (CQ).

4. Wait for Completion Events: The program waits for completion events by repeatedly calling io_uring_wait_cqe(). The corresponding buffer becomes safe to access whenever a completion event arrives. Finally, io_uring_cqe_seen() marks the completion queue entry as consumed.

   for (int i = 0; i < QUEUE_SIZE; i++) {
       int blockNum;
       int buff_id;
       struct io_uring_cqe* cqe;
   
       /* Wait for one io_uring completion event */
       io_uring_wait_cqe(&ring, &cqe);
   
       /* Mark io_uring completion event as consumed */
       io_uring_cqe_seen(&ring, cqe);
   
       /* Output the results */
       blockNum = (int)cqe->user_data;
       buff_id = dest[blockNum];
       printf("CQ[%d]: BlockNum_%d -> BufferPool[%d] offset=%3d bytes=%2d: [%s]\n",
              i+1, blockNum, buff_id, blockNum * PAGE_SIZE, PAGE_SIZE, BufferPool[buff_id]);
   }

The following output was produced by one execution of the program.

$ ./aio_read
CQ[1]: BlockNum_0 -> BufferPool[3] offset=  0 bytes= 8: [A0000000]
CQ[2]: BlockNum_1 -> BufferPool[7] offset=  8 bytes= 8: [B0000001]
CQ[3]: BlockNum_2 -> BufferPool[1] offset= 16 bytes= 8: [C0000010]
CQ[4]: BlockNum_3 -> BufferPool[0] offset= 24 bytes= 8: [D0000011]

A.2: The Vector-Read Approach

io_uring supports Vector I/O (Scatter-Gather I/O), leading to the effective reading of multiple continuous pages with one read request.

The following example demonstrates how io_uring operates on multiple continuous pages. The complete source code is shown below:

$ cat rel.data
A0000000B0000001C0000010D0000011

$ cat aio_readv.c
#include <fcntl.h>
#include <liburing.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

#define BUFFER_SIZE 8
#define PAGE_SIZE 8
#define QUEUE_SIZE 4
#define REL_FILE "rel.data"

int
main()
{
    int fd;
    struct io_uring ring;
    char BufferPool[BUFFER_SIZE][PAGE_SIZE + 1];
    struct iovec iov[QUEUE_SIZE];

    /*
     * Mapping Table: Logical Block -> Physical Buffer Slot
     * Defines the destination slot in BufferPool for each sequential file block.
     * - Block 0 (Bytes 0-7)   -> BufferPool[3]
     * - Block 1 (Bytes 8-15)  -> BufferPool[7]
     * - Block 2 (Bytes 16-23) -> BufferPool[1]
     * - Block 3 (Bytes 24-31) -> BufferPool[0]
     */
    int dest[QUEUE_SIZE] = { 3, 7, 1, 0 };

    /* Initialize io_uring environment and open target relation file */
    io_uring_queue_init(QUEUE_SIZE, &ring, 0);
    if ((fd = open(REL_FILE, O_RDONLY)) < 0)
        return 1;

    memset(BufferPool, 0, sizeof(BufferPool));

    /* Set up iovec array to bind scattered buffer pool addresses to the read stream */
    for (int blockNum = 0; blockNum < QUEUE_SIZE; blockNum++) {
        int buff_id = dest[blockNum];
        iov[blockNum].iov_base = BufferPool[buff_id];
        iov[blockNum].iov_len = PAGE_SIZE;
    }

    /*
     * Prepare and submit a single scatter-read request (readv)
     */
    struct io_uring_sqe* sqe = io_uring_get_sqe(&ring);
    io_uring_prep_readv(sqe, fd, iov, QUEUE_SIZE, 0 /* file offset */);
    io_uring_submit(&ring);

    /*
     * Wait until the single CQE (representing the entire readv completion) returns
     */
    struct io_uring_cqe* cqe;
    io_uring_wait_cqe(&ring, &cqe);

    /* Mark the completion event as consumed */
    io_uring_cqe_seen(&ring, cqe);

    /* Output the results scattered into the buffer pool */
    printf("CQ: readv completed: %d bytes read\n", cqe->res);
    for (int blockNum = 0; blockNum < QUEUE_SIZE; blockNum++) {
        int buff_id = dest[blockNum];
        printf("\tBlockNum_%d -> BufferPool[%d] offset=%3d bytes=%2d: [%s]\n",
               blockNum, buff_id, blockNum * PAGE_SIZE, PAGE_SIZE, BufferPool[buff_id]);
    }

    close(fd);
    io_uring_queue_exit(&ring);
    return 0;
}

$ gcc -o aio_readv aio_readv.c -luring

$ ./aio_readv
CQ: readv completed: 32 bytes read
	BlockNum_0 -> BufferPool[3] offset=  0 bytes= 8: [A0000000]
	BlockNum_1 -> BufferPool[7] offset=  8 bytes= 8: [B0000001]
	BlockNum_2 -> BufferPool[1] offset= 16 bytes= 8: [C0000010]
	BlockNum_3 -> BufferPool[0] offset= 24 bytes= 8: [D0000011]

To use the Vector I/O feature, the program creates an array iov and sets the addresses of the BufferPool slots.

struct iovec iov[QUEUE_SIZE];

/* Set up iovec array to bind scattered buffer pool addresses to the read stream */
for (int blockNum = 0; blockNum < QUEUE_SIZE; blockNum++) {
    int buff_id = dest[blockNum];
    iov[blockNum].iov_base = BufferPool[buff_id];
    iov[blockNum].iov_len = PAGE_SIZE;
}

Figure 8.30 illustrates the subsequent flow to process a vector read request.

1. Prepare Read Request: The program prepares a single read request. Unlike the previous example, this single request reads four continuous blocks into the scattered BufferPool slots specified by the iov array.

   struct io_uring_sqe* sqe = io_uring_get_sqe(&ring);
   io_uring_prep_readv(sqe, fd, iov, QUEUE_SIZE, 0 /* file offset */);

2. Submit Read Request: io_uring_submit() submits the read request.

   io_uring_submit(&ring);

3. Read Blocks and Queue Completion Event: The kernel reads all four continuous blocks from the file. When the entire vector read operation completes, the kernel enqueues a single completion event into the CQ.

4. Wait for Completion Event: The program waits for the completion event of the read request. Unlike the previous example, although the program requested four blocks, the kernel enqueues only one event in the CQ because there is only one read request.

   struct io_uring_cqe* cqe;
   
   /* Wait for a completion event */
   io_uring_wait_cqe(&ring, &cqe);
   
   /* Mark the completion event as consumed */
   io_uring_cqe_seen(&ring, cqe);
   
   /* Output the results scattered into the buffer pool */
   printf("CQ: readv completed: %d bytes read\n", cqe->res);
   for (int blockNum = 0; blockNum < QUEUE_SIZE; blockNum++) {
       int buff_id = dest[blockNum];
       printf("\tBlockNum_%d -> BufferPool[%d] offset=%3d bytes=%2d: [%s]\n",
              blockNum, buff_id, blockNum * PAGE_SIZE, PAGE_SIZE, BufferPool[buff_id]);
   }

The following output was produced by one execution of the program.

$ ./aio_readv
CQ: readv completed: 32 bytes read
	BlockNum_0 -> BufferPool[3] offset=  0 bytes= 8: [A0000000]
	BlockNum_1 -> BufferPool[7] offset=  8 bytes= 8: [B0000001]
	BlockNum_2 -> BufferPool[1] offset= 16 bytes= 8: [C0000010]
	BlockNum_3 -> BufferPool[0] offset= 24 bytes= 8: [D0000011]

Using the Vector I/O feature aggregates the reading of continuous blocks into a single read request.