April 24, 2025 29 min to read

OS Cache and Disk I/O - MySQL and Redis Performance Analysis

Understanding the impact of operating system caches on database performance

Overview

Operating system caching mechanisms sit at the critical intersection between application performance and hardware capabilities. While database administrators often focus on optimizing database settings and upgrading hardware, the operating system’s caching layer often has the most significant impact on real-world performance.

The OS cache serves as a transparent intermediary that minimizes expensive disk operations by storing recently accessed data in memory. This automatic optimization dramatically improves read and write speeds, particularly for database systems like MySQL and Redis that perform intensive I/O operations.

What is OS Cache?

The operating system cache is a memory management mechanism that improves I/O performance by storing recently accessed data in RAM, providing faster access than reading from or writing to physical storage devices.

Types of OS Caches

How Page Cache Works

When an application reads a file, the operating system follows these steps:

1. Check if the requested data exists in the page cache
2. If found (cache hit), return data directly from memory
3. If not found (cache miss), read the data from disk into the page cache
4. Return the data from the cache to the application

This process is transparent to applications, which simply make standard file I/O calls without needing to know about the cache.

// This simple C code demonstrates how the OS transparently caches file reads
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

void measure_read_time(const char* filename) {
    FILE* file;
    char buffer[4096];
    clock_t start, end;
    double time_spent;
    size_t bytes_read;
    
    // First read (likely uncached)
    printf("First read (likely from disk):\n");
    start = clock();
    file = fopen(filename, "rb");
    if (file) {
        while ((bytes_read = fread(buffer, 1, sizeof(buffer), file)) > 0) {
            // Just read the file, doing nothing with the data
        }
        fclose(file);
    }
    end = clock();
    time_spent = (double)(end - start) / CLOCKS_PER_SEC;
    printf("Time: %f seconds\n", time_spent);
    
    // Second read (likely cached)
    printf("Second read (likely from cache):\n");
    start = clock();
    file = fopen(filename, "rb");
    if (file) {
        while ((bytes_read = fread(buffer, 1, sizeof(buffer), file)) > 0) {
            // Just read the file, doing nothing with the data
        }
        fclose(file);
    }
    end = clock();
    time_spent = (double)(end - start) / CLOCKS_PER_SEC;
    printf("Time: %f seconds\n", time_spent);
}

int main() {
    // Replace with a file that exists on your system (ideally 100MB+ for clear results)
    measure_read_time("/path/to/large/file");
    return 0;
}

Page Cache Management in Linux

Linux dynamically manages the page cache size based on memory pressure. It tries to use as much available memory as possible for caching, but will reclaim cache pages when applications need more memory.

# View current memory usage including cache
free -m

# Example output:
#               total        used        free      shared  buff/cache   available
# Mem:          16384        4096        2048          64       10240        8192
# Swap:          4096           0        4096

In this output:

• buff/cache shows memory used for the buffer cache and page cache
• available shows memory that can be allocated to processes without swapping

Cache Flushing Mechanisms

For data consistency and durability, Linux needs to write dirty pages back to disk. This happens through several mechanisms:

1. Background Flush: The pdflush (older kernels) or flusher threads periodically write dirty pages to disk
2. Explicit Sync: Applications can call fsync(), fdatasync(), or sync() to force data to be written to disk
3. Memory Pressure: When memory is needed for other purposes, dirty pages may be written to disk
4. Age-Based Flush: Pages are written after being dirty for a certain time period

# View current dirty page settings
cat /proc/sys/vm/dirty_background_ratio
cat /proc/sys/vm/dirty_ratio
cat /proc/sys/vm/dirty_expire_centisecs
cat /proc/sys/vm/dirty_writeback_centisecs

# Example: Manually trigger sync to flush dirty pages
sync

Direct I/O vs Cached I/O

Some applications bypass the page cache for specific reasons:

// Example of bypassing page cache with O_DIRECT flag
#include <fcntl.h>
#include <unistd.h>

void direct_io_example() {
    int fd = open("file.data", O_RDWR | O_DIRECT);
    if (fd >= 0) {
        // Perform direct I/O operations
        // Note: Buffer must be aligned for direct I/O
        close(fd);
    }
}

Bypassing the cache might be preferred when:

• The application maintains its own cache (like MySQL’s InnoDB Buffer Pool)
• Predictable I/O latency is more important than throughput
• Very large sequential scans would push useful data out of cache

When Caching Doesn’t Work Well

While the operating system’s caching mechanisms generally provide significant performance benefits, several scenarios can reduce or negate these advantages.

Case Study: Cache Clearing Impact on Database Performance

The following graph demonstrates the performance impact of clearing the OS cache on a MySQL database server:

TPS
^
|
|   *******                 *******
|  *       *               *       *
| *         *             *         *
|*           *           *           *
|              *       *               *       *
|               *     *                 *     *
|                *****                   *****
+-------------------------------------------------------> Time
                    ^                       ^
                Cache Clear             Cache Clear

When the cache is cleared, transaction performance drops dramatically until the working set is reloaded into memory.

Testing Cache Impact on Your System

#!/bin/bash
# Script to demonstrate cache impact on file read performance

FILE_PATH="/path/to/large/file"  # Should be at least 1GB
ITERATIONS=3

echo "Testing read performance with OS cache:"

# Drop caches first to start with clean state
echo "Dropping caches..."
sudo sync
sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"

for i in $(seq 1 $ITERATIONS); do
    echo "Read $i:"
    time cat $FILE_PATH > /dev/null
done

echo "Note the significant speed improvement after the first read"
echo "This demonstrates the OS cache effect"

Common Cache Tuning Parameters

For workloads where the default caching behavior isn’t optimal, Linux provides several tuning parameters:

# Control how aggressive the kernel reclaims memory 
# (0-100, higher = favor cache retention)
echo 10 > /proc/sys/vm/swappiness

# Control dirty page thresholds (as % of total memory)
echo 10 > /proc/sys/vm/dirty_background_ratio  # Start background flushing
echo 20 > /proc/sys/vm/dirty_ratio             # Force synchronous flushes

# Control how long dirty pages can stay in memory (in centiseconds)
echo 3000 > /proc/sys/vm/dirty_expire_centisecs  # 30 seconds

# Readahead setting for specific device (in 512-byte sectors)
blockdev --setra 256 /dev/sda  # 128KB readahead

Disk I/O vs Cache Access Speed Comparison

The performance gap between memory and storage devices is one of the largest disparities in computer architecture, making caching essential for system performance.

Visualizing the Speed Difference

To put these differences in perspective, if we were to scale these times to human-relatable units, where RAM access is represented as 1 second:

CPU L1 Cache: 10 milliseconds (quick eye blink)
RAM: 1 second (heartbeat)
NVMe SSD: 2-5 minutes (short break)
SATA SSD: 10-25 minutes (coffee break)
HDD: 14-28 hours (more than a day)

This enormous performance gap makes caching critical for system performance.

Measuring I/O vs Cache Performance

You can measure and compare disk I/O versus cache performance on your system using tools like:

# Install necessary tools
apt-get install fio sysstat

# Create test file and bypass cache for accurate disk timing
dd if=/dev/zero of=test_file bs=1M count=1024 oflag=direct

# Run disk read test (bypassing cache)
fio --name=disk_read --filename=test_file --direct=1 --rw=read --bs=4k --size=1G --numjobs=1 --time_based --runtime=60 --group_reporting

# Run cached read test (after file is cached)
# First read it to ensure it's in cache
cat test_file > /dev/null
# Then measure cached performance
fio --name=cached_read --filename=test_file --direct=0 --rw=read --bs=4k --size=1G --numjobs=1 --time_based --runtime=60 --group_reporting

The results will typically show:

• Disk read: 50-500 MB/s depending on storage type
• Cached read: 5,000-50,000+ MB/s depending on RAM speed and CPU

Database Performance Impact

For database systems like MySQL and Redis, this speed difference translates directly to performance metrics:

• Cold cache: TPS (transactions per second) can be 10-100x lower
• Warm cache: Operations are primarily memory-bound, not I/O-bound
• Cache hits: Can support thousands more concurrent users with the same hardware

The database performance curve is typically non-linear with cache hit rate - a small decrease in cache hit rate from 99% to 95% might halve overall performance, while a drop from 95% to 80% might cause a 10x performance degradation.

MySQL: Buffer Pool vs OS Cache

MySQL’s InnoDB storage engine implements a buffer pool that caches frequently accessed data and index pages in memory. However, this is just one layer in a multi-tiered caching system that ultimately includes the OS cache.

Buffer Pool Architecture

The InnoDB buffer pool is MySQL’s memory area for caching table and index data. It operates as follows:

1. Requested data pages are first looked up in the buffer pool
2. If not found, they’re read from disk and placed in the buffer pool
3. When the buffer pool is full, least recently used (LRU) pages are evicted

-- Check buffer pool size and settings
SHOW VARIABLES LIKE 'innodb_buffer_pool%';

-- Monitor buffer pool usage
SHOW ENGINE INNODB STATUS\G
SELECT * FROM information_schema.INNODB_BUFFER_POOL_STATS;

OS Cache Role in MySQL Performance

Even with a well-sized buffer pool, MySQL still relies on the OS cache for several reasons:

1. Data accessed through non-InnoDB storage engines (e.g., MyISAM)
2. Binary logs, redo logs, and other system files
3. Data not in the buffer pool (cold data or after buffer pool eviction)
4. Temporary tables and sort files

Disk Read Metrics Analysis

Monitoring disk read operations helps understand how your MySQL instance is utilizing caches:

-- Check buffer pool hit rate
SELECT
  (1 - (SELECT variable_value FROM performance_schema.global_status WHERE variable_name = 'Innodb_buffer_pool_reads') / 
   (SELECT variable_value FROM performance_schema.global_status WHERE variable_name = 'Innodb_buffer_pool_read_requests'))
   * 100 AS buffer_pool_hit_ratio;

A high hit ratio (>99%) suggests effective buffer pool usage, but doesn’t give the full picture of OS cache usage.

O_DIRECT Flag and Its Impact

MySQL can be configured to use direct I/O, bypassing the OS cache:

# MySQL configuration for direct I/O
[mysqld]
innodb_flush_method=O_DIRECT

This configuration has important trade-offs:

Optimizing MySQL with Proper Buffer Pool and OS Cache Configuration

The key to optimal MySQL performance is balancing buffer pool size and allowing sufficient memory for the OS cache:

# Example optimal memory allocation for a database server with 64GB RAM
# MySQL Buffer Pool: 48GB (75%)
# OS and other processes: 4GB
# OS Cache available: 12GB

# MySQL configuration
innodb_buffer_pool_size = 48G
innodb_buffer_pool_instances = 8  # One per CPU core up to 8

# Monitor actual memory usage
free -m
vmstat 1

Practical Case Studies and Solutions

Case Study 1: MySQL Performance Degradation After Midnight

Problem: A production MySQL database experienced significant performance degradation every night at midnight, with query latency increasing by 10x and disk I/O spikes.

Investigation:

• Monitoring showed an increase in both buffer pool misses and disk reads
• OS monitoring revealed a scheduled script executed echo 3 > /proc/sys/vm/drop_caches as part of cleanup automation
• This cleared both the OS cache and caused cold reads for data not in the buffer pool

Solution:

1. Removed the drop_caches command from the cleanup script
2. Implemented proper buffer pool dumping and loading:

# Added to MySQL configuration
innodb_buffer_pool_dump_at_shutdown = 1
innodb_buffer_pool_load_at_startup = 1

1. Added monitoring for buffer pool and OS cache hit rates

Result:

• Overnight performance stabilized with no noticeable latency spikes
• Cold starts after planned maintenance were also improved

# Script to warm up MySQL cache after planned maintenance
mysql -e "SELECT COUNT(*) FROM important_table WHERE last_accessed > DATE_SUB(NOW(), INTERVAL 1 DAY)"

Case Study 2: Optimizing for Limited Memory

Problem: A server with 16GB RAM running both MySQL and application services experienced high I/O wait times.

Investigation:

• MySQL was configured with 12GB buffer pool
• Application servers required 2-3GB, leaving only 1-2GB for OS cache
• High I/O wait showed insufficient OS caching

Solution:

1. Rebalanced memory allocation:

# Reduced buffer pool size
innodb_buffer_pool_size = 8G

1. Optimized MySQL query patterns to work better with smaller buffer pool
2. Added monitoring for OS cache usage

Result:

• OS Cache grew to 5-6GB
• Overall performance improved by 30%
• I/O wait decreased by 60%

Redis: All In-Memory, But Why Cache?

While Redis is primarily an in-memory database, it still interacts with disk for persistence features and can benefit significantly from OS cache management.

Redis Persistence Mechanisms

Redis offers two persistence options that interact with the OS cache:

1. RDB (Redis Database): Point-in-time snapshots saved to disk
2. AOF (Append-Only File): Log of all write operations

AOF and OS Cache Interaction

With AOF enabled, Redis writes operations to a file, but the actual disk flush depends on the appendfsync setting:

# Redis configuration options
appendonly yes
appendfsync everysec  # Options: no, everysec, always

The OS cache’s role in each setting:

1. appendfsync no: Redis never calls fsync, relying entirely on OS cache to flush dirty pages
2. appendfsync everysec: Redis calls fsync every second, but writes first go to OS cache
3. appendfsync always: Redis calls fsync after every write, minimizing OS cache role but dramatically reducing performance

Performance: no > everysec > always
Durability: always > everysec > no

Case Study: Redis AOF Write Delay

Problem: A Redis server with AOF enabled experienced periodic “freeze” moments where all operations would block for several seconds.

Investigation:

• AOF file had grown to several GB
• vm.dirty_ratio was set to default 20%
• During peak write periods, dirty pages accumulated until hitting the threshold
• When threshold was reached, the kernel forced synchronous flushes, blocking Redis

Solution:

1. Adjusted Linux kernel parameters:

# Lower threshold for background flushing
echo 5 > /proc/sys/vm/dirty_background_ratio

# Set longer expiration for dirty pages
echo 10000 > /proc/sys/vm/dirty_expire_centisecs

1. Implemented scheduled AOF rewrites and monitoring:

# Add to crontab
0 3 * * * redis-cli BGREWRITEAOF

# Monitor AOF size
redis-cli info persistence | grep aof_current_size

1. Added higher-performance storage for the AOF file

Result:

• Background flushing prevented accumulation of excessive dirty pages
• Redis operations remained responsive even during high write periods
• Scheduled rewrites kept AOF size manageable

Redis and Memory Pressure

Even though Redis is an in-memory database, system memory pressure can still affect performance:

1. When memory is scarce, the OS may prioritize application memory over cache
2. Background saves (BGSAVE) require additional memory, potentially causing swapping
3. AOF rewrites also need memory to create the new file

Monitoring Redis Memory and OS Cache Interaction:

# Check Redis memory usage
redis-cli info memory

# Check system memory including cache
free -m

# Watch for Redis swapping
redis-cli info stats | grep total_swap

  #!/bin/bash
  # MySQL cache warming script
  mysql -e "SELECT * FROM (SELECT id FROM popular_table ORDER BY last_accessed DESC LIMIT 10000) t"
  mysql -e "SELECT * FROM (SELECT id FROM common_reference_table) t"
  

Practical Example: Monitoring Cache Hit Rate

Monitoring tools play a crucial role in understanding cache behavior:

OS Cache Status:

# View memory usage including cache
free -h

# More detailed cache information
cat /proc/meminfo | grep -E 'Cached|Buffers|Dirty|Writeback'

# Monitor I/O wait percentage - indicator of cache efficiency
top
# or
vmstat 1

MySQL Cache Monitoring:

-- Monitor buffer pool efficiency
SELECT
  (SELECT variable_value FROM information_schema.global_status WHERE variable_name = 'Innodb_buffer_pool_read_requests') AS read_requests,
  (SELECT variable_value FROM information_schema.global_status WHERE variable_name = 'Innodb_buffer_pool_reads') AS disk_reads,
  (1 - ((SELECT variable_value FROM information_schema.global_status WHERE variable_name = 'Innodb_buffer_pool_reads') /
        (SELECT variable_value FROM information_schema.global_status WHERE variable_name = 'Innodb_buffer_pool_read_requests'))) * 100 AS hit_ratio;

Redis Cache Monitoring:

# Monitor Redis memory usage
redis-cli info memory | grep used_memory

# Monitor AOF status
redis-cli info persistence | grep aof

# Check background save status
redis-cli info persistence | grep rdb

Advanced Cache Optimization Techniques

Beyond the basic understanding of OS cache and its interaction with databases, several advanced techniques can further optimize system performance.

Cache Prefetching

Cache prefetching involves reading data into memory before it’s actually requested, improving performance for sequential operations:

# Enable and configure readahead for a specific block device
sudo blockdev --setra 4096 /dev/sda  # 2MB readahead (4096 * 512 bytes)

# Check current readahead settings
sudo blockdev --getra /dev/sda

Application-level prefetching:

-- MySQL query that prefetches data for upcoming operations
SELECT * FROM users WHERE last_active > DATE_SUB(NOW(), INTERVAL 1 DAY) LIMIT 10000;

Filesystem Selection for Caching Behavior

Different filesystems have different caching characteristics:

Numa Cache Considerations

On multi-socket servers, Non-Uniform Memory Access (NUMA) can significantly impact cache performance:

# View NUMA topology
numactl --hardware

# Run MySQL with NUMA awareness
numactl --interleave=all mysqld

# MySQL NUMA configuration
[mysqld]
innodb_numa_interleave=1

Cache Partitioning

For systems with multiple applications, cache partitioning can prevent one application from dominating the cache:

# Create separate mount points for different applications
mount -o size=4G -t tmpfs tmpfs /mnt/mysql_tmpfs
mount -o size=2G -t tmpfs tmpfs /mnt/redis_tmpfs

# Move specific files to the partitioned cache
mkdir -p /mnt/mysql_tmpfs/tmp
ln -s /mnt/mysql_tmpfs/tmp /var/lib/mysql/tmp

Conclusion

The operating system’s cache mechanism plays a pivotal role in database performance, serving as a critical layer between applications and storage devices. Our exploration has revealed how both MySQL and Redis leverage the OS cache in different ways, each with its own unique optimization considerations.

Key Insights

1. Layered Caching Architecture: Database systems implement multiple caching layers that work together:
   • Application-level caches (e.g., Redis itself)
   • Database engine caches (MySQL’s Buffer Pool)
   • OS cache
   • Storage controller caches

2. Performance Implications: The difference between memory and disk access speeds (nano vs. milliseconds) makes caching essential for performance. Even a small decrease in cache hit rates can cause dramatic performance degradation.

3. Cache Cooperation: For optimal performance, database settings should be configured with OS cache in mind. A well-tuned system balances memory allocation between application caches and OS cache.

4. Monitoring Importance: Regularly monitoring cache hit rates, dirty page ratios, and I/O patterns allows for early detection of caching issues before they impact users.

Practical Recommendations

As we’ve seen through case studies and examples, several practical steps can ensure optimal cache performance:

• Balance memory allocation between database buffer pools and OS cache
• Monitor cache metrics to identify potential issues
• Avoid unnecessary cache clearing through automated scripts
• Tune OS cache parameters (vm.dirty_ratio, vm.swappiness) based on workload
• Consider filesystem and storage options that complement your caching strategy
• Implement cache warming strategies for planned maintenance and restarts

Broader Implications

The principles of caching extend beyond databases to all computing systems. The fundamental tradeoff between speed and capacity drives the design of modern storage hierarchies from CPU registers to cloud storage.

Understanding how the OS cache interfaces with applications provides deeper insight into system behavior and performance optimization opportunities. For database administrators and system engineers, this knowledge is not just theoretical—it has direct, practical applications in designing high-performance, reliable systems.

As database technologies continue to evolve, the role of OS caching remains crucial. Even with in-memory databases, the OS cache continues to play an important role in persistence, recovery, and overall system performance.

Remember: “A well-used cache is often more important than fast storage.” By leveraging the OS cache effectively, you can achieve significant performance improvements without additional hardware investments.

References

• RedHat: Linux Page Cache
• MySQL Documentation: InnoDB Buffer Pool
• Redis Documentation: Persistence
• Kernel Documentation: VM Sysctl
• Percona: Understanding InnoDB Buffer Pool
• Brendan Gregg: Linux Performance
• RedHat: VM Subsystems Monitoring
• Netflix: Using eBPF to Understand Cache Behavior
• PostgreSQL Wiki: Tuning Your Server
• Oracle: MySQL Server System Variables
• RHEL Documentation: Filesystems
• Linux Kernel Documentation: Page Cache
• Amazon AWS: Optimizing MySQL Performance