In the TPC-C benchmark tests, SkySQL consistently outperformed AWS RDS Aurora in both throughput and latency across all workloads, delivering up to 3x higher throughput and 2.5x lower latency.

Also, SkySQL demonstrated up to 7.5x better price-performance on Cost per Million Transactions for the benchmark workloads.

Overview

When customers using AWS Aurora approach us, they typically focus on key attributes such as latency under high concurrency, availability during failures, and price-performance efficiency for their workloads. Although discussions often gravitate toward large-scale data volumes, most operational databases are actually modest in size, ranging from hundreds of gigabytes to a few terabytes.

Despite the wealth of information on Aurora’s storage engine, finding clear and straightforward third-party benchmark results has proven difficult. We discovered a simple TPC-C benchmark run that compares different RDS engine flavors, including MySQL and MariaDB, which provided some clarity. Additionally, we observed performance trends for MySQL and MariaDB over time, noting that modern MariaDB consistently outperforms modern MySQL in sysbench tests.

These insights spurred our curiosity about how MariaDB fares against MySQL in a fully managed cloud environment that includes several other components beyond just the DB engine, such as firewall, secure connections, network performance with encryption, compute-storage separation, continuous health monitoring. And, customers, crucially, want to understand price-to-performance ratios. Hence, it made sense to compare SkySQL (a fully managed DB service) with AWS Aurora.

In this post, we compare SkySQL running MariaDB 11.4 to AWS Aurora running MySQL 8.0.xx using Carnegie Mellon Database Group’s BenchBase tool running the TPC-C benchmark. TPC-C, developed by the TPC council, evaluates OLTP system performance by simulating the workload of an order-entry environment, akin to an e-commerce system. TPC-C remains one of the most renowned TPC benchmarks. We also conducted a simple sysbench comparison to assess pure read-only performance across both platforms.

Benchmarking Methodology

Our approach mirrors typical customer usage of “fully managed” databases: selecting an appropriate instance size and storage configuration, and relying on the database service to self-optimize – everything from max concurrent connections, various buffers like Buffer pool, Sort, Temp Buffers, etc. We aimed to evaluate out-of-the-box performance.

We controlled the data volume and concurrent client load using two parameters: ‘warehouse count’ and ‘terminals’. Both were scaled from 100 to 400 in our measurements. Additionally, all concurrent clients operated in “firehose” mode, meaning there was no think-time between transactions.

Configuration

SkySQL:

• Database Version: MariaDB 11.4 (latest LTS release)

• Instance Size: Scaled from 2 vCPU x 8 GB RAM for 100 concurrent clients to 16 vCPU x 128 GB RAM for 400 concurrent clients

• Storage Configuration: 500 GB gp3 storage, scaled from 10k IOPS and 500 MB/s throughput to 16k IOPS and 1000 MB/s throughput.

• Connection: Public IP endpoint for clients

Aurora:

• Database Version: Aurora MySQL 3.05.2 (Compatible with MySQL 8.0.32)

• Instance Size: Scaled from 2 vCPU x 16 GB RAM for 100 concurrent clients to 16 vCPU x 128 GB RAM for 400 concurrent clients

• Storage Configuration: Default Aurora storage for maximum IO speed.

• Connection: Public IP endpoint for clients

Both deployments were in AWS’s us-east-1 region, with clients and servers in the same Availability Zone to eliminate inter-AZ network jitter.

Results and Observations

We measured the total transactions per second (TPS) rate (throughput) and 95th percentile latency with an increasing number of warehouses and concurrent threads, ranging from 100 to 400.

We first measured TPS and latencies with increasing load – 100 threads to 400 threads on the same 8 core, 64GB server. The results are below

As expected, with increased load and threads bombarded with non-stop requests, the CPUs spend significant time in OS scheduling and increased IO contention resulting in reduced throughput.

Then, we scaled everything together – the load, the instance size and the data volume (warehouse count). We started with a small 2 core server for 100 concurrent clients and went up to 16 cores with 128 GB to handle 400 concurrent clients. The results are shown below.

SkySQL consistently outperformed Aurora in throughput and latency across all workloads, with the most significant difference observed at the lower end of the spectrum. The throughput gains ranged from over 3X to about 23% at the highest scale factor. Similarly, SkySQL maintained significantly lower latencies compared to Aurora.

The primary performance difference can be attributed to the use of MariaDB 11.4 in SkySQL compared to MySQL 8.0.x in Aurora. As we scale the threads, we notice that the performance delta begins to decrease. When operating in a “firehose” manner, lock contention in InnoDB begins to increase. This contention arises from several factors:

• Dirty Page Flushes: Transactions are locked out during the flushing process.

• Lock Escalation: Row-level locks escalate to page-level locks under high contention.

• Buffer Pool Management: While the BufferPool in SkySQL is sized proportionally to the available memory, it is conservatively configured. No tuning (e.g., adjusting InnoDB writer threads) was performed, adhering to the typical customer usage of fully managed databases, which involves minimal low-level tuning.

In contrast, Aurora delegates all Transaction log management to its distributed storage layer, which likely benefits from a larger, distributed memory management system with fewer lock contention issues. However, due to the opaque nature of Aurora’s storage layer, we cannot directly validate these assumptions.

It’s important to consider the Quality of Service (QoS) at the storage layer. Aurora creates four copies of each transaction for redundancy, while SkySQL also provides redundant replication at the storage level to safeguard against failures. The critical attribute for customers, however, is the full high availability (HA) of the database service. Redundant disk block copies are insufficient if the Aurora MySQL server itself crashes. In several comparative runs between SkySQL’s “replicated topology” and Aurora, we observed similar results, which we plan to detail in future updates.

‘Read only’ Test using a Microbenchmark

To understand how each product performs without the storage layer’s influence (ensuring a more apples-to-apples comparison), we conducted a simple “point select” test using sysbench with increasing concurrency. The data set was intentionally kept small enough to fit entirely into each server’s BufferPool.

Configuration

• SkySQL: 8×32 instance

• Aurora: 8×64 instance

The larger memory allocation for Aurora was not a factor, as the data set was much smaller. Specifically, we used 10 tables, each containing 200,000 rows. Each row in sysbench is relatively small, consisting of a few integers and small strings. No tuning was applied to either product to keep the test conditions as close to default configurations as possible.

Results and Observations

The results, as illustrated in the accompanying charts, aligned with our expectations. SkySQL demonstrated roughly 40% higher throughput compared to Aurora. This throughput remained consistent across all workloads, given that the test utilized the same capacity for each. Notably, 100 concurrent threads were sufficient to fully max out the available capacity.

In terms of latency, SkySQL exhibited a performance advantage, with latency measurements being approximately 50-60% lower than those of Aurora. As anticipated, latency degraded with increasing concurrency, but SkySQL maintained a superior performance profile throughout.

Price-Performance Comparison

Customers running their workloads on Aurora often express concerns about high and unpredictable costs, particularly related to IOPS. Since Aurora charges separately for I/O operations, workloads with high read and write demands can result in significantly higher cost. During our benchmark tests, we observed that write IOPS hovered around 500k to 1 million operations per 5-minute interval (See screenshot from Aurora monitoring below). Interestingly, volume reads were minimal, indicating that most read transactions were handled at the buffer cache level. However, in a real-world scenario, volume reads would also contribute to Aurora’s costs.

We priced the SkySQL and AWS Aurora configurations used for benchmarking as well as volume IOPS usage data for Aurora during the benchmark runs to estimate the costs of running the 5-min workloads. SkySQL Launch Estimator and AWS Calculator were used for respective price calculations. As illustrated in the chart on the left below, Aurora costs were at least 2 to 5 times higher compared to SkySQL for the various DB configurations used.

Combining the throughput benchmarks with the pricing estimates for running these tests, we calculated Cost per Million Transactions (CPMT) using the following formula. This gives the cost to process a million transactions, providing a direct comparison of how efficiently each platform handles workloads relative to their costs.

CPMT = (Cost per Second) / (Throughput per Second) x 1000000

As expected, SkySQL showed a significant advantage compared to AWS Aurora with cost per million transactions coming up to 7.5 times lower. Although Aurora’s CPMT reduces as instance size and concurrency scales, SkySQL still delivered CPMT 2.5 times lower than that of Aurora at the highest benchmark scale.

Conclusion

Our benchmarking results highlight the significant advantages of MariaDB SkySQL over AWS Aurora, both in terms of performance and cost-efficiency. SkySQL consistently delivered superior throughput, lower latency, and significantly reduced costs, making it an excellent choice for organizations looking to optimize their database operations.

If you’re ready to experience the benefits of SkySQL for yourself, we invite you to try it out here. Whether you’re migrating from another platform or starting a new project, SkySQL offers the performance and value that modern workloads demand.

We’re always here to help. If you have any questions or feedback, please don’t hesitate to reach out to us at [email protected].