Cloud Run SQLite Persistence: GCS Mount vs. Litestream Performance Comparison (Part 2)

Cloud Run の Google Cloud Storage マウントと Litestream のパフォーマンス比較(2回目)

English follows Japanese.

結論

Cloud Run 上で動く Rails 8 から SQLite を利用する際、書き込み性能においては Litestream が Google Cloud Storage マウント (GCS fuse) を上回る(約1.7倍)ことが分かった。

アプリケーションが書き込み処理を伴う場合、GCS fuse の利用は慎重に検討すべきであり、Litestream が有力な選択肢となる。

この2回目で一連の記事は終わりである。

本記事の構成

  • 結論
  • 本記事の構成
  • 参考文献
  • パフォーマンス比較
    • 仕様
    • Cloud Run の Google Cloud Storage マウント (GCS fuse)
    • Litestream
  • 考察
  • 生データ (RAW Data)
    • Cloud Run の Google Cloud Storage マウント (GCS fuse)
    • Litestream

参考文献

はじめに

前回は、Rails 8 において SQLite をバックエンドに持つアプリケーションを、Cloud Run 上で動かし、Google Cloud Storage マウント (GCS fuse) と Litestream のパフォーマンス比較を大雑把に行った。

今回は、1 vCPU でどのくらいのパフォーマンスが出るのか、もう少し詳細に検証する。

パフォーマンス比較

仕様

前回と同様、計測される側の Ruby on Rails の動作は以下である。

  • id, count の2カラムを持つ counters テーブルを作成
  • seeds.rbid = 'count_up', count = 0 のレコードを1件作成
  • /count_up への POST リクエスト毎に、id = 'count_up' のレコードの count を1増やす(update_all("count = count + 1"))
  • レスポンスには、count の値を {"count":4856} の形式で返す
  • /count_up への POST リクエストを多数回実行し、Request per second (RPS) を計測

パフォーマンス計測には k6 を用いた。
今回は、計測誤差が少なくなるよう、シナリオに変化を加えた。

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 2,
  duration: '120s',
};

export default function() {
  const res = http.post('https:// URL .asia-northeast1.run.app/count_up');
  check(res, { "status is 200": (res) => res.status === 200 });
}

コンテナの同時リクエスト受け付け数を 2 から 2 刻みに 20 まで変化させた。3回計測し、それぞれの指標について、3回のうちの中央値を採用した。

取得した指標は、med, avg, p(90), p(95), min, max である。

Cloud Run のコンテナには、1 vCPU と 1 GiB のメモリを割り当てた。

Cloud Run の Google Cloud Storage マウント (GCS fuse)

CPU utilization はほぼ 100 % で、メモリ使用量は 12 % 未満だった。

単位は ms である。

同時リクエスト数(vus)medavgp(90)p(95)minmax
210.9716.7938.7442.628.12629.35
418.2932.1161.564.077.81668.82
633.7749.372.4374.958.92685.2
873.265.7183.0586.158.59676.23
1085.0383.194.1497.758.99722.75
1294.2198.2104.83112.098.59748.89
14103.31114.14125.39149.317.91769.1
16114.58130.89158.2164.817.97754.52
18126.01146.7170.05176.238.19783.91
20164.94165.12184.681928.32783.74

Litestream

CPU utilization はほぼ 100 % で、メモリ使用量は 20 % 前後だった。

単位は ms である。

同時リクエスト数(vus)medavgp(90)p(95)minmax
28.8110.1412.6118.496.37101.62
411.1419.5152.0955.216.41159.78
617.0728.6360.0362.926.59169.82
825.4538.8766.6370.486.48247.1
1034.0347.3673.0877.276.7200.26
1268.556.3579.5584.16.49183.35
1476.9467.4688.0293.166.51219.32
1683.67794.6699.936.63189.46
1889.7285.77101.19107.726.39215.69
2095.2494.59106.96115.956.34215.97

考察

全体的なレイテンシは、Litestream の方が低く、Cloud Storage に書き込むためのレイテンシが高いためだと想定される。

min レイテンシを全体的に見たときに、ほぼ変わらないのは、最初の数回のリクエストが早く応答しているためと考えられる。Litestream と Cloud Storage マウントの差は約 2 ms で、これが Cloud Storage への書き込みの最短レイテンシと考えられる。Litestream は、ローカルに書き込みを完了した時点で次の処理に進み、転送はバックグラウンドで非同期にレプリケーションされるため、書き込みの待ち時間が発生しないと考えられる。

avg (相加平均)はどちらもほぼ線形に増加している。(生データに対して)y切片を0としたときの係数は、8.19 と 4.77 である。約1.7倍の差がある。これがパフォーマンスの差ほぼそのものと言って良いだろうか?

中央値 (med) と平均値 (avg) の差分については考察が難しい。Litestream については、vus = 12 で med が小さかったところから大きくなる側に逆転しているが、その後は med, avg が近づいている。Cloud Storage については、大体の場合 med の方が若干小さく、ロングテールになっているためだと考えると、妥当な結果である。

どちらも、純粋に CPU と書き込みの待ち時間が処理に影響しており、特別なボトルネックはないと考えて良さそうだ。

生データ (RAW Data)

Cloud Run の Google Cloud Storage マウント

vustimesavgminmedmaxp(90)p(95)
2116.488.0310.91629.3537.5741.74
2216.798.1210.97612.1139.1242.77
2317.338.1211.48656.1238.7442.62
4130.337.7116.65682.6361.263.57
4233.099.2719.05668.8261.7164.13
4332.117.8118.29662.9561.564.07
6143.558.0224.58647.8270.2472.28
6249.38.9233.77740.1972.4374.95
6350.859.0937.48685.272.8876.34
8157.987.4869.09663.8379.1582.02
8265.718.5973.2676.2383.0586.15
8367.889.8273.71714.9483.5887.24
10176.498.9981.41712.8890.5194.24
10283.18.9785.03722.7594.1497.75
10389.49.3787.47803.1998.5103.99
12189.28.5989.29684.3897.77102.06
12298.27.9494.21748.89104.83112.09
123100.979.2995.63750.84106.47114.56
141105.067.4598.19751.74107.53119.47
142114.147.91103.31769.1125.39149.31
143118.858.94105.81779.73145.42153.72
161121.47.9108.29731.85152.49157.6
162130.898.67114.58754.52158.2164.81
163134.97.97118.14832.44160.62166.12
181139.578.06120.03754.14166.18171.32
182146.78.19126.01817.21170.05176.23
183157.869.2147.54783.91176.8182.67
201160.138.1157.38773.16180.01185.73
202165.128.32164.94868.16184.68192
203179.599.37176.5783.74193.79200.92

Litestream

vustimesavgminmedmaxp(90)p(95)
2110.256.969.24101.6212.1816.5
2210.145.798.81109.7913.5519.96
2310.016.378.7999.1612.6118.49
4119.516.3410.01181.3657.2360.88
4219.596.4111.63115.6352.0755.21
4319.126.7211.14159.7852.0955.02
6129.516.5911.55279.9673.5978.47
6228.566.1917.07157.2359.9162.92
6328.637.2217.16169.8260.0362.77
8138.876.2720.34291.7984.7796.16
8239.16.8625.54247.166.6370.48
8338.636.4825.45169.7766.570.08
10147.366.6734.03214.7773.0876.84
10246.986.7832.54181.1372.9677.83
10347.556.734.44200.2673.1677.27
12156.226.6268.66165.7379.7184.11
12256.356.3968.17183.3579.5584.1
12356.366.4968.5211.9879.5583.65
14167.466.5177.03219.3288.0293.16
14267.636.3276.94184.4688.2493.2
14365.576.6576.06219.8986.2791.41
16178.086.6283.94199.2996.59102.79
162776.6383.6189.4694.6699.93
16375.836.6383.12180.2493.3498.1
18187.826.1990.41215.69103.28112.25
18285.576.7389.72215.08100.48106.3
18385.776.3989.34318.6101.19107.72
20195.666.6395.66213.93109.17134.28
20293.086.0994.51215.97105.93114.93
20394.596.3495.24250.34106.96115.95

Cloud Run SQLite Persistence: GCS Mount vs. Litestream Performance Comparison (Part 2)

Conclusion

When using SQLite with Rails 8 on Cloud Run, we found that for write performance, Litestream outperforms Google Cloud Storage mount (GCS fuse) by about 1.7 times.

If your application involves write operations, using GCS fuse should be carefully considered, making Litestream a strong alternative.

This is the end of the series of articles.

Table of Contents

  • Conclusion
  • Table of Contents
  • References
  • Introduction
  • Performance Comparison
    • Specifications
    • Cloud Run Google Cloud Storage Mount (GCS fuse)
    • Litestream
  • Analysis
  • Raw Data (See the Japanese part)

References

Introduction

In the previous article, we performed a rough performance comparison between Google Cloud Storage mount (GCS fuse) and Litestream for a Rails 8 application with a SQLite backend running on Cloud Run.

This time, we will conduct a more detailed investigation to see how much performance can be achieved with 1 vCPU.

Performance Comparison

Specifications

As before, the behavior of the Ruby on Rails application being measured is as follows:

  • A counters table is created with two columns: id and count.
  • seeds.rb creates one record: id = 'count_up', count = 0.
  • For each POST request to /count_up, the count of the id = 'count_up' record is incremented by 1 (update_all("count = count + 1")).
  • The response returns the value of count in the format {"count":4856}.
  • We execute numerous POST requests to /count_up to measure Requests Per Second (RPS).

We used k6 for performance measurement.
This time, we modified the scenario to reduce measurement errors.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 2,
  duration: '120s',
};

export default function() {
  const res = http.post('https:// URL .asia-northeast1.run.app/count_up');
  check(res, { "status is 200": (res) => res.status === 200 });
}

We varied the number of concurrent requests the container accepts from 2 to 20, in increments of 2. We ran the measurement 3 times for each setting and used the median of the three runs for each metric.

The metrics collected were: med, avg, p(90), p(95), min, and max.

The Cloud Run container was allocated 1 vCPU and 1 GiB of memory.

Cloud Run Google Cloud Storage Mount (GCS fuse)

CPU utilization was nearly 100%, and memory usage was less than 12%.

All units are in ms.

Concurrent Requests (vus)medavgp(90)p(95)minmax
210.9716.7938.7442.628.12629.35
418.2932.1161.564.077.81668.82
633.7749.372.4374.958.92685.2
873.265.7183.0586.158.59676.23
1085.0383.194.1497.758.99722.75
1294.2198.2104.83112.098.59748.89
14103.31114.14125.39149.317.91769.1
16114.58130.89158.2164.817.97754.52
18126.01146.7170.05176.238.19783.91
20164.94165.12184.681928.32783.74

Litestream

CPU utilization was nearly 100%, and memory usage was around 20%.

All units are in ms.

Concurrent Requests (vus)medavgp(90)p(95)minmax
28.8110.1412.6118.496.37101.62
411.1419.5152.0955.216.41159.78
617.0728.6360.0362.926.59169.82
825.4538.8766.6370.486.48247.1
1034.0347.3673.0877.276.7200.26
1268.556.3579.5584.16.49183.35
1476.9467.4688.0293.166.51219.32
1683.67794.6699.936.63189.46
1889.7285.77101.19107.726.39215.69
2095.2494.59106.96115.956.34215.97

Analysis

Overall latency is lower with Litestream, which is assumed to be due to the high latency of writing to Cloud Storage.

When looking at the overall min latency, there is almost no difference. This is likely because the first few requests respond quickly. The difference between Litestream and the Cloud Storage mount is about 2 ms, which can be considered the minimum write latency to Cloud Storage. With Litestream, it is thought that there is no write latency because it proceeds to the next operation as soon as the write to the local file is complete, and the transfer is replicated asynchronously in the background.

The avg (arithmetic mean) increases almost linearly for both. The coefficients, when the y-intercept is set to 0 (based on the raw data), are 8.19 and 4.77. This is a difference of about 1.7 times. Can we say this is essentially the performance difference?

The difference between the median (med) and the average (avg) is difficult to analyze. For Litestream, at vus = 12, the med flips from being smaller than avg to being larger, but they converge afterward. For the Cloud Storage mount, the med is generally slightly smaller, which is a reasonable result considering it has a long tail.

It seems safe to conclude that for both methods, the processing is purely affected by CPU and write latency, with no other specific bottlenecks.

Raw Data (See the Japanese part)

See the Japanese part above for the raw data.

Cloud Run,google cloud,Ruby

Posted by tako