Affordable & Performant S3 Compatible Storage Providers
Evaluating & Benchmarking BackBlaze (B2), CloudFlare (R2) & IDrive® (e2)
Some Background
I run a few projects that need
1-2 TB
of Storage along with5-10 TB
of Bandwidth per month. So I want a provider that won't cause me bankruptcy or be too slow1.Current Setup
I use Cloudflare R2 mostly because my website already uses Cloudflare & r2 has free egress along with globally distributed buckets2. They have a pricing calculator here: r2-calculator.cloudflare.com. With default Operations3, accounting only for storage, it's
~ $14.85/month/TB || ~ $1.35/month/100GB
What do I actually get charged? (For an average month)
The total comes to about ~ $57
. What's to note here is, Class A Operations (Write/Mutate)
are more expensive than Storage itself.
The Alternatives
I came across this reddit post on r/cloudstorage. Thanks to
u/CookieDelivery
who made this spreadsheet: https://docs.google.com/spreadsheets/d/1cEd65XDW3gBHnRsJ0rbq3V_B28mKySHiMPAZvArHiiA/edit#gid=0 , I was able to quite quickly narrow down my search.As I need s3 api compatible storage, providers like
Box
,DeGoo
,DropBox
,Google Drive
,iceDrive
,Jottacloud
,Koofr.eu
,MediaFire
,Mega
4,OneDrive
,pCloud
,Proton Drive
,Sync.com
,TeraBox
,Zoho
either didn't provide s3-api compatibility, or were too expensive.Azure Blob Storage
isn't s35.Hetzner Storage Box
auto blocks signups from third-world countries. ,STORJ
is decentralized.Wasabi
has a 90 days retention policy.Amazon S3
,Digital Ocean
,Google Cloud Storage
,IBM Cloud Object Storage
,Oracle Object Storage
are too expensive. This leaves me with BackBlaze (B2), CloudFlare (R2) & IDrive® (e2).
Cost
BackBlaze B2
Free Operations: https://www.backblaze.com/cloud-storage/transaction-pricing
Free Bandwidth: https://www.cloudflare.com/partners/technology-partners/backblaze/
-
Using numbers from my monthly average usage:
-
The total comes to about
~ $58
. A mere1$
more thanCloudflare R2
. Even though theAPI Transaction
fees are much, much higher, BackBlaze offers much cheaper storage. It's~ $6/month/TB || ~ $0.6/month/100GB
compared to Cloudflare's~ $14.85/month/TB || ~ $1.35/month/100GB
IDrive® (e2)
The best thing about
IDrive® (e2)
is that there's no charge for API Transactions. So only storage pricing exists. It's~ $4/month/TB || ~ $0.4/month/100GB
. However, they charge for egress bandwidth once it exceeds 3X of $STORAGE. They also aren't a part of bandwidth-alliance.Using numbers from my monthly average usage:
The total comes to about
~ $38
. Mostly due to bandwidth, a factor I can't easily control. If the Bandwidth($0.01/GB)
were to get~ 10TB
, I would pay~ 60$ (After subtracting 4140GB)
for bandwidth alone. Other than that,IDrive® (e2)
seems to be the cheapest option.
Performance
Disclaimer
Servers
Server-AWS-oregon
was of type t4g.small
hosted on Amazon
in Oregon, USA
. Server-DGO-nyc
was of type DO-Regular (2) @ 2.2GHz
hosted on Digital Ocean
in Newyork, USA
. Server-azure-Chicago
was of typeStandard_D4as_v5
hosted onAzure
in Chicago, USA
. Server-gcp-iowa
was of type t2a-large
hosted on Google
in Oregon, USA
. Server-contabo-oregon
was of type Cloud VPS 1
hosted onContabo
in Düsseldorf,Germany.
Finally, my local machine using WorldLink`s 300Mbps Down 150Mbps Up
located in Kathmandu,Nepal
. At the time of benchmarking, I will pick the server that's closest to the bucket region. If for some reason, I have to run the same test on multiple servers, then I will pick the fastest result, without averaging.Methodology
regions
to be closer to servers
running these tests, everything else (cmd/args/parameters passed to the tools) will remain the same. I will use default settings for these tools as much as possible.BackBlaze B2
-
$# Download warp: https://github.com/minio/warp/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" $# Setup warp export WARP_HOST="<BACKBLAZE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>" export WARP_ACCESS_KEY="<BACKBLAZE_APP_ID>" export WARP_SECRET_KEY="<BACKBLAZE_APP_KEY>" $# Finally Benchmark #Choose a unique bucket_name. (Decrease duration if concerned about bucket size) warp mixed --bucket="warp-<UNIQUE_BUCKET_NAME" --duration="5m" --analyze.v --tls ❯ #RESULTS (from a server closest to the bucket region) ---------------------------------------- Operation: DELETE - total: 1649, 10.0%, Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.313 +0000 UTC * Throughput:5.63 obj/s Requests considered: 1650: * Avg: 50ms, 50%: 39ms, 90%: 80ms, 99%: 159ms, Fastest: 24ms, Slowest: 1.176s, StdDev: 40ms ---------------------------------------- Operation: GET - total: 7438, 45.0%, Size: 10485760 bytes. Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.282 +0000 UTC * Throughput:253.01 MiB/s, 25.30 obj/s Requests considered: 7439: * Avg: 348ms, 50%: 282ms, 90%: 660ms, 99%: 1.227s, Fastest: 80ms, Slowest: 3.076s, StdDev: 248ms * TTFB:Avg: 175ms, Best: 20ms, 25th: 60ms, Median: 130ms, 75th: 223ms, 90th: 376ms, 99th: 770ms, Worst: 1.515s StdDev: 162ms * First Access: Avg: 423ms, 50%: 351ms, 90%: 753ms, 99%: 1.357s, Fastest: 115ms, Slowest: 1.831s, StdDev: 249ms * First Access TTFB: Avg: 231ms, Best: 23ms, 25th: 122ms, Median: 181ms, 75th: 283ms, 90th: 446ms, 99th: 873ms, Worst: 1.515s StdDev: 173ms * Last Access: Avg: 363ms, 50%: 299ms, 90%: 679ms, 99%: 1.297s, Fastest: 81ms, Slowest: 2.823s, StdDev: 250ms * Last Access TTFB: Avg: 189ms, Best: 21ms, 25th: 69ms, Median: 143ms, 75th: 235ms, 90th: 406ms, 99th: 838ms, Worst: 1.402s StdDev: 170ms ---------------------------------------- Operation: PUT - total: 2474, 15.0%, Size: 10485760 bytes. Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.284 +0000 UTC * Throughput:84.36 MiB/s, 8.44 obj/s Requests considered: 2475: * Avg: 1.231s, 50%: 693ms, 90%: 2.652s, 99%: 4.424s, Fastest: 162ms, Slowest: 8.422s, StdDev: 1.069s ---------------------------------------- Operation: STAT - total: 4966, 30.0%, Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.292 +0000 UTC * Throughput:16.90 obj/s Requests considered: 4967: * Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 76ms, Fastest: 19ms, Slowest: 256ms, StdDev: 12ms * First Access: Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 77ms, Fastest: 19ms, Slowest: 256ms, StdDev: 13ms * Last Access: Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 78ms, Fastest: 20ms, Slowest: 210ms, StdDev: 13ms Cluster Total: 337.34 MiB/s, 56.22 obj/s over 4m54s. ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)] ---------------------------------------- Operation: DELETE - total: 137, 9.9%, Concurrency: 20, Ran 4m29s, starting 2024-03-12 21:41:26.63 +0545 +0545 * Throughput:0.52 obj/s Requests considered: 138: * Avg: 324ms, 50%: 306ms, 90%: 343ms, 99%: 859ms, Fastest: 273ms, Slowest: 859ms, StdDev: 81ms ---------------------------------------- Operation: GET - total: 614, 44.2%, Size: 10485760 bytes. Concurrency: 20, Ran 4m35s, starting 2024-03-12 21:41:22.879 +0545 +0545 * Throughput:22.94 MiB/s, 2.29 obj/s Requests considered: 615: * Avg: 2.831s, 50%: 2.709s, 90%: 4.076s, 99%: 5.257s, Fastest: 1.375s, Slowest: 8.099s, StdDev: 911ms * TTFB:Avg: 704ms, Best: 265ms, 25th: 498ms, Median: 640ms, 75th: 847ms, 90th: 1.103s, 99th: 1.551s, Worst: 2.298s StdDev: 277ms * First Access: Avg: 2.899s, 50%: 2.755s, 90%: 4.144s, 99%: 5.533s, Fastest: 1.51s, Slowest: 8.099s, StdDev: 937ms * First Access TTFB: Avg: 733ms, Best: 317ms, 25th: 516ms, Median: 662ms, 75th: 881ms, 90th: 1.114s, 99th: 1.658s, Worst: 2.298s StdDev: 271ms * Last Access: Avg: 2.87s, 50%: 2.75s, 90%: 4.144s, 99%: 5.533s, Fastest: 1.375s, Slowest: 6.559s, StdDev: 913ms * Last Access TTFB: Avg: 708ms, Best: 265ms, 25th: 503ms, Median: 644ms, 75th: 857ms, 90th: 1.103s, 99th: 1.658s, Worst: 2.298s StdDev: 278ms ---------------------------------------- Operation: PUT - total: 190, 13.7%, Size: 10485760 bytes. Concurrency: 20, Ran 4m30s, starting 2024-03-12 21:41:22.82 +0545 +0545 * Throughput:7.76 MiB/s, 0.78 obj/s Requests considered: 191: * Avg: 16.034s, 50%: 14.231s, 90%: 25s, 99%: 41.969s, Fastest: 4.954s, Slowest: 45.698s, StdDev: 7.539s ---------------------------------------- Operation: STAT - total: 416, 29.9%, Concurrency: 20, Ran 4m35s, starting 2024-03-12 21:41:22.735 +0545 +0545 * Throughput:1.52 obj/s Requests considered: 417: * Avg: 300ms, 50%: 291ms, 90%: 317ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 61ms * First Access: Avg: 299ms, 50%: 291ms, 90%: 317ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 60ms * Last Access: Avg: 301ms, 50%: 291ms, 90%: 318ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 65ms Cluster Total: 30.36 MiB/s, 5.06 obj/s over 4m35s.
-
$# Download rclone: https://github.com/rclone/rclone/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" $# Download dperf: https://github.com/minio/dperf/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" # Configure rclone to use backblaze as remote: https://www.backblaze.com/docs/cloud-storage-integrate-rclone-with-backblaze-b2 $# Mount Bucket rclone mount "backblaze-b2:" "/tmp/mnt/backblaze-b2" \ --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.5.3 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36" \ --buffer-size="100M" \ --s3-upload-concurrency="500" \ --s3-chunk-size="100M" \ --multi-thread-streams="500" \ --checkers="2000" \ --transfers="1000" \ --vfs-cache-mode="full" \ --vfs-read-ahead="100M" \ --vfs-read-chunk-size-limit="off" \ --vfs-fast-fingerprint \ --cache-writes \ --check-first \ --checksum \ --allow-other \ --daemon $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs #DEFAULT ❯ dperf "/tmp/mnt/backblaze-b2" ┌───────────┬────────────┐ │ TotalREAD │ TotalWRITE │ │ 1.3 GiB/s │ 874 MiB/s │ └───────────┴────────────┘ # More realistic ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/backblaze-b2" ┌───────────┬────────────┐ │ TotalREAD │ TotalWRITE │ │ 160 MiB/s │ 447 MiB/s │ └───────────┴────────────┘
-
$# Download warp: https://github.com/minio/warp/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" $# Setup warp export WARP_HOST="<CLOUDFLARE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>" export WARP_ACCESS_KEY="<CLOUDFLARE_ACCESS_KEY>" export WARP_SECRET_KEY="<CLOUDFLARE_SECRET_KEY>" $# Finally Benchmark #Choose a unique bucket_name. (Decrease duration if concerned about bucket size) warp mixed --bucket="warp-cloudflare-r2" --duration="5m" --analyze.v --tls ❯ #RESULTS (I ran three tests, in three diff regions, results were similar) Mixed operations. ---------------------------------------- Operation: DELETE - total: 1137, 10.0%, Concurrency: 20, Ran 3m18s, starting 2024-03-12 14:00:58.193 +0000 UTC * Throughput:5.74 obj/s Requests considered: 1138: * Avg: 147ms, 50%: 137ms, 90%: 155ms, 99%: 388ms, Fastest: 120ms, Slowest: 1.184s, StdDev: 60ms ---------------------------------------- Operation: GET - total: 5130, 45.0%, Size: 10485760 bytes. Concurrency: 20, Ran 3m19s, starting 2024-03-12 14:00:58.265 +0000 UTC * Throughput:258.32 MiB/s, 25.83 obj/s Requests considered: 5131: * Avg: 277ms, 50%: 234ms, 90%: 402ms, 99%: 915ms, Fastest: 133ms, Slowest: 7.811s, StdDev: 196ms * TTFB:Avg: 158ms, Best: 81ms, 25th: 104ms, Median: 123ms, 75th: 167ms, 90th: 241ms, 99th: 622ms, Worst: 2.599s StdDev: 136ms * First Access: Avg: 358ms, 50%: 311ms, 90%: 476ms, 99%: 1.228s, Fastest: 156ms, Slowest: 7.811s, StdDev: 262ms * First Access TTFB: Avg: 212ms, Best: 87ms, 25th: 147ms, Median: 170ms, 75th: 226ms, 90th: 299ms, 99th: 840ms, Worst: 2.599s StdDev: 178ms * Last Access: Avg: 282ms, 50%: 237ms, 90%: 412ms, 99%: 1.019s, Fastest: 137ms, Slowest: 3.291s, StdDev: 182ms * Last Access TTFB: Avg: 162ms, Best: 81ms, 25th: 104ms, Median: 124ms, 75th: 171ms, 90th: 247ms, 99th: 695ms, Worst: 2.599s StdDev: 145ms ---------------------------------------- Operation: PUT - total: 1699, 14.9%, Size: 10485760 bytes. Concurrency: 20, Ran 3m18s, starting 2024-03-12 14:00:58.224 +0000 UTC * Throughput:85.84 MiB/s, 8.58 obj/s Requests considered: 1700: * Avg: 1.087s, 50%: 435ms, 90%: 612ms, 99%: 2.054s, Fastest: 293ms, Slowest: 1m40.634s, StdDev: 7.684s ---------------------------------------- Operation: STAT - total: 3413, 30.0%, Concurrency: 20, Ran 3m19s, starting 2024-03-12 14:00:58.202 +0000 UTC * Throughput:17.17 obj/s Requests considered: 3414: * Avg: 97ms, 50%: 89ms, 90%: 104ms, 99%: 368ms, Fastest: 74ms, Slowest: 1.178s, StdDev: 57ms * First Access: Avg: 98ms, 50%: 88ms, 90%: 104ms, 99%: 393ms, Fastest: 75ms, Slowest: 1.178s, StdDev: 57ms * Last Access: Avg: 98ms, 50%: 89ms, 90%: 104ms, 99%: 346ms, Fastest: 75ms, Slowest: 1.178s, StdDev: 63ms Cluster Total: 343.85 MiB/s, 57.29 obj/s over 3m19s. ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)] ---------------------------------------- Operation: DELETE - total: 103, 9.8%, Concurrency: 20, Ran 3m7s, starting 2024-03-12 20:21:28.219 +0545 +0545 * Throughput:0.57 obj/s Requests considered: 104: * Avg: 586ms, 50%: 523ms, 90%: 873ms, 99%: 1.085s, Fastest: 468ms, Slowest: 1.085s, StdDev: 148ms ---------------------------------------- Operation: GET - total: 458, 43.6%, Size: 10485760 bytes. Concurrency: 20, Ran 3m8s, starting 2024-03-12 20:21:28.244 +0545 +0545 * Throughput:25.11 MiB/s, 2.51 obj/s Requests considered: 459: * Avg: 5.132s, 50%: 3.028s, 90%: 11.056s, 99%: 46.666s, Fastest: 920ms, Slowest: 1m11.835s, StdDev: 7.38s * TTFB:Avg: 1.316s, Best: 420ms, 25th: 833ms, Median: 1.102s, 75th: 1.693s, 90th: 2.098s, 99th: 4.926s, Worst: 8.45s StdDev: 880ms * First Access: Avg: 5.89s, 50%: 3.524s, 90%: 14.287s, 99%: 51.437s, Fastest: 1.087s, Slowest: 1m11.835s, StdDev: 7.981s * First Access TTFB: Avg: 1.466s, Best: 442ms, 25th: 943ms, Median: 1.207s, 75th: 1.872s, 90th: 2.131s, 99th: 5.396s, Worst: 8.45s StdDev: 881ms * Last Access: Avg: 5.231s, 50%: 3.02s, 90%: 12.152s, 99%: 46.666s, Fastest: 920ms, Slowest: 1m11.835s, StdDev: 7.488s * Last Access TTFB: Avg: 1.32s, Best: 420ms, 25th: 843ms, Median: 1.102s, 75th: 1.697s, 90th: 2.109s, 99th: 4.926s, Worst: 8.45s StdDev: 885ms ---------------------------------------- Operation: PUT - total: 152, 14.5%, Size: 10485760 bytes. Concurrency: 20, Ran 3m5s, starting 2024-03-12 20:21:31.086 +0545 +0545 * Throughput:8.54 MiB/s, 0.85 obj/s Requests considered: 153: * Avg: 4.849s, 50%: 4.295s, 90%: 7.41s, 99%: 15.38s, Fastest: 2.109s, Slowest: 16.705s, StdDev: 2.19s ---------------------------------------- Operation: STAT - total: 322, 30.6%, Concurrency: 20, Ran 3m9s, starting 2024-03-12 20:21:27.924 +0545 +0545 * Throughput:1.73 obj/s Requests considered: 323: * Avg: 541ms, 50%: 474ms, 90%: 710ms, 99%: 1.526s, Fastest: 403ms, Slowest: 3.764s, StdDev: 240ms * First Access: Avg: 542ms, 50%: 476ms, 90%: 737ms, 99%: 1.526s, Fastest: 403ms, Slowest: 3.764s, StdDev: 249ms * Last Access: Avg: 543ms, 50%: 476ms, 90%: 705ms, 99%: 1.526s, Fastest: 411ms, Slowest: 3.764s, StdDev: 248ms Cluster Total: 33.18 MiB/s, 5.60 obj/s over 3m8s.
-
$# Download rclone: https://github.com/rclone/rclone/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" $# Download dperf: https://github.com/minio/dperf/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" # Configure rclone to use cloudflare as remote: https://developers.cloudflare.com/r2/examples/rclone/ $# Mount Bucket # Same mnt cmd as above $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs #DEFAULT ❯ dperf "/tmp/mnt/cloudflare-r2" ┌───────────┬────────────┐ │ TotalREAD │ TotalWRITE │ │ 1.5 GiB/s │ 1.0 GiB/s │ └───────────┴────────────┘ # More realistic ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/cloudflare-r2" ┌───────────┬────────────┐ │ TotalREAD │ TotalWRITE │ │ 88 MiB/s │ 440 MiB/s │ └───────────┴────────────┘
-
$# Download warp: https://github.com/minio/warp/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" $# Setup warp export WARP_HOST="<IDRIVE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>" export WARP_ACCESS_KEY="<IDRIVE_ACCESS_KEY>" export WARP_SECRET_KEY="<IDRIVE_SECRET_KEY>" $# Finally Benchmark #idrive has unique creds per bucket, here bucket means a dir warp mixed --bucket="warp-idrive-r2" --duration="5m" --analyze.v --tls ❯ #RESULTS (from a server closest to the bucket region) ---------------------------------------- Operation: DELETE - total: 1596, 10.0%, Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.77 +0000 UTC * Throughput:5.39 obj/s Requests considered: 1597: * Avg: 27ms, 50%: 28ms, 90%: 30ms, 99%: 35ms, Fastest: 20ms, Slowest: 48ms, StdDev: 3ms ---------------------------------------- Operation: GET - total: 7166, 44.9%, Size: 10485760 bytes. Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.863 +0000 UTC * Throughput:241.96 MiB/s, 24.20 obj/s Requests considered: 7167: * Avg: 590ms, 50%: 553ms, 90%: 885ms, 99%: 1.34s, Fastest: 128ms, Slowest: 3.695s, StdDev: 239ms * TTFB:Avg: 83ms, Best: 25ms, 25th: 69ms, Median: 79ms, 75th: 92ms, 90th: 108ms, 99th: 167ms, Worst: 465ms StdDev: 24ms * First Access: Avg: 589ms, 50%: 544ms, 90%: 892ms, 99%: 1.407s, Fastest: 128ms, Slowest: 2.019s, StdDev: 244ms * First Access TTFB: Avg: 82ms, Best: 25ms, 25th: 69ms, Median: 78ms, 75th: 91ms, 90th: 107ms, 99th: 157ms, Worst: 465ms StdDev: 25ms * Last Access: Avg: 588ms, 50%: 553ms, 90%: 873ms, 99%: 1.315s, Fastest: 128ms, Slowest: 3.695s, StdDev: 236ms * Last Access TTFB: Avg: 83ms, Best: 30ms, 25th: 70ms, Median: 79ms, 75th: 92ms, 90th: 108ms, 99th: 161ms, Worst: 309ms StdDev: 22ms ---------------------------------------- Operation: PUT - total: 2384, 14.9%, Size: 10485760 bytes. Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.779 +0000 UTC * Throughput:80.70 MiB/s, 8.07 obj/s Requests considered: 2385: * Avg: 627ms, 50%: 559ms, 90%: 1.011s, 99%: 1.846s, Fastest: 145ms, Slowest: 5.552s, StdDev: 347ms ---------------------------------------- Operation: STAT - total: 4793, 30.0%, Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.942 +0000 UTC * Throughput:16.16 obj/s Requests considered: 4794: * Avg: 30ms, 50%: 31ms, 90%: 33ms, 99%: 38ms, Fastest: 23ms, Slowest: 115ms, StdDev: 3ms * First Access: Avg: 30ms, 50%: 30ms, 90%: 33ms, 99%: 39ms, Fastest: 23ms, Slowest: 115ms, StdDev: 4ms * Last Access: Avg: 30ms, 50%: 31ms, 90%: 33ms, 99%: 38ms, Fastest: 23ms, Slowest: 56ms, StdDev: 3ms Cluster Total: 322.36 MiB/s, 53.77 obj/s over 4m57s. ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)] ---------------------------------------- Operation: DELETE - total: 45, 10.1%, Concurrency: 20, Ran 3m55s, starting 2024-03-12 20:55:33.285 +0545 +0545 * Throughput:0.20 obj/s Requests considered: 46: * Avg: 88ms, 50%: 86ms, 90%: 97ms, 99%: 102ms, Fastest: 81ms, Slowest: 102ms, StdDev: 5ms ---------------------------------------- Operation: GET - total: 183, 41.2%, Size: 10485760 bytes. Concurrency: 20, Ran 4m8s, starting 2024-03-12 20:55:23.679 +0545 +0545 * Throughput:8.57 MiB/s, 0.86 obj/s Requests considered: 184: * Avg: 19.836s, 50%: 19.91s, 90%: 27.916s, 99%: 31.819s, Fastest: 3.402s, Slowest: 35.719s, StdDev: 6.123s * TTFB:Avg: 161ms, Best: 89ms, 25th: 131ms, Median: 137ms, 75th: 205ms, 90th: 223ms, 99th: 284ms, Worst: 461ms StdDev: 47ms * First Access: Avg: 20.216s, 50%: 20.37s, 90%: 28s, 99%: 35.719s, Fastest: 3.402s, Slowest: 38.175s, StdDev: 6.231s * First Access TTFB: Avg: 162ms, Best: 89ms, 25th: 131ms, Median: 137ms, 75th: 206ms, 90th: 225ms, 99th: 284ms, Worst: 461ms StdDev: 47ms * Last Access: Avg: 20.455s, 50%: 20.757s, 90%: 28.085s, 99%: 35.719s, Fastest: 3.402s, Slowest: 38.175s, StdDev: 6.315s * Last Access TTFB: Avg: 160ms, Best: 89ms, 25th: 130ms, Median: 137ms, 75th: 205ms, 90th: 222ms, 99th: 284ms, Worst: 461ms StdDev: 47ms ---------------------------------------- Operation: PUT - total: 65, 14.6%, Size: 10485760 bytes. Concurrency: 20, Ran 3m59s, starting 2024-03-12 20:55:24.74 +0545 +0545 * Throughput:2.86 MiB/s, 0.29 obj/s Requests considered: 66: * Avg: 6.765s, 50%: 4.238s, 90%: 15.266s, 99%: 20.483s, Fastest: 1.919s, Slowest: 20.483s, StdDev: 5.207s ---------------------------------------- Operation: STAT - total: 128, 28.8%, Concurrency: 20, Ran 4m1s, starting 2024-03-12 20:55:26.167 +0545 +0545 * Throughput:0.54 obj/s Requests considered: 129: * Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms * First Access: Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms * Last Access: Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms Cluster Total: 11.01 MiB/s, 1.83 obj/s over 4m4s.
-
$# Download rclone: https://github.com/rclone/rclone/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" $# Download dperf: https://github.com/minio/dperf/releases/latest # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" # Configure rclone to use cloudflare as remote: https://www.idrive.com/s3-storage-e2/rclone $# Mount Bucket # Same mnt cmd as above $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs #DEFAULT ❯ dperf "/tmp/mnt/idrive-e2" ┌───────────┬────────────┐ │ TotalREAD │ TotalWRITE │ │ 1.3 GiB/s │ 1.1 GiB/s │ └───────────┴────────────┘ # More realistic ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/idrive-e2" ┌───────────┬────────────┐ │ TotalREAD │ TotalWRITE │ │ 87 MiB/s │ 444 MiB/s │ └───────────┴────────────┘
Conclusions
-
Cost: It cost me
$5
to run all these tests/benchmarks. This is to say, that even though they are a part of the bandwidth-alliance, that's only true for a pre-configured rclone, which currently doesn't work. See: https://github.com/backblaze-b2-samples/cloudflare-b2/issues/16 . So most s3-clients will still hit the bandwidth cap pretty quickly, and you will be charged at0.01$/GB.
And if you have to pay for egress anyway, BackBlaze's B2 will quickly become the most expensive provider of the three thanks to astronomically high cost ofAPI Transactions
.Performance: In terms of performance/speed, the benchmark seems promising. However, that will hold true only as long as your server is located near their servers. And they are so few and limited: https://www.backblaze.com/docs/cloud-storage-data-regions
-
Cost: It cost me
2$
to run all these tests/benchmarks. This is to say, that even though they are NOT a part of the bandwidth-alliance, at least they don't charge forAPI Transactions
.Performance: In terms of performance/speed, the benchmark lags behind a bit but unlike BackBlaze, you can still get higher speeds because they have more regions & server locations: https://www.idrive.com/s3-storage-e2/locations
-
Cost: It cost me
0.5$
to run all these tests/benchmarks. This is to say, that I only got charged for Storage. I didn't hit the super generousAPI Transactions
.
Bandwidth/egress is free.Performance: In terms of performance/speed, the benchmark stays stable, no matter what location I tested from. The difference in delta was
10-20 MiB/s
at most. Compare this to Backblaze's B2 that falls off by70-80%
and the delta gets100-150 MiB/s.
Or to Idrive, who despite having more locations than BackBlaze, still can't provide stable speed because they aren't a globally distributed CDN.
TLDR
-
I will be sticking with CloudFlare R2 because of no bandwidth charge and reasonable pricing for
API Transactions.
Personal Media & Misc Files
In the near future, I intend to host personal media, where I will likely use IDrive® (e2) because of cheap storage & no charge on
API Transactions
. In terms of Bandwidth/egress, IDrive® (e2) will still be cheaper as I don't think I will need to download 3X of my Storage.-
Unless they change their pricing model for
API Transactions
& expand to more regions, B2 is not a sane choice for any of my needs.
-
Slow means longer time to
read|write|download|upload
. Hence, my servers will have to run for longer times, incurring me additional costs. ↩As per some forum discussion this may not actually be true. Regardless,
R2+CDN
already sort of achieves the same.↩Cloudflare offers free
Write Operations (Class A): 1000000 (1 Million)
&Read Operations (Class B): 10000000 (10 Million)
per month. After which, you are charged$4.50 / million
for Class A &$0.36 / million
for Class B respectively.↩As of 2024-03-12, Mega's Object Storage is beta-only https://mega.io/objectstorage↩
Azure Blob Storageisn't s3 compatible. While yes, things like https://github.com/gaul/s3proxy exist, it's not native and impacts performance/reliability.↩