Affordable & Performant S3 Compatible Storage Providers

Some Background

I run a few projects that need 1-2 TB of Storage along with 5-10 TB of Bandwidth per month. So I want a provider that won't cause me bankruptcy or be too slow¹.
Current Setup

I use Cloudflare R2 mostly because my website already uses Cloudflare & r2 has free egress along with globally distributed buckets². They have a pricing calculator here: r2-calculator.cloudflare.com. With default Operations³, accounting only for storage, it's ~ $14.85/month/TB || ~ $1.35/month/100GB

What do I actually get charged? (For an average month)

The total comes to about ~ $57. What's to note here is, Class A Operations (Write/Mutate)are more expensive than Storage itself.

The Alternatives

I came across this reddit post on r/cloudstorage. Thanks to u/CookieDelivery who made this spreadsheet: https://docs.google.com/spreadsheets/d/1cEd65XDW3gBHnRsJ0rbq3V_B28mKySHiMPAZvArHiiA/edit#gid=0 , I was able to quite quickly narrow down my search.

As I need s3 api compatible storage, providers like Box, DeGoo, DropBox, Google Drive, iceDrive, Jottacloud, Koofr.eu, MediaFire, Mega⁴, OneDrive, pCloud, Proton Drive, Sync.com, TeraBox, Zoho either didn't provide s3-api compatibility, or were too expensive. Azure Blob Storage isn't s3⁵. Hetzner Storage Box auto blocks signups from third-world countries. , STORJ is decentralized. Wasabi has a 90 days retention policy. Amazon S3, Digital Ocean, Google Cloud Storage, IBM Cloud Object Storage, Oracle Object Storageare too expensive. This leaves me with BackBlaze (B2), CloudFlare (R2) & IDrive® (e2).

Cost

BackBlaze B2

Free Operations: https://www.backblaze.com/cloud-storage/transaction-pricing

Free Bandwidth: https://www.cloudflare.com/partners/technology-partners/backblaze/
Using numbers from my monthly average usage:
The total comes to about ~ $58. A mere 1$ more than Cloudflare R2. Even though the API Transaction fees are much, much higher, BackBlaze offers much cheaper storage. It's ~ $6/month/TB || ~ $0.6/month/100GB compared to Cloudflare's ~ $14.85/month/TB || ~ $1.35/month/100GB

IDrive® (e2)

The best thing about IDrive® (e2) is that there's no charge for API Transactions. So only storage pricing exists. It's ~ $4/month/TB || ~ $0.4/month/100GB. However, they charge for egress bandwidth once it exceeds 3X of $STORAGE. They also aren't a part of bandwidth-alliance.
Using numbers from my monthly average usage:

The total comes to about ~ $38. Mostly due to bandwidth, a factor I can't easily control. If the Bandwidth ($0.01/GB) were to get ~ 10TB, I would pay ~ 60$ (After subtracting 4140GB) for bandwidth alone. Other than that, IDrive® (e2) seems to be the cheapest option.

Performance

Disclaimer

This is in no way scientific, or foolproof. And tbh, I don't really care about a thorough statistically useful test anyway. I only want to test these for my use case, so I can decide what's the best in terms of performance & cost. You are free to disagree or call this entire blog as inaccurate & biased jargon. I am basing this section on: https://hackernoon.com/benchmarking-minio-with-warp-and-speedtest

Servers

Tests were performed on 6 Servers. Server-AWS-oregon was of type t4g.small hosted on Amazon in Oregon, USA. Server-DGO-nyc was of type DO-Regular (2) @ 2.2GHz hosted on Digital Ocean in Newyork, USA. Server-azure-Chicago was of typeStandard_D4as_v5hosted onAzure in Chicago, USA. Server-gcp-iowa was of type t2a-large hosted on Google in Oregon, USA. Server-contabo-oregon was of type Cloud VPS 1 hosted onContabo in Düsseldorf,Germany. Finally, my local machine using WorldLink`s 300Mbps Down 150Mbps Up located in Kathmandu,Nepal. At the time of benchmarking, I will pick the server that's closest to the bucket region. If for some reason, I have to run the same test on multiple servers, then I will pick the fastest result, without averaging.

Methodology

MinIO offers a set of tools that can be used for these tests. There may be better, more accurate tools/ways/techniques for this, but I have decided to use: warp (For Detailed Mixed Test) & dperf (For rClone Mounted Drive Performance Test). Other than changing the bucket regions to be closer to servers running these tests, everything else (cmd/args/parameters passed to the tools) will remain the same. I will use default settings for these tools as much as possible.

BackBlaze B2

warp

  $# Download warp: https://github.com/minio/warp/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp"

  $# Setup warp
  export WARP_HOST="<BACKBLAZE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>"
  export WARP_ACCESS_KEY="<BACKBLAZE_APP_ID>"
  export WARP_SECRET_KEY="<BACKBLAZE_APP_KEY>"

  $# Finally Benchmark #Choose a unique bucket_name. (Decrease duration if concerned about bucket size)
  warp mixed --bucket="warp-<UNIQUE_BUCKET_NAME" --duration="5m" --analyze.v --tls

  ❯ #RESULTS (from a server closest to the bucket region)
  ----------------------------------------
  Operation: DELETE - total: 1649, 10.0%, Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.313 +0000 UTC
   * Throughput:5.63 obj/s

  Requests considered: 1650:
   * Avg: 50ms, 50%: 39ms, 90%: 80ms, 99%: 159ms, Fastest: 24ms, Slowest: 1.176s, StdDev: 40ms

  ----------------------------------------
  Operation: GET - total: 7438, 45.0%, Size: 10485760 bytes. Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.282 +0000 UTC
   * Throughput:253.01 MiB/s, 25.30 obj/s

  Requests considered: 7439:
   * Avg: 348ms, 50%: 282ms, 90%: 660ms, 99%: 1.227s, Fastest: 80ms, Slowest: 3.076s, StdDev: 248ms
   * TTFB:Avg: 175ms, Best: 20ms, 25th: 60ms, Median: 130ms, 75th: 223ms, 90th: 376ms, 99th: 770ms, Worst: 1.515s StdDev: 162ms
   * First Access: Avg: 423ms, 50%: 351ms, 90%: 753ms, 99%: 1.357s, Fastest: 115ms, Slowest: 1.831s, StdDev: 249ms
   * First Access TTFB: Avg: 231ms, Best: 23ms, 25th: 122ms, Median: 181ms, 75th: 283ms, 90th: 446ms, 99th: 873ms, Worst: 1.515s StdDev: 173ms
   * Last Access: Avg: 363ms, 50%: 299ms, 90%: 679ms, 99%: 1.297s, Fastest: 81ms, Slowest: 2.823s, StdDev: 250ms
   * Last Access TTFB: Avg: 189ms, Best: 21ms, 25th: 69ms, Median: 143ms, 75th: 235ms, 90th: 406ms, 99th: 838ms, Worst: 1.402s StdDev: 170ms

  ----------------------------------------
  Operation: PUT - total: 2474, 15.0%, Size: 10485760 bytes. Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.284 +0000 UTC
   * Throughput:84.36 MiB/s, 8.44 obj/s

  Requests considered: 2475:
   * Avg: 1.231s, 50%: 693ms, 90%: 2.652s, 99%: 4.424s, Fastest: 162ms, Slowest: 8.422s, StdDev: 1.069s

  ----------------------------------------
  Operation: STAT - total: 4966, 30.0%, Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.292 +0000 UTC
   * Throughput:16.90 obj/s

  Requests considered: 4967:
   * Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 76ms, Fastest: 19ms, Slowest: 256ms, StdDev: 12ms
   * First Access: Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 77ms, Fastest: 19ms, Slowest: 256ms, StdDev: 13ms
   * Last Access: Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 78ms, Fastest: 20ms, Slowest: 210ms, StdDev: 13ms

  Cluster Total: 337.34 MiB/s, 56.22 obj/s over 4m54s.

  ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)]
  ----------------------------------------
  Operation: DELETE - total: 137, 9.9%, Concurrency: 20, Ran 4m29s, starting 2024-03-12 21:41:26.63 +0545 +0545
   * Throughput:0.52 obj/s

  Requests considered: 138:
   * Avg: 324ms, 50%: 306ms, 90%: 343ms, 99%: 859ms, Fastest: 273ms, Slowest: 859ms, StdDev: 81ms

  ----------------------------------------
  Operation: GET - total: 614, 44.2%, Size: 10485760 bytes. Concurrency: 20, Ran 4m35s, starting 2024-03-12 21:41:22.879 +0545 +0545
   * Throughput:22.94 MiB/s, 2.29 obj/s

  Requests considered: 615:
   * Avg: 2.831s, 50%: 2.709s, 90%: 4.076s, 99%: 5.257s, Fastest: 1.375s, Slowest: 8.099s, StdDev: 911ms
   * TTFB:Avg: 704ms, Best: 265ms, 25th: 498ms, Median: 640ms, 75th: 847ms, 90th: 1.103s, 99th: 1.551s, Worst: 2.298s StdDev: 277ms
   * First Access: Avg: 2.899s, 50%: 2.755s, 90%: 4.144s, 99%: 5.533s, Fastest: 1.51s, Slowest: 8.099s, StdDev: 937ms
   * First Access TTFB: Avg: 733ms, Best: 317ms, 25th: 516ms, Median: 662ms, 75th: 881ms, 90th: 1.114s, 99th: 1.658s, Worst: 2.298s StdDev: 271ms
   * Last Access: Avg: 2.87s, 50%: 2.75s, 90%: 4.144s, 99%: 5.533s, Fastest: 1.375s, Slowest: 6.559s, StdDev: 913ms
   * Last Access TTFB: Avg: 708ms, Best: 265ms, 25th: 503ms, Median: 644ms, 75th: 857ms, 90th: 1.103s, 99th: 1.658s, Worst: 2.298s StdDev: 278ms

  ----------------------------------------
  Operation: PUT - total: 190, 13.7%, Size: 10485760 bytes. Concurrency: 20, Ran 4m30s, starting 2024-03-12 21:41:22.82 +0545 +0545
   * Throughput:7.76 MiB/s, 0.78 obj/s

  Requests considered: 191:
   * Avg: 16.034s, 50%: 14.231s, 90%: 25s, 99%: 41.969s, Fastest: 4.954s, Slowest: 45.698s, StdDev: 7.539s

  ----------------------------------------
  Operation: STAT - total: 416, 29.9%, Concurrency: 20, Ran 4m35s, starting 2024-03-12 21:41:22.735 +0545 +0545
   * Throughput:1.52 obj/s

  Requests considered: 417:
   * Avg: 300ms, 50%: 291ms, 90%: 317ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 61ms
   * First Access: Avg: 299ms, 50%: 291ms, 90%: 317ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 60ms
   * Last Access: Avg: 301ms, 50%: 291ms, 90%: 318ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 65ms

  Cluster Total: 30.36 MiB/s, 5.06 obj/s over 4m35s.

dperf

  $# Download rclone: https://github.com/rclone/rclone/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone"

  $# Download dperf: https://github.com/minio/dperf/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf"

  # Configure rclone to use backblaze as remote: https://www.backblaze.com/docs/cloud-storage-integrate-rclone-with-backblaze-b2
  $# Mount Bucket
  rclone mount "backblaze-b2:" "/tmp/mnt/backblaze-b2" \
   --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.5.3 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36" \
   --buffer-size="100M" \
   --s3-upload-concurrency="500" \
   --s3-chunk-size="100M" \
   --multi-thread-streams="500" \
   --checkers="2000" \
   --transfers="1000" \
   --vfs-cache-mode="full" \
   --vfs-read-ahead="100M" \
   --vfs-read-chunk-size-limit="off" \
   --vfs-fast-fingerprint \
   --cache-writes \
   --check-first \
   --checksum \
   --allow-other \
   --daemon

  $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs
  #DEFAULT
  ❯ dperf "/tmp/mnt/backblaze-b2"
  ┌───────────┬────────────┐
  │ TotalREAD │ TotalWRITE │
  │ 1.3 GiB/s │ 874 MiB/s  │
  └───────────┴────────────┘
  # More realistic
  ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/backblaze-b2"
  ┌───────────┬────────────┐
  │ TotalREAD │ TotalWRITE │
  │ 160 MiB/s │ 447 MiB/s  │
  └───────────┴────────────┘

CloudFlare R2

warp

  $# Download warp: https://github.com/minio/warp/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp"

  $# Setup warp
  export WARP_HOST="<CLOUDFLARE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>"
  export WARP_ACCESS_KEY="<CLOUDFLARE_ACCESS_KEY>"
  export WARP_SECRET_KEY="<CLOUDFLARE_SECRET_KEY>"

  $# Finally Benchmark #Choose a unique bucket_name. (Decrease duration if concerned about bucket size)
  warp mixed --bucket="warp-cloudflare-r2" --duration="5m" --analyze.v --tls

  ❯ #RESULTS (I ran three tests, in three diff regions, results were similar)
  Mixed operations.
  ----------------------------------------
  Operation: DELETE - total: 1137, 10.0%, Concurrency: 20, Ran 3m18s, starting 2024-03-12 14:00:58.193 +0000 UTC
   * Throughput:5.74 obj/s

  Requests considered: 1138:
   * Avg: 147ms, 50%: 137ms, 90%: 155ms, 99%: 388ms, Fastest: 120ms, Slowest: 1.184s, StdDev: 60ms

  ----------------------------------------
  Operation: GET - total: 5130, 45.0%, Size: 10485760 bytes. Concurrency: 20, Ran 3m19s, starting 2024-03-12 14:00:58.265 +0000 UTC
   * Throughput:258.32 MiB/s, 25.83 obj/s

  Requests considered: 5131:
   * Avg: 277ms, 50%: 234ms, 90%: 402ms, 99%: 915ms, Fastest: 133ms, Slowest: 7.811s, StdDev: 196ms
   * TTFB:Avg: 158ms, Best: 81ms, 25th: 104ms, Median: 123ms, 75th: 167ms, 90th: 241ms, 99th: 622ms, Worst: 2.599s StdDev: 136ms
   * First Access: Avg: 358ms, 50%: 311ms, 90%: 476ms, 99%: 1.228s, Fastest: 156ms, Slowest: 7.811s, StdDev: 262ms
   * First Access TTFB: Avg: 212ms, Best: 87ms, 25th: 147ms, Median: 170ms, 75th: 226ms, 90th: 299ms, 99th: 840ms, Worst: 2.599s StdDev: 178ms
   * Last Access: Avg: 282ms, 50%: 237ms, 90%: 412ms, 99%: 1.019s, Fastest: 137ms, Slowest: 3.291s, StdDev: 182ms
   * Last Access TTFB: Avg: 162ms, Best: 81ms, 25th: 104ms, Median: 124ms, 75th: 171ms, 90th: 247ms, 99th: 695ms, Worst: 2.599s StdDev: 145ms

  ----------------------------------------
  Operation: PUT - total: 1699, 14.9%, Size: 10485760 bytes. Concurrency: 20, Ran 3m18s, starting 2024-03-12 14:00:58.224 +0000 UTC
   * Throughput:85.84 MiB/s, 8.58 obj/s

  Requests considered: 1700:
   * Avg: 1.087s, 50%: 435ms, 90%: 612ms, 99%: 2.054s, Fastest: 293ms, Slowest: 1m40.634s, StdDev: 7.684s

  ----------------------------------------
  Operation: STAT - total: 3413, 30.0%, Concurrency: 20, Ran 3m19s, starting 2024-03-12 14:00:58.202 +0000 UTC
   * Throughput:17.17 obj/s

  Requests considered: 3414:
   * Avg: 97ms, 50%: 89ms, 90%: 104ms, 99%: 368ms, Fastest: 74ms, Slowest: 1.178s, StdDev: 57ms
   * First Access: Avg: 98ms, 50%: 88ms, 90%: 104ms, 99%: 393ms, Fastest: 75ms, Slowest: 1.178s, StdDev: 57ms
   * Last Access: Avg: 98ms, 50%: 89ms, 90%: 104ms, 99%: 346ms, Fastest: 75ms, Slowest: 1.178s, StdDev: 63ms

  Cluster Total: 343.85 MiB/s, 57.29 obj/s over 3m19s.

  ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)]
  ----------------------------------------
  Operation: DELETE - total: 103, 9.8%, Concurrency: 20, Ran 3m7s, starting 2024-03-12 20:21:28.219 +0545 +0545
   * Throughput:0.57 obj/s

  Requests considered: 104:
   * Avg: 586ms, 50%: 523ms, 90%: 873ms, 99%: 1.085s, Fastest: 468ms, Slowest: 1.085s, StdDev: 148ms

  ----------------------------------------
  Operation: GET - total: 458, 43.6%, Size: 10485760 bytes. Concurrency: 20, Ran 3m8s, starting 2024-03-12 20:21:28.244 +0545 +0545
   * Throughput:25.11 MiB/s, 2.51 obj/s

  Requests considered: 459:
   * Avg: 5.132s, 50%: 3.028s, 90%: 11.056s, 99%: 46.666s, Fastest: 920ms, Slowest: 1m11.835s, StdDev: 7.38s
   * TTFB:Avg: 1.316s, Best: 420ms, 25th: 833ms, Median: 1.102s, 75th: 1.693s, 90th: 2.098s, 99th: 4.926s, Worst: 8.45s StdDev: 880ms
   * First Access: Avg: 5.89s, 50%: 3.524s, 90%: 14.287s, 99%: 51.437s, Fastest: 1.087s, Slowest: 1m11.835s, StdDev: 7.981s
   * First Access TTFB: Avg: 1.466s, Best: 442ms, 25th: 943ms, Median: 1.207s, 75th: 1.872s, 90th: 2.131s, 99th: 5.396s, Worst: 8.45s StdDev: 881ms
   * Last Access: Avg: 5.231s, 50%: 3.02s, 90%: 12.152s, 99%: 46.666s, Fastest: 920ms, Slowest: 1m11.835s, StdDev: 7.488s
   * Last Access TTFB: Avg: 1.32s, Best: 420ms, 25th: 843ms, Median: 1.102s, 75th: 1.697s, 90th: 2.109s, 99th: 4.926s, Worst: 8.45s StdDev: 885ms

  ----------------------------------------
  Operation: PUT - total: 152, 14.5%, Size: 10485760 bytes. Concurrency: 20, Ran 3m5s, starting 2024-03-12 20:21:31.086 +0545 +0545
   * Throughput:8.54 MiB/s, 0.85 obj/s

  Requests considered: 153:
   * Avg: 4.849s, 50%: 4.295s, 90%: 7.41s, 99%: 15.38s, Fastest: 2.109s, Slowest: 16.705s, StdDev: 2.19s

  ----------------------------------------
  Operation: STAT - total: 322, 30.6%, Concurrency: 20, Ran 3m9s, starting 2024-03-12 20:21:27.924 +0545 +0545
   * Throughput:1.73 obj/s

  Requests considered: 323:
   * Avg: 541ms, 50%: 474ms, 90%: 710ms, 99%: 1.526s, Fastest: 403ms, Slowest: 3.764s, StdDev: 240ms
   * First Access: Avg: 542ms, 50%: 476ms, 90%: 737ms, 99%: 1.526s, Fastest: 403ms, Slowest: 3.764s, StdDev: 249ms
   * Last Access: Avg: 543ms, 50%: 476ms, 90%: 705ms, 99%: 1.526s, Fastest: 411ms, Slowest: 3.764s, StdDev: 248ms

  Cluster Total: 33.18 MiB/s, 5.60 obj/s over 3m8s.

dperf

  $# Download rclone: https://github.com/rclone/rclone/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone"

  $# Download dperf: https://github.com/minio/dperf/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf"

  # Configure rclone to use cloudflare as remote: https://developers.cloudflare.com/r2/examples/rclone/
  $# Mount Bucket
  # Same mnt cmd as above

  $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs
  #DEFAULT 
  ❯ dperf "/tmp/mnt/cloudflare-r2"
  ┌───────────┬────────────┐
  │ TotalREAD │ TotalWRITE │
  │ 1.5 GiB/s │ 1.0 GiB/s  │
  └───────────┴────────────┘

  # More realistic
  ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/cloudflare-r2"
  ┌───────────┬────────────┐
  │ TotalREAD │ TotalWRITE │
  │ 88 MiB/s  │ 440 MiB/s  │
  └───────────┴────────────┘

IDrive® (e2)

warp

  $# Download warp: https://github.com/minio/warp/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp"

  $# Setup warp
  export WARP_HOST="<IDRIVE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>"
  export WARP_ACCESS_KEY="<IDRIVE_ACCESS_KEY>"
  export WARP_SECRET_KEY="<IDRIVE_SECRET_KEY>"

  $# Finally Benchmark #idrive has unique creds per bucket, here bucket means a dir
  warp mixed --bucket="warp-idrive-r2" --duration="5m" --analyze.v --tls

  ❯ #RESULTS (from a server closest to the bucket region)
  ----------------------------------------
  Operation: DELETE - total: 1596, 10.0%, Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.77 +0000 UTC
   * Throughput:5.39 obj/s

  Requests considered: 1597:
   * Avg: 27ms, 50%: 28ms, 90%: 30ms, 99%: 35ms, Fastest: 20ms, Slowest: 48ms, StdDev: 3ms

  ----------------------------------------
  Operation: GET - total: 7166, 44.9%, Size: 10485760 bytes. Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.863 +0000 UTC
   * Throughput:241.96 MiB/s, 24.20 obj/s

  Requests considered: 7167:
   * Avg: 590ms, 50%: 553ms, 90%: 885ms, 99%: 1.34s, Fastest: 128ms, Slowest: 3.695s, StdDev: 239ms
   * TTFB:Avg: 83ms, Best: 25ms, 25th: 69ms, Median: 79ms, 75th: 92ms, 90th: 108ms, 99th: 167ms, Worst: 465ms StdDev: 24ms
   * First Access: Avg: 589ms, 50%: 544ms, 90%: 892ms, 99%: 1.407s, Fastest: 128ms, Slowest: 2.019s, StdDev: 244ms
   * First Access TTFB: Avg: 82ms, Best: 25ms, 25th: 69ms, Median: 78ms, 75th: 91ms, 90th: 107ms, 99th: 157ms, Worst: 465ms StdDev: 25ms
   * Last Access: Avg: 588ms, 50%: 553ms, 90%: 873ms, 99%: 1.315s, Fastest: 128ms, Slowest: 3.695s, StdDev: 236ms
   * Last Access TTFB: Avg: 83ms, Best: 30ms, 25th: 70ms, Median: 79ms, 75th: 92ms, 90th: 108ms, 99th: 161ms, Worst: 309ms StdDev: 22ms

  ----------------------------------------
  Operation: PUT - total: 2384, 14.9%, Size: 10485760 bytes. Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.779 +0000 UTC
   * Throughput:80.70 MiB/s, 8.07 obj/s

  Requests considered: 2385:
   * Avg: 627ms, 50%: 559ms, 90%: 1.011s, 99%: 1.846s, Fastest: 145ms, Slowest: 5.552s, StdDev: 347ms

  ----------------------------------------
  Operation: STAT - total: 4793, 30.0%, Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.942 +0000 UTC
   * Throughput:16.16 obj/s

  Requests considered: 4794:
   * Avg: 30ms, 50%: 31ms, 90%: 33ms, 99%: 38ms, Fastest: 23ms, Slowest: 115ms, StdDev: 3ms
   * First Access: Avg: 30ms, 50%: 30ms, 90%: 33ms, 99%: 39ms, Fastest: 23ms, Slowest: 115ms, StdDev: 4ms
   * Last Access: Avg: 30ms, 50%: 31ms, 90%: 33ms, 99%: 38ms, Fastest: 23ms, Slowest: 56ms, StdDev: 3ms

  Cluster Total: 322.36 MiB/s, 53.77 obj/s over 4m57s.

  ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)]
  ----------------------------------------
  Operation: DELETE - total: 45, 10.1%, Concurrency: 20, Ran 3m55s, starting 2024-03-12 20:55:33.285 +0545 +0545
   * Throughput:0.20 obj/s

  Requests considered: 46:
   * Avg: 88ms, 50%: 86ms, 90%: 97ms, 99%: 102ms, Fastest: 81ms, Slowest: 102ms, StdDev: 5ms

  ----------------------------------------
  Operation: GET - total: 183, 41.2%, Size: 10485760 bytes. Concurrency: 20, Ran 4m8s, starting 2024-03-12 20:55:23.679 +0545 +0545
   * Throughput:8.57 MiB/s, 0.86 obj/s

  Requests considered: 184:
   * Avg: 19.836s, 50%: 19.91s, 90%: 27.916s, 99%: 31.819s, Fastest: 3.402s, Slowest: 35.719s, StdDev: 6.123s
   * TTFB:Avg: 161ms, Best: 89ms, 25th: 131ms, Median: 137ms, 75th: 205ms, 90th: 223ms, 99th: 284ms, Worst: 461ms StdDev: 47ms
   * First Access: Avg: 20.216s, 50%: 20.37s, 90%: 28s, 99%: 35.719s, Fastest: 3.402s, Slowest: 38.175s, StdDev: 6.231s
   * First Access TTFB: Avg: 162ms, Best: 89ms, 25th: 131ms, Median: 137ms, 75th: 206ms, 90th: 225ms, 99th: 284ms, Worst: 461ms StdDev: 47ms
   * Last Access: Avg: 20.455s, 50%: 20.757s, 90%: 28.085s, 99%: 35.719s, Fastest: 3.402s, Slowest: 38.175s, StdDev: 6.315s
   * Last Access TTFB: Avg: 160ms, Best: 89ms, 25th: 130ms, Median: 137ms, 75th: 205ms, 90th: 222ms, 99th: 284ms, Worst: 461ms StdDev: 47ms

  ----------------------------------------
  Operation: PUT - total: 65, 14.6%, Size: 10485760 bytes. Concurrency: 20, Ran 3m59s, starting 2024-03-12 20:55:24.74 +0545 +0545
   * Throughput:2.86 MiB/s, 0.29 obj/s

  Requests considered: 66:
   * Avg: 6.765s, 50%: 4.238s, 90%: 15.266s, 99%: 20.483s, Fastest: 1.919s, Slowest: 20.483s, StdDev: 5.207s

  ----------------------------------------
  Operation: STAT - total: 128, 28.8%, Concurrency: 20, Ran 4m1s, starting 2024-03-12 20:55:26.167 +0545 +0545
   * Throughput:0.54 obj/s

  Requests considered: 129:
   * Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms
   * First Access: Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms
   * Last Access: Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms

  Cluster Total: 11.01 MiB/s, 1.83 obj/s over 4m4s.

dperf

  $# Download rclone: https://github.com/rclone/rclone/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone"

  $# Download dperf: https://github.com/minio/dperf/releases/latest
  # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" 
  # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf"

  # Configure rclone to use cloudflare as remote: https://www.idrive.com/s3-storage-e2/rclone
  $# Mount Bucket
  # Same mnt cmd as above

  $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs
  #DEFAULT
  ❯ dperf "/tmp/mnt/idrive-e2"
  ┌───────────┬────────────┐
  │ TotalREAD │ TotalWRITE │
  │ 1.3 GiB/s │ 1.1 GiB/s  │
  └───────────┴────────────┘

  # More realistic
  ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/idrive-e2"
  ┌───────────┬────────────┐
  │ TotalREAD │ TotalWRITE │
  │ 87 MiB/s  │ 444 MiB/s  │
  └───────────┴────────────┘

Conclusions

BackBlaze B2
- Cost: It cost me $5 to run all these tests/benchmarks. This is to say, that even though they are a part of the bandwidth-alliance, that's only true for a pre-configured rclone, which currently doesn't work. See: https://github.com/backblaze-b2-samples/cloudflare-b2/issues/16 . So most s3-clients will still hit the bandwidth cap pretty quickly, and you will be charged at 0.01$/GB.And if you have to pay for egress anyway, BackBlaze's B2 will quickly become the most expensive provider of the three thanks to astronomically high cost of API Transactions.
- Performance: In terms of performance/speed, the benchmark seems promising. However, that will hold true only as long as your server is located near their servers. And they are so few and limited: https://www.backblaze.com/docs/cloud-storage-data-regions
IDrive® (e2)
- Cost: It cost me 2$ to run all these tests/benchmarks. This is to say, that even though they are NOT a part of the bandwidth-alliance, at least they don't charge for API Transactions.
- Performance: In terms of performance/speed, the benchmark lags behind a bit but unlike BackBlaze, you can still get higher speeds because they have more regions & server locations: https://www.idrive.com/s3-storage-e2/locations
CloudFlare R2
- Cost: It cost me 0.5$ to run all these tests/benchmarks. This is to say, that I only got charged for Storage. I didn't hit the super generous API Transactions. Bandwidth/egress is free.
- Performance: In terms of performance/speed, the benchmark stays stable, no matter what location I tested from. The difference in delta was 10-20 MiB/s at most. Compare this to Backblaze's B2 that falls off by 70-80% and the delta gets 100-150 MiB/s. Or to Idrive, who despite having more locations than BackBlaze, still can't provide stable speed because they aren't a globally distributed CDN.

TLDR
- Projects
  
  I will be sticking with CloudFlare R2 because of no bandwidth charge and reasonable pricing for API Transactions.
- Personal Media & Misc Files
  
  In the near future, I intend to host personal media, where I will likely use IDrive® (e2) because of cheap storage & no charge on API Transactions. In terms of Bandwidth/egress, IDrive® (e2) will still be cheaper as I don't think I will need to download 3X of my Storage.
- BackBlaze B2
  
  Unless they change their pricing model for API Transactions & expand to more regions, B2 is not a sane choice for any of my needs.

Slow means longer time to read|write|download|upload. Hence, my servers will have to run for longer times, incurring me additional costs. ↩
As per some forum discussion this may not actually be true. Regardless, R2+CDN already sort of achieves the same.↩
Cloudflare offers free Write Operations (Class A): 1000000 (1 Million) & Read Operations (Class B): 10000000 (10 Million) per month. After which, you are charged $4.50 / million for Class A & $0.36 / million for Class B respectively.↩
As of 2024-03-12, Mega's Object Storage is beta-only https://mega.io/objectstorage ↩
Azure Blob Storageisn't s3 compatible. While yes, things like https://github.com/gaul/s3proxy exist, it's not native and impacts performance/reliability.↩

SecLogs

SecLogs

Affordable & Performant S3 Compatible Storage Providers

Evaluating & Benchmarking BackBlaze (B2), CloudFlare (R2) & IDrive® (e2)

Table of contents

Some Background

Current Setup

The Alternatives

Cost

BackBlaze B2

IDrive® (e2)

Performance

BackBlaze B2

Conclusions

TLDR