# Affordable & Performant S3 Compatible Storage Providers

* ### Some Background
    
    I run a few [projects](https://ajam.dev/projects) that need `1-2 TB` of Storage along with `5-10 TB` of [Bandwidth](https://www.backblaze.com/blog/cloud-101-data-egress-fees-explained/) per month. So I want a provider that won't cause me bankruptcy or be too slow[<sup>1</sup>](#ft1).
    
* ### Current Setup
    
    I use [Cloudflare R2](https://www.cloudflare.com/developer-platform/r2/) mostly because my website already uses Cloudflare & r2 has **free egress** along with *globally distributed buckets*[<sup>2</sup>](#ft2). They have a [pricing calculator](https://r2-calculator.cloudflare.com) here: https://r2-calculator.cloudflare.com/. With default Operations[<sup>3</sup>](#ft3), accounting only for storage, it's `~ $14.85/month/TB || ~ $1.35/month/100GB`
    

What do I actually get charged? (For an average month)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1710225706149/edc8e971-6ad8-4d22-b10d-ea8dca4af90f.png align="center")

The total comes to about `~ $57`. What's to note here is, `Class A Operations (Write/Mutate)`**are more expensive than Storage itself**.

---

### The Alternatives

* I came across this [reddit post](https://www.reddit.com/r/cloudstorage/comments/1ajmvcd/i_made_a_massive_cloud_storage_comparison_table/) on [r/cloudstorage](https://www.reddit.com/r/cloudstorage/comments/1ajmvcd/i_made_a_massive_cloud_storage_comparison_table/). Thanks to `u/CookieDelivery` who made this spreadsheet: [https://docs.google.com/spreadsheets/d/1cEd65XDW3gBHnRsJ0rbq3V\_B28mKySHiMPAZvArHiiA/edit#gid=0](https://docs.google.com/spreadsheets/d/1cEd65XDW3gBHnRsJ0rbq3V_B28mKySHiMPAZvArHiiA/edit#gid=0) , I was able to quite quickly narrow down my search.
    
    As I need [s3 api compatible](https://www.youtube.com/watch?v=gY090GEDdu8) storage, providers like [`Box`](https://www.box.com/pricing), [`DeGoo`](https://degoo.com/), [`DropBox`](https://www.dropbox.com/plans), [`Google Drive`](https://one.google.com/about/plans), [`iceDrive`](https://icedrive.net/plans), [`Jottacloud`](https://jottacloud.com/en/), [`Koofr.eu`](https://koofr.eu/pricing/), [`MediaFire`](https://www.mediafire.com/upgrade/), [`Mega`](https://mega.io/pricing)[<sup>4</sup>](#ft4), [`OneDrive`](https://www.microsoft.com/en-us/microsoft-365/onedrive/compare-onedrive-plans), [`pCloud`](https://www.pcloud.com/cloud-storage-pricing-plans.html?period=lifetime), [`Proton Drive`](https://proton.me/drive/pricing), [`Sync.com`](https://www.sync.com/pricing/), [`TeraBox`](https://www.terabox.com/cloud-storage-pricing-plans), [`Zoho`](https://www.zoho.com/workdrive/pricing.html) either didn't provide s3-api compatibility, or were too expensive. [`Azure Blob Storage`](https://azure.microsoft.com/en-us/products/storage/blobs) isn't s3[<sup>5</sup>](#ft5). [`Hetzner Storage Box`](https://www.hetzner.com/storage/storage-box/) auto [blocks signups from third-world countries](https://www.reddit.com/r/hetzner/search/?q=disabled&type=link&sort=new). , [`STORJ`](https://www.storj.io/pricing) is [decentralized](https://randomoracle.wordpress.com/2019/12/07/filecoin-storj-and-the-problem-with-decentralized-storage/). [`Wasabi`](https://wasabi.com/cloud-storage-pricing/#three-info) has a [90 days retention policy](https://wasabi.com/paygo-pricing-faq/#minimum-storage-duration). [`Amazon S3`](https://aws.amazon.com/s3/pricing/), [`Digital Ocean`](https://docs.digitalocean.com/products/spaces/details/pricing/), [`Google Cloud Storage`](https://cloud.google.com/storage/pricing), [`IBM Cloud Object Storage`](https://cloud.ibm.com/objectstorage/create#pricing), [`Oracle Object Storage`](https://www.oracle.com/cloud/storage/object-storage/)**are too expensive**. This leaves me with [BackBlaze (B2)](https://www.backblaze.com/cloud-storage/pricing), [CloudFlare (R2)](https://www.cloudflare.com/developer-platform/r2/) & [IDrive® (e2)](https://www.idrive.com/s3-storage-e2/).
    

---

### Cost

#### [BackBlaze B2](https://www.backblaze.com/cloud-storage/pricing)

* **Free** Operations: [https://www.backblaze.com/cloud-storage/transaction-pricing](https://www.backblaze.com/cloud-storage/transaction-pricing)
    
    **Free** Bandwidth: [https://www.cloudflare.com/partners/technology-partners/backblaze/](https://www.cloudflare.com/partners/technology-partners/backblaze/)
    
* ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1710231522059/6cb0ef8e-4fbe-4006-a848-999161b2d220.png align="center")
    
    Using numbers from my monthly average usage:
    
* ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1710232196903/a6d8ef8e-ba66-465c-8b5f-3bafa1982b43.png align="center")
    
    The total comes to about `~ $58`. A mere `1$` more than `Cloudflare R2`. Even though the `API Transaction` fees are much, much higher, BackBlaze offers much cheaper storage. It's `~ $6/month/TB || ~ $0.6/month/100GB` compared to Cloudflare's `~ $14.85/month/TB || ~ $1.35/month/100GB`
    

#### [IDrive® (e2)](https://www.idrive.com/s3-storage-e2/)

* The best thing about `IDrive® (e2)` is that there's [no charge for API Transactions](https://www.idrive.com/s3-storage-e2/e2-transactions-price). So only storage pricing exists. It's `~ $4/month/TB || ~ $0.4/month/100GB`. However, they [charge for egress bandwidth once it exceeds 3X of $STORAGE.](https://www.idrive.com/s3-storage-e2/faq-account-management#egress-policy) They also aren't a part of [bandwidth-alliance](https://www.cloudflare.com/bandwidth-alliance/).
    
* Using numbers from my monthly average usage:
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1710234061029/a14dfa36-7a9e-4093-a468-ba685b06d21c.png align="center")
    
    The total comes to about `~ $38`. Mostly due to bandwidth, a factor I can't easily control. If the Bandwidth `($0.01/GB)` were to get `~ 10TB`, I would pay `~ 60$ (After subtracting 4140GB)` for bandwidth alone. Other than that, `IDrive® (e2)` seems to be **the cheapest option**.
    

---

### Performance

<details data-node-type="hn-details-summary"><summary>Disclaimer</summary><div data-type="detailsContent">This is in no way scientific, or foolproof. And tbh, I don't really care about a thorough statistically useful test anyway. I only want to test these for my use case, so I can decide what's the best in terms of performance &amp; cost. You are free to disagree or call this entire blog as inaccurate &amp; biased jargon. I am basing this section on: <a target="_blank" rel="noopener noreferrer nofollow" href="https://hackernoon.com/benchmarking-minio-with-warp-and-speedtest" style="pointer-events: none">https://hackernoon.com/benchmarking-minio-with-warp-and-speedtest</a></div></details><details data-node-type="hn-details-summary"><summary>Servers</summary><div data-type="detailsContent">Tests were performed on 6 Servers. <code>Server-AWS-oregon</code> was of type <a target="_blank" rel="noopener noreferrer nofollow" href="https://aws.amazon.com/ec2/instance-types/t4/" style="pointer-events: none"><code>t4g.small</code></a> hosted on <code>Amazon</code> in <code>Oregon, USA</code>. <code>Server-DGO-nyc</code> was of type <code>DO-Regular (2) @ 2.2GHz</code> hosted on <code>Digital Ocean</code> in <code>Newyork, USA</code>. <code>Server-azure-Chicago</code> was of type<a target="_blank" rel="noopener noreferrer nofollow" href="https://learn.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series" style="pointer-events: none"><code>Standard_D4as_v5</code></a>hosted on<code>Azure</code> in <code>Chicago, USA</code>. <code>Server-gcp-iowa</code> was of type <a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.google.com/compute/docs/instances/arm-on-compute" style="pointer-events: none"><code>t2a-large</code></a> hosted on <code>Google</code> in <code>Oregon, USA</code>. <code>Server-contabo-oregon</code> was of type <a target="_blank" rel="noopener noreferrer nofollow" href="https://contabo.com/en/vps/" style="pointer-events: none"><code>Cloud VPS 1</code></a> hosted on<code>Contabo</code> in <code>Düsseldorf,Germany. </code>Finally, my local machine using <a target="_blank" rel="noopener noreferrer nofollow" href="https://worldlink.com.np/internet-plans/residential-broadband/" style="pointer-events: none"><code>WorldLink`s 300Mbps Down 150Mbps Up</code></a> located in <code>Kathmandu,Nepal</code>. At the time of benchmarking, I will pick the server that's closest to the bucket region. If for some reason, I have to run the same test on multiple servers, then I will pick the fastest result, without averaging.</div></details><details data-node-type="hn-details-summary"><summary>Methodology</summary><div data-type="detailsContent"><a target="_blank" rel="noopener noreferrer nofollow" href="https://min.io/" style="pointer-events: none">MinIO</a> offers a <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/orgs/minio/repositories" style="pointer-events: none">set of tools</a> that can be used for these tests. There may be better, more accurate tools/ways/techniques for this, but I have decided to use: <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/minio/warp" style="pointer-events: none">warp</a> (For Detailed <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/minio/warp?tab=readme-ov-file#mixed" style="pointer-events: none">Mixed</a> Test) &amp; <a target="_blank" rel="noopener noreferrer nofollow" href="https://github.com/minio/dperf" style="pointer-events: none">dperf</a> (For <a target="_blank" rel="noopener noreferrer nofollow" href="https://rclone.org/commands/rclone_mount/" style="pointer-events: none">rClone Mounted</a> Drive Performance Test). Other than changing the bucket <code>regions</code> to be closer to <code>servers</code> running these tests, everything else (cmd/args/parameters passed to the tools) will remain the same. I will use default settings for these tools as much as possible.</div></details>

#### [BackBlaze B2](https://www.backblaze.com/cloud-storage/pricing)

* [warp](https://github.com/minio/warp?tab=readme-ov-file#mixed)
    
    ```bash
    $# Download warp: https://github.com/minio/warp/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp"
    
    $# Setup warp
    export WARP_HOST="<BACKBLAZE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>"
    export WARP_ACCESS_KEY="<BACKBLAZE_APP_ID>"
    export WARP_SECRET_KEY="<BACKBLAZE_APP_KEY>"
    
    $# Finally Benchmark #Choose a unique bucket_name. (Decrease duration if concerned about bucket size)
    warp mixed --bucket="warp-<UNIQUE_BUCKET_NAME" --duration="5m" --analyze.v --tls
    
    ❯ #RESULTS (from a server closest to the bucket region)
    ----------------------------------------
    Operation: DELETE - total: 1649, 10.0%, Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.313 +0000 UTC
     * Throughput:5.63 obj/s
    
    Requests considered: 1650:
     * Avg: 50ms, 50%: 39ms, 90%: 80ms, 99%: 159ms, Fastest: 24ms, Slowest: 1.176s, StdDev: 40ms
    
    ----------------------------------------
    Operation: GET - total: 7438, 45.0%, Size: 10485760 bytes. Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.282 +0000 UTC
     * Throughput:253.01 MiB/s, 25.30 obj/s
    
    Requests considered: 7439:
     * Avg: 348ms, 50%: 282ms, 90%: 660ms, 99%: 1.227s, Fastest: 80ms, Slowest: 3.076s, StdDev: 248ms
     * TTFB:Avg: 175ms, Best: 20ms, 25th: 60ms, Median: 130ms, 75th: 223ms, 90th: 376ms, 99th: 770ms, Worst: 1.515s StdDev: 162ms
     * First Access: Avg: 423ms, 50%: 351ms, 90%: 753ms, 99%: 1.357s, Fastest: 115ms, Slowest: 1.831s, StdDev: 249ms
     * First Access TTFB: Avg: 231ms, Best: 23ms, 25th: 122ms, Median: 181ms, 75th: 283ms, 90th: 446ms, 99th: 873ms, Worst: 1.515s StdDev: 173ms
     * Last Access: Avg: 363ms, 50%: 299ms, 90%: 679ms, 99%: 1.297s, Fastest: 81ms, Slowest: 2.823s, StdDev: 250ms
     * Last Access TTFB: Avg: 189ms, Best: 21ms, 25th: 69ms, Median: 143ms, 75th: 235ms, 90th: 406ms, 99th: 838ms, Worst: 1.402s StdDev: 170ms
    
    ----------------------------------------
    Operation: PUT - total: 2474, 15.0%, Size: 10485760 bytes. Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.284 +0000 UTC
     * Throughput:84.36 MiB/s, 8.44 obj/s
    
    Requests considered: 2475:
     * Avg: 1.231s, 50%: 693ms, 90%: 2.652s, 99%: 4.424s, Fastest: 162ms, Slowest: 8.422s, StdDev: 1.069s
    
    ----------------------------------------
    Operation: STAT - total: 4966, 30.0%, Concurrency: 20, Ran 4m54s, starting 2024-03-12 10:20:19.292 +0000 UTC
     * Throughput:16.90 obj/s
    
    Requests considered: 4967:
     * Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 76ms, Fastest: 19ms, Slowest: 256ms, StdDev: 12ms
     * First Access: Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 77ms, Fastest: 19ms, Slowest: 256ms, StdDev: 13ms
     * Last Access: Avg: 28ms, 50%: 24ms, 90%: 38ms, 99%: 78ms, Fastest: 20ms, Slowest: 210ms, StdDev: 13ms
    
    Cluster Total: 337.34 MiB/s, 56.22 obj/s over 4m54s.
    
    ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)]
    ----------------------------------------
    Operation: DELETE - total: 137, 9.9%, Concurrency: 20, Ran 4m29s, starting 2024-03-12 21:41:26.63 +0545 +0545
     * Throughput:0.52 obj/s
    
    Requests considered: 138:
     * Avg: 324ms, 50%: 306ms, 90%: 343ms, 99%: 859ms, Fastest: 273ms, Slowest: 859ms, StdDev: 81ms
    
    ----------------------------------------
    Operation: GET - total: 614, 44.2%, Size: 10485760 bytes. Concurrency: 20, Ran 4m35s, starting 2024-03-12 21:41:22.879 +0545 +0545
     * Throughput:22.94 MiB/s, 2.29 obj/s
    
    Requests considered: 615:
     * Avg: 2.831s, 50%: 2.709s, 90%: 4.076s, 99%: 5.257s, Fastest: 1.375s, Slowest: 8.099s, StdDev: 911ms
     * TTFB:Avg: 704ms, Best: 265ms, 25th: 498ms, Median: 640ms, 75th: 847ms, 90th: 1.103s, 99th: 1.551s, Worst: 2.298s StdDev: 277ms
     * First Access: Avg: 2.899s, 50%: 2.755s, 90%: 4.144s, 99%: 5.533s, Fastest: 1.51s, Slowest: 8.099s, StdDev: 937ms
     * First Access TTFB: Avg: 733ms, Best: 317ms, 25th: 516ms, Median: 662ms, 75th: 881ms, 90th: 1.114s, 99th: 1.658s, Worst: 2.298s StdDev: 271ms
     * Last Access: Avg: 2.87s, 50%: 2.75s, 90%: 4.144s, 99%: 5.533s, Fastest: 1.375s, Slowest: 6.559s, StdDev: 913ms
     * Last Access TTFB: Avg: 708ms, Best: 265ms, 25th: 503ms, Median: 644ms, 75th: 857ms, 90th: 1.103s, 99th: 1.658s, Worst: 2.298s StdDev: 278ms
    
    ----------------------------------------
    Operation: PUT - total: 190, 13.7%, Size: 10485760 bytes. Concurrency: 20, Ran 4m30s, starting 2024-03-12 21:41:22.82 +0545 +0545
     * Throughput:7.76 MiB/s, 0.78 obj/s
    
    Requests considered: 191:
     * Avg: 16.034s, 50%: 14.231s, 90%: 25s, 99%: 41.969s, Fastest: 4.954s, Slowest: 45.698s, StdDev: 7.539s
    
    ----------------------------------------
    Operation: STAT - total: 416, 29.9%, Concurrency: 20, Ran 4m35s, starting 2024-03-12 21:41:22.735 +0545 +0545
     * Throughput:1.52 obj/s
    
    Requests considered: 417:
     * Avg: 300ms, 50%: 291ms, 90%: 317ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 61ms
     * First Access: Avg: 299ms, 50%: 291ms, 90%: 317ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 60ms
     * Last Access: Avg: 301ms, 50%: 291ms, 90%: 318ms, 99%: 806ms, Fastest: 258ms, Slowest: 853ms, StdDev: 65ms
    
    Cluster Total: 30.36 MiB/s, 5.06 obj/s over 4m35s.
    ```
    
* [dperf](https://github.com/minio/dperf)
    
    ```bash
    $# Download rclone: https://github.com/rclone/rclone/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone"
    
    $# Download dperf: https://github.com/minio/dperf/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf"
    
    # Configure rclone to use backblaze as remote: https://www.backblaze.com/docs/cloud-storage-integrate-rclone-with-backblaze-b2
    $# Mount Bucket
    rclone mount "backblaze-b2:" "/tmp/mnt/backblaze-b2" \
     --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.5.3 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36" \
     --buffer-size="100M" \
     --s3-upload-concurrency="500" \
     --s3-chunk-size="100M" \
     --multi-thread-streams="500" \
     --checkers="2000" \
     --transfers="1000" \
     --vfs-cache-mode="full" \
     --vfs-read-ahead="100M" \
     --vfs-read-chunk-size-limit="off" \
     --vfs-fast-fingerprint \
     --cache-writes \
     --check-first \
     --checksum \
     --allow-other \
     --daemon
    
    $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs
    #DEFAULT
    ❯ dperf "/tmp/mnt/backblaze-b2"
    ┌───────────┬────────────┐
    │ TotalREAD │ TotalWRITE │
    │ 1.3 GiB/s │ 874 MiB/s  │
    └───────────┴────────────┘
    # More realistic
    ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/backblaze-b2"
    ┌───────────┬────────────┐
    │ TotalREAD │ TotalWRITE │
    │ 160 MiB/s │ 447 MiB/s  │
    └───────────┴────────────┘
    ```
    

[CloudFlare R2](https://www.cloudflare.com/developer-platform/r2/)

* [warp](https://github.com/minio/warp?tab=readme-ov-file#mixed)
    
    ```bash
    $# Download warp: https://github.com/minio/warp/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp"
    
    $# Setup warp
    export WARP_HOST="<CLOUDFLARE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>"
    export WARP_ACCESS_KEY="<CLOUDFLARE_ACCESS_KEY>"
    export WARP_SECRET_KEY="<CLOUDFLARE_SECRET_KEY>"
    
    $# Finally Benchmark #Choose a unique bucket_name. (Decrease duration if concerned about bucket size)
    warp mixed --bucket="warp-cloudflare-r2" --duration="5m" --analyze.v --tls
    
    ❯ #RESULTS (I ran three tests, in three diff regions, results were similar)
    Mixed operations.
    ----------------------------------------
    Operation: DELETE - total: 1137, 10.0%, Concurrency: 20, Ran 3m18s, starting 2024-03-12 14:00:58.193 +0000 UTC
     * Throughput:5.74 obj/s
    
    Requests considered: 1138:
     * Avg: 147ms, 50%: 137ms, 90%: 155ms, 99%: 388ms, Fastest: 120ms, Slowest: 1.184s, StdDev: 60ms
    
    ----------------------------------------
    Operation: GET - total: 5130, 45.0%, Size: 10485760 bytes. Concurrency: 20, Ran 3m19s, starting 2024-03-12 14:00:58.265 +0000 UTC
     * Throughput:258.32 MiB/s, 25.83 obj/s
    
    Requests considered: 5131:
     * Avg: 277ms, 50%: 234ms, 90%: 402ms, 99%: 915ms, Fastest: 133ms, Slowest: 7.811s, StdDev: 196ms
     * TTFB:Avg: 158ms, Best: 81ms, 25th: 104ms, Median: 123ms, 75th: 167ms, 90th: 241ms, 99th: 622ms, Worst: 2.599s StdDev: 136ms
     * First Access: Avg: 358ms, 50%: 311ms, 90%: 476ms, 99%: 1.228s, Fastest: 156ms, Slowest: 7.811s, StdDev: 262ms
     * First Access TTFB: Avg: 212ms, Best: 87ms, 25th: 147ms, Median: 170ms, 75th: 226ms, 90th: 299ms, 99th: 840ms, Worst: 2.599s StdDev: 178ms
     * Last Access: Avg: 282ms, 50%: 237ms, 90%: 412ms, 99%: 1.019s, Fastest: 137ms, Slowest: 3.291s, StdDev: 182ms
     * Last Access TTFB: Avg: 162ms, Best: 81ms, 25th: 104ms, Median: 124ms, 75th: 171ms, 90th: 247ms, 99th: 695ms, Worst: 2.599s StdDev: 145ms
    
    ----------------------------------------
    Operation: PUT - total: 1699, 14.9%, Size: 10485760 bytes. Concurrency: 20, Ran 3m18s, starting 2024-03-12 14:00:58.224 +0000 UTC
     * Throughput:85.84 MiB/s, 8.58 obj/s
    
    Requests considered: 1700:
     * Avg: 1.087s, 50%: 435ms, 90%: 612ms, 99%: 2.054s, Fastest: 293ms, Slowest: 1m40.634s, StdDev: 7.684s
    
    ----------------------------------------
    Operation: STAT - total: 3413, 30.0%, Concurrency: 20, Ran 3m19s, starting 2024-03-12 14:00:58.202 +0000 UTC
     * Throughput:17.17 obj/s
    
    Requests considered: 3414:
     * Avg: 97ms, 50%: 89ms, 90%: 104ms, 99%: 368ms, Fastest: 74ms, Slowest: 1.178s, StdDev: 57ms
     * First Access: Avg: 98ms, 50%: 88ms, 90%: 104ms, 99%: 393ms, Fastest: 75ms, Slowest: 1.178s, StdDev: 57ms
     * Last Access: Avg: 98ms, 50%: 89ms, 90%: 104ms, 99%: 346ms, Fastest: 75ms, Slowest: 1.178s, StdDev: 63ms
    
    Cluster Total: 343.85 MiB/s, 57.29 obj/s over 3m19s.
    
    ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)]
    ----------------------------------------
    Operation: DELETE - total: 103, 9.8%, Concurrency: 20, Ran 3m7s, starting 2024-03-12 20:21:28.219 +0545 +0545
     * Throughput:0.57 obj/s
    
    Requests considered: 104:
     * Avg: 586ms, 50%: 523ms, 90%: 873ms, 99%: 1.085s, Fastest: 468ms, Slowest: 1.085s, StdDev: 148ms
    
    ----------------------------------------
    Operation: GET - total: 458, 43.6%, Size: 10485760 bytes. Concurrency: 20, Ran 3m8s, starting 2024-03-12 20:21:28.244 +0545 +0545
     * Throughput:25.11 MiB/s, 2.51 obj/s
    
    Requests considered: 459:
     * Avg: 5.132s, 50%: 3.028s, 90%: 11.056s, 99%: 46.666s, Fastest: 920ms, Slowest: 1m11.835s, StdDev: 7.38s
     * TTFB:Avg: 1.316s, Best: 420ms, 25th: 833ms, Median: 1.102s, 75th: 1.693s, 90th: 2.098s, 99th: 4.926s, Worst: 8.45s StdDev: 880ms
     * First Access: Avg: 5.89s, 50%: 3.524s, 90%: 14.287s, 99%: 51.437s, Fastest: 1.087s, Slowest: 1m11.835s, StdDev: 7.981s
     * First Access TTFB: Avg: 1.466s, Best: 442ms, 25th: 943ms, Median: 1.207s, 75th: 1.872s, 90th: 2.131s, 99th: 5.396s, Worst: 8.45s StdDev: 881ms
     * Last Access: Avg: 5.231s, 50%: 3.02s, 90%: 12.152s, 99%: 46.666s, Fastest: 920ms, Slowest: 1m11.835s, StdDev: 7.488s
     * Last Access TTFB: Avg: 1.32s, Best: 420ms, 25th: 843ms, Median: 1.102s, 75th: 1.697s, 90th: 2.109s, 99th: 4.926s, Worst: 8.45s StdDev: 885ms
    
    ----------------------------------------
    Operation: PUT - total: 152, 14.5%, Size: 10485760 bytes. Concurrency: 20, Ran 3m5s, starting 2024-03-12 20:21:31.086 +0545 +0545
     * Throughput:8.54 MiB/s, 0.85 obj/s
    
    Requests considered: 153:
     * Avg: 4.849s, 50%: 4.295s, 90%: 7.41s, 99%: 15.38s, Fastest: 2.109s, Slowest: 16.705s, StdDev: 2.19s
    
    ----------------------------------------
    Operation: STAT - total: 322, 30.6%, Concurrency: 20, Ran 3m9s, starting 2024-03-12 20:21:27.924 +0545 +0545
     * Throughput:1.73 obj/s
    
    Requests considered: 323:
     * Avg: 541ms, 50%: 474ms, 90%: 710ms, 99%: 1.526s, Fastest: 403ms, Slowest: 3.764s, StdDev: 240ms
     * First Access: Avg: 542ms, 50%: 476ms, 90%: 737ms, 99%: 1.526s, Fastest: 403ms, Slowest: 3.764s, StdDev: 249ms
     * Last Access: Avg: 543ms, 50%: 476ms, 90%: 705ms, 99%: 1.526s, Fastest: 411ms, Slowest: 3.764s, StdDev: 248ms
    
    Cluster Total: 33.18 MiB/s, 5.60 obj/s over 3m8s.
    ```
    
* [dperf](https://github.com/minio/dperf)
    
    ```bash
    $# Download rclone: https://github.com/rclone/rclone/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone"
    
    $# Download dperf: https://github.com/minio/dperf/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf"
    
    # Configure rclone to use cloudflare as remote: https://developers.cloudflare.com/r2/examples/rclone/
    $# Mount Bucket
    # Same mnt cmd as above
    
    $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs
    #DEFAULT 
    ❯ dperf "/tmp/mnt/cloudflare-r2"
    ┌───────────┬────────────┐
    │ TotalREAD │ TotalWRITE │
    │ 1.5 GiB/s │ 1.0 GiB/s  │
    └───────────┴────────────┘
    
    # More realistic
    ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/cloudflare-r2"
    ┌───────────┬────────────┐
    │ TotalREAD │ TotalWRITE │
    │ 88 MiB/s  │ 440 MiB/s  │
    └───────────┴────────────┘
    ```
    

[IDrive® (e2)](https://www.idrive.com/s3-storage-e2/)

* [warp](https://github.com/minio/warp?tab=readme-ov-file#mixed)
    
    ```bash
    $# Download warp: https://github.com/minio/warp/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/warp" -o "/usr/local/bin/warp" && sudo chmod +x "/usr/local/bin/warp"
    
    $# Setup warp
    export WARP_HOST="<IDRIVE_ENDPOINT_WITHOUT_HTTPS_SCHEMA>"
    export WARP_ACCESS_KEY="<IDRIVE_ACCESS_KEY>"
    export WARP_SECRET_KEY="<IDRIVE_SECRET_KEY>"
    
    $# Finally Benchmark #idrive has unique creds per bucket, here bucket means a dir
    warp mixed --bucket="warp-idrive-r2" --duration="5m" --analyze.v --tls
    
    ❯ #RESULTS (from a server closest to the bucket region)
    ----------------------------------------
    Operation: DELETE - total: 1596, 10.0%, Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.77 +0000 UTC
     * Throughput:5.39 obj/s
    
    Requests considered: 1597:
     * Avg: 27ms, 50%: 28ms, 90%: 30ms, 99%: 35ms, Fastest: 20ms, Slowest: 48ms, StdDev: 3ms
    
    ----------------------------------------
    Operation: GET - total: 7166, 44.9%, Size: 10485760 bytes. Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.863 +0000 UTC
     * Throughput:241.96 MiB/s, 24.20 obj/s
    
    Requests considered: 7167:
     * Avg: 590ms, 50%: 553ms, 90%: 885ms, 99%: 1.34s, Fastest: 128ms, Slowest: 3.695s, StdDev: 239ms
     * TTFB:Avg: 83ms, Best: 25ms, 25th: 69ms, Median: 79ms, 75th: 92ms, 90th: 108ms, 99th: 167ms, Worst: 465ms StdDev: 24ms
     * First Access: Avg: 589ms, 50%: 544ms, 90%: 892ms, 99%: 1.407s, Fastest: 128ms, Slowest: 2.019s, StdDev: 244ms
     * First Access TTFB: Avg: 82ms, Best: 25ms, 25th: 69ms, Median: 78ms, 75th: 91ms, 90th: 107ms, 99th: 157ms, Worst: 465ms StdDev: 25ms
     * Last Access: Avg: 588ms, 50%: 553ms, 90%: 873ms, 99%: 1.315s, Fastest: 128ms, Slowest: 3.695s, StdDev: 236ms
     * Last Access TTFB: Avg: 83ms, Best: 30ms, 25th: 70ms, Median: 79ms, 75th: 92ms, 90th: 108ms, 99th: 161ms, Worst: 309ms StdDev: 22ms
    
    ----------------------------------------
    Operation: PUT - total: 2384, 14.9%, Size: 10485760 bytes. Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.779 +0000 UTC
     * Throughput:80.70 MiB/s, 8.07 obj/s
    
    Requests considered: 2385:
     * Avg: 627ms, 50%: 559ms, 90%: 1.011s, 99%: 1.846s, Fastest: 145ms, Slowest: 5.552s, StdDev: 347ms
    
    ----------------------------------------
    Operation: STAT - total: 4793, 30.0%, Concurrency: 20, Ran 4m56s, starting 2024-03-12 14:20:32.942 +0000 UTC
     * Throughput:16.16 obj/s
    
    Requests considered: 4794:
     * Avg: 30ms, 50%: 31ms, 90%: 33ms, 99%: 38ms, Fastest: 23ms, Slowest: 115ms, StdDev: 3ms
     * First Access: Avg: 30ms, 50%: 30ms, 90%: 33ms, 99%: 39ms, Fastest: 23ms, Slowest: 115ms, StdDev: 4ms
     * Last Access: Avg: 30ms, 50%: 31ms, 90%: 33ms, 99%: 38ms, Fastest: 23ms, Slowest: 56ms, StdDev: 3ms
    
    Cluster Total: 322.36 MiB/s, 53.77 obj/s over 4m57s.
    
    ❯ #RESULTS (from my Home region (Kathmandu,Nepal)) [MAX DOWNLOAD: 300Mbps(35 MiB/s) || UPLOAD 150Mbps (17-18 MiB/s)]
    ----------------------------------------
    Operation: DELETE - total: 45, 10.1%, Concurrency: 20, Ran 3m55s, starting 2024-03-12 20:55:33.285 +0545 +0545
     * Throughput:0.20 obj/s
    
    Requests considered: 46:
     * Avg: 88ms, 50%: 86ms, 90%: 97ms, 99%: 102ms, Fastest: 81ms, Slowest: 102ms, StdDev: 5ms
    
    ----------------------------------------
    Operation: GET - total: 183, 41.2%, Size: 10485760 bytes. Concurrency: 20, Ran 4m8s, starting 2024-03-12 20:55:23.679 +0545 +0545
     * Throughput:8.57 MiB/s, 0.86 obj/s
    
    Requests considered: 184:
     * Avg: 19.836s, 50%: 19.91s, 90%: 27.916s, 99%: 31.819s, Fastest: 3.402s, Slowest: 35.719s, StdDev: 6.123s
     * TTFB:Avg: 161ms, Best: 89ms, 25th: 131ms, Median: 137ms, 75th: 205ms, 90th: 223ms, 99th: 284ms, Worst: 461ms StdDev: 47ms
     * First Access: Avg: 20.216s, 50%: 20.37s, 90%: 28s, 99%: 35.719s, Fastest: 3.402s, Slowest: 38.175s, StdDev: 6.231s
     * First Access TTFB: Avg: 162ms, Best: 89ms, 25th: 131ms, Median: 137ms, 75th: 206ms, 90th: 225ms, 99th: 284ms, Worst: 461ms StdDev: 47ms
     * Last Access: Avg: 20.455s, 50%: 20.757s, 90%: 28.085s, 99%: 35.719s, Fastest: 3.402s, Slowest: 38.175s, StdDev: 6.315s
     * Last Access TTFB: Avg: 160ms, Best: 89ms, 25th: 130ms, Median: 137ms, 75th: 205ms, 90th: 222ms, 99th: 284ms, Worst: 461ms StdDev: 47ms
    
    ----------------------------------------
    Operation: PUT - total: 65, 14.6%, Size: 10485760 bytes. Concurrency: 20, Ran 3m59s, starting 2024-03-12 20:55:24.74 +0545 +0545
     * Throughput:2.86 MiB/s, 0.29 obj/s
    
    Requests considered: 66:
     * Avg: 6.765s, 50%: 4.238s, 90%: 15.266s, 99%: 20.483s, Fastest: 1.919s, Slowest: 20.483s, StdDev: 5.207s
    
    ----------------------------------------
    Operation: STAT - total: 128, 28.8%, Concurrency: 20, Ran 4m1s, starting 2024-03-12 20:55:26.167 +0545 +0545
     * Throughput:0.54 obj/s
    
    Requests considered: 129:
     * Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms
     * First Access: Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms
     * Last Access: Avg: 94ms, 50%: 90ms, 90%: 100ms, 99%: 380ms, Fastest: 83ms, Slowest: 380ms, StdDev: 26ms
    
    Cluster Total: 11.01 MiB/s, 1.83 obj/s over 4m4s.
    ```
    
* [dperf](https://github.com/minio/dperf)
    
    ```bash
    $# Download rclone: https://github.com/rclone/rclone/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/rclone" -o "/usr/local/bin/rclone" && sudo chmod +x "/usr/local/bin/rclone"
    
    $# Download dperf: https://github.com/minio/dperf/releases/latest
    # arm64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/aarch64_arm64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf" 
    # x86-64 Linux : sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/dperf" -o "/usr/local/bin/dperf" && sudo chmod +x "/usr/local/bin/dperf"
    
    # Configure rclone to use cloudflare as remote: https://www.idrive.com/s3-storage-e2/rclone
    $# Mount Bucket
    # Same mnt cmd as above
    
    $# Finally Benchmark #This auto creates a unique bucket, might have to delete to not incur costs
    #DEFAULT
    ❯ dperf "/tmp/mnt/idrive-e2"
    ┌───────────┬────────────┐
    │ TotalREAD │ TotalWRITE │
    │ 1.3 GiB/s │ 1.1 GiB/s  │
    └───────────┴────────────┘
    
    # More realistic
    ❯ dperf --filesize "5GiB" --blocksize "50MiB" "/tmp/mnt/idrive-e2"
    ┌───────────┬────────────┐
    │ TotalREAD │ TotalWRITE │
    │ 87 MiB/s  │ 444 MiB/s  │
    └───────────┴────────────┘
    ```
    

---

### Conclusions

* [BackBlaze B2](https://www.backblaze.com/cloud-storage/pricing)
    
    * **Cost**: It cost me `$5` to run all these tests/benchmarks. This is to say, that even though they are a part of the [bandwidth-alliance](https://www.cloudflare.com/bandwidth-alliance/), that's only true for a pre-configured rclone, which currently doesn't work. See: [https://github.com/backblaze-b2-samples/cloudflare-b2/issues/16](https://github.com/backblaze-b2-samples/cloudflare-b2/issues/16) . So most s3-clients will still hit the bandwidth cap pretty quickly, and you will be charged at `0.01$/GB.`And if you have to pay for egress anyway, BackBlaze's B2 will quickly become the **most expensive** provider of the three thanks to astronomically high cost of [`API Transactions`](https://www.backblaze.com/cloud-storage/transaction-pricing).
        
    * **Performance**: In terms of performance/speed, the benchmark seems promising. However, that will hold true only as long as your server is located near their servers. And they are so few and limited: [https://www.backblaze.com/docs/cloud-storage-data-regions](https://www.backblaze.com/docs/cloud-storage-data-regions)
        
* [IDrive® (e2)](https://www.idrive.com/s3-storage-e2/)
    
    * **Cost:** It cost me `2$` to run all these tests/benchmarks. This is to say, that even though they are NOT a part of the [bandwidth-alliance](https://www.cloudflare.com/bandwidth-alliance/), at least they don't charge for `API Transactions`.
        
    * **Performance**: In terms of performance/speed, the benchmark lags behind a bit but unlike BackBlaze, you can still get higher speeds because they have more regions & server locations: [https://www.idrive.com/s3-storage-e2/locations](https://www.idrive.com/s3-storage-e2/locations)
        
* [CloudFlare R2](https://www.cloudflare.com/developer-platform/r2/)
    
    * **Cost:** It cost me `0.5$` to run all these tests/benchmarks. This is to say, that I only got charged for Storage. I didn't hit the super generous [`API Transactions`](https://developers.cloudflare.com/r2/pricing/#data-storage)`.` Bandwidth/egress is free.
        
    * **Performance**: In terms of performance/speed, the benchmark stays stable, no matter what location I tested from. The difference in delta was `10-20 MiB/s` at most. Compare this to ***Backblaze's B2*** that falls off by `70-80%` and the delta gets `100-150 MiB/s.` Or to ***Idrive***, who despite having more locations than ***BackBlaze***, still can't provide stable speed because [they aren't a globally distributed CDN](https://www.cloudflare.com/network/).
        

---

* ### TLDR
    
    * [**Projects**](https://ajam.dev/projects)
        
        I will be sticking with [CloudFlare R2](https://www.cloudflare.com/developer-platform/r2/) because of no bandwidth charge and reasonable pricing for [`API Transactions.`](https://developers.cloudflare.com/r2/pricing/#data-storage)
        
    * **Personal Media & Misc Files**
        
        In the near future, I intend to host personal media, where I will likely use [IDrive® (e2)](https://www.idrive.com/s3-storage-e2/) because of cheap storage & no charge on `API Transactions`. In terms of Bandwidth/egress, [IDrive® (e2)](https://www.idrive.com/s3-storage-e2/) will still be cheaper as I don't think I will need to download 3X of my Storage.
        
    * [BackBlaze B2](https://www.backblaze.com/cloud-storage/pricing)
        
        Unless they change their pricing model for [`API Transactions`](https://www.backblaze.com/cloud-storage/transaction-pricing) & expand to more regions, B2 is not a sane choice for any of my needs.
        
    

---

1. <span id="ft1"></span>Slow means longer time to `read|write|download|upload`. Hence, my servers will have to run for longer times, incurring me additional costs. [↩](#1)
    
2. <span id="ft2"></span>As per some [forum discussion](https://community.cloudflare.com/t/is-cloudflare-r2-actually-distributing-objects/451909) this may not actually be true. Regardless, `R2+CDN` already sort of achieves the same.[↩](#2)
    
3. <span id="ft3"></span>Cloudflare offers free `Write Operations (Class A): 1000000 (1 Million)` & `Read Operations (Class B): 10000000 (10 Million)` per month. After which, you are charged `$4.50 / million` for Class A & `$0.36 / million` for Class B respectively.[↩](#3)
    
4. <span id="ft4"></span>As of 2024-03-12, Mega's Object Storage is beta-only [https://mega.io/objectstorage](https://mega.io/objectstorage)[↩](#4)
    
5. <span id="ft5"></span>**Azure Blob Storage**[isn't s3 compatible](https://learn.microsoft.com/en-us/answers/questions/1183760/s3-api-support-over-azure-blob-storage). While yes, things like [https://github.com/gaul/s3proxy](https://github.com/gaul/s3proxy) exist, it's not native and impacts performance/reliability.[↩](#5)
    

---