Globally-distributed storage (beta)

​

Features

• Store Data Near Users: Data stored close to the users automatically ensuring low latency everywhere
• S3 Compatible API: Global and fast object storage with familiar AWS S3 tools, libraries, and extensions.
• Zero Egress Fees: Free data egress ensures seamless and unrestricted access to your data wherever you need it.
• Fast Small Object Retrieval: Access small objects at close to Redis speed, ensuring swift, efficient retrieval.
• Object Caching: Get the benefits of Content Delivery Network (CDN) with no work on your part.
• Public Buckets: Public buckets allows you to share data with anyone, anywhere.

Hydra offers a globally distributed S3-compatible object storage service that provides low latency anywhere in the world. Hydra enables you to quickly and easily store and access any amount of data for a wide range of use cases. Hydra automatically distributes your data close to the users, and removes the complexities of data replication, and caching. As Hydra supports the S3 API, you can use the wide range of available S3 tools, libraries, and extensions. Hydra Storage is built on top of Tigris Data infrastructure and can be configured separately if desired.

​

Single Global Endpoint

In Hydra, storage is an inherently global entity. This means that the objects within your bucket are stored in the region where the initial requests are made. To optimize performance and reduce latency, these objects are intelligently distributed to other regions based on the access patterns observed over time.

This means that you can access your bucket from any region by using a single global endpoint, fly.storage.tigris.dev.

The global endpoint provides a unified entry point for accessing your Hydra storage globally, while the dynamic distribution of objects based on access patterns, results in optimized latency, providing faster access to your data.

​

Store Data Near Users

Hydra automatically stores the data close to the users and distributes it to regions based on the request pattern. There is no configuration required to enable this feature.

What this means is, if a request comes from a user in Frankfurt, Germany, the data is stored in the closest region in EU. If a request comes from a user in New York, US, the data is stored in the closest US region.

Furthermore, if the user now starts accessing the data from a region where the data is not stored, Hydra transparently fetches the data from the region where the data is stored and caches it in the region where the user is accessing it. Eventually the data gets moved to the region where the user is accessing it.

​

Fast Small Object Retrieval

Hydra provides significantly lower latency for small objects as compared to S3. This allows you to use the same data storage for both small and large objects.

Small objects storage is optimized by using a combination of inlining and coalescing techniques. Access to small objects is further optimized by caching frequently accessed objects in a LSM-based on-disk cache.

You don’t need to do anything special to take advantage of this feature. It is enabled by default for all buckets.

​

Caching

Hydra behaves like a Content Delivery Network (CDN) with no work on your part. Unlike traditional CDNs, though, it handles dynamic data in a way that provides strong guarantees around freshness of data.

Hydra transparently caches the objects close to the user to provide low-latency access. The region chosen for caching the objects depends on the request pattern from the users. Objects stored in San Jose but requested frequently from Sydney will result in getting cached in the Sydney region. Caching is provided through a distributed global caching layer with cache nodes deployed in regions globally. This ensures that user requests can be served from the region closest to the user.

Object caching requires no configuration and is enabled by default on all buckets.

​

Cache Headers

By default, Hydra honors the cache headers set by the user when writing the object, and returns those headers as part of the response when the object is fetched. This allows the user to control the caching behavior of the object.

​

Public Buckets

Hydrasets default cache header for public buckets for static assets if no cache headers are provided by the user. The default cache header is set to Cache-Control: public, max-age=3600. This applies to the following static assets:

​

Caching on PUT (Eager Caching)

In addition to the default caching behavior, Hydra provides a way to eagerly cache objects on write. This is useful when you know that an object will be frequently accessed from a region different from the region where the object is written.

The AWS CLI and SDKs can be used to enable eager caching on write. The following example shows how to enable eager caching on write using the AWS CLI

aws s3api put-bucket-accelerate-configuration \
	--bucket foo-bucket \
	--accelerate-configuration Status=Enabled

​

Caching on List (Eager Caching)

Hydra also supports eager caching while listing the objects. This can be achieved by setting a header x-tigris-prefetch during list API request. This indicates Hydra to initiate automatic caching of listed objects in proximity to the list request’s region. Subsequent Get requests for these objects will then take advantage of the cache.

​

Prefetch Request Syntax

GET /?list-type=2&continuation-token=ContinuationToken&delimiter=Delimiter&encoding-type=EncodingType&fetch-owner=FetchOwner&max-keys=MaxKeys&prefix=Prefix&start-after=StartAfter HTTP/1.1
Host: bucket.fly.storage.tigris.dev
x-tigris-prefetch: true

The supported header is x-tigris-prefetch, applicable during both ListObjects/ListObjectsV2 API requests. The AWS SDKs can be used to use this feature as AWS SDKs supports passing custom headers.

​

Public Bucket

Sometimes you want to share your bucket with the world. You can do this by creating a public bucket. This will allow anyone to read the contents of your bucket. You can still control who can write to your bucket.

​

Creating a public bucket using AWS CLI​

Assuming you have the AWS CLI configured as shown in the AWS CLI guide, you can create a public bucket as follows:

aws s3api --endpoint-url https://fly.storage.tigris.dev create-bucket --bucket foo-public-bucket --acl public-read

$ aws s3api --endpoint-url https://fly.storage.tigris.dev create-bucket --bucket foo-public-bucket --acl public-read
{
    "Location": "/foo-public-bucket"
}

The key here is the --acl public-read flag. This will allow anyone to read the contents of the bucket foo-public-bucket.

​

Accessing objects in a public bucket​

Objects in a public bucket (by default) can be read without any authentication. However, only those with access to the bucket can write objects.

Let’s upload a file to our public bucket:

$ aws s3api --endpoint-url https://fly.storage.tigris.dev put-object --bucket foo-public-bucket --key bar.txt --body bar.txt
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}

Now, anyone can read this file without authentication.

​

Path-style request​

Path-style URLs use the following format:

https://fly.storage.tigris.dev/bucket-name/key-name

So for the object we just uploaded, the path-style URL would be:

$ wget https://fly.storage.tigris.dev/foo-public-bucket/bar.txt -O- -q
bar

​

Virtual-hosted–style request​

In a virtual-hosted–style URI, the bucket name is part of the domain name in the URL.

Virtual-hosted–style URLs use the following format:

https://bucket-name.fly.storage.tigris.dev/key-name

So for the object we just uploaded, the virtual-hosted–style URL would be:

$ wget https://foo-public-bucket.fly.storage.tigris.dev/bar.txt -O- -qbar

​

Pricing table​

​

Bucket-level S3 APIs​

Below is the list of standard Bucket-level S3 APIs and their compatibility status within Hydra.

​

Object-level S3 APIs​

Below is the list of standard Object-level S3 APIs and their compatibility status within Hydra.

​

CloudFront APIs​