S3/GCS to Hydra Lake
Migrating data to Hydra Lake results in a substantial improvement in performance and eliminates ingress and egress costs to your database.
If you would like to migrate your data to Hydra Lake, please contact support with your request. Migrations to S3-compatible storage are managed by the Hydra team directly and are accomplished with zero downtime.
How we accomplish a zero-downtime migration
We can configure any Hydra bucket to use a shadow bucket, which is any underlying S3-compatible bucket you would like to migrate.
This is how the process works:
- When requesting an object, it is first checked if the object exists in the Hydra bucket. If it does, it’s returned. If it doesn’t, the shadow bucket is checked and the object is returned from there. Then the object is copied to the Hydra bucket so that it is available for future requests.
- When uploading a new object to the Hydra bucket, it is first uploaded to the shadow bucket and then copied to the Hydra bucket. This ensures that the data is always available in the shadow bucket and is eventually consistent in the Hydra bucket.
- When deleting an object, it is deleted from the Hydra bucket, as well as the shadow bucket.
In summary, your Hydra bucket acts just like your original bucket and automatically copies any data fetched from that bucket.
Shadow buckets can be configured for any S3-compatible source, including Google Cloud Storage (GCS).
Configuration options
Please contact us if you would like to modify your configuration:
- Write pass-through can be disabled. If disabled, uploads and deletes will not be written to your shadow bucket, but instead only be written to Hydra.
- Once you’re confident all data has beeen migrated, we can disable the shadow bucket feature, at which point the bucket will be completely independent from the shadow bucket.