Using StorageGRID Webscale with AWS S3 CLI

One of the great things about StorageGRID Webscale is that it offers comprehensive S3 support, which is 100% compatible with the AWS S3 SDKs. This makes application code portable, as it does not rely on proprietary object storage APIs. And that applies to the command line too. In this post, I’ll show you how to access StorageGRID’s S3 endpoint via Amazon’s S3 CLI.

Setting up the AWS S3 CLI

For accessing StorageGRID S3 non-programmatically, AWS S3 CLI is an easy tool to use. First, install the AWS CLI on either Windows, MacOS or Linux by following this guide. Next, persist the StorageGRID S3 Tenant credentials in ~/.aws/credentials on the host:

Finally, create a profile called “webscale” and copy/paste your keys:

Accessing S3 via the CLI

Let’s give it a test run. I am using the –no-verify-ssl option, to disable SSL certificate checking. Obviously this is not a good idea in production:

To secure the connection, you can either specify either a CA bundle or persist the certificate in the trusted store of your OS:

Specific resources are accessed via “s3://” URIs:

Syncing a whole local folder from disk to S3 is just one command away:

Check out the full AWS S3 CLI documentation for more.

Shorter Alias

Since the mentioned commands are quite long, it’s probably a lot easier to just put a short alias in your ~/.bashrc on your host:

Now, the usage becomes quite simple:

Doing more with s3api

Using AWS CLI via s3api instead of just S3  allows you to execute arbitrary S3 API commands, such as setting bucket policies, enabling versioning, or copying objects with new metadata directly inside StorageGRID:

The full list of supported s3api commands can be found here. The mentioned alias obviously also works with the “sgws” alias.

Summary

Accessing S3 via CLI is a convenient tool if you do not want to access objects via a higher-level programming language or an UI. This is particularly helpful as part of a job in a pipeline or other scheduled workflow, connectivity testing, etc.

If you have any questions please use the comments below or join us in Slack! We love hearing from you and learning about your challenges, so please don’t be bashfull!

Clemens Siebler on GithubClemens Siebler on LinkedinClemens Siebler on Twitter
Clemens Siebler
Manager Solution Architects EMEA
Clemens is leading a technical team of Solution Architects in EMEA. In his current role, he and his team are evangelizing upcoming market trends like Containers, Object Storage, OpenStack, and NFV. His current passion is enabling customers to transition their large scale workloads to Object Storage. Before, he worked as a Software Engineer on NetApp’s software products, where he published multiple patents on plug-in frameworks.

Leave a Reply