One of the great things about StorageGRID Webscale is that it offers comprehensive S3 support, which is 100% compatible with the AWS S3 SDKs. This makes application code portable, as it does not rely on proprietary object storage APIs. And that applies to the command line too. In this post, I’ll show you how to access StorageGRID’s S3 endpoint via Amazon’s S3 CLI.

Setting up the AWS S3 CLI

For accessing StorageGRID S3 non-programmatically, AWS S3 CLI is an easy tool to use. First, install the AWS CLI on either Windows, MacOS or Linux by following this guide. Next, persist the StorageGRID S3 Tenant credentials in ~/.aws/credentials on the host:

$ mkdir ~/.aws
$ vi ~/.aws/credentials

Finally, create a profile called “webscale” and copy/paste your keys:

[webscale]
aws_access_key_id = <Paste your S3 Access Key here>
aws_secret_access_key = <Paste your S3 Secret Access Key here>

Accessing S3 via the CLI

Let’s give it a test run. I am using the –no-verify-ssl option, to disable SSL certificate checking. Obviously this is not a good idea in production:

$ aws s3 ls --profile webscale --endpoint-url https://s3.mycompany.com:8082 --no-verify-ssl
/usr/local/lib/python2.7/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
2016-10-27 18:40:20 fileshareapp
2016-08-11 11:17:10 versioned-bucket
2016-11-28 15:47:32 worm-bucket

To secure the connection, you can either specify either a CA bundle or persist the certificate in the trusted store of your OS:

$ aws s3 ls --profile webscale --endpoint-url https://s3.mycompany.com:8082 --ca-bundle RootCA.pem

Specific resources are accessed via “s3://” URIs:

$ aws s3 ls s3://fileshareapp --profile webscale --endpoint-url https://s3.mycompany.com:8082

Syncing a whole local folder from disk to S3 is just one command away:

$ aws s3 sync myfolder/ s3://targetbucket --profile webscale --endpoint-url https://s3.mycompany.com:8082

Check out the full AWS S3 CLI documentation for more.

Shorter Alias

Since the mentioned commands are quite long, it’s probably a lot easier to just put a short alias in your ~/.bashrc on your host:

# Alias for SGWS
sgws_cmd() {
  CMD="aws ${@:1} --profile webscale --endpoint-url https://s3.mycompany.com:8082 --ca-bundle /path/to/RootCA.pem"
  echo "Executing \"${CMD}\""
  eval ${CMD}
}
alias sgws=sgws_cmd

Now, the usage becomes quite simple:

$ sgws s3 ls
2016-10-27 18:40:20 fileshareapp
2016-08-11 11:17:10 versioned-bucket
2016-11-28 15:47:32 worm-bucket

Doing more with s3api

Using AWS CLI via s3api instead of just S3  allows you to execute arbitrary S3 API commands, such as setting bucket policies, enabling versioning, or copying objects with new metadata directly inside StorageGRID:

$ aws s3api copy-object --bucket fileshareapp --metadata "foo=bar,cust=42" \
                        --copy-source fileshareapp/test.pdf --key test_copy.pdf \
                        --metadata-directive "REPLACE" --profile webscale \
                        --endpoint-url https://s3.mycompany.com:8082

The full list of supported s3api commands can be found here. The mentioned alias obviously also works with the “sgws” alias.

Summary

Accessing S3 via CLI is a convenient tool if you do not want to access objects via a higher-level programming language or an UI. This is particularly helpful as part of a job in a pipeline or other scheduled workflow, connectivity testing, etc.

If you have any questions please use the comments below or join us in Slack! We love hearing from you and learning about your challenges, so please don’t be bashfull!

Clemens Siebler on GithubClemens Siebler on LinkedinClemens Siebler on Twitter
Clemens Siebler
Manager Solution Architects EMEA
Clemens is leading a technical team of Solution Architects in EMEA. In his current role, he and his team are evangelizing upcoming market trends like Containers, Object Storage, OpenStack, and NFV. His current passion is enabling customers to transition their large scale workloads to Object Storage. Before, he worked as a Software Engineer on NetApp’s software products, where he published multiple patents on plug-in frameworks.