hakopako

Full-stack Engineer's blog

Use goofys to mount S3 bucket on EC2 of AWS [centos6]

日本語 | English

Once you mount s3 bucket to EC2, you can use it by linux commands which you must be familiar with, such as cd, mv and cp. Actually, it helped me collecting a lot of server logs to 1 bucket.

What I use here is goofys which is written in golang. Before it became popular, s3fs was a major option and is written in C++. It seems goofys is much faster than s3fs. According to golang documents, CentOS/RHEL 5.x not supported. In the case your server is CentOS/RHEL 5.x, you must use s3fs.

Versions

  • CentOS 6.6
  • fuse 2.8.3
  • Go 1.6.2
  • goofys 0.0.5

Let's get started

install

User: root (not recommend...)

fuse

$ yum install fuse

golang

$ cd ~/
$ wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
$ tar xvf go1.6.2.linux-amd64.tar.gz
$ mv go .go
$ rm go1.6.2.linux-amd64.tar.gz
$ mkdir bin
$ mkdir gocode
$ ln -s ~/.go/bin/go ~/bin/go
$ vim .bashrc
export GOROOT=$HOME/.go   // add this line
export GOPATH=$HOME/gocode   // add this line as well

goofys

$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys

mount

export HTTP_PROXY=<if needed>
export HTTPS_PROXY=<if needed>
$ mkdir s3          // <-- mount-point
$ vim .aws/credentials
[s3]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>

$ $GOPATH/bin/goofys --region ap-northeast-1 --profile s3 <BUCKET_NAME> ~/s3

Let's see how it goes....

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       30G  8.6G   20G  31% /
tmpfs           498M     0  498M   0% /dev/shm
<BUCKET_NAME>   256P     0  256P   0% /root/s3

S3 capacity is 256P !!!!!!!!! Wow......

Tips

Check command options.
As long as I tried mount, -f (run in foreground) is good for debugging.
Other options are down below.

   --help, -h                   Print this help text and exit successfuly.
   -o [-o option -o option]     Additional system-specific mount options. Be careful!
   --dir-mode "493"             Permissions bits for directories. (default: 0755)
   --file-mode "420"            Permission bits for files (default: 0644)
   --uid "0"                    UID owner of all inodes.
   --gid "0"                    GID owner of all inodes.
   --endpoint                   The non-AWS endpoint to connect to. Possible values: http://127.0.0.1:8081/
   --region "us-west-2"         The non-AWS endpoint to connect to. Possible values: us-east-1, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, cn-north-1
   --storage-class "STANDARD"   The type of storage to use when writing objects. Possible values: REDUCED_REDUNDANCY, STANDARD (default), STANDARD_IA.
   --use-path-request           Use a path-style request instead of virtual host-style. (deprecated, always on)
   --profile                    Use a named profile from $HOME/.aws/credentials instead of "default"
   --stat-cache-ttl "1m0s"      How long to cache StatObject results and inode attributes.
   --type-cache-ttl "1m0s"      How long to cache name -> file/dir mappings in directory inodes.
   --debug_fuse                 Enable fuse-related debugging output.
   --debug_s3                   Enable S3-related debugging output.
   -f                           Run goofys in foreground.
   --version, -v                print the version