S3: A Flexible Approach to Uploading in Rails

S3, Amazon’s Simple Storage Service, is a cheap, scalable, highly available, and overall great solution for storing huge bits of data and thus reducing your database size. For a while now, we’ve realized that our database is on its path to becoming expensive and unmanageable due its massive size. So we decided to migrate some of that data over to S3.

In this article, I explain the approach we took to make any attribute S3able

We created a concern that could be included in any model and automatically upload the contents of a field to s3 on save and read from s3 on access. carrierwave gem is used to abstract API calls in addition to carrierwave-aws gem to setup AWS credentials.

module S3able
  extend ActiveSupport::Concern

  def self.included(klass)
    def klass.s3able(column_name)

      class_eval <<-RUBY, __FILE__, __LINE__ + 1
        self.mount_uploader :s3_#{column_name}, FileUploader
        def #{column_name}
          self.s3_#{column_name}&.read.to_s
        end

        def #{column_name}=(contents)
          current_time_txt_file = Time.now.to_f.to_s + ".txt"
          file = Tempfile.new(current_time_txt_file)
          file.write(contents)
          self.s3_#{column_name} = file
          super(contents)
        end
      RUBY

    end
  end
end

The code basically hijacks the setter and getter for whatever field you pass into mount :field inside your model. The getter uses the carrierwave object to retrieve and read the file. The setter creates a temp file and, again, uses carrierwave to upload the data into s3.

FileUploader used inside the S3able module just sets the store_dir which directs carrierwave on how to create the path and your file in the s3 bucket. Here is how it’s written:

class FileUploader < CarrierWave::Uploader::Base
  def store_dir
    "#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  def extension_white_list
    %w(txt)
  end
end

Now, if you drop s3able into any model like this:

class YourModel
  include S3able
  s3able :your_field
end

You turn the setter of that field into a text file uploader and the getter into a file reader. It’s that simple.

Not that this approach would only work for new fields. It can also work for existing fields if you don’t care about the data. You need to tweak the S3able concern to accommodate uploading existing data, but I hope this gives you a good starting point.