The GraphQL logo overlayed over a Japanese shrine
The GraphQL logo overlayed over a Japanese shrine

GraphQL file upload with Shrine

At the moment of writing there is no officially supported way to do file upload through GraphQL. Here is a roundup of all available methods to do file upload through it, their pros and cons.

This post grew out of a request on the Shrine issue tracker — you can find the original issue here. It's still useful for other libraries, but the code examples will only apply to Shrine.

Direct file upload

Handling uploads is a resource intensive task for the server (using up IO and bandwidth). To resolve this, we can upload the file directly from the client to our block-storage server (AWS S3 or DigitalOcean Spaces) by using a presigned URL generated on our server.

This is in contrast to the classic approach where we would first upload the client's file to our server, and then from our server to the block-storage server.

The benefit is that our server doesn't have to process the file (which uses up IO) and it doesn't have to upload the file to the block-storage server (more IO usage). It also solves scaling issues related to distributed file caches and distributed resource management.

The downside to this method is apparent when the uploaded file needs to be processed. To process a file our server needs to get the client's form submission containing the URL of the file, download the file, process it and re-upload it to the block-storage server. This uses up bandwidth and IO but it's necessary.

This method should be satisfactory for most use-cases. If your application doesn't require immediate file processing then the downloading and re-uploading can be done in the background or on a separate machine ( using an AWS lambda). On AWS you can get the best of both world using the Serverless image handler.

Shrine supports this feature out-of-the-box through the presign plugin. To get it running you'll need to add the AWS S3 SDK to your Gemfile, configure a new storage for Shrine, and expose the file presign endpoint — this is explained in detail in the documentation.

Once the client uploads the file it needs to send some kind of reference to the server — the URL of the file, or an ID, or the path of the file (I usually send the original file name as well as the URL).

mutation SetAvatar($url: String, $filename: String) {
  updateUser(
    id: "some_user_uuid"
    input: {
      avatar: {
        filename: $filename
        url: $url
      }
    }
  ) {
    avatar_url
  }
}

The server uses this reference to build a Shrine object and attach it.

# This code is license under the MIT license
# Full text: https://opensource.org/licenses/MIT
# Author Stanko K.R. <[email protected]>

# Usage:
# ```ruby
# user.avatar = S3UrlToShrineObjectConverter.call(
#   arguments[:input][:avatar][:url],
#   name: arguments[:input][:avatar][:filename]
# )

require 'uri'

class S3UrlToShrineObjectConverter
  def self.call(*args)
    new(*args).call
  end

  def initialize(url, name: nil)
    @url = url
    @name = name
  end

  def call
    return unless object&.exists?

    {
      id: id,
      storage: storage,
      metadata: {
        size: object.content_length,
        filename: name || id,
        mime_type: object.content_type
      }
    }
  end

  protected

  attr_reader :url
  attr_reader :name

  private

  def object
    @object ||= bucket.object(uri.path)
  end

  def uri
    @uri ||= URI(url)
  end

  def bucket
    @bucket ||= begin
      s3 = AWS::S3.new({}) # TODO: configure this
      s3.buckets['your_bucket_name'] # TODO: load this from the app's config
    end
  end

  def id
    @id ||= uri.path.split('/', 2).last
  end

  def storage
    @storage ||= uri.path.split('/', 2).first
  end
end

Note that you can use the upload_endpoint plugin instead of the presign_endpoint plugin. I advise against this as it enables anyone to upload any content, at any time, to your block-storage. By presiging we solve the "anyone” and "anytime” issues — we have control over who can request presigned URLs to upload or download files.

Base64 encoding

This method is the simplest, yet it's the worst for real-world applications.

Your client can encode the file's data as Base64 and set it as the value of a mutation's field.

mutation SetAvatar($data: String, $filename: String) {
  updateUser(
    id: "some_user_uuid"
    input: {
      avatar: {
        filename: $filename
        data: $data
      }
    }
  ) {
    avatar_url
  }
}

While on the surface this seems harmless, since the image has to be uploaded to the server in one way or another, this approach has many unwanted side-effects.

The biggest issue is memory consumption. Since the image is now part of your request's JSON body it will be parsed as a string, which will drive your memory consumption up. E.g. If you upload ten 5MiB images in one mutation you will have a hash that's at least 50MiB in memory. Storing whole files in memory can and will crash your application.

I advise against using this method. If you wish to implement it, you can do so using Shrine's data_uri plugin. After you add the plugin to your uploader you will need to set the <field>_data_uri of your file in the resolver.

user.avatar_data_uri = arguments[:input][:avatar][:data]
user.save

If you decide to use this method be careful to remove the image upload field from your logs, otherwise your log file will contain a copy of all uploaded images.

Multipart upload

Multipart uploads are the standard way of uploading files through HTTP. They are used by most browsers to do file uploads through forms. Since GraphQL is just a JSON request we can pass files alongside our request, but we need logic to interpret them.

I'll be referencing jaydenseric's multipart request specification as it's supported by several client and server implementations at this moment.

A typical GraphQL request consists of three fields query, variables and operationName. This spec adds a fourth — map. The map contains indices and JSON pointers, each index represents an image and each pointer represents the location of the ActionDispatch::Http::UploadedFile that holds the corresponding image.

Say we want to upload two images to a gallery using this method, our GraphQL mutation might look like this:

mutation UploadGalleryImages {
  uploadGalleryImages(
    galleryID: "some_gallery_id"
    images: [null, null]
  ) {
    id
    filename
    url
  }
}

Since the server knows that the images field is an array of File types it will match all null values in the array, in order of appearance, with their index in the map field and dereference their JSON pointer. If we were to inspect the input params hash we would see the following:

{
  galleryID: "some_gallery_id",
  images: [
    #<ActionDispatch::Http::UploadedFile:0x000055fb90c6dbd0 @tempfile=#<Tempfile:/tmp/RackMultipart20180131-31-6zkb3i.txt>, @original_filename="b.txt", @content_type="text/plain", @headers="Content-Disposition: form-data; name=\"0\"; filename=\"b.txt\"\r\nContent-Type: text/plain\r\n">,
    #<ActionDispatch::Http::UploadedFile:0x000055fb90c6da18 @tempfile=#<Tempfile:/tmp/RackMultipart20180131-31-13i0pio.txt>, @original_filename="c.txt", @content_type="text/plain", @headers="Content-Disposition: form-data; name=\"1\"; filename=\"c.txt\"\r\nContent-Type: text/plain\r\n">
  ]
}

From here on you can assign those UploadedFile objects to any of your own objects as usual. No Shrine plugins needed.

gallery = Gallery.find(arguemnts[:gallery_id])

arguments[:images].map do |file|
  # Image has a Shrine uploader attached to `file_data`
  Image.create(file: file, gallery: gallery)
end

Conclusion

My suggestion is to use the direct upload method if possible. It's the only method that doesn't require modifications to the GraphQL protocol, it's well understood, and it's used outside of the GraphQL ecosystem (better interoperability with client libraries).

If direct uploads won't work for you I would take a gamble on multipart uploads.

Base64 encoding of upload files is problematic at best and should be avoided in my opinion. Though there are limited use-cases where it's implementation speed out ways the issues it has. Though, those use-cases are rare.


Analytics & backlinks Analytics & backlinks

Other popular articles
Title image for "Do you really need WebSockets?"

Do you really need WebSockets?

Over the years I've had this conversation a couple of times. This post will explain why we use WebSockets, how they can be used, what alternatives exist and when to use them. Why WebSockets? Every time I worked on a project where we had to implement any kind of a "real-time" component, usuall...

Continue reading ...

Title image for "RabbitMQ is more than a Sidekiq replacement"

RabbitMQ is more than a Sidekiq replacement

I've had gripes with Sidekiq because of which I switched to RabbitMQ. Here are my thoughts and experiences after a year of using it in production. I got inspired to write this post by the overwhelming response I received for my talk at the local Ruby user group. Why do we need Sidekiq or Rabb...

Continue reading ...