Export large file in Rails with Active Job

Introduction In web development, sometimes you'll have to do long-running tasks. If doing these kinds of tasks in a normal Rails controller, you'll have some problems with request timeout, memory spikes, or bad UX. Therefore, these tasks should be moved into a background job for processing outside the life cycle of a single HTTP request. However, when work is moved to a background job, the code runs outside of a single request-response life cycle, and you need a way to report progress and send information once the job is completed. The solution is to use a Web Socket, a bidirectional communication method, to broadcast information from our background job. It is so good that Rails supports tools for us to work with both Background Job and Web Socket. Pick your tools Active Job vs Solid Queue - for working with background jobs Action Cable and Solid Cable - for working with web socket. I suppose those who read this post are familiar with the Rails ecosystem and some tools such as Active Job and Action Cable, so won't introduce more. Solid Queue and Solid Cable were released in 2024 and came with Rails 8, they are the database-backed for Active Job and Action Cable In the next part, I will use these tools to work with the practical example. Example: Export large CSV file Firstly, we need to write the export csv file logic, instead of writing this logic directly into the controller, we create the job ExportPhotosJob and write this logic into this class. Then, call the job inside the controller. require "csv" class ExportPhotosJob { this.statusTarget.classList.remove("hidden"); this.statusTarget.textContent = "Exporting ..."; this.subscribe_channel() }) .catch((error) => { console.error('Error:', error); }); } subscribe_channel() { this.channel = createConsumer().subscriptions.create("ExportCsvChannel", { connected() { console.log("hello") }, disconnected() { }, received(data) { if (data["jid"] != null) { window.location.href = `/photos/exports/download.csv?id=${data["jid"]}` } } }); } } When the exporting progress is finished, the client will receive the job ID, and we will open the downloadable link for the user. Progress bar Now our demo works well, users can receive the link to download the CSV file they want. But for better UX, we should display a progress bar for the user to know the progress of the export file. Instead of broadcasting once when the export job is finished, we broadcast periodically class ExportPhotosJob

Feb 28, 2025 - 09:16

Export large file in Rails with Active Job

Introduction

In web development, sometimes you'll have to do long-running tasks. If doing these kinds of tasks in a normal Rails controller, you'll have some problems with request timeout, memory spikes, or bad UX. Therefore, these tasks should be moved into a background job for processing outside the life cycle of a single HTTP request.

However, when work is moved to a background job, the code runs outside of a single request-response life cycle, and you need a way to report progress and send information once the job is completed. The solution is to use a Web Socket, a bidirectional communication method, to broadcast information from our background job.

It is so good that Rails supports tools for us to work with both Background Job and Web Socket.

Pick your tools

Active Job vs Solid Queue - for working with background jobs
Action Cable and Solid Cable - for working with web socket.

I suppose those who read this post are familiar with the Rails ecosystem and some tools such as Active Job and Action Cable, so won't introduce more. Solid Queue and Solid Cable were released in 2024 and came with Rails 8, they are the database-backed for Active Job and Action Cable
In the next part, I will use these tools to work with the practical example.

Example: Export large CSV file

Firstly, we need to write the export csv file logic, instead of writing this logic directly into the controller, we create the job ExportPhotosJob and write this logic into this class. Then, call the job inside the controller.

require "csv"

class ExportPhotosJob < ApplicationJob
  queue_as :default

  def perform(*args)
    photos = Photo.all
    filename = Rails.root.join("tmp", "all_photos.csv")

    CSV.open(filename, 'w') do |csv|
      csv << Photo::CSV_ATTRIBUTES
      photos.each_with_index do |photo, idx|
        csv << Photo::CSV_ATTRIBUTES.map{ |field| photo.send(field) }

        sleep 0.2 # for demonstrating taking long time
      end
    end
  end
end

Our controller will now be like this:

class Photos::ExportsController < ApplicationController
  def index
    ExportPhotosJob.perform_later
    head :accepted
  end

  def download
    job_id = params[:id]
    # ...
  end
end

However, there was a problem: the job is executed in the background so it doesn’t inform the main server when it is completed. So we executed the job, got the exported file, and cannot give the file back to the server to return it to our client? The solution is using Action Cable to broadcast for the client when the job is completed.

Broadcasting with Action Cable

In order to return the CSV file to the client, we will create a channel using Action Cable and broadcast the file back when it’s ready.

class ExportCsvChannel < ApplicationCable::Channel
  def subscribed
    stream_from "export_csv"
  end

  def unsubscribed
    # Any cleanup needed when channel is unsubscribed
  end
end

Then we will add the code to broadcast our file in the job so it will return the CSV when the job is performed.

class ExportPhotosJob < ApplicationJob
  queue_as :default

  def perform(*args)
    CSV.open(filename, 'w') do |csv|
      # code
    end
    ActionCable.server.broadcast "export_user", self.jid
  end
end

On the client side, it needs to subscribe to the export_csv channel and receive the message from Background Job broadcasting. Let’s add the Stimulus controller for that purpose:

import { Controller } from "@hotwired/stimulus"
import { createConsumer } from "@rails/actioncable"

export default class extends Controller {
  static targets = ["status"];

  export() {
    fetch('/photos/exports')
      .then(() => {
        this.statusTarget.classList.remove("hidden");
        this.statusTarget.textContent = "Exporting ...";
        this.subscribe_channel()
      })
      .catch((error) => {
        console.error('Error:', error);
      });
  }

  subscribe_channel() {
    this.channel = createConsumer().subscriptions.create("ExportCsvChannel", {
      connected() {
        console.log("hello")
      },

      disconnected() {
      },

      received(data) {
        if (data["jid"] != null) {
          window.location.href = `/photos/exports/download.csv?id=${data["jid"]}`
        }
      }
    });
  }
}

When the exporting progress is finished, the client will receive the job ID, and we will open the downloadable link for the user.

Progress bar

Now our demo works well, users can receive the link to download the CSV file they want. But for better UX, we should display a progress bar for the user to know the progress of the export file.

Instead of broadcasting once when the export job is finished, we broadcast periodically

class ExportPhotosJob < ApplicationJob
  queue_as :default

  def perform(*args)
    # ...

    CSV.open(filename, "w") do |csv|
      csv << Photo::CSV_ATTRIBUTES
      photos.each_with_index do |photo, idx|
        csv << Photo::CSV_ATTRIBUTES.map { |field| photo.send(field) }

        sleep 0.2

        if idx % 5 == 0
          percentage = (idx.to_f / photos.size * 100).round
          ActionCable.server.broadcast "export_csv", { progress: percentage }
        end
      end
    end

    ActionCable.server.broadcast "export_csv", { jid: self.job_id }
  end
end

And edit the Stimulus code that is responsible for receiving data from Action Cable like that:

export default class extends Controller {
  export() {}

  subscribe_channel(progressTarget, progressWrapperTarget) {
      # ...
      received(data) {
        if (data["jid"] == null) {
          progressTarget.style.width = `${data["progress"]}%`;
        } else {
          window.location.href = `/photos/exports/download.csv?id=${data["jid"]}`
          progressWrapperTarget.classList.add("hidden");
        }
      }
    });
  }
}

You can find the full source code for this guide on this repo