Real-Time Lambda Log Streaming to Your Web UI - Without CloudWatch Logs

CloudWatch Logs are sluggish, limited, and can get expensive fast — especially if you want real-time Lambda output in a web dashboard or dev tool. Here’s how to stream AWS Lambda logs directly to a browser UI in real time, without CloudWatch. We’ll use Kinesis, Lambda destinations, and a simple WebSocket server to build a powerful, cost-effective logging pipeline. Step 1: Set Up a Kinesis Firehose Stream Kinesis Firehose is the simplest way to buffer log messages and route them downstream. In the AWS Console: Go to Kinesis → Firehose → Create Delivery Stream Choose Direct PUT Select destination: Amazon S3, or just Discard records for dev/testing Name it lambda-logs-firehose Take note of the stream name. Step 2: Modify Your Lambda to Log to Firehose Instead of writing to console.log, use the AWS SDK to put logs into Firehose. import { FirehoseClient, PutRecordCommand } from "@aws-sdk/client-firehose"; const firehose = new FirehoseClient({ region: "us-east-1" }); export const handler = async (event) => { const log = { time: new Date().toISOString(), message: "Lambda did a thing", event, }; await firehose.send( new PutRecordCommand({ DeliveryStreamName: "lambda-logs-firehose", Record: { Data: Buffer.from(JSON.stringify(log) + "\n"), }, }) ); return { statusCode: 200 }; }; Step 3: Set Up a Tiny WebSocket Server for Streaming Spin up a simple Node.js WebSocket server to push log data to connected clients. import WebSocket from "ws"; import http from "http"; const server = http.createServer(); const wss = new WebSocket.Server({ server }); let clients = []; wss.on("connection", (ws) => { clients.push(ws); ws.on("close", () => { clients = clients.filter((c) => c !== ws); }); }); function broadcastLog(data) { clients.forEach((ws) => { if (ws.readyState === WebSocket.OPEN) { ws.send(data); } }); } server.listen(3000, () => { console.log("WebSocket server on port 3000"); }); Step 4: Pipe Firehose Logs into the WebSocket Server You can use a second Lambda function to listen to Firehose and broadcast logs. In your Firehose settings, set a Lambda destination: Trigger: Firehose → Lambda → WebSocket Here’s the WebSocket Lambda: import WebSocket from "ws"; const ws = new WebSocket("ws://your-websocket-server:3000"); export const handler = async (event) => { for (const record of event.records) { const logLine = Buffer.from(record.data, "base64").toString("utf8"); ws.send(logLine); } return { records: event.records.map((r) => ({ recordId: r.recordId, result: "Ok", })), }; }; Step 5: Build a Real-Time Frontend Log Viewer Here’s a simple browser UI to consume the logs: const ws = new WebSocket("ws://your-websocket-server:3000"); ws.onmessage = (event) => { const log = JSON.parse(event.data); const pre = document.createElement("pre"); pre.textContent = JSON.stringify(log, null, 2); document.body.appendChild(pre); }; You now have real-time Lambda logs in the browser, streamed directly from Kinesis → WebSocket → UI. ✅ Pros: Near-instant log visibility in the browser — ideal for dashboards or dev tools Avoids CloudWatch retention and UI limitations Logs stay structured and JSON-native Easy to replay, transform, or fork log data in real time ⚠️ Cons: More moving parts (Kinesis, Lambda, WebSocket server) Not ideal for very high log volume without buffering You'll need to manage authentication for production Summary If you’ve ever waited 30 seconds for CloudWatch Logs to load, this setup is for you. By combining AWS Firehose and a WebSocket stream, you can create a real-time pipeline for Lambda logs — no CloudWatch needed. It’s fast, flexible, and perfect for internal tools, dashboards, or dev visibility. You control the format, the storage, and the delivery — all while saving time and costs. If this was helpful, you can support me here: Buy Me a Coffee ☕

Apr 29, 2025 - 07:32

Real-Time Lambda Log Streaming to Your Web UI - Without CloudWatch Logs

CloudWatch Logs are sluggish, limited, and can get expensive fast — especially if you want real-time Lambda output in a web dashboard or dev tool.

Here’s how to stream AWS Lambda logs directly to a browser UI in real time, without CloudWatch. We’ll use Kinesis, Lambda destinations, and a simple WebSocket server to build a powerful, cost-effective logging pipeline.

Step 1: Set Up a Kinesis Firehose Stream

Kinesis Firehose is the simplest way to buffer log messages and route them downstream.

In the AWS Console:

Go to Kinesis → Firehose → Create Delivery Stream
Choose Direct PUT
Select destination: Amazon S3, or just Discard records for dev/testing
Name it lambda-logs-firehose

Take note of the stream name.

Step 2: Modify Your Lambda to Log to Firehose

Instead of writing to console.log, use the AWS SDK to put logs into Firehose.

import { FirehoseClient, PutRecordCommand } from "@aws-sdk/client-firehose";

const firehose = new FirehoseClient({ region: "us-east-1" });

export const handler = async (event) => {
  const log = {
    time: new Date().toISOString(),
    message: "Lambda did a thing",
    event,
  };

  await firehose.send(
    new PutRecordCommand({
      DeliveryStreamName: "lambda-logs-firehose",
      Record: {
        Data: Buffer.from(JSON.stringify(log) + "\n"),
      },
    })
  );

  return { statusCode: 200 };
};

Step 3: Set Up a Tiny WebSocket Server for Streaming

Spin up a simple Node.js WebSocket server to push log data to connected clients.

import WebSocket from "ws";
import http from "http";

const server = http.createServer();
const wss = new WebSocket.Server({ server });

let clients = [];

wss.on("connection", (ws) => {
  clients.push(ws);
  ws.on("close", () => {
    clients = clients.filter((c) => c !== ws);
  });
});

function broadcastLog(data) {
  clients.forEach((ws) => {
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(data);
    }
  });
}

server.listen(3000, () => {
  console.log("WebSocket server on port 3000");
});

Step 4: Pipe Firehose Logs into the WebSocket Server

You can use a second Lambda function to listen to Firehose and broadcast logs.

In your Firehose settings, set a Lambda destination:

Trigger: Firehose → Lambda → WebSocket

Here’s the WebSocket Lambda:

import WebSocket from "ws";

const ws = new WebSocket("ws://your-websocket-server:3000");

export const handler = async (event) => {
  for (const record of event.records) {
    const logLine = Buffer.from(record.data, "base64").toString("utf8");
    ws.send(logLine);
  }

  return {
    records: event.records.map((r) => ({
      recordId: r.recordId,
      result: "Ok",
    })),
  };
};

Step 5: Build a Real-Time Frontend Log Viewer

Here’s a simple browser UI to consume the logs:

You now have real-time Lambda logs in the browser, streamed directly from Kinesis → WebSocket → UI.

✅ Pros:

Near-instant log visibility in the browser — ideal for dashboards or dev tools
Avoids CloudWatch retention and UI limitations
Logs stay structured and JSON-native
Easy to replay, transform, or fork log data in real time

⚠️ Cons:

More moving parts (Kinesis, Lambda, WebSocket server)
Not ideal for very high log volume without buffering
You'll need to manage authentication for production

Summary

If you’ve ever waited 30 seconds for CloudWatch Logs to load, this setup is for you.

By combining AWS Firehose and a WebSocket stream, you can create a real-time pipeline for Lambda logs — no CloudWatch needed. It’s fast, flexible, and perfect for internal tools, dashboards, or dev visibility. You control the format, the storage, and the delivery — all while saving time and costs.

If this was helpful, you can support me here: Buy Me a Coffee ☕