Inter-Process Communication between NodeJS and Python

I'm not a fan of Python, but it is the languange of choice for most Machine Learning code. I'm currently working on a NodeJS based project where I need to call the HuggingFace Diffusers python module. So I came up with an IPC wrapper around the the Python script. It works by sending JSON messages back and forth over stdin/out using U+0000 (NULL,\0) as a delimeter. This is one of the few characters that must be escaped in valid JSON. So we can be confident that no response should contain this character. Since the Python process is blocking, we can send an arbitraty number of messages to it and they will be automatically buffered until the process can handle them. Responses from the Python process are manually buffered until we receive the null byte and then decoded. The JS Side import { spawn } from 'child_process'; import { dirname, join } from 'path'; import { fileURLToPath } from 'url'; const dirname = dirname(fileURLToPath(import.meta.url)); const pythonProcess = spawn('python3', [join(dirname, 'example.py')]); // Buffer for incomplete messages let buffer = ''; // Handle incoming messages from Python pythonProcess.stdout.on('data', (data) => { // Split incoming data on null bytes const messages = (buffer + data.toString()).split('\0'); // Store incomplete message buffer = messages.pop(); // Process complete messages for (const message of messages) { try { const response = JSON.parse(message); console.log('Received from Python:', response); } catch (error) { console.error('Parse error:', error); } } }); // Handle Python errors pythonProcess.stderr.on('data', (data) => { console.error('Python stderr:', data.toString()); }); // Send messages to Python function sendToPython(message) { pythonProcess.stdin.write(JSON.stringify(message) + '\n'); } // Test the communication sendToPython({ type: "test", message: "Hello Python!" }); The Python Side import sys import json import time def log_message(message): # Write messages with null terminator as delimiter sys.stdout.buffer.write(json.dumps(message).encode() + b'\0') sys.stdout.buffer.flush() if name == "main": log_message({"status": "ready"}) while True: try: # Read input from Node.js command = input() data = json.loads(command) # Echo back with timestamp response = { "status": "success", "received": data, "timestamp": time.time() } # Send response log_message(response) except Exception as e: log_message({ "status": "error", "message": str(e) }) This is bare-bones example of how to accomplish this. Stay tuned for more details about this project in the near future.

May 2, 2025 - 19:24

Inter-Process Communication between NodeJS and Python

I'm not a fan of Python, but it is the languange of choice for most Machine Learning code. I'm currently working on a NodeJS based project where I need to call the HuggingFace Diffusers python module. So I came up with an IPC wrapper around the the Python script.

It works by sending JSON messages back and forth over stdin/out using U+0000 (NULL,\0) as a delimeter. This is one of the few characters that must be escaped in valid JSON. So we can be confident that no response should contain this character.

Since the Python process is blocking, we can send an arbitraty number of messages to it and they will be automatically buffered until the process can handle them.

Responses from the Python process are manually buffered until we receive the null byte and then decoded.

The JS Side

import { spawn } from 'child_process';
import { dirname, join } from 'path';
import { fileURLToPath } from 'url';

const __dirname = dirname(fileURLToPath(import.meta.url));
const pythonProcess = spawn('python3', [join(__dirname, 'example.py')]);

// Buffer for incomplete messages
let buffer = '';

// Handle incoming messages from Python
pythonProcess.stdout.on('data', (data) => {
    // Split incoming data on null bytes
    const messages = (buffer + data.toString()).split('\0');

    // Store incomplete message
    buffer = messages.pop();

    // Process complete messages
    for (const message of messages) {
        try {
            const response = JSON.parse(message);
            console.log('Received from Python:', response);
        } catch (error) {
            console.error('Parse error:', error);
        }
    }
});

// Handle Python errors
pythonProcess.stderr.on('data', (data) => {
    console.error('Python stderr:', data.toString());
});

// Send messages to Python
function sendToPython(message) {
    pythonProcess.stdin.write(JSON.stringify(message) + '\n');
}

// Test the communication
sendToPython({ type: "test", message: "Hello Python!" });

The Python Side

import sys
import json
import time

def log_message(message):
    # Write messages with null terminator as delimiter
    sys.stdout.buffer.write(json.dumps(message).encode() + b'\0')
    sys.stdout.buffer.flush()

if __name__ == "__main__":
    log_message({"status": "ready"})

    while True:
        try:
            # Read input from Node.js
            command = input()
            data = json.loads(command)

            # Echo back with timestamp
            response = {
                "status": "success",
                "received": data,
                "timestamp": time.time()
            }

            # Send response
            log_message(response)

        except Exception as e:
            log_message({
                "status": "error",
                "message": str(e)
            })

This is bare-bones example of how to accomplish this. Stay tuned for more details about this project in the near future.