Python child process loop over stdin from Node.js parent process

ghz 3days ago ⋅ 2 views

My Node.js parent program executes a child Python process and send to it text data - see here for the implementation. This works ok, the node parent process writes data as:

child.stdin.setEncoding('utf-8');
child.stdin.write(data + '\r\n');
child.stdin.end();

the Python child process echoes it back:

for line in sys.stdin:
    ofp.write(line)

and it gets back the data to the parent:

child.stdout.on('data', function (_data) {
    var data = Buffer.from(_data, 'utf-8').toString().trim();
    res += data;
});

I want my parent process to fork once the child process, and make subsequent calls the child.stdin.write(); without closing the stream. If in the parent process I do like

setInterval(() => {
    child.stdin.write(data + '\r\n');
    child.stdin.end();
}, 1000);

this will cause, at the second very execution a Error [ERR_STREAM_WRITE_AFTER_END]: write after end error, because of the end() call. While if I omits the child.stdin.end(); I will get any output. According to here sys.stdin.readline will read line-by-line until Ctrl + D

def read_stdin():
    readline = sys.stdin.readline()
    while readline:
        yield readline
        readline = sys.stdin.readline()

for line in read_stdin():
    ofp.write(line)

the other approach was

for line in iter(sys.stdin.readline, ''):
    print line

In both cases I get no data back to the parent Node process.

Answer

The issue you're encountering is related to how the stdin stream is being handled in the Node.js parent process. Specifically, the error occurs because you're calling child.stdin.end() every time you write to the stream, which closes the stdin stream after the first write. Once the stream is closed, you cannot write to it again, causing the error you observed (Error [ERR_STREAM_WRITE_AFTER_END]: write after end).

Solution

To keep the stdin stream open for subsequent writes, you should not call end() every time you write to stdin. Instead, simply omit the child.stdin.end() call and keep writing to stdin as needed. The Python process will continue to read from stdin line by line.

Here’s how you can modify both your Node.js and Python code:

Node.js (Parent Process)

Instead of calling end() after each write, you should only call end() when you are done with all interactions. This means in your setInterval, remove the end() call:

setInterval(() => {
    child.stdin.write(data + '\r\n'); // Just write to stdin
    // child.stdin.end();  // Don't call end() here
}, 1000);

Now, the Node.js process can continue sending data to the Python child process without closing the stdin stream after each write.

Python (Child Process)

In the Python code, you can use the iter() function to read continuously from stdin line-by-line, without needing to close stdin. Here's how you can modify the Python code:

import sys

# Using iter to keep reading from stdin until it's closed
for line in iter(sys.stdin.readline, ''):  # This reads line by line
    sys.stdout.write(line)  # Echo the line back to stdout

In this Python snippet:

  • iter(sys.stdin.readline, '') will continuously read from stdin until it reaches the end (i.e., when stdin is closed). Since we're not closing stdin in the Node.js parent process (because we're omitting end()), Python will keep reading and processing the input.
  • The sys.stdout.write(line) sends the data back to the Node.js parent process.

Handling Multiple Writes to stdin in Node.js

If you need to write multiple pieces of data to the child process in an interval, simply keep the stdin open without calling end(). The Python process will continue to receive data until the Node.js parent process explicitly closes the stdin stream.

For example, if you want to send data continuously every second:

const child = require('child_process').spawn('python', ['your_python_script.py']);

child.stdin.setEncoding('utf-8');
child.stdout.on('data', function (_data) {
    var data = Buffer.from(_data, 'utf-8').toString().trim();
    console.log(data);  // Do something with the data from Python
});

setInterval(() => {
    let data = "Some data to send";
    child.stdin.write(data + '\r\n'); // Keep writing without closing stdin
}, 1000);

// You can close the stdin stream when you're done with the process
// child.stdin.end(); // Call this only when you're done sending all data

Additional Considerations:

  • Child Process Lifecycle: Make sure that you're not inadvertently closing or terminating the child process unless you're done with all communication. If you do need to terminate the child process, call child.stdin.end() and child.kill() appropriately.
  • Handling Errors: Ensure you handle potential errors in the child process properly, especially in the stdout event handler of the parent process.

With this setup, your Node.js parent process can send multiple pieces of data to the Python child process, and the Python process will continue to read and process the input as it comes in.

#py