[HELP]How to feed stdin into os.exec?

[thenoobtester]

Hi,

first of all thank you for this it's really a big help for me. I have the following code which allows me to use the linux base64 utility to decode a base64 encoded file:

`'use strict';

import * as std from 'std';
import * as os from 'os';

var result = os.exec(["/bin/sh", "-c", "base64 -d file"]);

console.log(result);`

It works really well and is fast (I had a base64 decode js function but it was to slow that's why I use exec), however I was wondering how to feed a variable containing the base64 data directly to the base64 utility? It says that it takes a file as input and when no file is specified takes data from stdin (on a terminal feeding data through a pipe 'echo "string" | base64 -d -' will work), how could I achieve this and write the result to a file with quickjs please? I looked for a solution online but just couldn't find anything that made sense to me.

Thank you for your help.

[ctn-malone]

You can create a pipe and pass one of its end as the stdin parameter of exec (I think there are some examples in the tests directory. You can also take a look at a small lib I worked on which provide some utilities such as async exec + easier input management, as well as more complete curl support (https://github.com/ctn-malone/qjs-ext-lib/blob/master/doc/process.md)

[ctn-malone]

Something like this should work

import * as std from 'std';
import * as os from 'os';

const TEXT = '123456';

let inPipe, outPipe, pid, file;

/*
    Encode
 */
outPipe = os.pipe();
inPipe = os.pipe();
pid = os.exec(['base64'], {
  stdout: outPipe[1],
  stdin: inPipe[0],
  block: false,
});
os.close(outPipe[1]);
os.close(inPipe[0]);

file = std.fdopen(inPipe[1], 'w');
file.puts(TEXT);
file.close();
file = std.fdopen(outPipe[0], 'r');
const base64Encoded = file.readAsString().trim();
file.close();
os.waitpid(pid, 0);
console.log(`Base64Encoded: ${base64Encoded}`);

/*
    Decode
 */
outPipe = os.pipe();
inPipe = os.pipe();
pid = os.exec(['base64', '-d'], {
  stdout: outPipe[1],
  stdin: inPipe[0],
  block: false,
});
os.close(outPipe[1]);
os.close(inPipe[0]);

file = std.fdopen(inPipe[1], 'w');
file.puts(base64Encoded);
file.close();
file = std.fdopen(outPipe[0], 'r');
const base64Decoded = file.readAsString().trim();
file.close();
os.waitpid(pid, 0);
console.log(`Base64Decoded: ${base64Decoded}`);

[thenoobtester]

Thank you very much for this I really appreciate. It works on short base64 encoded strings however when I try on a big file (see attached, it's a base64 encoded .ts video file from a m3u8 stream) it hangs. i tried strace to see what was going on but all I could see was :
fcntl(6, F_GETFL) = 0x1 (flags O_WRONLY) fstat(6, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 write(6, "R0ARGQBC8C8AAcEAAP8B/wAB/IAeSBwB"..., 1490944) = 159744
base64chunk.txt

Do you have any idea why?

Thank you.

[ctn-malone]

Might be related to the fact that you're expecting to decode binary data and not text. You should read from the pipe using ArrayBuffer and not utf8 string. Of you can just do os.exec(["/bin/sh", "-c", "base64 -d >decoded_file"]); and provide input using a pipe. Where does the base64encoded content come from ?

[thenoobtester]

It comes from a csv file with base64 encoded data that needs to be put back together to get the full base64 data te be decoded into a .ts video file. Attached is a file called sample_code.txt (rename to .js) which does all of that and spits out the full base64 data to be decoded which is where I am stuck. The data.csv file is the source for sample_code.js. It should be run as qjs -m sample_code.js > base64Encoded.
data.csv
sample_code.txt

EDIT 1:when I then do base64 -d base64Encoded>vid_chunk.ts it works so the base64Encoded data is sane.

EDIT 2: You mentioned 'You should read from the pipe using ArrayBuffer' this makes sense indeed since the data is binary as you noticed but how would I do that? Sorry for my poor if not non-existent programming skills. As I said doing qjs -m sample_code.js > base64Encoded and then base64 -d base64Encoded>vid_chunk.ts works so I have something that works but I'm just curious since this small js engine and your lib open many new possibilities for scripting small utilities and I'd like to see how to achieve this in one js script. Thanks again for your help 👍

[ctn-malone]

Try like this (ie: you can open a file and use it to redirect the output of the exec process)

sample_code.txt

[thenoobtester]

Thank you so much it worked!!!