Reading Stdout from a subprocess

Question

I am attempting to spawn a subprocess from Golang. The goal is to read and process the input line-by-line. Here is what I am trying to get working:

func readStuff(scanner *bufio.Scanner) {
    for scanner.Scan() {
        fmt.Println("Performed Scan")
        fmt.Println(scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Fprintln(os.Stderr, "reading standard input:", err)
    }
}

func main() {
    cmd := exec.Command("/usr/local/bin/pocketsphinx_continuous", "-inmic", "yes")
    out, err := cmd.StdoutPipe()

    err = cmd.Start()
    checkError(err)

    scanner := bufio.NewScanner(out)
    fmt.Println("Scanner created")

    defer cmd.Wait()
    go readStuff(scanner)
}

In this example, "Scanner created" is printed, but nothing happens after that.

Running this command however does result in what I am expecting to be printed to :

/usr/local/bin/pocketsphinx_continuous -inmic yes 1>out.txt

And modifying the code to directly copy to stdout works as well:

cmd := exec.Command("/usr/local/bin/pocketsphinx_continuous", "-inmic", "yes")
cmd.Stdout = os.Stdout

What am I missing that is keeping me from reading the output?

Answer 1

There are multiple things you may want to check.

1. The error code returned by cmd.StdoutPipe() is not checked. It should be.

2. The pocketsphinx_continuous command requires the -hmm and -dict arguments to be provided. Otherwise, it will fail, and all the output is actually sent to stderr and not stdout. Here, you read only stdout, but there is nothing to read.

3. You should not call cmd.Wait() before being sure all the data have been read from stdout. The result is non deterministic (actually, it is a race condition). Check the documentation about the os/exec package. If you absolutely need the parsing to be done in a goroutine, you need to synchronize with the end of the goroutine before cmd.Wait() is called. For instance, you could write the function as:

   func readStuff(scanner *bufio.Scanner, stop chan bool) {
       // Scanning code
       // ...
       stop<-true
   }
   

   and the main code as:

   stop := make(chan bool)
   go readStuff(scanner,stop)
   <-stop
   cmd.Wait()
   

Answer 2

Seems you need not

go readStuff(scanner)

because of

cmd.Start()

do system fork itself so just

readStuff(scanner)

would be enough to my mind (not spawning gorouting for that)

Answer 3

This seems to work fine, and it works with go readStuff(scanner) and also with just readStuff(scanner) - I don't think that this usage actually calls for a goroutine, but don't know the actual context.:

package main

import (
    "bufio"
    "fmt"
    "os"
    "os/exec"
)

func readStuff(scanner *bufio.Scanner) {
    for scanner.Scan() {
        fmt.Println("Performed Scan")
        fmt.Println(scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Fprintln(os.Stderr, "reading standard input:", err)
    }
}

func main() {
    cmd := exec.Command("/Users/rfay/bin/junk.sh")
    out, err := cmd.StdoutPipe()

    err = cmd.Start()
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to start err=%v", err)
        os.Exit(1)
    }

    scanner := bufio.NewScanner(out)
    fmt.Println("Scanner created")

    defer cmd.Wait()

    go readStuff(scanner)
}

This is the junk.sh I used for testing.

#!/bin/bash
for i in `seq 1 10`;
do
  echo $i
  sleep 1
done