How to have multiple goroutines read the lines of a single file? : <p>I want to read a huge file, say > 1 GB, and have its lines processed by multiple worker goroutines.</p>

<p>I'm worried that using a single goroutine (main) for reading the input lines will impose a bottleneck, when using a huge number of worker goroutines.</p>

<p>How can I safely have multiple goroutines read the lines of the file? Is it possible split the input file into several chuncks and have each goroutine operate on a separate chunk individually?</p>

<p>The following is sample code of having one goroutine read input lines with several worker goroutines processing them:</p>

<pre><code>package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    file, err := os.Open("/path/to/file.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    lines := make(chan string)

    for i := 0; i &lt; 100; i++ {
        // start 100 workers to process input lines.
        // the workers terminate once 'lines' is closed.
        go worker(lines)
    }

    scanner := bufio.NewScanner(file)
    go func() {
        defer close(lines)
        for scanner.Scan() {
            lines &lt;- scanner.Text()
        }

        if err := scanner.Err(); err != nil {
            log.Fatal(err)
        }
    }()

    ...
}
</code></pre>
