io.Reader/io.Writer, os and bufio in Go: File Operations, Composing Streams and embed

Version note: The code in this post was built and run with Go 1.27.4; the Bookstore module's go line is 1.26.0, and the code also compiles and passes its tests with Go 1.26.11. embed arrived in Go 1.16, errors.Join in Go 1.20. The 1 GB log measurements were taken on a developer machine that was also running other work; the timings show direction only.

Guess First

One day a 70 KB line shows up in the log of Bookstore, the online bookstore used as the shared example across this site's series: a client sent a very long URL. The code that reads the log line by line looks like this:

func main() {
    log := "line 1\n" + strings.Repeat("x", 70_000) + "\nline 3\n" // a 70 KB line in the middle
    sc := bufio.NewScanner(strings.NewReader(log))
    n := 0
    for sc.Scan() {
        n++
    }
    fmt.Println("lines read:", n) // looks like a short file
    fmt.Println("sc.Err():", sc.Err())

    sc = bufio.NewScanner(strings.NewReader(log))
    sc.Buffer(make([]byte, 0, 64<<10), 1<<20) // allow lines up to 1 MB
    n = 0
    for sc.Scan() {
        n++
    }
    fmt.Println("lines read:", n, "err:", sc.Err())
}

How many lines does the first loop count in this three-line input, and does it report an error?

$ go run ./labs/18-scanner
lines read: 1
sc.Err(): bufio.Scanner: token too long
lines read: 3 err: <nil>

It counted one line and stopped silently. By default, bufio.Scanner keeps a line in a buffer of at most 64 KB; when a longer line comes, Scan returns false and the reason is only in Err. A report that does not look at Err says "done" without ever reading the rest of the log. This post covers files and streams in Go: io.Reader and io.Writer, opening and closing files with os, bufio, encoding/csv, composing readers and writers, and the embed promised in post 11. Bookstore gets a package that turns a 1 GB order log into a daily sales report with constant memory, and an e-mail template embedded in the binary.

io.Reader and io.Writer: Two Small Interfaces

In .NET, Stream is a single abstract class that carries Read, Write, Seek, CanRead, Length and more. Go has single-method interfaces instead:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

io.Closer, io.Seeker and io.ReaderAt are separate; when needed, combinations such as io.ReadCloser are used. This is the most successful example in the standard library of the "small interfaces" idea we discussed in post 04: *os.File, *gzip.Reader, *bytes.Buffer, *strings.Reader, an HTTP request body and a TCP connection all satisfy the same Read. The contract of Read is like in .NET: it may return fewer bytes than asked for, and it may return io.EOF together with the last chunk, so you process the n bytes first and look at the error afterwards. The short-read problem from the "Stream Model" post of the I/O in .NET series applies here too; when you need exactly len(p) bytes, there is io.ReadFull.

Opening and Closing Files, and the Close Error

The lab that runs the report from file to file wraps one reader in another depending on whether the input ends in .gz:

func run(in, out string) (err error) {
    f, err := os.Open(in)
    if err != nil {
        return err
    }
    defer f.Close() // read-only: a Close error loses nothing

    var r io.Reader = f
    if strings.HasSuffix(in, ".gz") {
        gz, err := gzip.NewReader(f) // a Reader that wraps a Reader
        if err != nil {
            return err
        }
        defer gz.Close() // does not close f
        r = gz
    }

    w, err := os.Create(out)
    if err != nil {
        return err
    }
    defer func() { err = errors.Join(err, w.Close()) }() // the write is not done until Close succeeds

    ist, err := time.LoadLocation("Europe/Istanbul")
    if err != nil {
        return err
    }
    return report.DailySales(r, w, ist)
}

For a file opened for reading, defer f.Close() is enough. A file opened for writing is different: the operating system may keep the data in its own buffer, and an error (disk full, network drive gone) may only surface in Close. defer w.Close() throws that error away; a named return with errors.Join is the shortest way to add it to the function's error. Note that gzip.Reader.Close does not close the file underneath; .NET's leaveOpen: true is Go's default, and you close every layer yourself. For small files, os.ReadFile and os.WriteFile are one line each; for paths, filepath.Join uses backslashes on Windows and forward slashes on Linux, and os.MkdirAll creates a folder tree.

A 1 GB Log with bufio.Scanner

Since this post, the Bookstore API writes an "order placed" line to the JSON log from post 13 for every order:

{"time":"2026-11-15T06:21:35.098623+03:00","level":"INFO","msg":"order placed","order_id":"ORD-1001","items":1,"total":"699.80","request_id":"XKEL4VWWA5GM"}

The new internal/report package turns these lines into a daily sales summary in CSV on the warehouse clock. Its signature has no file in it; it takes an io.Reader and writes to an io.Writer:

// DailySales reads JSON log lines from r and writes one CSV row per day to
// w. Days are calendar days in loc. It reads one line at a time, so memory
// use does not grow with the size of the log.
func DailySales(r io.Reader, w io.Writer, loc *time.Location) error {
    days := make(map[string]*day)
    sc := bufio.NewScanner(r)
    sc.Buffer(make([]byte, 0, 64<<10), MaxLineBytes)
    line := 0
    for sc.Scan() {
        line++
        if !bytes.Contains(sc.Bytes(), marker) {
            continue
        }
        var e orderPlaced
        if err := json.Unmarshal(sc.Bytes(), &e); err != nil {
            return fmt.Errorf("line %d: %w", line, err)
        }
        total, err := catalog.ParseMoney(e.Total)
        if err != nil {
            return fmt.Errorf("line %d: %w", line, err)
        }
        key := e.Time.In(loc).Format(time.DateOnly)
        d := days[key]
        if d == nil {
            d = &day{}
            days[key] = d
        }
        d.orders++
        d.items += e.Items
        d.revenue += total
    }
    if err := sc.Err(); err != nil { // without this, a long line ends the loop silently
        return fmt.Errorf("after line %d: %w", line, err)
    }

    cw := csv.NewWriter(w)
    if err := cw.Write([]string{"date", "orders", "items", "revenue"}); err != nil {
        return err
    }
    for _, key := range slices.Sorted(maps.Keys(days)) {
        d := days[key]
        row := []string{key, strconv.Itoa(d.orders), strconv.Itoa(d.items), d.revenue.Decimal()}
        if err := cw.Write(row); err != nil {
            return err
        }
    }
    cw.Flush() // csv.Writer buffers too
    return cw.Error()
}

The Buffer call raises the line limit to 1 MB. The bytes.Contains pre-check skips three out of every four lines without ever decoding them as JSON; sc.Bytes() is not a copy but a slice that looks into the Scanner's buffer and changes on the next Scan, so it has to be copied if you keep it. The tests hand this function a strings.NewReader and a bytes.Buffer; no files, folders or mocks are needed. With labs/18-genlog I generated a synthetic 1 GB log (6,144,588 lines, 1,536,147 orders) and ran the report on it:

$ ./bin/report orders.log daily.csv
done in 1.5s, memory from the OS: 13 MB
$ head -4 daily.csv
date,orders,items,revenue
2026-11-01,94530,189043,50592794.25
2026-11-02,108120,216189,57829449.78
2026-11-03,108063,216529,57921928.43
$ ./bin/report orders.log.gz daily-gz.csv
done in 2s, memory from the OS: 17 MB
$ ./bin/readall orders.log
1536147 orders in 900ms, memory from the OS: 1181 MB

The 1 GB file was processed with 13 MB of memory; its 36 MB gzipped form produced the same CSV with 17 MB, and the only difference is the gzip.NewReader layer. labs/18-readall only counts the orders in the same file and never decodes JSON, which is why it is faster, but because of os.ReadFile and bytes.Split it took more than 1.1 GB from the operating system. That is more than four times a container limit such as the --memory=256m we used in post 13. November 1 having fewer orders than the other days is a check too: the synthetic log starts at 00:00 UTC, which is 03:00 on the Istanbul clock.

encoding/csv and Flush

encoding/csv writes and reads quotes and escapes correctly, following the format defined in RFC 4180; the strings.Split we used in post 15 would break on a quoted field. To read the semicolons of a Turkish Excel export, r.Comma = ';' is enough. There is one trap: csv.Writer buffers internally.

for _, flush := range []bool{false, true} {
    path := filepath.Join(dir, fmt.Sprintf("daily-%v.csv", flush))
    f, err := os.Create(path)
    if err != nil {
        panic(err)
    }
    cw := csv.NewWriter(f) // buffers in a bufio.Writer of its own
    _ = cw.Write([]string{"date", "orders"})
    _ = cw.Write([]string{"2026-11-14", "108330"})
    if flush {
        cw.Flush()
        if err := cw.Error(); err != nil {
            panic(err)
        }
    }
    if err := f.Close(); err != nil { // closing the file does not flush the csv.Writer
        panic(err)
    }
    fmt.Printf("flush=%v: %d bytes on disk\n", flush, size(path))
}

$ go run ./labs/18-flush
flush=false: 0 bytes on disk
flush=true: 30 bytes on disk

Closing the file does not flush the csv.Writer; if you forget Flush, the file is created but stays empty, and you see no error anywhere. Because Flush itself returns no error, you have to check cw.Error() afterwards; the last two lines of DailySales do exactly that. The same rule holds for bufio.Writer: labs/18-genlog uses a 1 MB bufio.Writer, and its last line is return w.Flush().

Composing Readers and Writers

The real gain of small interfaces is composition. Think of an endpoint that uploads a book cover: while writing the body to a file you want to limit its size and compute its SHA-256 digest, reading the data only once:

// saveCover copies body into dir/name, at most maxCover bytes, and returns
// the size and the SHA-256 of what it wrote, in a single pass.
func saveCover(dir, name string, body io.Reader) (n int64, sum string, err error) {
    f, err := os.Create(filepath.Join(dir, name))
    if err != nil {
        return 0, "", err
    }
    defer func() { err = errors.Join(err, f.Close()) }()

    h := sha256.New()
    limited := io.LimitReader(body, maxCover+1) // one byte more, to notice a body that is too big
    n, err = io.Copy(io.MultiWriter(f, h), limited)
    if err != nil {
        return n, "", err
    }
    if n > maxCover {
        return n, "", errTooLarge
    }
    return n, fmt.Sprintf("%x", h.Sum(nil))[:16], nil
}

$ go run ./labs/18-compose
4000 fedafb0efa4bfcb8 <nil>
1048577  cover is larger than 1 MB
5 "Dune\n" "Dune\n"

io.LimitReader reads one byte more than 1 MB; that is how you accept a file of exactly 1 MB and still notice a larger one. io.MultiWriter sends every write both to the file and to the hash, and io.Copy copies to the end with a 32 KB buffer. io.TeeReader on the last line copies everything it reads into a writer as well, for a "side output" into a log or a hash. io.MultiReader chains several readers one after another. On the HTTP side, http.MaxBytesReader from post 10 does the same job. They are all the counterpart of the decorator chain in .NET (GZipStream around a FileStream), but each piece does a single job.

embed: Putting the Template into the Binary

In post 11, sqlstore embedded its schema file into a string:

//go:embed schema.sql
var schema string

For several files, the same mechanism gives you an embed.FS. Bookstore's order confirmation e-mail now lives in internal/notifications/templates/order_confirmation.txt and is rendered with text/template:

// templates is compiled into the binary; there is no file to deploy.
//
//go:embed templates/*.txt
var templates embed.FS

var mails = template.Must(template.ParseFS(templates, "templates/*.txt"))

Subject: Your Bookstore order {{.ID}}

Hello {{.CustomerID}},

thank you for your order. We received it on {{.Placed}}.
{{range .Items}}
  {{.Quantity}} x {{.Title}} ({{.ISBN}})  {{.Total}}
{{- end}}

Total: {{.Total}}
Estimated delivery: {{.Delivery}}

Bookstore

$ go run ./labs/18-email
Subject: Your Bookstore order ORD-1001

Hello ada,

thank you for your order. We received it on 13.11.2026 14:59.

  2 x Dune (9780441172719)  699.80 TRY
  1 x Emma (9780141439587)  199.00 TRY

Total: 898.80 TRY
Estimated delivery: 17.11.2026

Bookstore

OrderConfirmation(w io.Writer, ...) writes the e-mail to a writer: os.Stdout in the lab, a strings.Builder in the test, an SMTP client later. The delivery estimate and the warehouse clock from post 16 are used here too. template.Must parses the template when the package loads; if the template has a syntax error, the program stops at start-up, not at the first e-mail. //go:embed has a few rules. Patterns are relative to the package folder and use forward slashes even on Windows; you cannot leave the module with ... A pattern that matches no file stops the build:

$ go build .
main.go:8:12: pattern templates/*.html: no matching files found

When you name a folder, files starting with . or _ are left out; templates/* and all:templates take them as well. I tried the three patterns with a .draft.txt in the folder:

func main() {
    for name, f := range map[string]embed.FS{"templates": dir, "templates/*": star, "all:templates": all} {
        var files []string
        fs.WalkDir(f, ".", func(p string, d fs.DirEntry, err error) error {
            if !d.IsDir() {
                files = append(files, p)
            }
            return nil
        })
        fmt.Println(name, files)
    }
}

$ go run . | sort
all:templates [templates/.draft.txt templates/order_confirmation.txt]
templates [templates/order_confirmation.txt]
templates/* [templates/.draft.txt templates/order_confirmation.txt]

To keep temporary files left by editors, or hidden files such as .env, out of the binary, naming the folder or narrowing the pattern by file extension is the safe choice; Bookstore uses templates/*.txt.

Production Notes

Put io.Reader/io.Writer in function signatures, not file names. DailySales handles a file, a gzipped file, an HTTP body and the string in a test with the same code; opening and closing the file is the job of main or the handler.

Keep memory constant on large input. Reading line by line, or streaming with io.Copy, gives memory use that does not depend on the size of the input; leave os.ReadFile and io.ReadAll to small inputs whose size you know, and always limit input from outside with io.LimitReader or http.MaxBytesReader.

Do not update the file you write in place. If the program crashes while writing the report, half a CSV is left behind; write to a temporary file in the same folder first, check the Close error, then move it to its real name with os.Rename.

Common Pitfalls

Not looking at sc.Err(). A long line, a read error or a broken gzip ends the loop silently; the report is incomplete but looks "successful".

Ignoring the Close error of a written file. defer f.Close() is fine for reading; for writing, add the error to the function's result.

Leaving a csv.Writer or a bufio.Writer unflushed. The file is created but stays empty or incomplete; after Flush, check Error() too.

Keeping sc.Bytes(). The next Scan overwrites the same buffer; if you keep it, use bytes.Clone or sc.Text().

Embedding a folder wholesale with //go:embed. templates/* also takes hidden files; narrow it by extension and list the embedded files in a test with fs.WalkDir.

A Small Experiment

★☆☆ Change labs/18-report to write to a temporary file and move it with os.Rename. Stop the program with Ctrl+C while it writes and look at what is left in the folder.

★★☆ Add the five best-selling ISBNs to DailySales. For that the log line has to carry the ISBNs too; try slog.Group or a slice attribute, update orderPlaced accordingly, and produce a sorted result as in post 17.

★★★ Write the POST /api/books/{isbn}/cover endpoint with saveCover: save the body into a folder opened with os.Root from post 14, return 413 for a request over the size limit, and check the first 512 bytes with http.DetectContentType, rejecting anything that is not JPEG. Hint: you will need to read the first 512 bytes and join them back to the rest of the body with io.MultiReader.

We have seen files and streams; in the last post of the series the same io.Reader shows up as the body of an HTTP response. The next post writes an HTTP client in Go: http.Client's timeouts, cancellation with context, reusing connections, and retries.