Concurrent File Downloader

Learn concurrent file downloader through an executable example, then connect the small demonstration to production code.

A concurrent downloader requests independent files in parallel while respecting a worker limit and request timeouts. Always stream or cap large bodies and safely choose output filenames.

This demonstration downloads two small files from a local HTTP test server into an isolated temporary directory, avoiding external-network dependencies.

Follow this diagram before executing the example. Check where a value or decision changes what runs next.

Hardening: enforce URL allowlists where appropriate, connect/read timeouts, maximum size, temporary-file-and-rename writes, retries with backoff and per-job error handling. Do not trust Content-Length as the only size safeguard.

This example is a complete program. Save it as main.go or paste it into the Go Playground.

Go code

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"os"
	"path/filepath"
	"sync"
	"time"
)

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "data:"+r.URL.Path) }))
	defer server.Close()
	dir, err := os.MkdirTemp("", "go-download-")
	if err != nil {
		panic(err)
	}
	defer os.RemoveAll(dir)
	names := []string{"a.txt", "b.txt"}
	sizes := make([]int64, len(names))
	client := &http.Client{Timeout: 2 * time.Second}
	var wg sync.WaitGroup
	for i, name := range names {
		wg.Add(1)
		go func(index int, filename string) {
			defer wg.Done()
			response, err := client.Get(server.URL + "/" + filename)
			if err != nil {
				panic(err)
			}
			defer response.Body.Close()
			file, err := os.Create(filepath.Join(dir, filename))
			if err != nil {
				panic(err)
			}
			size, err := io.Copy(file, response.Body)
			closeErr := file.Close()
			if err != nil || closeErr != nil {
				panic("download failed")
			}
			sizes[index] = size
		}(i, name)
	}
	wg.Wait()
	for i, name := range names {
		fmt.Println(name, "bytes:", sizes[i])
	}
}

Output

a.txt bytes: 11
b.txt bytes: 11

Two goroutines fetch independent local paths. Each writes to a separate file and stores its byte count by index; Wait ensures that the printed results are complete.

• In a production downloader, return worker errors through channels instead of panicking in goroutines.
• Use an explicit concurrency limit, response-status checks, content-size bounds and safe filenames before accepting arbitrary URLs.

Add a worker pool and error channel, reject non-200 responses and compute hashes for downloaded content.

Tip: Edit the example above, predict the output, and run the modified code to test your reasoning.

You have practiced Concurrent File Downloader in a small, self-contained program. The important habit is to predict what runs, inspect the actual output and distinguish the language rule from the particular example.

Before continuing, make sure you can explain the program to someone who has not seen it, including the line that changes its result.