Tag Archives: Go

Goroutines don’t panic if channel they write to is gone

Couple of months ago I wrote an article about error handling in concurrent Go programs. One concern I had about it was if other goroutines would panic if some goroutine produces an error value which causes return from function when the channel is iterated with range construct.

So I made another program that mimics scenario. It spawns couple of goroutines, each taking a second more to complete than previous (for the sake of test being easier to observe). When done, they write to channel. 5th goroutine returns an error (after 5 seconds) via channel which is iterated using range. In case of an error, it’s returned from function.

package main

import (
    "errors"
    "sync"
    "time"
)

type data struct {
    err  error
    data int
}

func someWork(dataChan chan<- *data, collection []int) {
    var wg sync.WaitGroup
    l := len(collection)
    wg.Add(l)
    for i, e := range collection {
        go func(i, e int) {
            time.Sleep(time.Duration(i) * time.Second)
            var err error
            if i == 5 {
                err = errors.New("error")
            }
            dataChan <- &data{err, e}
            defer wg.Done()
        }(i, e)
    }
    wg.Wait()
    close(dataChan)
}

func someFunc() {
    collection := []int{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}
    dataChan := make(chan *data)

    go someWork(dataChan, collection)

    for c := range dataChan {
        if c.err != nil {
            return
        }
    }

    time.Sleep(2 * time.Second)
}

func main() {
    someFunc()
    time.Sleep(20 * time.Second)
}

It surprised me a little other goroutines which still aren't finished don't cause panic because channel is gone even. For more realistic test I've also added some work after someFunc is finished.

Don't know why, it sure has to do with the way garbage collection works in Go (would be a great read if there's detailed explanation of it). I'm glad though I don't have to deal with goroutines panicking myself.

One thing I didn't do in example is cancelling the work running goroutines are still doing so they stop wasting additional memory. One idiomatic way to do that is with the use of context package from standard library, which provides cancellation and timeout mechanisms.

One-liner for serving static files in Go explained

Whenever I want to set up a server for serving static files in Go, which is not often, I always have to spend some time to figure out the following one-liner.

http.Handle("/images/", http.StripPrefix("/images/", http.FileServer(http.Dir("./images"))))

What confuses me probably the most here is all three strings being practically the same.

So in this post I will explain what this does for my future-self so he won’t have to google about it and will have it explained here in a way he’ll like it. 🙂

For this particular example (where one-liner above applies) let’s say we are storing images in ./images folder on a server and accessing them from outside of the server with <server's domain>/images/article2.jpeg

  • We shouldn’t set up a file server at root endpoint if our server accepts other endpoints.
  • http.FileServer(http.Dir("./images"))) exposes folder ./images as a handler.
  • http.StripPrefix is used because images are stored in ./images` folder and not in it’s subfolder images (we want ./images/article2.jpeg not ./images/images/article2.jpeg). It strips prefix from request’s URL.

A friendly reminder

Serving static files is more efficient using dedicated web server (Nginx, Apache,..). This approach also aids to better security of your application.

Libraries over frameworks

Lately (in term of years) I noticed I prefer using libraries over frameworks. Or in other words I like to use packages that each do their one thing well rather than one package with all batteries included. I recognize both libraries and frameworks have their pros and cons.

You can see my preference for libraries if you are reading my blog where I write a lot about web development, specifically about React and Go. Well, Go is a programming language but because it has an excellent standard library, there’s actually no need for a framework.

The most obvious advantage of using libraries over frameworks I have are:

  • I have a freedom of putting all pieces of an application as I see fit and only those necessarily.
  • I can easily replace one piece after another, better option appears.
  • they take less time to learn.
  • feeling I have a control over them not other way around.
  • process of updating a library is less painful.

The biggest pain point is writing boilerplate after starting every new project. But this is solvable by maintaining your own starter/base/boilerplate. I don’t like using boilerplates from other people because they don’t fit my mental model (but that doesn’t mean I don’t like inspecting them for fresh ideas on how to I improve my own boilerplate).

Disadvantages of using frameworks I’ve experienced so far are:

  • hard to learn
  • developing in it I often don’t feel like a [insert a programming language] developer
  • learning specifics which are in general useless instead of core programming concepts, computer science that can be reapplied to every new project.
  • using only small subset of features or fixing a bug? Too bad, you have to learn all of it so that you might not break anything.
  • having to conform to specific mindset and way of doing things because framework is designed for doing things THAT way (convention over configuration).

I only enjoyed using/learning frameworks at the beginning of my career because I didn’t exactly know what a modern applications consist of. After years of learning how to architecture an application (folder structure, design patterns etc) on my own or from popular frameworks I became confident in doing them from scratch with the help of libraries that aren’t limiting my creativeness.

Context package in Golang

Go has a package named context in its standard library. It’s responsibilities are cancelations and carrying request-scoped data (but that doesn’t mean it’s only used in HTTP handlers). I’ve learned it in two parts. For a long time I used it only for storing request-scoped data in HTTP middlewares, but recently I’ve learned the rest of it – (1) how to set up canceling context and (2) acting upon receiving canceling signal from context.

Setting up canceling context

Setting up canceling context is easy. We can use context.WitCancel(), context.WithTimeout() or context.WithDeadline(). All those functions return resulting context and cancel function. When calling, cancel function will cancel context:

  • immediatelly if created with WithCancel or
  • after elapsed time if created with WithTimeout and WithDeadline.

WithDeadline and WithTimeout differ on how to specify time when cancel function sends signal to context’s Done() channel. WithTimeout expects time.Duration while WithDeadline expects time.Time.

Another important note is that when new context is created from existing, a parent-child link between contexts is created. This allows us if parent is canceled, its childrens are canceled too. So that’s why when we are creating a context we need to specify a base context (in most cases that would be context.Background() or in HTTP handlers, r.Context() where r is of *http.Request).

Acting upon receiving canceling signal from context

Here we acknowledge contex.Done() signal and cancel function/service that could take a long time to execute and for which we are doing all this context thing.

Acknowledging is easy because for this we use built-in select with at least two cases. One case is reserved for Done() and other for our function/service singaling that it’s done (as in it’s result is sent into dedicated custom channel). If our function/service is done before canceling signal then we return its return.

If Done() is triggered first, we also need to cancel our function/service so it stops eating system’s resources. This might be the most challenging part of all. For example canceling a request requires us to get a reference to its transport instance and call its cancelRequest. If we our service/function is implemented with third-party package, explore its API if some sort of cancelation is offered.

Simple example

Below is an example of canceling a request that takes too long to complete and we’d like to cancel it after timeout. We have a server that simulates long running task by sleeping

Than we have a client which sends request to server. On line 17 we create context with WithTimeout and then defering call to cancel. Function httpDo performs the actual canceling part.

You can start client with different timeout to see how client’s response differ if timeout is greater/lower than server’s response time.

A Movie Downloader (a weekend project in Go)

Last weekend’s project I made was a CLI downloader, named A Movie Downloader (movie-dl), that searches for a movie and downloads both magnet link and a matching English subtitles. A magnet is then opened with Transmission that handles its download. Subtitles are placed in a folder where a movie will be downloaded. That meant that somehow I needed to figure it out where and how this folder will be called. I did it with downloading magnet’s metadata – extracted its name and movie filename. A name is used to create a folder which will contain resulting magnet’s files. A filename is used to rename subtitles which are downloaded as a ZIP file and extracted so that video players can detect them.

Both movie magnet and subtitles ZIP file are downloaded via requesting websites’ search URLs and then scraping for the appropriate links. Links are also requested and scraped just as a user had to click a number of times for getting to the appropriate download buttons.

Workflow BEFORE this program for getting a movie and subtitles was:

  1. visit a page with movies,
  2. click search page,
  3. enter a search term, hit enter,
  4. click on a movie that is the first search result,
  5. click download button which shows a popup where you can pick a quality that you want to watch (magnet is being downloaded then),
  6. visit a page with subtitles,
  7. enter a search term, hit enter,
  8. click on a movie that is the first search result,
  9. click a link for English subtitles that has the highest approval rating,
  10. click a download button (ZIP file is being downloaded then),
  11. extract ZIP file,
  12. rename it according to video file’s name.

Workflow for getting a movie (say La La Land) AFTER this program (movie-dl) was made and subtitles is:

  1. open up the terminal and execute movie-dl "la la land"

The number of steps in BEFORE section can be reduced if you are searching for a movie that exists. You then don’t have to visit a search page, because the first page offers search box that takes you straight to movie’s page if a movie is found in the database (you continue from the 5th and 9th step). This is probably always the case for me, I just listed all the steps that a program is actually doing. In AFTER section you don’t actually have to specify whole movie name, just enough to make it uniquely identifiable.

I coded movie downloader coded in Go. It was a pleasant experience because I didn’t have to deal with a lot of bugs. Every time I (manually) tested a functionality I’ve just coded it resulted in desired output. A reason why is probably in using Go because it offers great tools that spot trivial bugs even before you compile the code. I also didn’t have to solve all problems myself. Reading a magnet’s info was done with torrent package, searching the website for links with go-libxml2 package – I just needed to learn how to use their API and refresh my XPath skills.

This movie downloader works as expected but there’s still work to be done. I want to make it more efficient and more maintainable, efficiency being my primary focus. That’s because I love Go’s concurrency and searching for movie’s magnet and its subtitles can certainly be done in parallel. Both steps are requesting websites and scraping them so there’s a room for improvement.