Tag Archives: Golang

One-liner for serving static files in Go explained

Whenever I want to set up a server for serving static files in Go, which is not often, I always have to spend some time to figure out the following one-liner.

http.Handle("/images/", http.StripPrefix("/images/", http.FileServer(http.Dir("./images"))))

What confuses me probably the most here is all three strings being practically the same.

So in this post I will explain what this does for my future-self so he won’t have to google about it and will have it explained here in a way he’ll like it. 🙂

For this particular example (where one-liner above applies) let’s say we are storing images in ./images folder on a server and accessing them from outside of the server with <server's domain>/images/article2.jpeg

  • We shouldn’t set up a file server at root endpoint if our server accepts other endpoints.
  • http.FileServer(http.Dir("./images"))) exposes folder ./images as a handler.
  • http.StripPrefix is used because images are stored in ./images` folder and not in it’s subfolder images (we want ./images/article2.jpeg not ./images/images/article2.jpeg). It strips prefix from request’s URL.

A friendly reminder

Serving static files is more efficient using dedicated web server (Nginx, Apache,..). This approach also aids to better security of your application.

Error handling in concurrent programs in Golang

Error handling in concurrent programs in Go consists of little more work than

if err != nil {
   return err
}

because the return value doesn’t reach intended receiver (for example in parent function where function was fired as a goroutine using go keyword). Just as we use channel for sending resulting data, we must also use channel for sending error. And we can do this using single channel of type Data which is a struct that will hold data and error for the data. And than when we range over channel we first check if there’s an error (and when we are done, when all tasks have been completed, we close the channel).

Using same channel for data and error simplifies concurrency, we don’t have to handle scenarios of completed tasks end error handling separately.

So here’s a simple example of what I have in mind (with sending HTTP request as a concurrent task).

type Data struct {
    URL      string
    IsAvailable bool
    Err      error
}

func getAvailableSites(urls []string) ([]string, error) {
    var wg sync.WaitGroup
    urlChan := make(chan *Data)

    go func() {
        wg.Wait()
        close(urlChan)
    }()

    for _, url := range urls {
        go checkIfSiteAvailable(&wg, urlChan, url)
    }

    availableSites := []string{}
    for data := range urlChan {
        if data.Err != nil {
            return nil, fmt.Errorf("problem getting url: %v", data.Err)
        }
        if (data.IsAvailable) {
            availableSites = append(availableSites, data.URL)
        }
    }
    return availableSites, nil
}

func checkIfSiteAvailable(wg *sync.WaitGroup, urlChan chan<- *Data, url string) {
    defer wg.Done()

    resp, err := http.Head(url)
    if err != nil {
        urlChan <- &Data{Err: err}
        return
    }

    if resp.StatusCode != http.StatusOK {
        urlChan <- &Data{Err: fmt.Errorf("status code not OK for %s", url)}
        return
    }

    urlChan <- &Data{URL: url, Available: true}
}

I won’t be describing logic here just a steps related to concurrency and error handling in it. In getAvailableSites we

  • set up a channel we’ll be using for sending the data for specific URL (with error as part of data),
  • a WaitGroup to specify how many goroutines we’ll fire and after all finish, Go will know to close channel,
  • define and fire a goroutine which waits for all goroutines to finish and than close a channel,
  • for each URL we call a function checkIfSiteAvailable as a goroutine (alonside URL we pass channel so we can send resulting data to it and a WaitGroup to be able to indicate task is finished)
  • iterate over a channel, which will finish after we close the channel (from 3rd point). Iterating over a channel is a blocking operation so we don’t have to worry data from channel will get lost. In for loop we check for error and return it as if no concurrency was involved.

In checkIfSiteAvailable we defer call to wg.Done to indicate task is finished. And whenever there’s an error we send instance of Data with only Err field specified because in this simple example we don’t need other fields. Lastly, we send the actual data (wrapped in the Data struct).

Libraries over frameworks

Lately (in term of years) I noticed I prefer using libraries over frameworks. Or in other words I like to use packages that each do their one thing well rather than one package with all batteries included. I recognize both libraries and frameworks have their pros and cons.

You can see my preference for libraries if you are reading my blog where I write a lot about web development, specifically about React and Go. Well, Go is a programming language but because it has an excellent standard library, there’s actually no need for a framework.

The most obvious advantage of using libraries over frameworks I have are:

  • I have a freedom of putting all pieces of an application as I see fit and only those necessarily.
  • I can easily replace one piece after another, better option appears.
  • they take less time to learn.
  • feeling I have a control over them not other way around.
  • process of updating a library is less painful.

The biggest pain point is writing boilerplate after starting every new project. But this is solvable by maintaining your own starter/base/boilerplate. I don’t like using boilerplates from other people because they don’t fit my mental model (but that doesn’t mean I don’t like inspecting them for fresh ideas on how to I improve my own boilerplate).

Disadvantages of using frameworks I’ve experienced so far are:

  • hard to learn
  • developing in it I often don’t feel like a [insert a programming language] developer
  • learning specifics which are in general useless instead of core programming concepts, computer science that can be reapplied to every new project.
  • using only small subset of features or fixing a bug? Too bad, you have to learn all of it so that you might not break anything.
  • having to conform to specific mindset and way of doing things because framework is designed for doing things THAT way (convention over configuration).

I only enjoyed using/learning frameworks at the beginning of my career because I didn’t exactly know what a modern applications consist of. After years of learning how to architecture an application (folder structure, design patterns etc) on my own or from popular frameworks I became confident in doing them from scratch with the help of libraries that aren’t limiting my creativeness.

Context package in Golang

Go has a package named context in its standard library. It’s responsibilities are cancelations and carrying request-scoped data (but that doesn’t mean it’s only used in HTTP handlers). I’ve learned it in two parts. For a long time I used it only for storing request-scoped data in HTTP middlewares, but recently I’ve learned the rest of it – (1) how to set up canceling context and (2) acting upon receiving canceling signal from context.

Setting up canceling context

Setting up canceling context is easy. We can use context.WitCancel(), context.WithTimeout() or context.WithDeadline(). All those functions return resulting context and cancel function. When calling, cancel function will cancel context:

  • immediatelly if created with WithCancel or
  • after elapsed time if created with WithTimeout and WithDeadline.

WithDeadline and WithTimeout differ on how to specify time when cancel function sends signal to context’s Done() channel. WithTimeout expects time.Duration while WithDeadline expects time.Time.

Another important note is that when new context is created from existing, a parent-child link between contexts is created. This allows us if parent is canceled, its childrens are canceled too. So that’s why when we are creating a context we need to specify a base context (in most cases that would be context.Background() or in HTTP handlers, r.Context() where r is of *http.Request).

Acting upon receiving canceling signal from context

Here we acknowledge contex.Done() signal and cancel function/service that could take a long time to execute and for which we are doing all this context thing.

Acknowledging is easy because for this we use built-in select with at least two cases. One case is reserved for Done() and other for our function/service singaling that it’s done (as in it’s result is sent into dedicated custom channel). If our function/service is done before canceling signal then we return its return.

If Done() is triggered first, we also need to cancel our function/service so it stops eating system’s resources. This might be the most challenging part of all. For example canceling a request requires us to get a reference to its transport instance and call its cancelRequest. If we our service/function is implemented with third-party package, explore its API if some sort of cancelation is offered.

Simple example

Below is an example of canceling a request that takes too long to complete and we’d like to cancel it after timeout. We have a server that simulates long running task by sleeping

Than we have a client which sends request to server. On line 17 we create context with WithTimeout and then defering call to cancel. Function httpDo performs the actual canceling part.

You can start client with different timeout to see how client’s response differ if timeout is greater/lower than server’s response time.

A Movie Downloader (a weekend project in Go)

Last weekend’s project I made was a CLI downloader, named A Movie Downloader (movie-dl), that searches for a movie and downloads both magnet link and a matching English subtitles. A magnet is then opened with Transmission that handles its download. Subtitles are placed in a folder where a movie will be downloaded. That meant that somehow I needed to figure it out where and how this folder will be called. I did it with downloading magnet’s metadata – extracted its name and movie filename. A name is used to create a folder which will contain resulting magnet’s files. A filename is used to rename subtitles which are downloaded as a ZIP file and extracted so that video players can detect them.

Both movie magnet and subtitles ZIP file are downloaded via requesting websites’ search URLs and then scraping for the appropriate links. Links are also requested and scraped just as a user had to click a number of times for getting to the appropriate download buttons.

Workflow BEFORE this program for getting a movie and subtitles was:

  1. visit a page with movies,
  2. click search page,
  3. enter a search term, hit enter,
  4. click on a movie that is the first search result,
  5. click download button which shows a popup where you can pick a quality that you want to watch (magnet is being downloaded then),
  6. visit a page with subtitles,
  7. enter a search term, hit enter,
  8. click on a movie that is the first search result,
  9. click a link for English subtitles that has the highest approval rating,
  10. click a download button (ZIP file is being downloaded then),
  11. extract ZIP file,
  12. rename it according to video file’s name.

Workflow for getting a movie (say La La Land) AFTER this program (movie-dl) was made and subtitles is:

  1. open up the terminal and execute movie-dl "la la land"

The number of steps in BEFORE section can be reduced if you are searching for a movie that exists. You then don’t have to visit a search page, because the first page offers search box that takes you straight to movie’s page if a movie is found in the database (you continue from the 5th and 9th step). This is probably always the case for me, I just listed all the steps that a program is actually doing. In AFTER section you don’t actually have to specify whole movie name, just enough to make it uniquely identifiable.

I coded movie downloader coded in Go. It was a pleasant experience because I didn’t have to deal with a lot of bugs. Every time I (manually) tested a functionality I’ve just coded it resulted in desired output. A reason why is probably in using Go because it offers great tools that spot trivial bugs even before you compile the code. I also didn’t have to solve all problems myself. Reading a magnet’s info was done with torrent package, searching the website for links with go-libxml2 package – I just needed to learn how to use their API and refresh my XPath skills.

This movie downloader works as expected but there’s still work to be done. I want to make it more efficient and more maintainable, efficiency being my primary focus. That’s because I love Go’s concurrency and searching for movie’s magnet and its subtitles can certainly be done in parallel. Both steps are requesting websites and scraping them so there’s a room for improvement.