I recently wrote a simple CLI tool called Tiwi that could convert Markdown files to HTML pages. I borrowed some ideas from how modern frameworks like Angular, React or Django support hot reload. This would allow Tiwi to watch a project directory and generate HTML pages on the fly as you wrote your Markdown.

tiwi live reload demo

Fsnotify is a Golang library which provides file system notifications. This means it enables you to watch a directory and receive notifications of different file system operations which occur in that directory i.e CREATE, WRITE, DELETE, RENAME etc

This is useful as it provides Tiwi with the ability to watch a directory and when a new file is created or updated, the method to generate HTML from Markdown is executed.

Fsnotify makes uses of channels to send these file system notifications as events. For instance, when a file is created, a CREATE event is sent through the channel.

The library provides an events channel through which file system events that occur in a specific directory are sent.

But what are channels?

A channel is a data structure that provides a way to communicate within a program. They can be thought of as a pipe through which data can be passed through. In built support for channels within programming languages is rare as only Golang, Kotlin and Clojure ship with channels.

messages := make(chan string)

Channels are defined using the make and chan keyword. A type is also specified. The type represents the data-type of which is expected to be passed through the channel

Channels support two primary operations: sending and receiving.

// sending to a channel
messages <- "hello"

// receiving from a channel
// and storing the read data in a variable
msg :=  <- messages

fmt.Println(msg) // prints "hello" to the terminal

Receiving from a channel blocks until there is data available to be read.

events := make(chan string)

// this line will block execution 
// since it is attempting to read a channel 
// yet no data has been sent through the channel
msg := <- events

My goal was to implement hot reload within Tiwi. Below is a snippet of how Fsnotify can be used to watch directories (actual implementation in Tiwi)

directory := "./"
watcher, err := fsnotify.NewWatcher()

if err != nil {
  fmt.Printf("failed to create file watcher: %v", err)
  os.Exit(1)
}

defer watcher.Close()

done := make(chan bool)

go func() {
  for {
    select {
      case event := <-watcher.Events:
        log.Println(event)
      case err := <-watcher.Errors:
        log.Println(err)
    }
  }
}()

if err := watcher.Add(directory); err != nil {
  fmt.Printf("failed to watch directory (%s): %v", directory, err)
}
<-done

Let’s break this down.

First, define the directory or path you want to watch for changes. Then fsnotify is initialized with the NewWatcher() method. It is also a good practice to defer the method which stops watching the directory.

directory := "./"
watcher, err := fsnotify.NewWatcher()

if err != nil {
  fmt.Printf("failed to create file watcher: %v", err)
  os.Exit(1)
}

defer watcher.Close()

In this section there are three interesting things that happen.

done := make(chan bool)

go func() {
  for {
    select {
      case event := <- watcher.Events:
        log.Println(event)
      case err := <- watcher.Errors:
        log.Println(err)
    }
  }
}()

if err := watcher.Add(directory); err != nil {
  fmt.Printf("failed to watch directory (%s): %v", directory, err)
}

<-done

A channel called done is created with the type of boolean. Remember earlier I mentioned that, the receive operation on channels blocks execution until data is available to be read. For Tiwi to watch the given directory indefinitely, I used this feature of channels. At the bottom of the snippet there is an attempt to read the value from the done channel yet no data has been sent to it. This causes execution on the main thread to block.

Fsnotify is then provided with the directory (path) to watch for events using the watcher.Add() method.

go func() {
  for {
    select {
      case event := <- watcher.Events:
        log.Println(event)
      case err := <- watcher.Errors:
        log.Println(err)
      }
  }
}()

In the above snippet, an infinite for loop is ran inside a goroutine. The infinite loops makes it possible to listen for events continuously.

go func() {
  select {
    case event := <- watcher.Events:
      log.Println(event)
    case err := <- watcher.Errors:
      log.Println(err)
    }
}()

If the for loop wasn’t used, this code would only catch the first event or error to be sent.

Within the for loop, select is used to receive values from two channels provided by Fsnotify: Event channel and Errors channel.

The infinite loop is defined inside a goroutine so as to not block the main thread of execution. In Go, the main function runs in a what’s referred to as the main thread. The program exits once the main thread completes execution.

Goroutines is a broad topic which I might write about later on, but in summary, they are a lightweight thread also known as a green thread. Instead of being scheduled by the operating system, they are scheduled by the Go runtime. This makes it more performant since context switching green threads is computationally cheaper than if they were actual threads.

gorutines in go

The fact that the logic for listening for file system events is in a goroutine, the program would exit since it only waits for the main thread to finish execution. This is why the done channel is used to block the main thread, preventing the program from exiting.

select {
  case event := <- watcher.Events:
    log.Println(event)
  case err := <- watcher.Errors:
    log.Println(err)
}

The ability to listen to multiple channels at once is an incredibly useful feature of Go. In the above snippet, select is used to literally ‘select’ values from channels. Whichever channel sends out data first, the respective case will be triggered and the value will be logged on the console. In this scenario, either a file system event (CREATE REMOVE WRITE) will be sent or an error event.

To test out this snippet of watching for file system events, ran the code and create a file on the same directory. You’ll notice a number of events logged to the terminal.

This is a sample output of when the following actions are taken:

  • Create a file named ‘demo.txt’
  • Type “Hello World” on the file and save it
➜ go run main.go

2023/06/20 10:55:15 CREATE   "./demo.txt"
2023/06/20 10:55:17 WRITE    "./demo.txt"
2023/06/20 10:55:17 WRITE    "./demo.txt"

As you can see there is a strange behavior. Only two operations were performed: creating of a file, and editing of a file. However, the program received three events. The WRITE event was fired twice at the same time. VsCode code editor was used to create and edit the file.

touch demo.txt

nano demo.txt

Using the terminal to create a file and write to it showcases an even more interesting behavior.

➜ go run main.go

2023/06/20 10:59:46 CREATE  "./demo.txt"
2023/06/20 10:59:46 CHMOD   "./demo.txt"
2023/06/20 11:00:02 CREATE  "./.demo.txt.swp"
2023/06/20 11:00:02 WRITE   "./.demo.txt.swp"
2023/06/20 11:00:04 REMOVE  "./.demo.txt.swp"
2023/06/20 11:00:04 CREATE  "./.demo.txt.swp"
2023/06/20 11:00:04 WRITE   "./.demo.txt.swp"
2023/06/20 11:00:10 WRITE   "./demo.txt"
2023/06/20 11:00:10 WRITE   "./demo.txt"
2023/06/20 11:00:10 REMOVE  "./.demo.txt.swp"

Depending on the editor used, lots of events from the file system can be emitted. Though notice the WRITE event is also called twice when using the nano shell editor. This behavior is called atomic save. The Sublime text editor used to ship with atomic save enabled by default.

2023/06/20 11:00:10 WRITE   "./demo.txt"
2023/06/20 11:00:10 WRITE   "./demo.txt"

The idea is that when a change is made to a file, a copy is created with the new content and then the old file is deleted. This happens in the background and results in the double firing of the WRITE event.

Double firing of the WRITE event can also be due to the underlying implementation of the editor used.

This can lead to inefficiency if your program relies on the WRITE event to perform some task.

select {
  case event := <-watcher.Events:
    if event.Op == fsnotify.Write:
      // doSomething()
  case err := <-watcher.Errors:
    log.Println(err)
}

For Tiwi, the WRITE event is used to trigger the rebuild of the HTML pages from Markdown files. Depending on the number of markdown files, it could get expensive performing that operation twice. I had to figure out a way to filter out the extra event to ensure the operation is only carried out once.That’s where Timers came in handy.

Thanks for sticking around. Next, I’ll dive into what timers are in Go and how I used them in Tiwi to prevent HTML files from being generated twice.

Cheers.