15 Oct 2018

Hacktoberfest - Week Two

Hacktoberfest week two! Week one was all about getting into the groove of Hacktoberfest, making a small change to contribute, and finding out about other awesome projects in open source and my area of interest.

My contribution for this week had a bit more self-interest involved; I contributed a change to dplesca/purehugo, the Hugo blog theme I use for this blog, to normalize URLs that use Hugo’s .Site.BaseURL variable.

The Problem

Using the dplesca/purehugo theme, I was noticing that many links to other parts of the site - that is, links that were not external links to e.g. GitHub or Twitter - contained multiple slashes after the site root. Modern web browsers are really good at dealing with partially-malformed URLs so there was no functional problem with any of the links as the browser’s URL parser took over and cleaned them up into a value it could resolve, but it would still be better if the URLs were correctly formatted when Hugo generates the static site content.

The malformed URLs looked something like this:

<a class="pure-button" href="http://localhost:1313//index.xml">[...]</a>
<a class="post-category post-category-open source" href="http://localhost:1313//categories/open-source">open source</a>

Delving into the template code revealed how dplesca/purehugo was generating the URLs:

<a class="pure-button" href="{{ .Site.BaseURL }}/index.xml">[...]</a>
<a class="post-category post-category-{{ . }}" href="{{ $baseUrl }}/categories/{{ . | urlize }}">{{ . }}</a>

The template string used .Site.BaseURL and appended a slash before the next URL fragment. Since this results in a double slash in the generated output, I presume but have not confirmed that .Site.BaseURL always ends in a trailing slash that needs to be taken into consideration when generating URLs relative the the site’s root.

The Solution

In a similar way to how we join file paths in NodeJS with path.join(), we almost never want to be responsible for building URL paths ourselves. Luckily, Hugo provides many built-in helper functions that cover a whole range of use cases, including the one I decided to use to normalize URL paths throughout dplesca/purehugo: absURL

From the Hugo documentation, absURL

Creates an absolute URL based on the configured baseURL.

where baseURL is the hostname (and path) to the root of the site which is set as part of Hugo’s site configuration.

This function was a perfect solution in most of the places where static files need to be linked and served to the client, such as JavaScript or CSS content. In those cases it was a simple matter of removing .Site.BaseURL and the subsequent concatenation and replace it with a call to the absURL function, like so:

<a class="pure-button" href='{{ "index.xml" | absURL }}'>[...]</a>

Normalizing the post category links turned out to be a bit more complicated than this simple substitution because these links needed to include more URL fragments to create the complete path. In order to correctly concatenate and then normalize these URLs, I turned to another of Hugo’s built-in functions: printf.

Hugo’s templating system is built almost entirely on top of Go’s built-in template package, which makes things very familiar if you have worked with Go templates before. It also means that Hugo can expose a lot of Go functions as Hugo functions available inside templates. In this case, Hugo’s printf function exposes Go’s fmt.Sprintf function which works much like printf in C.

This is the solution I came up with for building the more complicated post category link URLs while still being able to normalize the resulting URL using absURL:

<a class="post-category post-category-{{ . }}" href='{{ ( printf "categories/%s" . ) | absURL }}'>{{ . }}</a>

Unlike in NodeJS, in this case it’s safe to do a simple string concatenation of categories/<category-name> because unlike with file systems, the fragment separator in URLs is always the / character.

To be Continued

I am very thankful to purehugo’s author, @dplesca, for responding so quickly to my pull request with this change and for merging the pull request within a few hours of my submission. While working on this change I noticed a few other areas where I think dplesca/purehugo could be improved, and I’m looking forward to working on it more as I continue using it to generate my blog and host my site content.

Hacktoberfest continues on, and I need to start thinking about my next contribution. My school is on a break starting next week, so I’m hoping to find the time to dig into something a bit larger and more challenging based on the projects I’ve seen so far or had recommended to me by classmates.

04 Oct 2018

Hacktoberfest - Week One

Hacktoberfest, start! In my previous post I mentioned that I was hoping to find some interesting projects to contribute to this month, and so far I’ve come up with a pretty good shortlist of contributions I want to make.

To start the month off, I lent a hand to lk-geimfari/awesomo to reorganize their list of Rust projects. They had already reorganized the one for Python and suggested contributors use that as a model for the rest of the reorganizations. There are still a bunch of languages on the list in the tracking issue, so head on over and give them a hand!

The work itself was straight forward; contributors were asked to alphabetize the projects in the list, and provide a table of contents and headings based on the example. I think this was a great first issue for Hacktoberfest because my goal for this month is to contribute to at least two projects that use Rust, and what better way to find out about cool projects than working with a list chock full of them?


Alacritty was one of the first projects to catch my eye, not just because it was originally at the top of the list, but also because I saw an announcement post that Alacritty now supports terminal scrollback which introduced me to the existence of the project in the first place. Alacritty claims to be “the fastest terminal emulator in existence”, and uses the GPU for rendering to enable optimizations that aren’t possible using other terminal emulators. Alacritty is still very much in its infancy, with version 0.2.1 as the most recent release at the time of writing. Windows support is planned before a 1.0 release, so if this intrigues you and you’re a Windows developer have a look at the tracking issue and see if you can help out.


Another project I was aware of before working with this awesome list of Rust projects is Diesel, a safe, extensible object-relational mapper (ORM) for Rust. When deciding on the technology stack for my capstone project this year, I investigated using the Rocket web framework written in Rust, and Diesel is one of the ORMs recommended in the getting started guide. Diesel provides a comfortable CLI experience for developers familiar with tools like ActiveRecord from Ruby on Rails and Sequelize in NodeJS, but drastically improves on both by leveraging Rust’s type safety and by “eliminat[ing] the possibility of incorrect database interactions at compile time.”


exa is a modern replacement for the built-in unix ls command which aims to have better defaults and more features. I have been using exa myself for a few weeks and I am really enjoying the experience so far. With ls I set up a number of shell aliases and functions to do things like enable human-readable sizes and always show colours; in exa these features are turned on automatically without any setup. I am all for customizability in those cases where it’s absolutely necessary but sane defaults is almost always a better solution, especially with a tool that most of us use hundreds of times a day.

To be Continued

I definitely have some ideas about more projects that I’d like to contribute to after seeing the lk-geimfari/awesomo list, either this month for Hacktoberfest or on an ongoing basis. If Rust isn’t your language of choice, there are similar lists of projects in pretty much every popular language as well as some more obscure ones, so take a look, be inspired, and happy hacking!

29 Sep 2018

A Return to Open Source

Today marks the first month of my return to open source. I say return even though I never really left, it’s more of a return to active participation in the community. As I wrote about earlier, one of my courses right now focuses on open source development, so I have been able to contribute much more regularly and I hope to be able to continue to do so even after the course is over.

This past month has been all about getting back into the swing of things by contributing to filerjs/filer, a port of NodeJS' fs module to the web using backends like IndexedDB. My first contribution to filer, which I have already blogged about, was to update a development dependency to resolve package vulnerabilities as reported by npm audit. This issue was logged as filer#384 and fixed in filer#385 by updating the vulnerable dependency. This issue also spawned some discussion as filer#386 about automating the process of keeping project dependencies up to date so it is much easier to stay on top of new versions of dependencies.

Open source isn’t just about contributing code and documentation though. It’s also about the community, interacting with other developers, discussing issues and potential solutions, as well as reviewing and helping out other members of the community. In addition to helping out my classmates over Slack and in-class throughout the month, I also reviewed a few pull requests opened against the filer repository.

The first issue I reviewed, filer#478, is a pull request that adds a test to fs.watch() to verify the event type emitted when renaming an existing file. Other than the boilerplate mocha test code, the test logic itself is fairly straight forward. However, fs.watch() and related watching functions are an interesting set of methods because they are not guaranteed to be consistent due to the underlying implementation details. I actually had to go back to this pull request and re-review it because it turns out that filer doesn’t currently have support for rename events at all. This issue was also interesting because I picked out a minor style nitpick in the form of a missing semi-colon that the Travis build didn’t seem to pick up for some reason. Rerunning the build resolved the issue, but it’s always worrying when you run into issues with your build infrastructure and tests because it can shake your faith in them a bit.

Another issue that I reviewed was filer#472. This pull request adds a test to ensure that reading from a nonexistent file results in an error. The submission is well done overall, with one minor nitpick that lead to me filing filer#498 to reduce small nitpick-type comments on reviews. Nitpicks are generally a waste of everyone’s time because style should be automatically enforced by some kind of automated tool before code is pushed; JavaScript doesn’t have anything quite like gofmt or rustfmt, but tools like xo and prettier can help. In this review I also cautioned the original author of the pull request about changing multiple variables at a time in a test, which we ideally would like to avoid as we can easily end up testing more code paths than we either need or want to, thereby complicating the test or making it fragile against future changes to the code.

Looking ahead, October marks the beginning of Hacktoberfest. I’m really excited to be participating this year; I participated once before but I either missed the cutoff or something happened to my package because I never received my t-shirt from DigitalOcean. I’ll be writing a lot more short blog posts throughout October as I chronicle my participation in this event, and I’m hoping to find some interesting projects to contribute to for at least the rest of this year and hopefully well into the next.

29 Sep 2018

It's Dependencies All the Way Down

NodeJS has experienced a lot of growth over the years. Like any platform, programming or otherwise, much of that growth and user adoption depends not only on the quality of the platform itself, but also the ecosystem that builds up around it.

In this respect, NodeJS has possibly one of the largest ecosystems available. With over 700,000 packages published to npm, the NodeJS platform offers developers a lot of choice when it comes to 3rd-party modules. This choice lets developers choose and compose modules from many different authors when writing software, saving time, consolidating work, and reducing bugs instead of constantly reinventing the wheel.

The NodeJS ecosystem by and large follows the Unix philosophy, in particular the primary tenant to

Write programs [and libraries] that do one thing and do it well

— Peter H. Salus, A Quarter-Century of Unix (1994)

Keeping a module’s focus small is an important aspect of many programming design patterns such as the single-responsibility principle, as well as improving code reuse and composability. However, including these modules doesn’t come for free. Every project dependency is an added burden that the maintainers need to keep up to date and have to modify their code as the API of the module changes. Sometimes serious bugs are found in these modules that can expose application secrets, user data, or unauthorized access, among other classes of vulnerabilities, that the module authors need to fix before dependent projects can update to fix the bug.

Staying on top of vulnerability disclosures and keeping track of all of the changes from version to version in a project’s dependencies is a challenging and resource-draining task. Even more challenging, authors must not only be concerned with the vulnerabilities in modules they depend on directly, but also those in all of the transitive dependencies that NodeJS is (in)famous for.

npm audit

In the April 2018 announcement of npm@6, the headlining feature the npm team showcased was npm audit, a powerful tool designed to provide automatic security warnings when using code with a known security issue. The npm audit tool allows developers to completely analyze a project’s dependency tree - including all transitive dependencies - to identify known vulnerable versions of dependencies based on data from the Node Security Platform database that npm acquired in April 2018.

The npm team made two really important decisions when designing npm audit. First, all install requests issued through npm install are automatically subjected to the same analysis that the npm audit command performs, analyzing the package to be installed as well as any additional dependencies that package would pull into the project. Second, npm audit suggests resolutions for vulnerabilities it identifies thereby giving clear actionable feedback to developers on the next steps they can take to resolve issues identified in the audit. These two decisions are critical because not only does npm take a security-conscious approach by default, but it tries to make resolving issues as painless as possible. Developers don’t want to spend hours combing through changelogs or reviewing vulnerability reports just to find the package version that includes the vulnerability fix, they want to resolve the problem and get on with writing code.

As of [email protected], npm includes the npm audit fix command to automatically run all the package installations recommended by npm audit, lowering the pain of updating vulnerable packages even further.

The npm team shares their vision with the inclusion of tools like npm audit and a security-conscious approach to ecosystem packages:

By alerting the entire community to security vulnerabilities within a tool you already use, we can make JavaScript development safer for everyone.

npm, Inc.

Auditing filer

The first thing to do after making a fresh clone of any project is to install its dependencies. After grabbing the filer git url and cloning the repository, a quick npm install pulls in all the dependencies we should need to get started.

added 1079 packages from 1531 contributors and audited 9518 packages in 36.358s
found 6 vulnerabilities (1 low, 5 moderate)
  run `npm audit fix` to fix them, or `npm audit` for details

It looks like filer@353290a has some vulnerable packages, which is a perfect opportunity to dig into the auditing process.

npm@6 is pretty in-your-face about this information, which is important. It lists the total number of vulnerabilities, as well as breaks down the number of vulnerabilities of varying severity, and provides an actionable next step: running a full audit.

Running npm audit produces the following report:

                        === npm audit security report ===

    # Run  npm install --save-dev [email protected]  to resolve 6 vulnerabilities
    SEMVER WARNING: Recommended action is a potentially breaking change
    │ Moderate      │ Memory Exposure                                              │
    │ Package       │ tunnel-agent                                                 │
    │ Dependency of │ karma [dev]                                                  │
    │ Path          │ karma > log4js > loggly > request > tunnel-agent             │
    │ More info     │ https://nodesecurity.io/advisories/598                       │

    │ Moderate      │ Prototype pollution                                          │
    │ Package       │ hoek                                                         │
    │ Dependency of │ karma [dev]                                                  │
    │ Path          │ karma > log4js > loggly > request > hawk > boom > hoek       │
    │ More info     │ https://nodesecurity.io/advisories/566                       │

    │ Moderate      │ Prototype pollution                                          │
    │ Package       │ hoek                                                         │
    │ Dependency of │ karma [dev]                                                  │
    │ Path          │ karma > log4js > loggly > request > hawk > cryptiles > boom  │
    │               │ > hoek                                                       │
    │ More info     │ https://nodesecurity.io/advisories/566                       │

    │ Moderate      │ Prototype pollution                                          │
    │ Package       │ hoek                                                         │
    │ Dependency of │ karma [dev]                                                  │
    │ Path          │ karma > log4js > loggly > request > hawk > hoek              │
    │ More info     │ https://nodesecurity.io/advisories/566                       │

    │ Moderate      │ Prototype pollution                                          │
    │ Package       │ hoek                                                         │
    │ Dependency of │ karma [dev]                                                  │
    │ Path          │ karma > log4js > loggly > request > hawk > sntp > hoek       │
    │ More info     │ https://nodesecurity.io/advisories/566                       │

    │ Low           │ Regular Expression Denial of Service                         │
    │ Package       │ timespan                                                     │
    │ Dependency of │ karma [dev]                                                  │
    │ Path          │ karma > log4js > loggly > timespan                           │
    │ More info     │ https://nodesecurity.io/advisories/533                       │

    found 6 vulnerabilities (1 low, 5 moderate) in 9518 scanned packages
    6 vulnerabilities require semver-major dependency updates.

The audit provides a lot of information, but it’s all broken down for the developer in a consistent format so even those who aren’t familiar with vulnerability mitigation or announcement procedures can understand what’s going on.

One of the most important parts of the audit output is the path. This shows how the project and the vulnerable package are related, and is a subset of the total dependency graph for the project. This is really useful if there is no clear update path for the package your project depends on directly and you need to report (or ideally, submit a pull request) that a dependency in one of your direct dependencies should update one (or more) of its packages. Without this information clearly displayed, you would spend a lot of time chasing dependencies down by manually reviewing package.json and package-lock.json files trying to figure out who needed to update their dependencies so you can update yours.

In the case of Filer we’re pretty lucky: not only are all of the vulnerable packages related to a single direct dependency, but that dependency is one that is only used during development; resolving the problems is a simple as updating to [email protected]. Since this is a simple resolution, we could either run npm audit fix and have npm take care of the version updates for us, or we can update the karma version in our package.json and install the updated package.

After updating karma, re-running npm audit to ensure we haven’t introduced any new vulnerabilities, and running the project’s tests to ensure we haven’t accidentally broken any functionality, we should take the time to contribute this fix back to the main project. The fix was small and simple, but security is everyone’s responsibility; if you notice a project using outdated or vulnerable packages, take some time to help the maintainers out by submitting a pull request. Also, projects should consider looking into automating the process of keeping project dependencies up to date through various tools like Greenkeeper and Snyk, which is a conversation you can start in your project today!

13 Sep 2018

Filesystem Events in NodeJS

I have a personal interest in automation, mostly because I believe that we should be trying to offload everything we possibly can onto computing systems so that we can spend our time thinking about interesting problems rather than manually repeating the same tasks over and over again. Good programmers, coders, and software engineers (depending on how you identify) should be trying to automate as much of their jobs as possible for all kinds of reasons: to help mitigate the bus factor, to get rid of boring and repetitive parts of their jobs, and even to better document internal and external processes (if the process has been put into code - especially well-commented code - that is a form of documentation about the process!) that can then be tracked and versioned in a version control system.

The Programming Loop

Why am I talking about automation when the title of this post is about filesystem events? Before we talk about those kinds of events, we need to take a look at a simplified but typical example of the programming loop when writing code. Such a loop might look something like the following:

  1. Make a change to some code
  2. Check the editor or IDE for feedback about the code you just wrote
  3. Run tools that check for programming errors, outright bugs, and stylistic errors
  4. Run the project’s tests (the project has tests, right?)
  5. Review tool and test feedback, and resolve issues
  6. Return to Step 1 and repeat

That’s a lot to do every time the code changes, but the tools are there to help write better code and catch a lot of little or subtle issues that people aren’t as good at noticing as computers can be. To take advantage of all of these tools and their output, we want to make this loop from writing some code to reviewing feedback as quick as possible so the programmer isn’t waiting around for feedback, gets annoyed at how long the process takes, and stops using the tools. Some projects also use a lot of different tools, and it can be a lot to remember how to run each tool individually and what command line arguments are required for the current project.

Instead of having to remember all this information, it would be great to have to remember fewer commands to get feedback from all the tools. Automation to the rescue! Instead of running each tool separately ourselves, we can use a tool that runs other tools; in this case we want to use a task runner. make is a well-known tool for automating this process, and can be used for many different kinds of projects:

C programs […] are most common, but you can use make with any programming language whose compiler can be run with a shell command. In fact, make is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change.

make manpage

Using a tool like make can simplify the loop to something like the following:

  1. Make a change to some code
  2. Run make
  3. Review tool and test feedback from each tool that make runs, and resolve issues
  4. Return to Step 1 and repeat

This significantly reduces the steps the programmer has to take every time they make a change as well as the number of commands they need to remember: all they have to do is run make! But do we really need to remember to run a command? Is there some way that we can have make or something like it run whenever we make a change to the code?

Enter: Filesystem Events

Filesystem events are a great way to trigger other tasks. Other kinds of task runners such as grunt and gulp in NodeJS, as well as guard in Ruby, let the programmer set up a process to “watch” individual files or entire directories for changes. When a file changes, these task runners pick up the change and perform whatever tasks the programmer has specified.

NodeJS core exposes this functionality to the JavaScript runtime through functions in the fs module. These functions include:

These functions respond to changes in the filesystem and communicate data about the detected changed file back to their listener functions.

A Simple Example

The following is an example of how to use the fs.watch() function in NodeJS. This is the recommended function to use because it supports single files as well as directories where fs.watchFile() only watches individual files, and it is more efficient than both fs.watchFile() and fs.unwatchFile().

'use strict' // enforce JavaScript strict mode

const fs = require('fs'); // import the core module

const path = '.' // start watching in the current directory

fs.watch(path, (eventType, filename) => {
    if (eventType === 'change') {
        console.log(`${filename} changed!`)

Some Caveats

The fs.watch() function does come with some caveats due to the state of this kind of filesystem event handling on various platforms; specifically, the API is not 100% consistent across platforms and some options are not universally available due to the underlying implementations.

Also, because fs.watch() is event-driven, there is no matching set of functions in the experimental fs Promise API introduced in Node 10.0.0 and there is no synchronous equivalent like there are for many other functions in the fs module because the task is inherently asynchronous.

However, this is still really useful functionality when working with files and filesystems, and is of particular interest in our pursuit of efficient automation.

One Last Twist

After looking a bit more into how grunt and gulp actually implement their file-watching functionality, it turns out that neither one of these popular test runners uses any of the fs.watch() functions directly and don’t even use the same dependency to provide watching functionality!

Grunt’s file-watching functionality is provided in a Grunt plugin called grunt-contrib-watch, which uses the gaze package to interact with filesystem events.

Gulp, on the other hand, relies on gulp-watch for watching files, which uses a package called chokidar.

Chokidar goes into some detail as to why fs.watch() is insufficient, and gaze lists a number of alternative projects, so there are clearly some community opinions on the quality of the built-in functions.

Some notable functionality fs.watch() lacks:

  • no wildcard support (e.g. cannot watch only for files that end with a particular extension, like *.js)
  • does not support recursive watching on platforms other than Windows or macOS (see the fs watch caveats)
  • eventType is only one of 'rename' or 'change', no support for other file events such as 'removed'

As in everything, there are trade-offs when implementing functionality. For simple use cases, fs.watch() may be sufficient and will not require including additional project dependencies. However, if file watching is a core piece of functionality, it might be worth evaluating some of the other options available on npm to find one with the necessary functionality for the project at hand, while also keeping you from tearing your hair out tracking down odd edge cases.

13 Sep 2018

Third Time's The Charm

I’ve been meaning to start blogging for a while. Not even blogging consistently, just writing some long-form-ish pieces on projects that I’ve been working on, things I’ve learned, or things that I might want for my own reference. I’ve tried at least twice before - once when I purchased this domain back in December of 2013 and again at the end of 2016 - and as evidenced by the utter lack of content it’s pretty clear I haven’t been successful.

I made a real attempt to begin publishing at the end of 2016 with the release of that year’s Advent of Code, but coursework kept me pretty busy and I never managed to finish and publish any of the posts I was working on at the time. I’ll probably revisit some of them, clean them up, and post them - if I still remember what I was writing about - and the rest I may throw into a backlog so that if I ever run into a similar topic again I can incorporate my draft.

The reason I’m giving it another try now is I’m taking a course that specifically focuses on open source and includes a requirement that the students blog about their work and experiences as they gain exposure to the open source community. I’m not exactly a stranger to open source work, but I haven’t had time to be as active as I would like to be so this course is giving me a good excuse to commit more time to getting involved and getting my hands dirty with some contributions.

On the technical side, this site is built using the static site generator Hugo. I’ve been learning a lot about the ins and outs of Hugo’s project structure as I set this blog up, but so far I’m pretty happy with the results. Hugo is written in Go, the Google-sponsored C/++ replacement, and Hugo has a very active community that keeps improving the project.

One issue that I did run into was some confusion over themes in Hugo. Each Hugo site has a config.toml that sets parameters for how the static site content should be generated. Those parameters also include information like social media links that get embedded in certain parts of the Hugo theme that structures the site content and controls the look and feel. Unfortunately, there don’t seem to be standard keys for many of these parameters, so the key name for your GitHub or StackOverflow account information needs to be updated if you decide to change your site theme. You can see some of the work I had to do in this commit to update these values when I changed my theme from hyde-x to purehugo.

I’ve been keeping my eye on other static site generators as well, but Hugo is very popular and has a lot of great community contributions and support so I’ll be using it for a while yet. Now that things are up and running I might take a stab at customizing my theme a little bit more, an opportunity I wouldn’t have if Hugo and its public themes weren’t open source. Where many static site generators use things like Handlebars or Liquid for templating, Hugo actually delegates straight to the built-in text.template package in Go. This makes it immediately familiar to most Go developers, but I haven’t had the opportunity to use Go much - especially recently - so I’ll likely be spending a lot of time reading the documentation.

Hopefully this third attempt will be the one that sticks. Stay tuned for more posts on topics including open source, NodeJS, Rust, functional programming, and anything else I might get up to.