Let's Build a Thing: NPM Dependency Checker - Part 4

Picture shows Jason A. Martin - Software Engineer, Indie Game Developer, Tech Evangelist, Entrepreneur.

note: Series start along with dev notes: Let's Build: NPM Dependency Checker

Previous part: Let's Build: NPM Dependency Checker - Part 3

Core Coding

GitHub: NPM Dependency Checker branch: testing-and-refactor

So at this point we have a "working" application and we've coded many core functions.

The main thing we're missing is testing. We should have at least a few basic tests so we know our functions generally operate the way we expect. Ideally we should have many tests that thoroughly test each function, but for this series I plan on doing basic testing.

Additionally, we should probably add some brief documentation to each function. Let's get going on this right now.

Pre-Test Refactor

Before we get to the tests, let's do a little bit of refactoring for the code we do have. For starters, we should put in a documentation line for each function.


To help ensure our function works as intended, let's put a guard up. We will only use the main function when the argument being passed is a map. For all others, we will send them to a function that returns false.

We will also put in some @doc information.

@doc """
Iterates through a map and prints out the key:value pairs
Returns `:ok`.
def iterate_dependencies(map) when is_map(map) do
  ## More work to do.
  Enum.map(map, fn {k, v} -> IO.inspect [k,v] end)

@doc """
Returns false when a non-map argument is passed in
def iterate_dependencies(_anything_else), do: false


We have a similar situation for the npm_view function. If we were to pass in something other than a string (binary), the function will break. Let's add another guard along with another catch-all-else function.

@doc """
Takes a String and runs the system command "npm view <package>"
Returns: the output of that command
def npm_view(package) when is_binary(package) do
  System.cmd("npm", ["view", package])

@doc """
Handles all non-string arguments and returns false since we're not doing anything with it
def npm_view(_anything), do: false


For fetch_package_json we are going to modify it to raise an exception in the case of the response not being a 200 or 404.

@doc """
Fetches a url, which should be a JSON file and sends it to decode_body for decoding to a map
def fetch_package_json(repo) do
  case HTTPoison.get(repo) do
    {:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
    {:ok, %HTTPoison.Response{status_code: 404}} ->
      "Your princess is in another castle."
    {:error, %HTTPoison.Error{reason: reason}} ->
      raise reason


Again we need to add a function guard.

@doc """
Takes a GitHub url and transforms it to point to the package.json in the master branch
def transform_repo_raw_json_url(repo) when is_binary(repo) do
  ## we will do a little replace therapy to fetch the raw json.
  ## end result is: https://raw.githubusercontent.com/someuser/somerepo/master/package.json
  String.replace(repo, "#readme","")
    |> String.replace("github","raw.githubusercontent")
    |>  (fn x -> x <> "/master/package.json" end).()

@doc """
Returns false when a non-binary is passed in
def transform_repo_raw_json_url(_anything_else), do: false

That does it for the refactor changes for now.


With our function refactored with guarding, we can now create some simple tests.

Rather than paste out all of the tests, I will just do one. Please see the branch for this part in the repo for a full listing of tests I created.

You can run the tests by running the following in your project directory:

$ mix test

The test below will check that the get_package_repo_url does take the output from our npm_view function and returns the appropriate URL for the repo.

test "get_package_repo_url: returns a url when used in conjunction with npm_view" do
  assert Ndc.get_package_repo_url(Ndc.npm_view("microlibrary")) == "https://github.com/JasonAMartin/microlibrary#readme"

Continuing Refactor

We now have documentation, tests setup and our code is a little more bulletproof. Let's turn our attention to a major issue.

As I was testing the application, I noticed that when I ran it with --pkg=eslint the application blew up. Why? It turns out that grabbing the "homepage" key from npm view is a bad idea. In this case (and others), the homepage link is to a main site and not GitHub.

Therefore, we're going to refactor to grab the repository.url key and work with that.


We're going to add an attribute for pulling out the repository node in the JSON response. Put this at the top of the file with the other attribute.

@npm_view_fields ~w(

NPM view

Next, we need to adjust out call to npm view so that the response is in JSON. This is simply done by adding --json to our Systen.cmd call.

def npm_view(package) when is_binary(package) do
    System.cmd("npm", ["view", "--json", package])


Next we're going to refactor the decode_body function to take an addition argument so we can pass in the attribute we want since the function will be used in multiple places.

def decode_body(body, fields) do
  IO.inspect body
  |> Poison.decode!
  |> Map.take(fields)
  |> Enum.map(fn({k, v}) -> {String.to_atom(k), v} end)


We need to alter the url in a different way now, so transform_repo_raw_json_url needs to be refactored. Additionally, we're now going to pass in a List instead of a binary, so we'll update the guard too.

The new function looks like this:

def transform_repo_raw_json_url(repo) when is_list(repo) do
  ## we will do a little replace therapy to fetch the raw json.
  ## expected: git+https://github.com/user/repo.git
  ## end result is: https://raw.githubusercontent.com/someuser/somerepo/master/package.json
  String.replace(repo[:repository]["url"], "git+","")
    |> String.replace(".git", "")
    |> String.replace("github","raw.githubusercontent")
    |>  (fn x -> x <> "/master/package.json" end).()

Cleanup Pipe

After this refactor, we're left with a function that we no longer need (get_package_repo_url). We also need to add something to our main pipe. When the dust settles, here's what it looks like:

      |> npm_view
      |> elem(0)
      |> decode_body(@npm_view_fields)
      |> transform_repo_raw_json_url
      |> fetch_package_json
      |> parse_dependencies

Testing Again

After all the refactoring, we need to re-run our tests and potential add new ones.

  1. Remove get_package_repo_url tests.

  2. Update test for decode_body to pass the additional argument.

  3. Update test for transform_repo_raw_json_url to pass list instead of string.

I also took out one additional test for now.

We're at a good stopping point for this part. In the next part we'll start working on refactoring the parse_dependencies function so that it iterates fully and we'll begin to explore how we're going to tackle the issue of drilling down until there are no repositories left to check.

Let's continue forward: Let's Build: NPM Dependency Checker - Part 5