How to test HTTP requests in Elixir with ExVCR
  • Elixir

How to test HTTP requests in Elixir with ExVCR

As your app grows so does its integrations base. At some point, you may need to consume API from third-party services. Today you'll learn how to test them properly with the exvcr library.

Demo app

To illustrate why testing 3-rd party services is a problem, let's create a simple wrapper within the sample Address Converter app that will return the latitude and longitude of a geographical object. We're going to consume https://nominatim.openstreetmap.org free API for this purpose.

For simplicity, let's assume that the Address Converter app is already created. To interact easily with external API let's use the HTTPoison and Jason libraries.

In mix.exs:

defp deps do
  ...
  {:jason, "~> 1.0"},
  {:httpoison, "~> 1.6"}
  ...
end

and after that:

mix deps.get

The next step is to implement Nominatim API wrapper. In lib/address_converter/nominatim.ex file, let's add few lines of code:

defmodule AddressConverter.Nominatim do
  @base_url "https://nominatim.openstreetmap.org/search?format=json&q="
  @headers [{"Content-Type", "application/json"}]

  def fetch_coordinates(query) do
    with %{ body: body } <- HTTPoison.get!(@base_url <> query, @headers),
         response <- Jason.decode!(body) do
      response
      |> Enum.map(&%{ lat: &1["lat"], lon: &1["lon"] })
      |> Enum.at(0)
    else
      _ -> %{}
    end
  end
end

As you can see we are fetching data from https://nominatim.openstreetmap.org with proper query and JSON Content-Type header to easily interact with a response later on.

Thanks to the Jason library we can decode the returned body and use Enum.map to map each result's latitude and longitude. As nominatim.openstreetmap.org lists all possible geographical places that match the given query, there might a lot of mapped coordinates. For simplicity, let's assume that we only need the first one, and that's exactly why we use Enum.at(0).

Quick demo of how it works:

iex> AddressConverter.Nominatim.fetch_coordinates("Poznan, Poland")
%{lat: "52.4082663", lon: "16.9335199"}

Testing external API problem

As our wrapper is ready to rock the world, we can proceed to the test phase.

In test/address_converter/nominatim_test.exs let's create the following test case scenario:

defmodule AddressConverter.NominatimTest do
  use ExUnit.Case, async: true

  alias AddressConverter.Nominatim

  describe "fetch_coordinates/1" do
    test "for given query it should return proper coordinates" do
      cords = %{lat: "52.4082663", lon: "16.9335199"}

      assert Nominatim.fetch_coordinates("Poznan, Poland") == cords
    end
  end
end

It's as easy as checking whether the coordinates match proper values. Let's check it out in action:

$ mix test

.

Finished in 0.4 seconds
1 test, 0 failures

Side note: If you want to make sure that our code is making a real HTTP request, turn off the internet connection and run this test once again.

Ok, our test works as intended. So what's the problem?

Well, with this solution, each time we run this test it's executing a real HTTP request to fetch the data. It's not a good idea, and there are at least five reasons why:

  • request/response cycle might take some time, you don't want to slow down your tests,
  • it might not be a free API, you may pay for requests that are being executed during tests,
  • some APIs have rate limits,
  • you need an internet connection all the time even if you didn't change test and code,
  • response for a given request in most cases should always be the same (we can mock it).

Meet exvcr library

In this case, the solution to our problem is to record the response in a file. We don't have to do it manually, there is a lib for that, and it's called exvcr.

Let's add this lib to deps:

defp deps do
  ...
  {:exvcr, "~> 0.11", only: :test}
  ...
end

As you can see we're only adding it to test dependencies, as most likely you'll not gonna use it in the dev/prod environment.

Let's fetch our new dependency:

mix deps.get

If you take a look at the documentation, you'll notice that currently exvcr works well with three HTTP clients:

  • hackney,
  • httpc,
  • ibrowse.

It means that if you're trying to test a code that makes an HTTP request using one of these clients under the hood then you're good to go. In our case, we're using HTTPoison which is built on top of Hackney.

Here is an updated test code that takes leverage of exvcr:

defmodule AddressConverter.NominatimTest do
  use ExUnit.Case, async: true
  use ExVCR.Mock, adapter: ExVCR.Adapter.Hackney

  alias AddressConverter.Nominatim

  setup do
    ExVCR.Config.cassette_library_dir("fixture/vcr_cassettes")
    :ok
  end

  describe "fetch_coordinates/1" do
    test "for given query it should return proper coordinates" do
      use_cassette "nominatim" do
        response = %{lat: "52.4082663", lon: "16.9335199"}

        assert Nominatim.fetch_coordinates("Poznan, Poland") == response
      end
    end
  end
end

Let's break it down.

First, we use the ExVCR.Mock module with a proper adapter. As already mentioned, HTTPoison uses Hackney under the hood:

use ExVCR.Mock, adapter: ExVCR.Adapter.Hackney

Next, we need to configure ExVCR cassettes path:

setup do
  ExVCR.Config.cassette_library_dir("fixture/vcr_cassettes")
  :ok
end

Once we run the test, it'll create a file with recorded HTTP request/response data and save it in the fixture/vcr_cassettes folder. Can we choose the name for this file? We can, and that's exactly what happens here:

use_cassette "nominatim" do
  ...
end

use_cassette wraps the block of code to ensure that all HTTP requests and responses will be saved into the nominatim file cassette.

Ok, since we're now familiar with ExVCR and how it can be used within a test case scenario let's run our test:

$ mix test

.

Finished in 0.7 seconds
1 test, 0 failures

It works! As you should notice, there is a new file in your repo - fixture/vcr_cassettes/nominatim.json:

[
  {
    "request": {
      "body": "",
      "headers": {
        "Content-Type": "application/json"
      },
      "method": "get",
      "options": [],
      "request_body": "",
      "url": "https://nominatim.openstreetmap.org/search?format=json&q=Poznan, Poland"
    },
    "response": {
      "binary": false,
      "body": ..., // The body is a bit long to paste it here
      "headers": {
        "Server": "nginx",
        "Date": "Thu, 29 Oct 2020 17:51:44 GMT",
        "Content-Type": "application/json; charset=UTF-8",
        "Transfer-Encoding": "chunked",
        "Connection": "keep-alive",
        "Keep-Alive": "timeout=20",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "OPTIONS,GET"
      },
      "status_code": 200,
      "type": "ok"
    }
  }
]

In the end, ExVCR runs a real HTTP request to save request and response data, and then when you run the test again, it's trying to match the pattern of request to deliver response without performing HTTP request again. If it finds a match, it returns a saved response. If there is no match, again, a real HTTP request will be performed to save data into a cassette file.

To make sure that our test doesn't depend on internet connection anymore (so there is no real HTTP request), let's turn it off and run our test once again:

$ mix test

.

Finished in 0.7 seconds
1 test, 0 failures

Great! This is how you can improve tests that depend on external services with the ExVCR cassette mechanism. There is more configuration stuff in this library, as well as little details that you may want to use. Makes sure to check the documentation.

A girl receiving new message

Sign to our Newsletter

We're committed to your privacy. Curiosum uses the information you provide to us to contact you about our relevant content. You may unsubscribe from these communications at any time. For more information, check out our privacy policy.

Szymon Soppa Web Developer
Szymon Soppa Curiosum Founder & CEO

Read more
on #curiosum blog

Phoenix LiveView Tutorial: Adding Phoenix PubSub and Pow Authentication to Messenger
  • Elixir

Phoenix LiveView Tutorial: Adding Phoenix PubSub and Pow Authentication to Messenger

We've already bootstrapped our Phoenix LiveView-based Messenger app's database structure and a first LiveView page.

This time, we're going to improve real-time communication between the app's users using Phoenix PubSub, and use the Pow library to add secure user authentication.

As of November 2020, the latest Phoenix LiveView version is 0.14.8 - and the series has been updated to match it!

5 top-tier companies that use Elixir
  • Elixir

5 top-tier companies that use Elixir

Elixir is a pretty capable language - and it consistently ranks near the top of most loved and wanted languages rankings. It has a large following and some very persuasive preachers as well. But that would not be enough to make me like it – what I need as real proof of its strengths is real businesses that strive with Elixir.

That’s what this list is all about – a bunch of stories from top companies that chose Elixir and never looked back. Let us show you how its power and versatility shows in practice.