Keep on Mocking with a Key, Girrrrl

(with apologies to Neil Young) tl;dr - a story is told about how the author tests a module against a third-party web API when that service is not always available and without leaking sensitive authentication tokens You find yourself to be an aspiring CPAN author of a web API and as a righteous follower of Test Driven Development you want to write tests to verify that your API works as advertised. Testing is a big part of Perl culture, so a skeleton module usually comes with a t/ directory to hold your tests. Once uploaded to CPAN, the CPAN Testing Service will run your tests on every OS and version of Perl possible, but you shouldn't require an active network connection on either end. After all, has your API failed because the end service is down for annual maintenance or the local internet company van has turned up on the CPANTS volunteer's street? No, of course not! So you mock the service. Time to Mock and Roll When things get difficult to test, you could run up a tiny working version of the object in question or intercept the calls your module makes to the object or module and return simulated responses. A mock is a bit like a cardboard cutout of what you want to test. It looks just like the real thing ... from the right angle. Here's a longer explanation for the curious. There are a few different ways of mocking in Perl. I really like LWP::UserAgent::Mockable for testing web services. It lets you record a live version of the network conversation and "playback" the response afterwards, so you don't need that connection anymore. This module runs on environment variables so you set up your defaults in a BEGIN block. My recorded filename default is .mock.out tacked on to the end of the test file name in the same directory. There's an option of skipping the mock with the LWP_UA_MOCK=passthrough option. You'll need that when you add a new network query that you haven't recorded yet. Having been inspired by this post, you'll now run off and add L::U::Mockable to all your tests and record them. Go look at the .mock.out files. They're all the plain text-ish traffic to and from the service. Here are some of mine. But what if all you see in the file is only this? pt0 Go and check the UserAgent. If instead of the standard Perl web module, LWP, you're using Mojolicious, the mock has just been sitting there twiddling its thumbs. You'll need to use Mojo::UserAgent::Mockable instead, but don't despair! You haven't lost all that effort getting L::U::Mockable to work. Mojo::UserAgent::Mockable has a mode=lwp-ua-mock option to make it behave the same way the LWP module does. You can even remove the END block which you don't need now. Wait, what key are we in? Some APIs are for services that restrict access to registered users to prevent resource abuse. To access them requires an authorisation token or developer key. Ooops! When you looked at the .mock.out files, did you see your SECRET_DEV_KEY in the Authorisation header? Well you certainly don't want to upload that to a public repository! Scrub the .mock.out files with something like this substitution. s/Bearer \w{10,}/Bearer TOKEN_REMOVED/g and set the Mockable option ignore-headers because our recorded test doesn't care about actually authenticating. You can still test how your API handles an authorisation failure by recording this subtest with local $ENV{SECRET_DEV_KEY} = 'BAD'; Don't forget to run your scrub_mock_headers.pl script every single time before committing the recorded mocks. You don't want that Key getting out "in the wild" for naughty children to misuse. The Hook brings you back Or should I say DO forget about running the script, because you're going to save it as .git/commit/pre-commit, and maybe check that there isn't an existing pre-commit hook already. A git commit hook will run every time you commit so you can get on with coding that API. Just make sure the pre-commit file is executable. Try it with a minor commit before you commit the mocks and look for any error messages during the commit process. Happy Mocking! Image remixed from "Neil Young, Heart of Gold" by Stoned59 and "Neil Young (Crazy Horse) + Sonic Youth + Social Distorion May 15, 1991" by Howdy, I'm H. Michael Karshis , licensed under CC BY 2.0. The Perl logo is Copyright (c) 2024 Olaf Alders, licensed under the CC-BY License, Version 4.0.

May 7, 2025 - 10:32

(with apologies to Neil Young)

tl;dr - a story is told about how the author tests a module against a third-party web API when that service is not always available and without leaking sensitive authentication tokens

You find yourself to be an aspiring CPAN author of a web API and as a righteous follower of Test Driven Development you want to write tests to verify that your API works as advertised. Testing is a big part of Perl culture, so a skeleton module usually comes with a t/ directory to hold your tests.

Once uploaded to CPAN, the CPAN Testing Service will run your tests on every OS and version of Perl possible, but you shouldn't require an active network connection on either end. After all, has your API failed because the end service is down for annual maintenance or the local internet company van has turned up on the CPANTS volunteer's street?

No, of course not! So you mock the service.

Time to Mock and Roll

When things get difficult to test, you could run up a tiny working version of the object in question or intercept the calls your module makes to the object or module and return simulated responses. A mock is a bit like a cardboard cutout of what you want to test. It looks just like the real thing ... from the right angle. Here's a longer explanation for the curious.

There are a few different ways of mocking in Perl. I really like LWP::UserAgent::Mockable for testing web services. It lets you record a live version of the network conversation and "playback" the response afterwards, so you don't need that connection anymore. This module runs on environment variables so you set up your defaults in a BEGIN block.

My recorded filename default is .mock.out tacked on to the end of the test file name in the same directory. There's an option of skipping the mock with the LWP_UA_MOCK=passthrough option. You'll need that when you add a new network query that you haven't recorded yet.

Having been inspired by this post, you'll now run off and add L::U::Mockable to all your tests and record them. Go look at the .mock.out files. They're all the plain text-ish traffic to and from the service. Here are some of mine.

But what if all you see in the file is only this?

pt0

Go and check the UserAgent. If instead of the standard Perl web module, LWP, you're using Mojolicious, the mock has just been sitting there twiddling its thumbs. You'll need to use Mojo::UserAgent::Mockable instead, but don't despair! You haven't lost all that effort getting L::U::Mockable to work. Mojo::UserAgent::Mockable has a mode=lwp-ua-mock option to make it behave the same way the LWP module does. You can even remove the END block which you don't need now.

Wait, what key are we in?

Some APIs are for services that restrict access to registered users to prevent resource abuse. To access them requires an authorisation token or developer key.

Ooops! When you looked at the .mock.out files, did you see your SECRET_DEV_KEY in the Authorisation header? Well you certainly don't want to upload that to a public repository!

Scrub the .mock.out files with something like this substitution.

s/Bearer \w{10,}/Bearer TOKEN_REMOVED/g

and set the Mockable option ignore-headers because our recorded test doesn't care about actually authenticating.

You can still test how your API handles an authorisation failure by recording this subtest with

    local $ENV{SECRET_DEV_KEY} = 'BAD';

Don't forget to run your scrub_mock_headers.pl script every single time before committing the recorded mocks. You don't want that Key getting out "in the wild" for naughty children to misuse.

The Hook brings you back

Or should I say DO forget about running the script, because you're going to save it as .git/commit/pre-commit, and maybe check that there isn't an existing pre-commit hook already. A git commit hook will run every time you commit so you can get on with coding that API. Just make sure the pre-commit file is executable. Try it with a minor commit before you commit the mocks and look for any error messages during the commit process.

Happy Mocking!

Image remixed from "Neil Young, Heart of Gold" by Stoned59 and "Neil Young (Crazy Horse) + Sonic Youth + Social Distorion May 15, 1991" by Howdy, I'm H. Michael Karshis , licensed under CC BY 2.0. The Perl logo is Copyright (c) 2024 Olaf Alders, licensed under the CC-BY License, Version 4.0.