Testing in production: using JSON Schema for 3rd party API response validation

..ensuring sound mocks and supplying Typescript types.

Submotion offers a central place to manage SaaS accounts and subscriptions. This is made possible by connecting to a bunch of third party API’s. It’s an amazing fact that so many companies expose API’s that allow you to integrate with their service, to mutual benefit. Using such API’s is fairly trivial and even though things like OAuth implementations quite often diverge from the standard, it’s usually not too hard to set up.

However, testing such integrations is painful.

Proper end-to-end testing is not feasible in a CI setup. It’s not only slow and relies on access to outside servers that could be down, it’s also next to impossible to set up useful fixtures.

Initially I tried to set up a record/playback system to alleviate this. That works well in some situations, but Submotion almost always connects via OAuth which is difficult to automate (by design), and even then, I still had the issue with fixtures. I would want to test that Submotion detects, say, that an account has been deleted. In order to do that, I had to actually create and subsequently delete an account in Slack or whatever, a very involved task.
So I dropped that approach and for a while I just accepted that mocks were the best I could do. And in fact, I am still writing manual mocks, but I did manage to improve the situation.

I was discussing this with my friend Lars who has put a lot of thought into these and similar issues. He suggested that I move the assertions from the test into production code. Now, I remember doing this sort of thing a lot in C++ where assertions are common in the debug build and then removed in production. However, this didn’t feel like a Node-native approach and it rubbed me the wrong way somehow. But, as he pointed out, such assertions could just log warnings in production, and when running locally or via tests, they could throw. That way, I would continuously run validation on real responses meaning that said validations are kept up to date and good enough to keep my mock data sound. Although this is strictly validating, it still feels to me like a variation on the parse, don’t validate philosophy coined by Alexis King. And while it’s still work in progress, so far it’s a big step forward.

Here is what I do in practice:

  1. Make a request or two to the 3rd party API in question, and save the JSON responses in a file. This is the sketch.
  2. Convert these into JSON Schemas using a tool like https://transform.tools/json-to-json-schema
  3. Clean up the schemas. They can rarely be inferred correctly from a single response, so some tweaking is required. Notably, I add format fields for things like emails, URLs or IP addresses where applicable.
  4. Then, use AJV (with ajv-formats) to validate the responses right after making a request. If validation fails, log an error or, if running tests, throw an exception
  5. Finally, use as-typed with the schema to get a typed response. It’s not perfect but much nicer than an untyped one and I can be confident that my runtime validation is aligned with the type check.

Now, I could replace JSON Schema validation with something like io-ts or Zod which I am playing with, but this is what I am doing for now. Here is an example and a simplified version of the approach:

Of course, schema validation/type checking only looks at the shape of the responses which means there is still lots of room for semantic errors. There is nothing stopping me from expanding this setup with more semantic assertions which would further improve the quality but I haven’t really found my sweet spot for that yet. Perhaps in a future post.




Bit of a geek..

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Reading 02: See You In My Nightmares

Cloud Native London 2019


Issues with deploying a Spring Boot application onto JBoss

Creating a ledge grab mechanic | Unity

Game Servers in Google Cloud with Valheim and Kubernetes

AMP for E-commerce — Intro & Setup

Best Azure Optimization Tools for 2022

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Kristian Dupont

Kristian Dupont

Bit of a geek..

More from Medium

Why GraphQL is better than REST API?

CORS policy Error In Working With Graphql, ReactJs,Apollo client.

Authentication & Authorization, Hashing in Web Applications

Micro Frontends in Action: Architecture and Integration Approaches