Testing in production: using JSON Schema for 3rd party API response validation

..ensuring sound mocks and supplying Typescript types.

Submotion offers a central place to manage SaaS accounts and subscriptions. This is made possible by connecting to a bunch of third party API’s. It’s an amazing fact that so many companies expose API’s that allow you to integrate with their service, to mutual benefit. Using such API’s is fairly trivial and even though things like OAuth implementations quite often diverge from the standard, it’s usually not too hard to set up.

However, testing such integrations is painful.

Proper end-to-end testing is not feasible in a CI setup. It’s not only slow and relies on access to outside servers that could be down, it’s also next to impossible to set up useful fixtures.

Initially I tried to set up a record/playback system to alleviate this. That works well in some situations, but Submotion almost always connects via OAuth which is difficult to automate (by design), and even then, I still had the issue with fixtures. I would want to test that Submotion detects, say, that an account has been deleted. In order to do that, I had to actually create and subsequently delete an account in Slack or whatever, a very involved task.
So I dropped that approach and for a while I just accepted that mocks were the best I could do. And in fact, I am still writing manual mocks, but I did manage to improve the situation.

I was discussing this with my friend Lars who has put a lot of thought into these and similar issues. He suggested that I move the assertions from the test into production code. Now, I remember doing this sort of thing a lot in C++ where assertions are common in the debug build and then removed in production. However, this didn’t feel like a Node-native approach and it rubbed me the wrong way somehow. But, as he pointed out, such assertions could just log warnings in production, and when running locally or via tests, they could throw. That way, I would continuously run validation on real responses meaning that said validations are kept up to date and good enough to keep my mock data sound. Although this is strictly validating, it still feels to me like a variation on the parse, don’t validate philosophy coined by Alexis King. And while it’s still work in progress, so far it’s a big step forward.

Here is what I do in practice:

1. Make a request or two to the 3rd party API in question, and save the JSON responses in a file. This is the sketch.
2. Convert these into JSON Schemas using a tool like https://transform.tools/json-to-json-schema
3. Clean up the schemas. They can rarely be inferred correctly from a single response, so some tweaking is required. Notably, I add format fields for things like emails, URLs or IP addresses where applicable.
4. Then, use AJV (with ajv-formats) to validate the responses right after making a request. If validation fails, log an error or, if running tests, throw an exception
5. Finally, use as-typed with the schema to get a typed response. It’s not perfect but much nicer than an untyped one and I can be confident that my runtime validation is aligned with the type check.

Now, I could replace JSON Schema validation with something like io-ts or Zod which I am playing with, but this is what I am doing for now. Here is an example and a simplified version of the approach:

Of course, schema validation/type checking only looks at the shape of the responses which means there is still lots of room for semantic errors. There is nothing stopping me from expanding this setup with more semantic assertions which would further improve the quality but I haven’t really found my sweet spot for that yet. Perhaps in a future post.