JSON-Schema everywhere. How to make web development simpler.

When I took on my first project at kodira I was excited but suffered at the same experience every developer has coming onto a new project developing web-apps. After all there is so much to consider and keep in mind: User-Input-Validation, Datastructure-Layout, typings, Backend-Data-Validation & documentation. Greater are only the technological possibilities to tackle each of these topics individually. You inescapably end up in a strugle to do it all right and to just finish the job. Nothing you can do about it, right? Wrong, theres JSON-Schema to the rescue.

JSON-Schema describes itself in one sentece on their website better than I can do:

“JSON Schema is a vocabulary that allows you to annotate and validate JSON documents.”

At the end of the day it is just a convention to describe the data that is used and could look like this:

{
  "$id": "CustomerOrderSchema",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "CustomerOrderSchema",
  "description": "An address similar to http://microformats.org/wiki/h-card",
  "type": "object",
  "properties": {
    "customer-id": {
        "type": "number"
    },
    "post-office-box": {
      "type": "string"
    },
    "street-address": {
      "type": "string"
    },
    "postal-code": {
      "type": "string"
    },
    "country-name": {
      "type": "string"
    }
  },
  "required": [ "customer-id", "street-address", "postal-code" ],
  "additionalProperties": false
}

This json defines the structure of the data which is an object that must contain the required properties which are defined in the required array in the schema. The data can contain the other properties defined under the “properties”-tab but must not contain other properties hence the “additionalProperties” is set to false. This is a very basic example and there are many more configurations to describe the data you use most exactly. This was pretty easy, right? The JSON-Schema is the building brick of the features to come so when you use it in your project you want to try your best in describing it best to the specification.

Generating interfaces with json-schema-to-typescript

So the first thing I as an Angular developer want is to use my data information in a typescript interface. Both the JSON-schemas and interfaces describe the structure of your data so it makes a lot of sense to generate interfaces from your JSON-Schemas. It turns out this is easy with a library like json-schema-to-typescript but there is a variety of others for you to choose from. We used a bash-script to use the libraries CLI to generate the interfaces for us. The generated interfaces are named after the schemas title property.

As a programmer I now know how the data looks like that I am dealing with. Lets use the full potential of JSON-Schema!

Frontend Validation in Angular using AJV

In our projects we usually use Angular and I like to give an example since it provides a robust implementation for forms and Form-Validation. We will use a very popular JSON-Schema-Validation library named AJV which is very easy to use and completly independent of any frontend framework so you can use React or any other framework you favor with it. Basically you can also use Angulars build-in-validation but AJV provides two major advantages:

It scales. Maybe you generate a form dynamically. You can also set validation dynamically but you dont want to, really. On the other side you are also prune to do mistakes. If AJV screams at you its clear you messed up either your form or your JSON-Schema is not the way you intend it to be.
It is really easy. If you want to change your data, which happens all the time, without AJV you would change your form and the validation which is again, prune to errors. Why not just change the JSON-Schema and your done.

There are also very fascinating libraries like formly that support JSON-Schema that far that it generates your forms automatically based on the JSON-Schema you provide. AJV deals with the validation and you are ready to go. This saves you a lot of angular-form code and validation errors in the long run.

The setup is quite easy too since Angular provides Validator functions for you. I usually used a AJV-Service:

Instantiate ajv => private ajv = new Ajv({ allErrors: true }); The “allErrors”-property is a AJV-option you can use to configure the behaviour of AJV. Check out their docs for more information.
load the schema => this.ajv.addSchema(urlPath, name) Define a function to upload your schema to AJV once you enter the angular-component you need the schema.
read the schema => this.ajv.getSchema(name) Define a funtion in the service you use in your Form-Validator once you need the schema validation.

check validity => Define a function that validates the input data against the uploaded schema in the Form-Validator

  const isValid = this.ajv.validate(name, data) as boolean;
  return { isValid, errorsText: this.ajv.errorsText() };

Now create a validator for angular. To do this checkout the angular docs or use the build-in angular schematic. After that use the “validateData” method we defined in the service. Then parse and return the result how you want it to be.

Last but not least go to your angular component:

Inject our AJV-Service.

Get the schema or load it initially.

 try { this.schemaLoader.getSchema('SchemaUploadName');} catch (err) {
 this.schemaLoader.loadSchema('SchemaUploadName', TheImportedSchema);
 console.log(`Error reading schema: ${err} Current schema has been LOADED.`);}

Add the Form-Validator as a Validator for your form inside your Formgroup => OurValidatorFunction(this.schemaLoader, 'SchemaUploadName')

Its done! Your form now gets checked every time the user changes his input and gets invalid if the input does not match your associated JSON-Schema. New form? Just repeat the last three steps in the component and you dont even have to think about input validation anymore. If you want to invalidate the specific Formcontroll you might have to do a little parsing with the error that AJV gives you back but that is up to the case.

Backend Validation in a Node.js-Middleware

Following the motto “better be safe than sorry” I find it generally a very good idea to do some kind of backend data validation before your request data enters any database. This not only enhaces your security but also gives you the chance to avoid unwanted behaviour and spot errors before production. Lets set AJV up for backend validation:

Basically we are following the same procedure as in the frontend of our app which is AJV-Initialization => Loading Schemas => validate Request-Data with the schema.

Initialize AJV and load the Schemas. After installing AJV as a dependency you create a new AJV-object and add the imported schemas as an option to it.
```
  export default new Ajv({
    schemas: [
 YourSchemaGoesHere,
 AnotherImportedSchema
    ]
  });
```

Define your middleware which uses the validate-method to check the request.body.

  export function isValid(schema: string) {
    const validate = ourAJVObject.getSchema(schema);
    if (!validate) {
 throw new Error(`No validation found for schema "${schema}"`);
    }
    return (ctx: Context, next: () => Promise<any>) => {
 if (validate(ctx.request.body)) {
   return next();
 } else if (validate.errors) {...}}

You let AJV to the business and return next() if everything is OK or throw an error.

Add the middleware to every REST-API that needs validation, all at best.

  router.post('/order', isValid('CustomerOrderSchema'), saveOrderToDatabase);

Our validation allready works. You can also use another very similar middleware to check outgoing data inside the response-bodies. The parameter ‘CustomerOrderSchema’ provided to our middleware comes from the $id property of our schema. You can identify a schema, or a part of your schema (a subschema) by using $id, and then you can reuse it somewhere else by using the $ref keyword. This will come in handy now when we build a documentation of our REST-APIs in just minutes with redoc.

Building a beautiful and easy documentation by using redoc.

This time we will utilize a toolbox named swagger which is often used in synonym with the OpenAPI-Specification. The specification is a guideline on how to describe Restfull APIs in a swagger.yaml file so they can be picked up by a great variety of tools of which one is redoc. Redoc takes your API description and builds a beautiful UI with all the information provided which is key for any new developer coming onto the project. It sounds more complex than it really is because the majority of the information is allready provided by your JSON-Schema. Lets build it.

Install redoc as a dependency. If you want to use redoc with React or Docker check out their README. Otherwise redoc-cli provides an easy to use alternative to start-up the documentation. Install it as a dev-dependency and add a start-script to your package.json:
```
    "start": "redoc-cli serve --watch true swagger.yaml"
```
Create the swagger.yaml and describe you APIs. Since the possibilities to describe your API with OpenAPI are great I will just provide a basic example. At this point please feel free to read more at their detailed documentation where you can find many more things to do with your swagger.yaml which could look like this:

  openapi: 3.0.0
  info:
    description: Your description of the project goes here
    version: 1.0.0
    title: Name of the App
  components:                                                     #Defince Components that can be reused inside your specification
    responses:
      UnauthorizedError:
        description: Access token is missing or invalid.

  paths:
    /api/auth/login:                                              #Path of the API-Endpoint
      post:                                                       #Method to use
        tags:                                                     #Sorts your endpoints in tag-groups
          - authorization
        summary: Logs User in with Credentials
        requestBody:
          content:
            application/json:
              schema:                                             #Here our request-data from our schema gets referenced
                $ref: schemas/login-post-request.schema.json 
          description: Username and password
          required: true                                          #Is the request.body required?
        responses:                                                #You can describe the possible responses yourself or use generic references over the whole specification
          "200":
            description: Success
            content:
              application/json:
                schema:
                  $ref: schemas/login-post-response.schema.json
          "401":
            $ref: "#/components/responses/UnauthorizedError"

The result of this simple description looks like this: example for redoc documentation

Conclusion

As you can see JSON-Schema can do a lot for you. Of course there is some preset to be done like writing schemas for your data. Some people like to skip that step by using libraries which generate JSON-Schemas from interfaces which is the other way to go. Personally I like writing the schemas myself, because like with testing the schemas force you to really think about the data you want to use. At the end it depends on you how conservative you will be with the data streaming in and out.

Using JSON-Schema more in small projects I can allready see the benefits when it comes to scaleablility of your validation. Tools like redoc just build on top of your allready described JSON-Schemas and there is much more to them then described in this blog post.

Really the best of it all is the structure that JSON-Schema and everything around it gives you. Once a Schema is carefully described interfaces, validation, documentation and maybe your UI depend on it. New developers can really trust these schemas and get a much easier start into the project, also because the documentation is so good and up to date. If there is a error chances are you might do something wrong outside of your predefinded rules. At the end more thougth and boundaries might save everyone a lot of time and trouble developing apps.