Project: OpenAPI definition generator

June 2018

This is a first attempt at documenting the objective, background and design of Zenc Labs projects before actually building them. I’m trying this for a few reasons:

It will give visibility to fellow developers on what I’m building, and hopefully they’ll give me useful feedback and let me know about similar/related projects.
It will help me “freeze” the medium/long-term vision of the project before getting my hands dirty. It’s too easy to forget your original goals and lose motivation when you’ve been working on a project for a few weeks. It never feels “complete”, and it is notoriously impossible for a human to remember exactly how they thought about a problem in the past. Unless they’ve got it written down.
Last but not least, documenting projects publicly will help me stay accountable, get them implemented and go through the extra work of publishing them.

Enough said, onto the actual project details!

Background

OpenAPI, formerly known as Swagger, is becoming an industry standard to document HTTP-based RESTful APIs.

Unless you use GraphQL or gRPC, your API can probably be described in detail with a YAML file following the official OpenAPI specification. Whether your endpoints handle JSON, XML or multipart forms, you can document their inputs, outputs and status codes with an OpenAPI definition. It can then be used to write tests, generate client libraries or run server stubs.

Here’s an example from the official OpenAPI 3.0 repository:

# Source: https://github.com/OAI/OpenAPI-Specification/blob/master/examples/v3.0/petstore-expanded.yaml

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  description: A sample API that uses a petstore as an example to demonstrate features in the OpenAPI 3.0 specification
  termsOfService: http://swagger.io/terms/
  contact:
    name: Swagger API Team
    email: apiteam@swagger.io
    url: http://swagger.io
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
servers:
  - url: http://petstore.swagger.io/api
paths:
  /pets:
    get:
      description: |
        Returns all pets from the system that the user has access to
        Nam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia.
        Sed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.
      operationId: findPets
      parameters:
        - name: tags
          in: query
          description: tags to filter by
          required: false
          style: form
          schema:
            type: array
            items:
              type: string
        - name: limit
          in: query
          description: maximum number of results to return
          required: false
          schema:
            type: integer
            format: int32
      responses:
        "200":
          description: pet response
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/Pet"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
    post:
      description: Creates a new pet in the store.  Duplicates are allowed
      operationId: addPet
      requestBody:
        description: Pet to add to the store
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/NewPet"
      responses:
        "200":
          description: pet response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pet"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
  /pets/{id}:
    get:
      description: Returns a user based on a single ID, if the user does not have access to the pet
      operationId: find pet by id
      parameters:
        - name: id
          in: path
          description: ID of pet to fetch
          required: true
          schema:
            type: integer
            format: int64
      responses:
        "200":
          description: pet response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pet"
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
    delete:
      description: deletes a single pet based on the ID supplied
      operationId: deletePet
      parameters:
        - name: id
          in: path
          description: ID of pet to delete
          required: true
          schema:
            type: integer
            format: int64
      responses:
        "204":
          description: pet deleted
        default:
          description: unexpected error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
components:
  schemas:
    Pet:
      allOf:
        - $ref: "#/components/schemas/NewPet"
        - required:
            - id
          properties:
            id:
              type: integer
              format: int64

    NewPet:
      required:
        - name
      properties:
        name:
          type: string
        tag:
          type: string

    Error:
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string

The OpenAPI format can be a bit intimidating for someone who isn’t familiar with its structure. Adding a new endpoint, defining a new type or editing an existing one can be tricky. The API definition can also get quite large, and splitting it into multiple files requires workarounds.

Developers can use tools like Swagger Editor to edit an API definition more easily. Such tools are essential to ensure that the produced OpenAPI definition is valid. However when editing the OpenAPI definition in a code editor, you’re on your own. It’s just YAML.

As of June 2018, a lot of tools in the OpenAPI ecosystem only support OpenAPI 2.0 (aka Swagger 2.0) definitions. Writing an OpenAPI 3.0 definition means that you’ll have limited tools for a while. For example, swagger-codegen isn’t yet stable with OpenAPI 3.0.

Objective

We’ll create a higher-level DSL to generate OpenAPI definitions which:

is easier to learn
is language-agnostic (independent of server language/framework)
can be split into multiple files easily
can output both OpenAPI 2.0 and 3.0 definitions on-demand
validates itself

In the interest of efficiency, and to avoid expanding unnecessary effort without first validating the tool’s value in practice, we’ll first limit support to a subset of OpenAPI: JSON-based RESTful APIs.

Overview

We’ll create our DSL with Ruby.

This doesn’t mean that you need to use Ruby on Rails for your server. Your API definition is independent from your server implementation. Whether you use Java, Golang, Node.js, Rails or even C++, you can still use this DSL to generate your API definition.

Here’s what a simple API definition could look like:

require 'apigen/rest'

# Start an API declaration.
api = Apigen::Rest::Api.new

api.endpoint :list_users do
  method :get
  path "/users"
  output :array do
    item :user
  end
end

api.endpoint :create_user do
  method :post
  path "/users"
  input :object do
    name :string
    email :string
    password :string
    captcha :string
  end
  output :success do
    status 200
    type :user
  end
  output :failure do
    status 401
    type :string
  end
end

api.endpoint :update_user do
  method :put
  path "/users/{id}" do
    id :string
  end
  input :object do
    name :string?
    email :string?
    password :string?
    captcha :string
  end
  output :success do
    status 200
    type :user
  end
  output :failure do
    status 401
    type :string
  end
end

api.endpoint :delete_user do
  method :delete
  path "/users/{id}" do
    id :string
  end
  output :success do
    status 200
    type :void
  end
  output :failure do
    status 401
    type :string
  end
end

api.model :user do
  type :object do
    id :int64
    name :string
  end
end

# Ensure that the API spec is valid.
api.validate

# Generate OpenAPI v2 definition.
api.generate_openapi_v2('openapi-v2.yaml')

# Generate OpenAPI v3 definition.
api.generate_openapi_v3('openapi-v3.yaml')

Example API definition using our Ruby DSL

You generate the OpenAPI definitions with:

$ gem install apigen
$ ruby example.rb

Detailed design

Language

We need to pick a language to implement our DSL. Ruby makes it easy to do just that, and it has a minimalistic syntax that shouldn’t scare away too many developers. See Alternatives below for an in-depth of discussion of possible substitutes.

Functionality

Our DSL will initially provide the minimal amount of functionality to describe a JSON API.

In particular, you start by declaring an API:

require 'apigen/rest'

api = Apigen::Rest::Api.new

You can then define as many endpoints as you’d like:

api.endpoint :list_users do
  method :get
  path "/users"
  ...
end

api.endpoint :create_user do
  method :post
  path "/users"
  ...
end

api.endpoint :update_user do
  method :put
  path "/users/{id}" do
    id :string
  end
  ...
end

api.endpoint :delete_user do
  method :delete
  path "/users/{id}" do
    id :string
  end
  ...
end

Each endpoint must also declare its input and output shape:

api.endpoint :list_users do
  method :get
  path "/users"
  # Because :list_users is a GET endpoint, it doesn't declare an input type.
  output :array do
    item :user
  end
end

api.endpoint :create_user do
  method :post
  path "/users"
  input :object do
    name :string
    email :string
    password :string
    captcha :string
  end
  output :success do
    status 200
    type :user
  end
  output :failure do
    status 401
    type :string
  end
end

...

You’ll notice that we haven’t defined the type :user. Here’s how to add it:

api.model :user do
  type :object do
    id :int64
    name :string
  end
end

Then, all that’s left is to validate the API and generate the OpenAPI definition:

api.validate
api.generate_openapi_v2('openapi-v2.yaml')
api.generate_openapi_v3('openapi-v3.yaml')

API validation

Given the above structure, validating the API is a relatively straightforward process. We need to ensure that:

Each endpoint has a valid input and output type
Each path parameter (e.g. /users/{id}) is defined with a valid type
Composite types (object and list) reference valid types

Generating the OpenAPI definition

To generate the OpenAPI definition, we need to create a hash (dictionary) and simply output it with the YAML module.

We’ll make sure that required fields such as contact can be set through the DSL. If they aren’t explicitly specified, we’ll show a warning and set them to a reasonable default (e.g. contact may be set to apigen@localhost).

Splitting up into multiple files

If you’d like to split up your API definition into multiple files, the shortest solution is to make api a global variable by prepending a $, then using Ruby’s require keyword:

# api.rb

require 'apigen/rest'

# Start an API declaration.
$api = Apigen::Rest::Api.new

require './endpoints_users'
require './models_user'

# Ensure that the API spec is valid.
$api.validate

# Generate OpenAPI v2 definition.
$api.generate_openapi_v2('openapi-v2.yaml')

# Generate OpenAPI v3 definition.
$api.generate_openapi_v3('openapi-v3.yaml')

# endpoint_users.rb

$api.endpoint :list_users do
  method :get
  path "/users"
  output :success do
    status 200
    type :array do
      item :user
    end
  end
end

$api.endpoint :create_user do
  method :post
  path "/users"
  input :object do
    name :string
    email :string
    password :string
    captcha :string
  end
  output :success do
    status 200
    type :user
  end
  output :failure do
    status 401
    type :string
  end
end

$api.endpoint :update_user do
  method :put
  path "/users/{id}" do
    id :string
  end
  input :object do
    name :string?
    email :string?
    password :string?
    captcha :string
  end
  output :success do
    status 200
    type :user
  end
  output :failure do
    status 401
    type :string
  end
end

$api.endpoint :delete_user do
  method :delete
  path "/users/{id}" do
    id :string
  end
  output :success do
    status 200
    type :void
  end
  output :failure do
    status 401
    type :string
  end
end

# models_user.rb

$api.model :user do
  type :object do
    id :int64
    name :string
  end
end

Alternatives considered

Alternative languages

Although writing an API definition with our DSL only requires rudimentary understanding of Ruby, it still may be a different language from what developers are used to.

JavaScript is another candidate, since 70% of developers are familiar with the language. The only hiccup: it’s not as concise as Ruby and would require a package.json and node_modules to work.

Here’s what it could look like in JS:

const Rest = require("apigen/rest");

const api = new Rest.Api();

api.endpoint("list_users", (endpoint) => {
  endpoint.method("GET");
  endpoint.path("/users");
  endpoint.output("array", "user");
});

// ...

api.model("user", "object", (object) => {
  object.required("id", "int64");
  object.required("name", "string");
});

// Ensure that the API spec is valid.
api.validate();

// Generate OpenAPI v2 definition.
api.generate_openapi_v2("openapi-v2.yaml");

// Generate OpenAPI v3 definition.
api.generate_openapi_v3("openapi-v3.yaml");

Another possibility is a made-up language. I actually built this before learning about Swagger and OpenAPI (here). Here’s an example:

endpoint createUser: POST /users CreateUserRequest
-> success 200 CreateUserResponse
-> failure 400 string

type CreateUserRequest = {
  name: string
  password: string
}

type CreateUserResponse = {
  id: string
}

There are a couple of issues with this approach:

developers need to learn a new syntax
code editors will lack syntax highlighting, making errors harder to catch
it becomes a purely descriptive syntax (no way to add any dynamic logic)
splitting an API into multiple files requires introducing a module/import system (possibly reinventing the wheel)

Future plans

Headers, authorisation and other formats

While the DSL described above is fairly limited, it should easily be extended to cover other concepts that OpenAPI provides, while keeping required effort from the developer at a minimal level.

Typing

Ruby may soon have typing, thanks to Stripe’s efforts. If code editor support follows, this may help provide helpful autocomplete suggestions to developers as they write their API definitions in Ruby.

Extension to non-REST APIs

The DSL could be extended to support non-RESTful HTTP endpoints, such as GraphQL schemas.

The same approach to API definition could also be used to promote modular code. You could describe the API of a local code module in an effort to split up a monolithic codebase into several components communicating locally, potentially via inter-process communication between different languages.

Implementation progress

Follow along at https://github.com/zenclabs/apigen. Feel free to send any feedback/ideas via Twitter or by email.

[ ✔ ] Reserve a Ruby gem (apigen) - 29 May 2018
[ ✔ ] DSL proof of concept in Ruby - 29 May 2018
[ ✔ ] Write a design doc - 5 June 2018
[ ✔ ] API validation - 8 June 2018
[ ✔ ] Unit testing - 8 June 2018
[ ✔ ] OpenAPI v2 generation - 9 June 2018
[ ✔ ] OpenAPI v3 generation - 9 June 2018
[ ✔ ] JSON Schema generation - 16 June 2018

Update from the future (2019)

This project wasn't quite the right idea, but it led to a much better one. It's the same idea, but using TypeScript as a DSL, which makes a lot more sense when you're typing JSON payloads.

Check it out here.

François Wouts

The developer happiness engineer