API build process description

Build(task) model

Apihub implement pull model for build processing. I.e. BE have a queue of builds and have an interface to take the build. In this schema BE is not aware of builders It allows to scale builders(browser, node services) horizontally without changes in BE configuration.

build config

Biold config is a metadata set for an object that will be created during the build.

Example:

{
    "files": [
        {
            "slug": "",
            "index": 0,
            "fileId": "swagger.json",
            "labels": [
            ],
            "publish": true
        }
    ],
    "status": "draft",
    "version": "2024.3",
    "metadata": {
    },
    "buildType": "build",
    "createdBy": "username",
    "packageId": "username.g1.p1",
    "publishId": "",
    "publishedAt": "0001-01-01T00:00:00Z",
    "previousVersion": "2024.1",
    "previousVersionPackageId": ""
}

“publish” for file defines is the file should present in result(builder output)

“publishId” is generated by Apihub BE when the build is accepted and stored to DB

“buildType” defines the logic of build process options:

• “build” - regular version publish
• “changelog” - changelog only
• “documentGroup” - export
• “reducedSourceSpecifications” - export
• “mergedSpecification” - export

Stages for buildType = build (in builder lib)

0. detect file type (swagger 2.0, OpenAPI 3.0+, graphql, protobuf) 0.1) convert swagger 2.0 to OpenAPI 3.0
1. bundling (resolve ext refs)
2. make operations from specs and make search scopes (set of words in a file excluding specification terms(e.x. description, summery, parameters, etc))
3. make comparison if previous version is defined

result

• “info.json” - main file which contains build(future version) parameters and metadata.
• documents.json - list of documents with metadata
• operations.json - list of operatrions with metadata
• comparisons.json - list of comparisons with metadata
• notifications.json - messages raised while building the version’s own documents and operations, each with a severity, a category and, when the message concerns a document, its slug
• comparison-notifications.json - messages raised while calculating a changelog, grouped by the version pair they belong to. Present for a build with a previousVersion and for a changelog build. A comparison reused from the backend carries no entry

notifications of a failed build

A failed build or changelog build writes no archive, so notifications.json and comparison-notifications.json are never produced for it. Instead, the builder sends the messages it had raised before it failed as an optional notifications part of the status=error request (POST /api/v3/packages/{packageId}/publish/{publishId}/status). The part is a JSON document with two lists, notifications and comparisonNotifications, in the integer-severity format of the archive files.

The backend stores both lists as one in the error_build_notification table, in the transaction that marks the build failed, and sets hasNotifications in the build metadata. The flag is returned by the publish status endpoints. The messages are served by GET /api/v2/packages/{packageId}/publish/{publishId}/notifications (same filters and paging as the version notification endpoints, sorted most severe first) and exported as xlsx by GET /api/v2/packages/{packageId}/publish/{publishId}/export/notifications. The workbook’s cover page is titled “Failed publication notifications” and shows the requested version name and status with the suffix “(not published)”, for example release (not published), so it is not mistaken for the report of a published version. Both endpoints require read permission on the package and answer 404 for a publish id that does not belong to the package.

Rules worth knowing:

• Only builds of packages of kind package store the part. A dashboard build compares one version pair per reference, and the flat part cannot tell the pairs apart, so the backend accepts the part and stores nothing.
• An invalid part (not a file part, not JSON, invalid against the contract, or sent more than once) does not fail the request: the error status is applied, the part is dropped with a WARN log line, and hasNotifications stays false. A builder does not retry a 400, so rejecting the request would lose the status itself.
• The messages live as long as the failed build’s sources: the builds cleanup job removes them after 30 days and clears the flag (see Data maintenance).

build sequences

Success cases

build type = build & client_build=false

publish request -> validation -> save build entity and sources(tables build and build_src ) -> build is in queue (status==none) -> build is taken by node service and bound to builder id (status=running) -> the build is processed on node service (node service sends keepalives(set build status=running) while the build is running) -> node service sends build status = success with build result data -> builder id is validated -> build result is stored in DB (table build_result) or in minio -> build result is validated -> build result is stored in DB via single transaction -> search indexes calculated asynchronously(somehow)

build type = build & client_build=true

publish request (with client_build=true and builderId!=null) -> validation -> save build entity and sources(tables build and build_src ) -> the build is bound to the builderId from request and status = running -> buildId is returned to the client -> the build is processed on client (client sends keepalives(set build status=running) while the build is running)-> client sends build status = success with build result data -> builder id is validated -> build result is stored in DB (table build_result) or in minio -> build result is validated -> build result is stored in DB via single transaction -> search indexes calculated asynchronously(somehow)

build type = changelog

almost the same, but only comparisons generated and stored

build type = documentGroup \ reducedSourceSpecifications \ mergedSpecification

TODO

Error cases

builder is restarted (state is lost)

For example due to OOM killed or browser(tab) is closed.

start -> processing -> keepalives stop going -> BE detects that the build is no longer processed by the following criterea: “now - last_active > {timeout}” -> the build is considered as free and goes to the queue again -> when node service is ready to take the build, restart_count is checked. If restart_count >= 2 then the build is failed with message “Restart count exceeded limit” -> if restart is available, build is taken by node service and bound to builder id (status=running) and restart_count is incremented -> build as usual

builder internal error (handled)

Internal error(exception) happens in builder logic and it’s caught during the processing.

Builder sets status=error and error details. New version/changelog/export is not created. Error build is not restarted.

problem with source documents

The error in source documents is detected by builder. Builder sets status=error and details regarding the problem. New version/changelog/export is not created. Error build is not restarted.

build statuses

none - build is not strarted running - build is started complete - successfully completed error - completed with error

DB

Related tables:

• build - list of build
• build_src - config and sources archive(zip)
• build_result or minio (depends on configuration) - result archive(zip)