Worker Manager API

Overview

Worker manager is a service which runs alongside every qserv worker or czar and manages many common operations:

  • database operations (table creating, data loading, etc.)

  • certain xrootd (plug-in) operations

  • management of other services’ lifetime

Primary motivation for implementing wmgr service was to facilitate data loading in integration tests. It is likely that some of the wmgr functions may be changed in the future once we implement production-level services for data loading and distribution. For more details on wmgr design consult pages:

This document describes wmgr HTTP-basedAPI in details. Note that there is also a Python client library implemented on top of HTTP API.

General API description

Wmgr API is implemented on top of HTTP protocol following RESTful principles (where possible). All communication with wmgr is performed over HTTP without encryption (implementing SSL/TLS is feasible but needs solution for certificate management).

Wmgr can be configured to require one of few authentication mechanisms - none, basic, or digest. Lack of SSL/TLS dictates use digest authentication for production services.

Whole wmgr API is split into three groups - database, xrootd, and process API. Below is description of individual API methods (actions).

Return codes and content types

Responses generated by wmgr service use regular HTTP status codes to indicate success or error. Codes in range 200-299 are used for success, 400-499 for errors (range 500-599 typically means service failure).

Normally responses generated by wmgr service include data in JSON format and have their Content-Type header set to application/json. This applies to both successful completion and error conditions. In some cases when error condition is returned by the transport/framework layer it will have different content type.

See also document which defines general structure of JSON objects for returned response data: https://confluence.lsstcorp.org/display/DM/REST+API+General+Guidelines

Database API

This section contains description of actions related to database operations - creating/deleting databases and tables, loading table data, etc.

GET /dbs

Return the list of all existing database names.

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for successful return

  • 500 - for database errors

Response body (for successful completion):
JSON object, “results” property is a list of database objects with keys:

  • name: database name

  • uri: URL for database operations

Request/response example:

GET /dbs HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "qservMeta",
      "uri": "/dbs/qservMeta"
    },
    {
      "name": "qservResult",
      "uri": "/dbs/qservResult"
    }
  ]
}

POST /dbs

Create new database. In addition to creating database itself this method also grants all privileges on this database to regular non-privileged account.

Request headers:

  • Content-Type: required as multipart/form-data

Form Parameters:

  • db: Database name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 201 - if database was successfully created

  • 400 - if database name is missing

  • 409 - if database already exists

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “result” property is a database object with keys:

  • name: database name

  • uri: URL for database operations

Request/response example:

POST /dbs HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------bb306714c15713c2

--------------------------bb306714c15713c2
Content-Disposition: form-data; name="db"

newDB
--------------------------bb306714c15713c2--

HTTP/1.0 201 CREATED
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/dbs/newDB"
  }
}

DELETE /dbs/<dbName>

Deletes database.

Parameters:

  • dbName: database name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - if database was successfully deleted

  • 400 - if parameters have invalid format

  • 404 - if database does not exist

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “result” property is a database object with keys:

  • name: database name

  • uri: URL for database operations

Request/response example:

DELETE /dbs/newDB HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/dbs/newDB"
  }
}

GET /dbs/<dbName>/tables

Returns the list of tables in a database.

Parameters:

  • dbName: database name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for successful return

  • 400 - if parameters have invalid format

  • 404 - if database does not exist

  • 500 - for database errors

Response body (for successful completion):
JSON object, “results” property is a list of table objects with keys:

  • name: table name

  • uri: URL for database operations

Request/response example:

GET /dbs/qservMeta/tables HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "QCzar",
      "uri": "/dbs/qservMeta/tables/QCzar"
    },
    {
      "name": "QInfo",
      "uri": "/dbs/qservMeta/tables/QInfo"
    },
    ...
  ]
}

POST /dbs/<dbName>/tables

Create new table.

If schemaSource (see below) is “request” then request must include schema parameter which is an SQL DDL statement starting with ‘CREATE TABLE TableName …’.

If schemaSource is “css” then table parameter must be specified. Table schema will be extracted from CSS in this case, schemaSource must not be given.

Parameters:

  • dbName: database name

Request headers:

  • Content-Type: required as multipart/form-data

Form Parameters:

  • table: Table name

  • schemaSource: source for schema name, possible values: “request”, “css”, (default: “request”)

  • schema: complete “CREATE TABLE …” statement (optional)

  • chunkColumns: boolean flag, false by default, accepted values: ‘0’, ‘1’, ‘yes’, ‘no’, ‘false’, ‘true’. If set to true then delete columns “_chunkId”, “_subChunkId” from table (if they exist) and add columns “chunkId”, “subChunkId” (if they don’t exist)

Response headers:

  • Content-Type: application/json

Status Codes:

  • 201 - if table was successfully created

  • 400 - if parameters have invalid format or if form parameters are missing or conflicting

  • 409 - if table already exists

  • 500 - if table is not defined in CSS or other database errors

Response body (for successful completion):
JSON object, “result” property is a table object with keys:

  • name: database name

  • uri: URL for database operations

Request/response example:

POST /dbs/newDB/tables HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------c5c44964f0f9add0

--------------------------c5c44964f0f9add0
Content-Disposition: form-data; name="schema"

CREATE TABLE newTable (I INT)
--------------------------c5c44964f0f9add0
Content-Disposition: form-data; name="table"

newTable
--------------------------c5c44964f0f9add0--

HTTP/1.0 201 CREATED
Content-Type: application/json

{
  "result": {
    "name": "newTable",
    "uri": "/dbs/newDB/tables/newTable"
  }
}

DELETE /dbs/<dbName>/tables/<tblName>

Drop a table and optionally all chunk/overlap tables.

Parameters:

  • dbName: database name

  • tblName: table name

Query Parameters:

  • dropChunks: boolean flag, false by default, accepted values: ‘0’, ‘1’, ‘yes’, ‘no’, ‘false’, ‘true’

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - if table was successfully deleted

  • 400 - if parameters have invalid format

  • 404 - if table does not exist

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “result” property is a table object with keys:

  • name: database name

  • uri: URL for database operations

Request/response example:

DELETE /dbs/newDB/tables/newTable HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newTable",
    "uri": "/dbs/newDB/tables/newTable"
  }
}

GET /dbs/<dbName>/tables/<tblName>/schema

Return result of SHOW CREATE TABLE statement for given table.

Parameters:

  • dbName: database name

  • tblName: table name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for successful return

  • 400 - if parameters have invalid format

  • 404 - if table does not exist

  • 500 - for database errors

Response body (for successful completion):
JSON object, “result” property is a string with resulting schema.

  • name: table name

  • uri: URL for database operations

Request/response example:

GET /dbs/newDB/tables/newTable/schema HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": "CREATE TABLE `newTable` (\n  `I` int(11) DEFAULT NULL\n) ENGINE=MyISAM DEFAULT CHARSET=latin1"
}

GET /dbs/<dbName>/tables/<tblName>/columns

Return result of SHOW COLUMNS statement for given table.

Parameters:

  • dbName: database name

  • tblName: table name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for successful return

  • 400 - if parameters have invalid format

  • 404 - if table does not exist

  • 500 - for database errors

Response body (for successful completion):

JSON object, “results” property is a list of column objects with keys: name, type, key, default, null

Request/response example:

GET /dbs/newDB/tables/newTable/columns HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "default": null,
      "key": "",
      "name": "I",
      "null": "YES",
      "type": "int(11)"
    }
  ]
}

GET /dbs/<dbName>/tables/<tblName>/chunks

Return the list of chunks in a table. For non-chunked table empty list is returned.

Parameters:

  • dbName: database name

  • tblName: table name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for successful return

  • 400 - if parameters have invalid format

  • 404 - if table does not exist

  • 500 - for database errors

Response body (for successful completion):
JSON object, “results” property is a list of chunk objects with keys:

  • chunkId: chunk number (integer)

  • chunkTable: true if chunk has regular chunk table (boolean)

  • overlapTable: true if chunk has overlap table (boolean)

  • uri: URL for chunk operations

Request/response example:

GET /dbs/qservTest_case01_qserv/tables/Object/chunks HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "chunkId": 7648,
      "chunkTable": true,
      "overlapTable": true,
      "uri": "/dbs/qservTest_case01_qserv/tables/Object/chunks/7648"
    },
    ...
  ]
}

POST /dbs/<dbName>/tables/<tblName>/chunks

Create new chunk.

Parameters:

  • dbName: database name

  • tblName: table name

Request headers:

  • Content-Type: required as multipart/form-data

Form Parameters:

  • chunkId: chunk ID, non-negative integer

  • overlapFlag: if true then create overlap table too (default is true), accepted values: ‘0’, ‘1’, ‘yes’, ‘no’, ‘false’, ‘true’

Response headers:

  • Content-Type: application/json

Status Codes:

  • 201 - if chunk tables were successfully created

  • 400 - if parameters have invalid format or if form parameters are missing or conflicting

  • 404 - if table is missing

  • 409 - if chunk table already exists

  • 500 - if table is not defined in CSS or other database errors

Response body (for successful completion):
JSON object, “result” property is a chunk object with keys:

  • chunkId: chunk number (integer)

  • chunkTable: true if chunk has regular chunk table (boolean)

  • overlapTable: true if chunk has overlap table (boolean)

  • uri: URL for chunk operations

Request/response example:

POST /dbs/newDB/tables/newTable/chunks HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------df029da2ec8387ce

--------------------------df029da2ec8387ce
Content-Disposition: form-data; name="chunkId"

1000
--------------------------df029da2ec8387ce--

HTTP/1.0 201 CREATED
Content-Type: application/json

{
  "result": {
    "chunkId": 1000,
    "chunkTable": true,
    "overlapTable": true,
    "uri": "/dbs/newDB/tables/newTable/chunks/1000"
  }
}

DELETE /dbs/<dbName>/tables/<tblName>/chunks/<chunkId>

Delete chunk from a table, both chunk data and overlap data is dropped.

Parameters:

  • dbName: database name

  • tblName: table name

  • chunkId: chunk number, non-negative integer

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - if table was successfully deleted

  • 400 - if parameters have invalid format

  • 404 - if table does not exist

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “result” property is a chunk object with keys:

  • chunkId: chunk number (integer)

  • chunkTable: true if chunk has regular chunk table (boolean)

  • overlapTable: true if chunk has overlap table (boolean)

  • uri: URL for chunk operations

Request/response example:

DELETE /dbs/newDB/tables/newTable/chunks/1000 HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "chunkId": 1000,
    "chunkTable": true,
    "overlapTable": true,
    "uri": "/dbs/newDB/tables/newTable/chunks/1000"
  }
}

POST /<dbName>/tables/<tblName>/data

Upload data into a table using file format supported by mysql command LOAD DATA [LOCAL] INFILE.

Parameters:

  • dbName: database name

  • tblName: table name

Request headers:

  • Content-Type: required as multipart/form-data

Form Parameters:

  • table-data: the data come in original LOAD DATA format with binary/octet-stream content type and binary encoding, and it may be compressed with gzip.

  • load-options: set of options encoded with usual application/x-www-form-urlencoded content type, options are: - delimiter - defaults to TAB - enclose - defaults to empty string (strings are not enclosed) - escape - defaults to backslash - terminate - defaults to newline - compressed - “0” or “1”, by default is guessed from file extension (.gz)

Response headers:

  • Content-Type: application/json

Status Codes:

  • 201 - if chunk tables were successfully created

  • 400 - if parameters have invalid format or if form parameters are missing or conflicting

  • 404 - if table is missing

  • 409 - if chunk table already exists

  • 500 - if table is not defined in CSS or other database errors

Response body (for successful completion):
JSON object, “result” property is an object with keys:

  • status: string “OK”

  • count: count of rows added to a table

Request/response example:

POST /dbs/newDB/tables/newTable/data HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------345ad77805210ac6

--------------------------345ad77805210ac6
Content-Disposition: form-data; name="table-data"; filename="table.dat.gz"
Content-Type: application/octet-stream

.....<.U..table.dat.3.2400.2.bS..;.......

--------------------------345ad77805210ac6
Content-Disposition: form-data; name="load-options"

compressed=1&delimiter=%2C
--------------------------345ad77805210ac6--

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "count": 4,
    "status": "OK"
  }
}

POST /<dbName>/tables/<tblName>/chunks/<chunkId>/data

Upload data into a chunk table using file format supported by mysql command LOAD DATA [LOCAL] INFILE.

This method works exactly as previous one taking the same form parameter but it loads data into a chunk and has additional URL parameter specifying chunk number.

POST /<dbName>/tables/<tblName>/chunks/<chunkId>/overlap

Upload data into overlap table using file format supported by mysql command LOAD DATA [LOCAL] INFILE.

This method works exactly as previous one taking the same form parameter but it loads data into an overlap table and has additional URL parameter specifying chunk number.

GET /dbs/<dbName>/tables/<tblName>/index

Return index data (array of (objectId, chunkId, subChunkId) triplets).

Parameters:

  • dbName: database name

  • tblName: table name

  • chunkId: chunk number (non-negative integer)

Query Parameters:

  • columns: specifies comma-separated list of three column names. Default column names are “objectId”, “chunkId”, “subChunkId”. Result returns columns in the same order as they are specified in ‘columns’ argument.

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for successful return

  • 400 - if parameters have invalid format

  • 404 - if table does not exist

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “result” property is an object with keys:

  • description: array of three objects describing columns, each with keys “name” (column name) and “type” (MySQL type name)

  • rows: array of arrays of integers

Request/response example:

GET /dbs/qservTest_case01_qserv/tables/Object/index HTTP/1.1

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "description": [
      {
        "name": "objectId",
        "type": "LONGLONG"
      },
      {
        "name": "chunkId",
        "type": "LONG"
      },
      {
        "name": "subChunkId",
        "type": "LONG"
      }
    ],
    "rows": [
      [
        386937898687249,
        6630,
        897
      ],
      [
        386942193651348,
        6630,
        660
      ],
      ...
    ]
  }
}

GET /dbs/<dbName>/tables/<tblName>/chunks/<chunkId>/index

Return index data (array of (objectId, chunkId, subChunkId) triplets) for single chunk.

Does the same as previous method but for one chunk from partitioned table. Useful when index for whole table may be too big.

Request/response example:

GET /dbs/qservTest_case01_qserv/tables/Object/chunks/7648/index HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "description": [
      {
        "name": "objectId",
        "type": "LONGLONG"
      },
      {
        "name": "chunkId",
        "type": "LONG"
      },
      {
        "name": "subChunkId",
        "type": "LONG"
      }
    ],
    "rows": [
      [
        433306365599363,
        7648,
        5
      ],
      [
        433314955527561,
        7648,
        10
      ],
      ...
    ]
  }
}

Xrootd API

This section contains description of actions related to xrootd operations - e.g. publishing database via xrootd.

GET /xrootd/dbs

Return the list of databases known to xrootd.

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for success

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “results” property is a list of database objects with keys:

  • name: database name

  • uri: URL for xrootd database operations

Request/response example:

GET /xrootd/dbs HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "qservTest_case01_qserv",
      "uri": "/xrootd/dbs/qservTest_case01_qserv"
    }
  ]
}

POST /xrootd/dbs

Register new database in xrootd chunk inventory.

Request headers:

  • Content-Type: required as multipart/form-data

Form Parameters:

  • db: database name (required)

  • xrootdRestart: if set to ‘no’ then do not restart xrootd (defaults to yes)

Response headers:

  • Content-Type: application/json

Status Codes:

  • 201 - if database was successfully registered

  • 400 - if parameters are missing or have invalid format

  • 409 - if database is already registered

  • 500 - on other database errors

Response body (for successful completion):
JSON object, “results” property is a database object with keys:

  • name: database name

  • uri: URL for xrootd database operations

Request/response example:

POST /xrootd/dbs HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------370e6e4d60b7499e

--------------------------370e6e4d60b7499e
Content-Disposition: form-data; name="db"

newDB
--------------------------370e6e4d60b7499e
Content-Disposition: form-data; name="xrootdRestart"

no
--------------------------370e6e4d60b7499e--

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/xrootd/dbs/newDB"
  }
}

DELETE /xrootd/dbs/<dbName>

Unregister database from xrootd chunk inventory.

Parameters:

  • dbName: database name

Query Parameters:

  • xrootdRestart: if set to ‘no’ then do not restart xrootd (defaults to yes)

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for success

  • 400 - if parameters have invalid format

  • 409 - if database is not registered

  • 500 - for other database errors

Response body (for successful completion):
JSON object, “results” property is a database object with keys:

  • name: database name

  • uri: URL for xrootd database operations

Request/response example:

DELETE /xrootd/dbs/newDB?xrootdRestart=no HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "newDB",
    "uri": "/xrootd/dbs/newDB"
  }
}

GET /xrootd/dbs/<dbName>

Return the list of chunk IDs in a database as known to xrootd.

Note

Not implemented yet.

Services API

This section contains description of actions related to operations on services - e.g. stopping and starting processes.

GET /services

Return the list of services.

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for success

Response body (for successful completion):
JSON object, “results” property is a list of service objects with keys:

  • name: service name

  • uri: URL for service operations

Request/response example:

GET /services HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "results": [
    {
      "name": "xrootd",
      "uri": "/services/xrootd"
    },
    {
      "name": "mysqld",
      "uri": "/services/mysqld"
    }
  ]
}

GET /services/<service>

Return service state.

Parameters:

  • service: service name

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for success

  • 404 - for invalid service name

Response body (for successful completion):
JSON object, “result” property is a service object with keys:

  • name: service name

  • state: one of “active” or “stopped”

  • uri: URL for service operations

Request/response example:

GET /services/mysqld HTTP/1.0

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "mysqld",
    "state": "active",
    "uri": "/services/mysqld"
  }
}

PUT /services/<service>

Execute some action on service, like “stop” or “restart”.

Parameters:

  • service: service name

Request headers:

  • Content-Type: required as multipart/form-data

Form Parameters:

  • action: action: one of ‘stop’, ‘start’, ‘restart’ (required)

Response headers:

  • Content-Type: application/json

Status Codes:

  • 200 - for success

  • 400 - if parameters are missing or have invalid format

  • 409 - if action has failed

Response body (for successful completion):
JSON object, “result” property is a service object with keys:

  • name: service name

  • state: state of service after action, one of “active” or “stopped”

  • uri: URL for service operations

Request/response example:

PUT /services/mysqld HTTP/1.0
Content-Type: multipart/form-data; boundary=------------------------48169e483bc7d12e

--------------------------48169e483bc7d12e
Content-Disposition: form-data; name="action"

restart
--------------------------48169e483bc7d12e--

HTTP/1.0 200 OK
Content-Type: application/json

{
  "result": {
    "name": "mysqld",
    "state": "active",
    "uri": "/services/mysqld"
  }
}