EMS - Self Hosted Guide

This section will provide information related to self-hosted deployment of Execution Management System API (EMS API) software.

Configuration Manual

This section will provide the necessary information how to configure the CoinAPI EMS software.

Configuration keys and values

Configuration keys:

• Are case-insensitive. For example, ExchangeId and exchangeid are treated as equivalent keys.

• If a key and value is set in more than one configuration providers, the value from the last provider added is used.

• Using array indices in configuration keys are allowed.

• Hierarchical keys

  • Within the Configuration API, a colon separator (:) works on all platforms.

  • In environment variables, a colon separator may not work on all platforms. A double underscore, __, is supported by all platforms and is automatically converted into a colon :.

Configuration values:

• Are strings.

• Null values can't be stored in configuration.

Configuration providers

You can set the values of configuration parameters for the components using the following providers (in case of collision for any specific key):

1. Command-line arguments

2. Environment variables

Environment variables

To configure specific parameters using host or container environment variable, you must replace all occurrences of : with __ in the configuration parameter key, this is because the : separator doesn't work with environment variable hierarchical keys on all platforms. __, the double underscore, is:

• Supported by all platforms. For example, the : separator is not supported by Bash, but __ is.

• Automatically replaced by a :

For example, to set the value of the configuration parameter OEML:ExchangeId to BITSTAMP you would need to set the environment variable named OMS__ExchangeId to BITSTAMP.

Command-line arguments

You can set the configuration parameters by providing the arguments to the application executable.

The following example sets keys and values using the =: OEML:ExchangeId=BITSTAMP The following example sets keys and values using the /: /OEML:ExchangeId=BITSTAMP The following example sets keys and values using the --: --OEML:ExchangeId=BITSTAMP or --OEML:ExchangeId BITSTAMP

The key value:

• Must follow =, or the key must have a prefix of -- or / when the value follows a space.

• Isn't required if = is used. For example, MySetting=.

Within the same command, don't mix command-line argument key-value pairs that use = with key-value pairs that use a space.

Configuration parameters

Exchange conditional parameters

The table below explains the conditional parameters by the OEML:ExchangeId value. For example when OEML:ExchangeId == "BITSTAMP" then OEML:Exchanges:BITSTAMP:CustomerId is required.

Exchange specific recommendations

Kraken Futures KRAKENFTS

Whitelist your IP addresses

Whitelisting your IP addresses will give you the lowest possible latency and most stable connection to exchange servers in AWS EU-West 1. Whitelisting works by allowing you to skip CloudFlare. Here is an article on how to whitelist your IP addresses (opens in a new tab).

Consul Configuration (optional)

The end result should look like this:

{
  "datacenter": "hq",
  "data_dir": "C:\\consul\\data-hq",
  "log_level": "INFO",
  "server": false,
  "retry_join": [ "server-a", "server-b", "server-c" ],
  "ui": true,
  "http_config": {
    "response_headers": {
      "Access-Control-Allow-Origin": "*"
    }
  }
}

If your deployment including optional Consul Cluster for the Service Discovery layer then it's required to change CORS settings of the local Consul client on the hosts where the EMS WebUI will be used. This step is necessary as we need to discover the address of the EMS API component.

"http_config": { "response_headers": { "Access-Control-Allow-Origin": "*" } }

Installation Manual

This section will provide the necessary information how to install the CoinAPI EMS software.

Service discovery

The CoinAPI EMS API is capable of automaticaly detecting all your instances of CoinAPI EMS Edge service. This service discovery works by the one of the following methods:

1. Using the CoinAPI EMS Central Cloud, every time you start CoinAPI EMS Edge, it will register automatically in our central cloud infrastructure.

2. Using the HashiCorp Consul Cluster (opens in a new tab).

Using the CoinAPI EMS Central Cloud is a default, simple, and straightforward method. Only one disadvantage of using the CoinAPI EMS Central Cloud over Consul Cluster is that we do not measure the latencies if one order destination is registered more than one time in the cluster for failover scenarios. Then CoinAPI EMS Central Cloud will connect to the last started exchange; however, when using the Consul Cluster, it will connect to the order destination in the closest proximity data center to the datacenter where CoinAPI EMS API is run.

Running using an executable

To install the precompiled binary, download the latest version of the executable matching your system architecture from http://coinapi-releases.s3-website-us-east-1.amazonaws.com/

Our software is currently packaged as a zip file, and we currently do not plan to distribute it another way. Once the zip is downloaded, unzip it into any directory. The self-contained binary inside is necessary to run EMS, and any additional files are not required to run the software.

If you intend to access it from the command-line, make sure to place the binary somewhere on your PATH. You may require to add the executable to the system startup with all necessary parameters if you want the software to be started automatically after restart.

Running on Docker

#  example using only command line parameters
docker run -d --name=HITBTC --restart=always --net=host coinapi/oeml-api --OEML:ExchangeId HITBTC \
	--OD:PublicApiKey _public_api_key_ --OD:PrivateApiKey _private_api_key_ \
	--CoinAPI:ApiKey _coinapi_api_key_ 
docker run -d --restart=always --net=host coinapi/oeml-webui
docker run --net=host coinapi/oeml-cli

# example with enviroment variables file
cat > oeml.env << EOF
CoinAPI__ApiKey=_coinapi_api_key_

OD__PublicApiKey=_public_api_key_
OD__PrivateApiKey=_private_api_key_
EOF

docker run -d --name=HITBTC --restart=always --net=host coinapi/oeml-api --env-file oeml.env --OEML:ExchangeId HITBTC
docker run -d --name=BITFINEX --restart=always --net=host coinapi/oeml-api --env-file oeml.env --OEML:ExchangeId BITFINEX
docker run -d --name=OEML_EDGE --restart=always --net=host coinapi/oeml-api --env-file oeml.env

Our software is listed in public docker repository. To run the software on docker, use the following images:

• coinapi/oeml-api

• coinapi/oeml-cli

• coinapi/oeml-webui

To pass the configuration parameters using the environment variables (more information about this feature is available in the Configuration Manual section of this documentation), the environment variables must be provided into the container and not set on the host machine.

It's a good practice to lock itself into the specific version. To do that you can supply the version after image name and : separator, e.g. coinapi/oeml-api:1.1255

Running on Kubernetes Cluster

Our software is compatible with Kubernetes, and you can quickly run it on the cluster using our (Helm Charts)[https://github.com/coinapi/helm-charts (opens in a new tab)]. Follow steps below to perform quickstart deployment.

Add the Repository to Helm

$ helm repo add coinapi-charts https://coinapi.github.io/helm-charts/
$ helm repo update

Helm chart configuration

The example config below will create two api pods, a composite pod, and a webui pod. One exchange account corresponds to one entry in the values.yaml file. That means, for multiple exchange accounts, please add one config entry with the matching API keys for each account i.e. binance-01, binance-02, kraken-01, kraken-02, etc. A complete list of all supported exchanges can be found in the API documentation (opens in a new tab).

cat > values.yaml << EOF

# Using WebUI (oeml-api/values.yaml)
webUI:
  enabled: true

# Using composite (oeml-api/values.yaml)
oemlCompositeAPI:
  enabled: true

# Setting the CoinAPI key (oeml-api/values.yaml)
extraEnv:
  - name: CoinAPI__ApiKey
    value: "YOUR_COINAPI_APIKEY"

# Specify listing of the accounts that EMS should manage (oeml-api/acct-dev.yaml)
accounts:
  # Binance spot market 
  - name: "binance"
    env:
      - name: OEML__ExchangeId
        value: "BINANCE"
      - name: OD__PublicApiKey
        value: "XXX"
      - name: OD__PrivateApiKey
        value: "YYY"
        
  # Binance spot testnet
  - name: "binanceuat"
    env:
      - name: OEML__ExchangeId
        value: "BINANCEUAT"
      - name: OD__PublicApiKey
        value: "XXX"
      - name: OD__PrivateApiKey
        value: "YYY"
        
EOF

Install ems-api latest version

$ helm install oeml-api coinapi-charts/oeml-api -f values.yaml

Install ems-api custom version

Version listing available here.

$ helm install oeml-api coinapi-charts/oeml-api -f values.yaml --set oemlAPI.tag={version}

Verify installation

kubectl get pods | grep oeml

Expected output:

oeml-api-binance-7754fc95cc-x2gkk        1/1     Running
oeml-api-binance-test-757c8bbd95-2mxd5   1/1     Running
oeml-api-composite-d5688f8f6-csbwh       1/1     Running
oeml-webui-685cd94d77-hq68j              1/1     Running 

Inspect

Inspect binance api pod:

kubectl logs oeml-api-binance-7754fc95cc-x2gkk

Inspect testnet api pod:

kubectl logs oeml-api-binance-test-757c8bbd95-2mxd5

Expected output for both api pods:

[02:22:04 WRN] Overriding address(es) 'http://+:80'. Binding
to endpoints defined in UseKestrel() instead.
Hosting environment: Production
Content root path: /app
Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.

List services

kubectl get svc | grep oeml

Expected output:

oeml-api-binance        ClusterIP   10.112.9.119    <none>        80/TCP
oeml-api-binance-test   ClusterIP   10.112.15.239   <none>        80/TCP
oeml-api-composite      NodePort    10.112.8.159    <none>        80:30578/TCP
oeml-webui              NodePort    10.112.4.195    <none>        80:30857/TCP              

Connect to EMS API

The composite pod gives unified access to all connected exchanges & accounts. Therefore, the oeml-api-composite is the only pod required to connect to the EMS system.

Connect from within the cluster

Deploy a pod with a shell

kubectl apply -f https://k8s.io/examples/application/shell-demo.yaml

Verify the shell pod is online

kubectl get pod shell-demo

Get api-composite IP address:

kubectl get svc | grep oeml-api-composite
>oeml-api-composite      NodePort    10.112.8.159  

Exec into the pod

kubectl exec --stdin --tty shell-demo -- /bin/bash

CURL IP of the composite pod.

curl 10.112.8.159/v1/balances --header 'Accept: application/json'

CURL DNS of the composite pod.

DNS name may vary depending on cloud provider.

curl oeml-api-composite.default.svc.cluster.local/v1/balances --header 'Accept: application/json'

Expected output:

[
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCE",
  },
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCEUAT",
    "data": [
      {
        "id": "XRP",
        "asset_id_exchange": "XRP",
        "asset_id_coinapi": null,
        "balance": 50000.00000000,
        "available": 50000.00000000,
        "locked": 0.00000000,
        "traded": 0.0,
        "last_updated_by": "EXCHANGE",
        "rate_usd": null
      },
      // more entries
      ]
  }
]

Exit the shell pod

exit 

Delete shell pod

kubectl delete pod shell-demo

Connect cluster to localhost

Port-forward from cluster port 80 to localhost port 8080

 kubectl port-forward service/oeml-api-composite 8080:80

Expected output:

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

CURL account balance.

curl localhost:8080/v1/balances --header 'Accept: application/json'

Expected output:

[
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCE",
  },
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCEUAT",
    "data": [
      {
      // more entries
      },
      ]
  }
]

Management Guide

This section will provide the necessary information on how to manage the CoinAPI EMS software.

To run the EMS cluster, you will need to:

1. (optional) Deploy the HashiCorp Consul Cluster for Service Discovery, if you don't want or can't use our included free of charge discovery service.

2. Run the CoinAPI EMS Edge software instance per each managed exchange account.

3. Run single or multiple instances of the CoinAPI EMS API software.

If the lowest latency is required, start the instance per location or server where the API will is used to minimize latency between API client and the EMS API Server.

5. (optional) Run the CoinAPI EMS WebUI to have GUI to the cluster.

6. Done! All the exchange accounts can be managed using the single API.

Installation

CoinAPI EMS can be installed using several approaches:

• Running using an executable

• Running on Docker

• Running on Kubernetes Cluster

Our software is compiled for the following architectures:

• linux-x64
• linux-arm64
• osx-x64
• osx-arm64
• win-x64
• win-arm64

Take a look at the Starter Guide of this documentation for more information how to run the software in the specific conditions.

Upgrading

Currently, all components are stateless. Upgrading stateless components is limited to replacing the binaries or running the new docker image.

Versioning & Compatibility

Our software is versioned using the Major.Minor(+branch) pattern. Major version number is incremented manually on our side, and its scope is for the EMS project. The Minor version number is incremented on every release automatically separately for every component.

You can assume that our software components will work together when their Major version numbers are the same.

Time synchronization

EMS highly depends on the current time to perform actions and correctly timestamp the data for real-time trading, risk management, compliance, or reporting.

For the time synchronization we strongly recommending:

• chrony (Linux, FreeBSD, NetBSD, Solaris, macOS)

• w32tm (Windows)

If you can't use any of the programs above for reasons like:

• Software has not been ported to my operating system.
• Software doesn't have a driver or it's not compatible with my hardware reference clock, and I can't use other programs like gpsd or ntp-refclock to provide reference time via the SHM or SOCK interface.

Then it's worth trying solutions below (ordered from the best to worst options):

1. ntp (Linux, FreeBSD, NetBSD, OpenBSD, Solaris, macOS, Windows)

2. openntpd (Linux, FreeBSD, NetBSD, OpenBSD, Solaris, macOS)

If need the sub-millisecond or sub-nanosecond accuracy take a look there:

1. Meinberg Radio Clocks (opens in a new tab)

2. White Rabbit (opens in a new tab)

Limits

This section will list all known software limitations.

Changelog

It's a log of: BREAKING CHANGES, FEATURES, IMPROVEMENTS, BUG FIXES or other NOTES per public version.

2.0.12829 (March 18, 2022)

BUG FIXES:

• BITSTAMP: Exchange error info is not reported properly

2.0.12813 (March 17, 2022)

FEATURES:

• Added more detailed order debug information.

2.0.12808 (March 17, 2022)

BUG FIXES:

• BINANCE*: Balances are wiped out after placing order.

2.0.12782 (March 16, 2022)

BUG FIXES:

• BINANCE*: Exchange container can be stuck after it lost connection to the exchange.

2.0.12325 (February 25, 2022)

BUG FIXES:

• Positions in some cases could not be loaded because of the exception.

2.0.12316 (February 25, 2022)

FEATURES:

• DERIBITUAT: Added new integration.

IMPROVEMENTS:

• FTX/FTXUS: Added RestApiUrl optional parameter.
• Split parameters into conditionally required and not required.

BUG FIXES:

• DERIBITUAT: Added new integration.
• BINANCE: AvgPx and Fills data for the order was not available.

1.9281 (September 16, 2021)

FEATURES:

• POLONIEX: WebSocket balances updates added

BUG FIXES:

• POLONIEX: Order update dataflow fixes

1.9136 (September 09, 2021)

BUG FIXES:

• LMAXDIGITAL/LMAXDIGITALUAT: Added Time In Force options: FOK, IOC

1.9066 (August 27, 2021)

FEATURES:

• FTXUS: Added FTX.US exchange on the FTXUS identifier.

1.8931 (August 11, 2021)

FEATURES:

• Ability to control the refresh rate behaviour of exchange rates from CoinAPI trough configuration parameter.

1.8893 (August 06, 2021)

IMPROVEMENTS:

• Extended error log info on all BINANCExxx exchanges

1.8889 (August 06, 2021)

BUG FIXES:

• BINANCEOPTV/BINANCEOPTVUAT fixed invalid balance calculations

1.8877 (August 05, 2021)

FEATURES:

• BINANCEOPTV: Added BINANCE Vanilla Options exchange on the BINANCEOPTV identifier.
• BINANCEOPTVUAT: Added BINANCE Vanilla Options testnet exchange on the BINANCEOPTVUAT identifier.

1.8821 (July 29, 2021)

FEATURES:

• BINANCEFTS: Added BINANCE USDⓈ-M Futures exchange on the BINANCEFTS identifier.
• BINANCEFTSUAT: Added BINANCE USDⓈ-M Futures testnet exchange on the BINANCEFTSUAT identifier.
• BINANCEFTSC: Added BINANCE COIN-M Futures testnet exchange on the BINANCEFTSC identifier.
• BINANCEFTSCUAT: Added BINANCE COIN-M Futures testnet exchange on the BINANCEFTSCUAT identifier.

1.8797 (July 23, 2021)

IMPROVEMENTS:

• Improved order parameters validation for all exchanges

1.8787 (July 22, 2021)

BREAKING CHANGES:

• Replaced MESSAGE with MESSAGE_REJECT.

IMPROVEMENTS:

• Improved the error reporting by introducing Message Rejection.

1.8781 (July 22, 2021)

FEATURES:

• BITMEXUAT: Added BITMEX testnet exchange on the BITMEXUAT identifier.

1.8713 (July 14, 2021)

BUG FIXES:

• COINBASE/DERIBIT/FTX: Fix in the startup procedure.
• BINANCE*/BITSTAMP/KRAKEN/POLONIEX: Fixed bug that prevented multiple accounts to coexist with each other.

1.8707 (July 13, 2021)

FEATURES:

• BINANCEUAT: Added BINANCE testnet exchange on the BINANCEUAT identifier.
• BINANCE: Migrated to the /sapi from the /wapi endpoints.
• BINANCEUS: Upgraded to the v3 version of the /sapi endpoints.

IMPROVEMENTS:

• Error reporting got better when the user provided invalid parameters
• When we cant detect the local IP address then it's properly communicated
• When the CoinAPI key is a valid Guid, but not allowed to use the service then the error is more descriptive.

Integrations & Plugins

This section is reserved for Integrations, plugins or other external tools or modules that make it easier to work with EMS API.

Prometheus metrics

In this section, we will describe the metrics exposed by the software on the /metrics endpoint.

If you are using our Helm chart then ServiceMonitor objects required by Prometheus to pull data are automatically installed by default version of the values file (helm-charts/charts/oeml-api/values.yaml).

Published metrics:

Deployment

In this section we are listing tools that could be used to deploy or manage our software.

• Helm charts (opens in a new tab)
• Artifact HUB (opens in a new tab)