Tutorial

Install

See Installation page to access Ackal's container images and to register.

Console

Ackal includes a web-based Console that provides administrative functionality including login and creating a subscription to Ackal.

Ackal uses Firebase Authentication. This service supports various OAuth providers and enables you to choose your preferred provider(s) to login to Ackal. Once you are signed in, you can download your credentials as a JSON file to configure ackalctl.

Ackal uses Stripe. This service is a trusted payments service and enables you to subscribe to Ackal. Even if signed in, you will not be able to use Ackal until you purchase a subscription.

Console displays user information and access to additional functionality:

Login

Ackal supports several federated identity providers

NOTE contact us if you want to associated different identities with your Customer ID.

Download

Clicking DOWNLOAD will download a Credentials file (credentials.json) to your machine.

ackalctl looks for this file in ${HOME}/.config/ackalctl/credentials.json.

Subscribe

Clicking SUBSCRIBE will redirect you to Stripe and to its subscription signup page.

You may enter your billing details on this page to subscribe to Ackal.

Billing

Clicking BILLING will redirect you to Stripe and to your subscription details page.

You may view details of your subscription and cancel your subscription on this page.

Credentials

Ackal's Console provides a way for you to Download your credentials as a JSON file (typically credentials.json).

ackalctl stores user credentials in ${HOME}/.config/ackalctl/credentials.json.

You will need a Customer ID. A Customer ID represents one (individual) or many (organizational) users.

After subscribing to Ackal, you will need to request a Customer ID before you may begin creating health checks against your gRPC services.

NOTE Once created in Console, your credentials will expire within 60 minutes. Once you begin using them with ackalctl, it will renew them for you indefinitely. If you need to manually refresh your Credentialsd in Console, log out and back in.

Domains

You are permitted to create health checks against servers on

Before using domains that you own/control, you will need to generate a verification code and then add a DNS record to your domain to verify the ownership to Ackal:


DOMAIN="[YOUR DOMAIN]"

# Create the domain's record
ackalctl \
--domain_name=${DOMAIN} \
create domain
                            

Either:


DOMAIN="[YOUR DOMAIN]"

# Get the domain's record
ackalctl \
--domain_name=${DOMAIN} \
get domain
                            

NOTE The result includes a field verified that will be false until the domain has been verified.

Or:


DOMAIN="[YOUR DOMAIN]"

# Get the domain's verification code
VERIFICATION=$(\
  ackalctl \
  --domain_name=${DOMAIN} \
  --format=json \
  get domain | jq -r .verification) \
&& echo ${VERIFICATION}
                            

And then ensure that your DOMAIN DNS records include:

Name Type Data
TXT "ackal-verfication={VERIFICATION}"
NOTE The requirement for NAME varies by DNS provider. Some permit the NAME field to be left blank, others require e.g. @ as a placeholder. Consult the provider's documentation.

You may use Linux tools (e.g. dig or nslookup) or Google's Admin Toolbox Dig to verify that the TXT record has been created correctly.

When you're confident the record has propagated, you can instruct Ackal to verify the domain:


DOMAIN="[YOUR DOMAIN]"

ackalctl \
--domain_name=${DOMAIN} \
verify domain
                            

You can confirm that the domain has been verified successfully:


DOMAIN="[YOUR DOMAIN]"

# Get the domain's record
ackalctl \
--domain_name=${DOMAIN} \
get domain
                            
NOTE The result includes a field verified that will be true if Ackal was able to verify the domain.

Health Checks

Once you have ackalctl installed, have logged in and downloaded your credentials, and have acquired a Customer ID, you're ready to begin creating health checks against your gRPC services.

You'll need gRPC services that implement the gRPC Healthcheck Protocol. If you do not have your own gRPC services to test, there is an example gRPC endpoint for you to use:

  • healthcheck-server.ack.al:443

You can list the service's implemented on this endpoint and you can invoke one of the Health Checking Protocol's methods using a general-purpose tool such as gRPCurl to confirm that it is working correctly:


ENDPOINT="healthcheck-server.ack.al:443"

grpcurl \
${ENDPOINT} \
list

grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
                            

ENDPOINT="healthcheck-server.ack.al:443"

grpcurl \
-d '{"service":"grpc.health.v1.Health"}'
${ENDPOINT} \
grpc.health.v1.Health/Check
                            
NOTE Per the Health Checking Protocol, you may optionally specify the gRPC service name to be checked. The endpoint healthcheck-server.ack.al not only implements the Health Check Protocol as grpc.health.v1.Health but -- as shown above -- for convenience, it uses grpc.health.v1.Health as a service that you can health check. You may also use the empty string ("") or omit the data (-d) entirely to check the endpoint's overall health.
NOTE When you reference these gRPC service (using --endpoint with ackalctl, it is important that you not prefix the host:port value with a scheme (i.e. no https://) and you must include the port (i.e. :443). Web browsers generally assume :443 when you're using https but this assumption is not valid with gRPC.

To test that ackalctl is working correctly:


ackalctl list locations
                            

Yields:


ID              Name
las-vegas       Las Vegas
london          London
los-angeles     Los Angeles
salt-lake-city  Salt Lake City
the-dalles      The Dalles
tokyo           Tokyo
                            
NOTE ackalctl supports JSON and YAML output formatting. Simply append --format=json or --format=yaml to any command, e.g. ackalctl --format=json list locations:

{
    "locations":[
      {"id":"las-vegas","location":{"name":"Las Vegas"}},
      {"id":"london","location":{"name":"London"}},
      {"id":"los-angeles","location":{"name":"Los Angeles"}},
      {"id":"salt-lake-city","location":{"name":"Salt Lake City"}},
      {"id":"the-dalles","location":{"name":"The Dalles"}},
      {"id":"tokyo","location":{"name":"Tokyo"}}
    ]
}
                            

Then create a health check:


ENDPOINT="healthcheck-server.ack.al:443"
PERIOD="120s"
LOCATION_IDS="las-vegas,the-dalles"
SERVICES="ServiceA,ServiceB"

ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
--location_ids=${LOCATION_IDS}
--services=${SERVICES}
create check
                            
NOTE Health checks are uniquely identified by the service endpoint and the period.

Returns:


id: {ID}
                            

Then:


ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
get check
                            

Returns:


endpoint: {ENDPOINT}
period: {PERIOD}
locationIDs: {LOCATION_IDS}
services: {SERVICES}
enabled: false
                            

Or:


ackalctl list checks
                            

Returns:


Endpoint     Period     Location(s)      Service(s)   Enabled
${ENDPOINT}  ${PERIOD}  ${LOCATION_IDS}  ${SERVICES}  ${ENABLED}
                            

You may delete health checks:


ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
delete check
                            
NOTE Because Healthchecks are uniquely identified by the service endpoint and the period, both values are required to delete a health check.

Metrics

Ackal publishes Prometheus metrics for the health checks that you create. The simplest way to interact with your health checks' metrics is to use Prometheus (and possibly AlertManager and Grafana) to record, alert and graph them.

Ackal provides 2 proxies (ackal-http-proxy and ackal-file-proxy) that perform service discovery for Prometheus. Running either (or both) of these proxies locally, they will leverage your credentials to securely access your Healthcheck so that their metrics may be scraped by Prometheus.

You will need to have at least one health check:


ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
get check
                            

Returns:


endpoint: {ENDPOINT}
period: {PERIOD}
locationIDs: {LOCATION_IDS}
services: {SERVICES}
enabled: false
                            
OPTIONAL

Ackal's proxies discover your health check endpoints using an Ackal service called listr. You may use listr directly using a tool like curl. The following commands use curl, jq and awk to show how you can interact with listr directly:


CUSTOMER="[YOUR CUSTOMER ID]"
TOKEN=$(more ${HOME}/.config/ackalctl/credentials.json | jq -r '.jwt')

LISTR="https://listr.ack.al:443"

curl \
--silent \
--request GET \
--header "Ackal-Customer: ${CUSTOMER}" \
--header "Authorization: Bearer ${TOKEN}" \
${LISTR}/?format=prometheus \
| jq -r .
                                

If you omit the query string (?format=prometheus), the resulting JSON will be in Ackal's native format.

This command lists your Ackal health check service endpoints in a Prometheus-friendly format:


[
  {
    "targets": [
      "{url}"
    ],
      "labels": {
      "endpoint": "{endpoint}",
      "location": "{location}",
      "period": "{period}"
    }
  }
]
                                

You can then curl one of the endpoint values returned, to query the specific health check for its metrics. This time we'll use awk to filter the Prometheus metrics to those prefixed ackal_exporter_code:


CUSTOMER="[YOUR CUSTOMER ID]"
TOKEN=$(more ${HOME}/.config/ackalctl/credentials.json | jq -r 'jwt')

ENDPOINT="[ONE OF THE ENDPOINT VALUES]"

curl \
--silent \
--request GET \
--header "Ackal-Customer: ${CUSTOMER}" \
--header "Authorization: Bearer ${TOKEN}" \
${ENDPOINT}/metrics \
| awk '\^ackal_exporter_check {print}
                                

It is more convenient to configure a Prometheus server (or equivalent) to perform service discovery of your health checks. To do this, you will need to:

  1. Run ackalctl-http-proxy
  2. Run Prometheus
  3. Optionally: run Grafana

Grafana

You can interact with Ackal's health check metrics through Grafana.


VERS="9.2.3"

docker run \
--interactive --rm --tty \
--net=host \
docker.io/grafana/grafana-oss:${VERS}
                        
NOTE In order for Grafana to access Prometheus, Grafana must (also) be bound to your host's network with --net=host.

You should now be able to browse Grafana (http://localhost:3000).

The default username|password are both admin.

You will want to add Prometheus as a datasource (http://localhost:3000/datasources). The URL will be the default value (http://localhost:9090) but you must type this in. The "Save & Test" to confirm that it works.

Then you can create a new Dashboard (http://localhost:3000/dashboard) and for the value of Metrics Browser, you can use:


avg
  without(
    endpoint,
    job,
    status,
  )
  (ackal_exporter_check_histogram_ms_bucket{
    job="ackalctl-http-proxy",
  })
                        

You can import an example Ackal Check Dashboard that includes dropdowns that enable you to choose which Check and Location you would like to view.

If you'd prefer to build something for yourself based on the query shown above, you'll want to filter by a specific instance and consider using Grafana's Histogram visualization tool.

Frequently Asked Questions (FAQ)

Public Domains

Ackal supports the creation of health checks against services running on public domains. These are domains -- often provided by cloud providers -- that are used to host services but which the developer does not own or control.

For this reason, health checks created against these domains are permitted without the user first needing to verify ownership of the domain.

For public (non-authenticated) services on these domains -- as with any other tool -- Ackal is able to access any service hosted on these domains.

Ackal supports the following public domains:

Domain Service Provider
appspot.com App Engine Google
cloudfunctions.net Cloud Functions Google
a.app.run Cloud Run Google
fly.dev fly.io Fly

Ackal Proxies and TLS

When you configure Prometheus to use ackalctl-file-proxy or ackalctl-http-proxy you must use http://listr.ack.al:443 (http but 443) as the proxy_urlvalue URL for Ackal’s listr service even though Ackal's listr service uses TLS (https).

This is to simplify your use of the proxies as it saves you having to configure between Prometheus and the proxy. Instead Prometheus uses plaintext http to communicate with ackalctl-http-proxy but the proxy itself communicates with Ackal using encrypted https.

For this reason, you should not run ackalctl-http-proxy outside your secure network perimeter.

What is gRPC Health Checking protocol?

Health checks are used by clients to determine the state of health of a server. The server's health check implementation should accurately report on the ability of the server to respond to client requests. For example, if the server depends on a database to respond to clients, if the database is down, the server is unhealthy and should report itself so. Often, particularly with HTTP/REST APIs, a server is deemed healthy if e.g. GET'ting a method's endpoint returns a 200.

Because gRPC is a protocol layer above HTTP, it's not possible to use HTTP to check on a gRPC server's health. Additionally, because gRPC servers implement arbitrary sets of methods, there's no guarantee that every gRPC server will implement a health checking method, let alone that they will agree on the method name and request and response message types.

For these reasons, the gRPC Health Checking protocol is a de facto standard. If gRPC servers implement the Health Checking protocol, then clients know that the server will implement a grpc.health.v1 package with a Health service and with Check and Watch methods both accepting HealthCheckRequest's and returning HealthCheckResponse's.

You can confirm whether a gRPC service implements the Health Checking protocol by listing its services or by invoking one of the Health Checking protocol methods directly:


ENDPOINT="healthcheck-server.ack.al:443"

grpcurl \
${ENDPOINT} \
list

grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
                                

ENDPOINT="healthcheck-server.ack.al:443"

grpcurl \
${ENDPOINT} \
list grpc.health.v1.Health

grpc.health.v1.Health.Check
grpc.health.v1.Health.Watch
                                

ENDPOINT="healthcheck-server.ack.al:443"

grpcurl \
-d '{"service":"grpc.health.v1.Health"}'
${ENDPOINT} \
grpc.health.v1.Health/Check
                                

Ackal requires the gRPC Health Check protocol be implemented by gRPC services so it can provide health checks.