Kubernetes Operator for API Management

11 min readOct 29, 2019

In two prior posts, API Microgateway and API Management and Service Mesh, I discussed how API management has evolved with developer-centric and cloud-native approaches.

In this article, I will discuss how to extend the Kubernetes platform to handle cloud-native API management.

Kubernetes

Kubernetes is an open-source platform for automating deployment, scaling, and managing containerized workloads across multiple host machines. Combined with best-of-breed ideas and practices from the community, it builds upon Google’s 15 years of experience in running production workloads. I was lucky to have been exposed to Kubernetes at an early stage, and we (Apache Stratos project) integrated with Kubernetes at very early stages (v0.5.1) back at the end of 2014. Since then, I have been advocating Kubernetes internally (at WSO2) and externally, envisioning Kubernetes as the future for distributed operating systems to manage workloads.

I am most passionate about Kubernetes’s extensible architecture and its community-driven governance. Custom resources (CR) and Custom Controllers are the key central extension mechanisms used throughout the Kubernetes ecosystem.

Custom Resources and Custom Controllers

A custom resource is an extension of the Kubernetes API; it is not shipped with the default Kubernetes installation. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources such as Pods. Custom resources have been available in every Kubernetes cluster since version 1.7. They are stored in the same etcd instance as the main Kubernetes API resources and are served by the same Kubernetes API server.

Custom controllers help to keep the current state of Kubernetes objects (defined in CR) in sync with the desired state. The controller interprets the structured data as a record of the user’s desired state, and continually maintains this state. So, when you combine a custom resource with a custom controller, custom resources provide a true declarative API.

Kubernetes Operators

Kubernetes is designed for automation. But when increasing the complexity of the applications, it leads to increase in the DevOps burden and increases the likelihood of error.

The operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides. Operators combine custom resources and custom controllers, while CR provides APIs the users’; controllers will implement the functionalities of these APIs. With this extensible architecture, we can extend the Kubernetes cluster’s behavior without modifying the code of Kubernetes itself.

WSO2 API Microgateway

The open API definition is considered the single source of truth to the WSO2 API Microgateway. This Open API definition contains all the required information regarding your API. By providing this definition to the WSO2 API Microgateway toolkit, you can create an API microgateway.

In my previous article (API Microgateway), I discussed different deployment patterns of a microgateway. Typically, it requires an Ops person to carry out a few more steps. After you receive the microgateway binary, you have to create a microgateway Docker image and relevant Kubernetes YAML files (deployment.yaml, service.yaml, configmap.yaml, secret.yaml). These manual steps will slow the production pipeline. To minimize this manual interference, the previous WSO2 Microgateway toolkit had the capability to generate the Docker image and required Kubernetes YAML files.

WSO2 APIM Operator

WSO2 APIM Operator provides a fully automated experience for cloud-native API management. When a user wants to expose an API for the service that has been created, he/she only needs to provide the Open API definition to the `apictl` tool.

APIM Operator introduced four new custom resource definitions related to the API management domain.

Custom resource: Security:

`Security` holds security-related information. You can see the API definition and data structure for `Security` here. Security supports different security types: basic-auth, OAuth2, JWT, etc. The following YAML shows a sample payload for Security with JWT.

Custom resource: RateLimiting:

`RateLimiting` holds rate-limiting related information. You can see the API definition and data structure for `RateLimiting` here. The following YAML shows sample payload.

Custom resource: TargetEndpoint:

`TargetEndpoint` holds endpoint related information. You can see the API definition and data for `TargetEndpoint` here. WSO2 API Microgateway can be deployed in three patterns: shared, private-jet, and sidecar (Refer to this API Microgateway article for more information). If your backend is already running and you need to expose it via a microgateway, you can define the target URL in the Swagger itself. If your backend service is not running, but you plan to run it in the same Kubernetes cluster, you can use `TargetEndpoint` with its relevant Docker image. Then APIM Operator will spin-up the corresponding Kubernetes deployment for the defined backend service itself with the microgateway. In shared and private-jet mode, the backend can be running in separate PODs, but in sidecar mode, the gateway will run in the same POD adjacent to the backend service. The following YAML shows a sample payload for TargetEndpoint.

Custom resource: API:

`API` holds API-related information. You can see the API definition and data structure for `API` here. API takes the Swagger definition as a configMap along with replica count and microgateway deployment mode. The following YAML shows sample payload for `API`.

Each custom API has corresponding custom controllers. Custom controllers are the “brains” behind the custom resources.

Custom Controller: Security

The security controller will store user-defined security policies corresponding to the Security API and creates a Security secret. It supports JWT, Oauth2, and basic security types out-of-the-box. When running the Kaniko job by the API controller, it will add to the Keystore and then the Keystore will be added to the microgateway Docker image. Refer to a Security controller implementation here.

Custom Controller: RateLimiting

The RateLimiting controller will store the user-defined policy corresponding to the RateLimit API in addition to default policies provided out-of-the-box. It also creates policy template configMaps. When a new RateLimiting policy is added, we update that policy template config map. When running the Kaniko job by the API controller, it takes this policy template configMap and uses it to build the Docker image. Please refer to a RateLimiting controller implementation here.

Custom Controller: TargetEndpoint

The TargetEndpoint controller will store target endpoint metadata corresponding to the TargetEndpoint API. If the mode of the target endpoint is privateJet, it will create Deployment, Service and PODs for relevant backend services. If the mode is sidecar, it will store the definition and when we add API definition with the TargetEndpoint, it will create PODs with the gateway attached as a sidecar to the service. You can see a TargetEndpoint controller implementation here.

Custom Controller: API

API controller is quite complex compared to other controllers. It has two main tasks.

Build an API microgateway container and push it to the Docker registry.
Create Kubernetes artifacts and deploy them into Kubernetes clusters.

When the API custom controller is triggered, it will receive a Swagger definition from the attached configMap and create a Kaniko job by attaching a multi-step Dockerfile along with the Swagger definition. This Dockerfile is used to pre-build the Docker image that has the API microgatewayy toolkit. The microgateway toolkit will generate the API microgateway runtime with the corresponding swagger file passed. Finally, Kaniko builds a new API microgateway docker image and pushes it to the configured docker registry.

After finishing step one, the API controller will start creating relevant Kubernetes artifacts corresponding to the API definition. Depending on defined API mode, it will create Kubernetes deployment for both API microgateway and backend services.

As you can see, API Controller has taken out all the complexity from DevOps and automates deployment with all the best practices required to deploy API microgateway along with microservices architecture.

Deploy APIM Controller

The most common way to deploy an Operator is to add the Custom Resource Definition and its associated Controller to your cluster. The Controller will normally run outside of the control plane, much as you would run any containerized application. For example, you can run the controller in your cluster as a Deployment.

Follow the steps below to configure APIM Operator in your Kubernetes cluster.

Issue following kubectl command
Deploying CRDs for API, TargetEndpoint, Security, RateLimiting

kubectl apply -f deploy/crds/

3. Deploying namespace, roles/role binding and service account associated with the operator

kubectl apply -f deploy/controller-artifacts/

4. Deploying controller level configurations.

Before deploying controller configs you need to change two config values.
You need to update the docker registry in the controller_conf.yaml. API controller will push created API microgateway docker images in this docker registry.
Enter the base 64 encoded username and password of the user’s docker registry into the docker_secret_template.yaml.
After you updates those two values you can deploy controller configs

kubectl apply -f deploy/controller-configs/

If you successfully configured APIM operator, you can find the following CRDs and controllers running in your Kubernetes cluster.

lakmal$ kubectl get crdNAME                       CREATED 
ATapis.wso2.com            2019–10–16T23:38:01Z
ratelimitings.wso2.com     2019–10–16T23:38:01Z
securities.wso2.com        2019–10–16T23:38:01Z
targetendpoints.wso2.com   2019–10–16T23:38:01Z lakmal$ kubectl get all -n wso2-system
NAME                               READY STATUS  RESTARTS AGE
pod/apim-operator-7c86cd5d7f-nr8sq 1/1   Running 0        29mNAME TYPE                  CLUSTER-IP EXTERNAL-IP   PORT(S)      AGE
service/apim-operator ClusterIP  10.110.93.134 8383/TCP     29mNAME                          READY UP-TO-DATE AVAILABLE AGE
deployment.apps/apim-operator 1/1   1          1         29mNAME                                     DESIRED CURRENT READY AGE
replicaset.apps/apim-operator-7c86cd5d7f 1       1       1     29m

Deploy an API in K8s cluster via APIM Operator

Now that you have successfully configured APIM Operator, let's see how we can expose our service via API microgateway with APIM operator support. You can find several samples in the scenarios folder in the git repo.

Let’s try the simplest scenario. In this scenario we will deploy a sample microservice separately and then apply API management.

Deploy sample microservice backend.

kubectl apply -f ./scenarios/scenario-1/products_dep.yaml
kubectl get services products
NAME     TYPE         CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
products LoadBalancer 10.83.1.131  104.197.114.248  80:30475/TCP  27m

Issue following commands to test our microservice.

curl -X GET http://localhost:80/products{“products”:[{“name”:”Apples”, “id”:101, “price”:”$1.49 / lb”}, {“name”:”Macaroni & Cheese”, “id”:151, “price”:”$7.69"}, {“name”:”ABC Smart TV”, “id”:301, “price”:”$399.99"}, {“name”:”Motor Oil”, “id”:401, “price”:”$22.88"}, {“name”:”Floral Sleeveless Blouse”, “id”:501, “price”:”$21.50"}]}curl -X GET http://localhost:80/products/101{“name”:”Apples”, “id”:101, “price”:”$1.49 / lb”, “reviewScore”:”0", “stockAvailability”:false}

Let’s now deploy an API for our microservice. The definition of the API can be found in the scenario/scenario-1/products-swagger.yaml. The endpoint of our microservice has been referred to in the API definition.

`apictl` is provided to use for APIM Operator related operations. You can get the installation of `apictl` from here. Set the `apictl` mode to Kubernetes to be compatible with kubectl commands

apictl set --mode k8sapictl add api -n online-store — from-file=scenarios/scenario-1/products_swagger.yamlcreating configmap with swagger definition
configmap/online-store-swagger created
api.wso2.com/online-store created

Let's verify the deployment.

kubectl get allNAME                          COMPLETIONS DURATION AGE
job.batch/online-store-kaniko 0/1         47s      47s

First, you may notice a Kaniko job that is running. In that stage, the Operator will create a Docker image corresponding to the petstore API microgateway and then it will upload into the docker registry. The Docker image creation only happens at the first API microgateway deployment. Thereafter, it will use the existing Docker image to perform the pod autoscaling.

After completion above step, the Operator will deploy API microgateway into the Kubernetes cluster.

lakmal$ kubectl get allNAME                                   READY  STATUS    RESTARTS AGE pod/online-store-kaniko-c6vnk          0/1    Completed 0        87s pod/products-deployment-74d757bb-2grp9 1/1    Running   0        14m NAME               TYPE      CLUSTER-IP EXTERNAL-IP PORT(S)     AGE
service/kubernetes ClusterIP 10.96.0.1  <none>      443/TCP     48d
service/products   LoadBalancer 10.104.110.143 localhost 80:30462/TCP   14m NAME                                  READY UP-TO-DATE AVAILABLE AGE deployment.apps/products-deployment   1/1   1          1         14m NAME                                     DESIRED  CURRENT  READY AGE
replicaset.apps/products-deployment-84757bb   1    1       1    14m NAME   REFERENCE TARGETS         MINPODS   MAXPODS   REPLICAS   AGE hpa.autoscaling/online-store-hpa Deployment/online-store   <unknown>/50%   1         5         0          87s NAME                              COMPLETIONS   DURATION   AGE
job.batch/online-store-kaniko      1/1           84s       87s

As you can see, it has deployed the online-store deployment, K8s service with LB type and HPA.

Since the API is secured now, you need a valid access token to invoke the API. You can find a sample token below.

TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IlpqUm1ZVE13TlRKak9XVTVNbUl6TWpnek5ESTNZMkl5TW1JeVkyRXpNamRoWmpWaU1qYzBaZz09In0.eyJhdWQiOiJodHRwOlwvXC9vcmcud3NvMi5hcGltZ3RcL2dhdGV3YXkiLCJzdWIiOiJhZG1pbkBjYXJib24uc3VwZXIiLCJhcHBsaWNhdGlvbiI6eyJvd25lciI6ImFkbWluIiwidGllciI6IlVubGltaXRlZCIsIm5hbWUiOiJzYW1wbGUtY3JkLWFwcGxpY2F0aW9uIiwiaWQiOjMsInV1aWQiOm51bGx9LCJzY29wZSI6ImFtX2FwcGxpY2F0aW9uX3Njb3BlIGRlZmF1bHQiLCJpc3MiOiJodHRwczpcL1wvd3NvMmFwaW06MzIwMDFcL29hdXRoMlwvdG9rZW4iLCJ0aWVySW5mbyI6e30sImtleXR5cGUiOiJQUk9EVUNUSU9OIiwic3Vic2NyaWJlZEFQSXMiOltdLCJjb25zdW1lcktleSI6IjNGSWlUM1R3MWZvTGFqUTVsZjVVdHVTTWpsUWEiLCJleHAiOjM3MTk3Mzk4MjYsImlhdCI6MTU3MjI1NjE3OSwianRpIjoiZDI3N2VhZmUtNTZlOS00MTU2LTk3NzUtNDQwNzA3YzFlZWFhIn0.W0N9wmCuW3dxz5nTHAhKQ-CyjysR-fZSEvoS26N9XQ9IOIlacB4R5x9NgXNLLE-EjzR5Si8ou83mbt0NuTwoOdOQVkGqrkdenO11qscpBGCZ-Br4Gnawsn3Yw4a7FHNrfzYnS7BZ_zWHPCLO_JqPNRizkWGIkCxvAg8foP7L1T4AGQofGLodBMtA9-ckuRHjx3T_sFOVGAHXcMVwpdqS_90DeAoT4jLQ3darDqSoE773mAyDIRz6CAvNzzsWQug-i5lH5xVty2kmZKPobSIziAYes-LPuR-sp61EIjwiKxnUlSsxtDCttKYHGZcvKF12y7VF4AqlTYmtwYSGLkXXXwcurl -X GET “https://localhost:9095/store/v1.0.0/products" -H “Authorization:Bearer $TOKEN” -k{“products”:[{“name”:”Apples”, “id”:101, “price”:”$1.49 / lb”}, {“name”:”Macaroni & Cheese”, “id”:151, “price”:”$7.69"}, {“name”:”ABC Smart TV”, “id”:301, “price”:”$399.99"}, {“name”:”Motor Oil”, “id”:401, “price”:”$22.88"}, {“name”:”Floral Sleeveless Blouse”, “id”:501, “price”:”$21.50"}]}curl -X GET “https://localhost:9095/store/v1.0.0/products/101" -H “Authorization:Bearer $TOKEN” -k{“name”:”Apples”, “id”:101, “price”:”$1.49 / lb”, “reviewScore”:”0", “stockAvailability”:false}

If you wish to use production-grade full lifecycle API management (API discovery, secure token management etc) then you can install WSO2 APIM manager into the Kubernetes cluster.

Deploy WSO2 APIM Manager

If you wish to use full lifecycle management (API discovery, token management etc) then you can install WSO2 APIM manager into the Kubernetes cluster.

The following kubectl commands will deploy the WSO2 API Manager deployment into the Kubernetes cluster. The following command will deploy API portal & token service under a namespace called “wso2”.

kubectl apply -f k8s-artifacts/api-portal/namespace "wso2" created
configmap "apim-conf" created
deployment.apps "wso2apim" created
service "wso2apim" created

Pushing the API to the API Portal

To make the API discoverable for other users and get the access tokens, we need to push the API to API portal.

apimcli add-env -e k8s --registration https://wso2apim:32001/client-registration/v0.15/register --apim https://wso2apim:32003 --token https://wso2apim:32003/token --admin https://wso2apim:32001/api/am/admin/v0.15 --api_list https://wso2apim:32001/api/am/publisher/v0.15/apis --app_list https://wso2apim:32001/api/am/store/v0.15/applicationapictl init online-store -oas=./scenarios/scenario-1/products_swagger.yamlInitializing a new WSO2 API Manager project in /Users/lakmal/k8s-apim-operator/online-storeProject initializedOpen README file to learn moreSuccessfully added environment ‘k8s’

Import the API to the API portal. (You need to change the API life cycle status to PUBLISHED before importing the API. You can edit the api.yaml file located in online-store/Meta-information/)

apimcli import-api -f online-store/ -e k8s -kLogin to k8s
Username:admin
Password:
Logged into k8s environment
Successfully imported API

If you navigate the API Portal (https://wso2apim:32001/devportal/apis), you can see our online store API is listed in the API devportal.

Now let’s generate an access token to access our API.

By default, the API is secured with JWT. Hence a valid JWT token is needed to invoke the API. You can obtain a JWT token using the `apictl` command as below.

apictl set token-type JWTToken type set to: JWTapictl get-keys -n OnlineStore -v v1.0.0 -e k8s — provider admin -kAPI name: OnlineStore & version: v1.0.0 exists
API OnlineStore : v1.0.0 subscribed successfully.
Access Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IlpqUm1ZVE13TlRKak9XVTVNbUl6TWpnek5ESTNZMkl5TW1JeVkyRXpNamRoWmpWaU1qYzBaZz09In0=.eyJhdWQiOiJodHRwOlwvXC9vcmcud3NvMi5hcGltZ3RcL2dhdGV3YXkiLCJzdWIiOiJhZG1pbkBjYXJib24uc3VwZXIiLCJhcHBsaWNhdGlvbiI6eyJvd25lciI6ImFkbWluIiwidGllciI6IlVubGltaXRlZCIsIm5hbWUiOiJkZWZhdWx0LWFwaW1jbGktYXBwIiwiaWQiOjJ9LCJzY29wZSI6ImFtX2FwcGxpY2F0aW9uX3Njb3BlIGRlZmF1bHQiLCJpc3MiOiJodHRwczpcL1wvd3NvMmFwaW06OTQ0M1wvb2F1dGgyXC90b2tlbiIsInRpZXJJbmZvIjp7fSwia2V5dHlwZSI6IlBST0RVQ1RJT04iLCJzdWJzY3JpYmVkQVBJcyI6W3sic3Vic2NyaWJlclRlbmFudERvbWFpbiI6ImNhcmJvbi5zdXBlciIsIm5hbWUiOiJPbmxpbmVTdG9yZSIsImNvbnRleHQiOiJcL3N0b3JlXC92MS4wLjBcL3YxLjAuMCIsInB1Ymxpc2hlciI6ImFkbWluIiwidmVyc2lvbiI6InYxLjAuMCIsInN1YnNjcmlwdGlvblRpZXIiOm51bGx9XSwiY29uc3VtZXJLZXkiOiJLMnBTbXFsOXZqYmxQS0I4Zk9nUUw2YWZrSjBhIiwiZXhwIjoxNTcxODU2MzU2LCJpYXQiOjE1NzE4NTI3NTYsImp0aSI6IjcwMTAwMmZmLTczZWQtNDRkYy1hMDE1LTMwMzhlYTZlM2IwZiJ9.Vhe8VIuqKF_4q41U8Yo5wEvAAz3gjj_UxdmsUelg16cWEr8l7aFVV-KUcPIhtTtXAGqUCtyAYSjqRVKcCi_EaGJASg56W1cOcYgJg_bfm9xQcMJ416qk6H1cPXNoUuk0zZhdW7WIHFVHwRjD8oadvZqVFJB702Fev02OQ1WnosjMrKmb52oph8IuLYmyM2HOAmBEt5a_ENr1TUQ7KLg29m8k0xP48RpGxiSzEDGAXiTKAgx0PhPYy9Fbloy1a2wyKBAENUV9AMU7oEZpftPR-JpPdTdW7wAcfKrAHPHZoUbDq551sf9nBc0O0Gix9ezi8ZLW0l9x3vy_52-P55oPfw==

Note: You also have the option to generate an access token by logging into the devportal.

You can find several sample artifacts for many other user scenarios in the APIM Operator git repo.

Sample 1: Expose a K8s service as an API
Sample 2: Basic Petstore Sample
Sample 3: Secure an API with basic authentication
Sample 4: Secure an API with JWT
Sample 5: Secure an API with OAuth2 tokens
Sample 6: Apply rate-limiting for an API
Sample 7: Private jet mode for API and Endpoint
Sample 8: Sidecar mode for API and Endpoint
Sample 9: Expose an API with multiple service endpoints