AWS STS and Kubernetes: How to develop locally?

Kubernetes, OpenShift, Containers, AWS, STS
Page content

Introduction

AWS STS-based Kubernetes clusters are common practice and a widely known concept today. Application platforms such as Red Hat OpenShift Service on AWS (or even self-managed OpenShift) and Managed Kubernetes distributions such as Amazon Elastic Kubernetes Service (EKS) use this concept. The concept allows for the use short-lived tokens (as opposed to access keys) by assigning identities to different components of the cluster.

For workloads that run in the cluster, ROSA/EKS both use a project called pod-identity-webhook which allows for workloads in the cluster to assume an IAM role configured with a specific set of permissions for that workload. It is a great concept which allows workloads to only be permitted permissions that they need. This is known as IAM Roles for Service Accounts.

The challenge when tested against this (ensure your workloads have proper permissions) is that you usually need to spin up an STS cluster in AWS to begin testing. This can be both expensive and time-consuming. To help with this issue, I was curious if there was a way to use something like a local KIND cluster, and use the Pod Identity Webhook in order to move some of the testing workflow locally.

I took most of my steps from https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/SELF_HOSTED_SETUP.md but this aims to make the walkthrough a bit simpler and specific to KIND for local development (as opposed to any Kubernetes distro).

Walkthrough

Setup

  1. Setup the environment. These are variables used in other steps throughout the following walkthrough steps:
# signing key configuration
export CONFIG_DIR="./config"
export PRIV_KEY="${CONFIG_DIR}/sa-signer.key"
export PUB_KEY="${CONFIG_DIR}/sa-signer.key.pub"
export PKCS_KEY="${CONFIG_DIR}/sa-signer-pkcs8.pub"
export AUDIENCE="kubernetes.default.svc"

# aws configuration
export AWS_REGION="us-east-2"
export S3_BUCKET="dscott-test-s3"
export ACCOUNT_ID="$(aws sts get-caller-identity --query 'Account' --output text)"
export HOSTNAME=s3.$AWS_REGION.amazonaws.com
export ISSUER_HOSTPATH=$HOSTNAME/$S3_BUCKET
export ROLE_NAME="dscott-test-s3"
export POLICY_NAME="dscott-test-s3"

# kubernetes configuration
export NAMESPACE="default"
export SERVICE_ACCOUNT="default"
  1. Install the KIND binary (full instructions here if you prefer a different installation method - I use brew)
brew install kind
  1. Clone the pod identity hook repo. This repo has a small go program that generates OIDC configuration in a specific format needed to configure OIDC.
git clone git@github.com:aws/amazon-eks-pod-identity-webhook.git

Configure OIDC

  1. Generate a keypair (used for signing and verifying projected service account tokens):
mkdir -p ${CONFIG_DIR}
ssh-keygen -t rsa -b 2048 -f $PRIV_KEY -m pem
ssh-keygen -e -m PKCS8 -f $PUB_KEY > $PKCS_KEY
  1. Configure S3:
# Create the bucket if it doesn't exist
_bucket_name=$(aws s3api list-buckets  --query "Buckets[?Name=='$S3_BUCKET'].Name | [0]" --out text)
if [ $_bucket_name == "None" ]; then
    if [ "$AWS_REGION" == "us-east-1" ]; then
        aws s3api create-bucket --bucket $S3_BUCKET
    else
        #aws s3api create-bucket --bucket $S3_BUCKET --create-bucket-configuration LocationConstraint=$AWS_REGION --acl public-read-write --object-ownership BucketOwnerPreferred
        aws s3api create-bucket --bucket $S3_BUCKET --create-bucket-configuration LocationConstraint=$AWS_REGION
    fi
fi
aws s3api put-bucket-ownership-controls --bucket $S3_BUCKET --ownership-controls="Rules=[{ObjectOwnership=BucketOwnerPreferred}]"
aws s3api put-public-access-block --bucket $S3_BUCKET --public-access-block-configuration "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=true,RestrictPublicBuckets=true"
  1. Create the OIDC discovery file (here is the spec if you are curious):
cat <<EOF > ${CONFIG_DIR}/discovery.json
{
    "issuer": "https://$ISSUER_HOSTPATH",
    "jwks_uri": "https://$ISSUER_HOSTPATH/keys.json",
    "authorization_endpoint": "urn:kubernetes:programmatic_authorization",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256"
    ],
    "claims_supported": [
        "sub",
        "iss"
    ]
}
EOF
  1. Create the keys file. This assumes you cloned the pod-identity-webhook repo during setup and it is available in your current working directory:
pushd amazon-eks-pod-identity-webhook
go run ./hack/self-hosted/main.go -key ../$PKCS_KEY  | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > ../${CONFIG_DIR}/keys.json
popd
  1. Upload the OIDC configuration files to the S3 bucket:
aws s3 cp --acl public-read ${CONFIG_DIR}/discovery.json s3://$S3_BUCKET/.well-known/openid-configuration
aws s3 cp --acl public-read ${CONFIG_DIR}/keys.json s3://$S3_BUCKET/keys.json
  1. Retrieve the certificate thumbprint from the S3 services. This was far more complicated than it needed to be, but openssl commands would consistently not print out the last cert in the chain, which is needed, so I scripted it out:
CERT=""
while IFS= read -r line; do
    if [ -n "$(echo $line | grep 'BEGIN CERTIFICATE')" ]; then
        # reset cert
        CERT="${line}\n"
        ADD_LINE="true"
        continue
    fi

    if [ -n "$(echo $line | grep 'END CERTIFICATE')" ]; then
        # end cert line addition
        CERT+="${line}\n"
        ADD_LINE="false"
        continue
    fi

    if [ "${ADD_LINE}" == "true" ]; then
        CERT+="${line}\n"
        continue
    fi
done < <(openssl s_client -connect s3.${AWS_REGION}.amazonaws.com:443 -showcerts -servername s3.${AWS_REGION}.amazonaws.com </dev/null 2>&1)

# print fingerprint
export FINGERPRINT=$(echo -e "${CERT}" | openssl x509 -fingerprint -sha1 -noout -text 2>&1 | grep Fingerprint | awk -F'=' '{print $NF}' | sed 's/://g')
  1. Create the OIDC provider:
export OIDC_PROVIDER_ARN=$(aws iam create-open-id-connect-provider \
    --url "https://$ISSUER_HOSTPATH" \
    --client-id-list="${AUDIENCE}" \
    --thumbprint-list="${FINGERPRINT}" --query 'OpenIDConnectProviderArn' --output text)
export OIDC_PROVIDER="$(echo $OIDC_PROVIDER_ARN | awk -F'oidc-provider/' '{print $NF}')"
echo $OIDC_PROVIDER_ARN
echo $OIDC_PROVIDER

Create a Test Policy/Role to Simulate

  1. The following policy simply allows us to list S3 buckets so we can see the bucket above you just created. It is valid for any role that needs to access an AWS service. We are creating a simple one for ease of understanding.
cat <<EOF > ${CONFIG_DIR}/iam-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*"
            ],
            "Resource": "*"
        }
    ]
}
EOF

POLICY_ARN=$(aws iam create-policy \
    --policy-name $POLICY_NAME \
    --policy-document file://$CONFIG_DIR/iam-policy.json \
    --query 'Policy.Arn' --output text)
  1. Create the trust policy. This will allow a service account in the cluster to assume a web identity with our configured OIDC provider:
cat <<EOF > ${CONFIG_DIR}/iam-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "$OIDC_PROVIDER_ARN"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "$OIDC_PROVIDER:aud": "kubernetes.default.svc",
                    "$OIDC_PROVIDER:sub": "system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT"
                }
            }
        }
    ]
}
EOF
  1. Finally, create the role and attach the policies. This is the role that we will assume as a Kubernetes service account:
ROLE_ARN=$(aws iam create-role \
    --role-name $ROLE_NAME \
    --assume-role-policy-document file://$CONFIG_DIR/iam-trust-policy.json \
    --query 'Role.Arn' --output text)
aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn $POLICY_ARN

Configure Local Cluster

  1. Create the configuration. This configuration takes the keys that we previously related and configures the API server to use them:
cat <<EOF > ${CONFIG_DIR}/kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: ${CONFIG_DIR}
        containerPath: /etc/kubernetes/pki/identity-keys
        readOnly: true
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            service-account-key-file: "/etc/kubernetes/pki/identity-keys/sa-signer-pkcs8.pub"
            service-account-signing-key-file: "/etc/kubernetes/pki/identity-keys/sa-signer.key"
            api-audiences: "$AUDIENCE"
            service-account-issuer: "https://$ISSUER_HOSTPATH"
EOF
  1. Create the cluster with the configuration we just used:
kind create cluster --config ${CONFIG_DIR}/kind.yaml
  1. Install cert-manager. This is needed by the pod-identity-webhook. Here is a simple install method:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml
  1. Install the pod-identity-webhook:
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-eks-pod-identity-webhook/master/deploy/auth.yaml
curl https://raw.githubusercontent.com/aws/amazon-eks-pod-identity-webhook/master/deploy/deployment-base.yaml | sed 's/IMAGE/amazon\/amazon-eks-pod-identity-webhook:latest/g' | sed "s/--token-audience=sts.amazonaws.com/--token-audience\=$AUDIENCE/g" | kubectl apply -f -
kubectl patch deployment pod-identity-webhook -p '{"spec": {"template": {"spec": {"containers": [{"name": "pod-identity-webhook", "image": "amazon/amazon-eks-pod-identity-webhook:latest"}]}}}}'
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-eks-pod-identity-webhook/master/deploy/service.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/amazon-eks-pod-identity-webhook/master/deploy/mutatingwebhook.yaml
kubectl patch deployment pod-identity-webhook -p '{"spec": {"template": {"spec": {"containers": [{"name": "pod-identity-webhook", "image": "amazon/amazon-eks-pod-identity-webhook:latest"}]}}}}'
  1. Finally, we need to tell our service account to use our role. I will use the default service account for this demo, but that is not a good idea in practice:
kubectl patch serviceaccount $SERVICE_ACCOUNT -n $NAMESPACE -p "{\"metadata\": {\"annotations\": {\"eks.amazonaws.com/role-arn\": \"$ROLE_ARN\"}}}"
  1. And create a pod to test with. This will exec into the container so that we can run a command:
kubectl -n $NAMESPACE run -it dscott-s3-test --image=public.ecr.aws/aws-cli/aws-cli:latest --command=true bash
  1. We can see that the pod-identity-webhook injected a session token that we use to authenticate against:
echo $AWS_WEB_IDENTITY_TOKEN_FILE
cat $AWS_WEB_IDENTITY_TOKEN_FILE
  1. Finally, let’s make sure we can list S3 buckets as per our policy which should succeed:
aws s3api list-buckets --region us-east-2

...
{
    "Buckets": [
...
  1. And to make sure there is no magic here, let’s attempt another action which will fail because we did not include necessary permissions in the policy:
aws route53 list-hosted-zones

...
An error occurred (AccessDenied) when calling the ListHostedZones operation: User: arn:aws:sts::660250927410:assumed-role/dscott-test-s3/botocore-session-1711750306 is not authorized to perform: route53:ListHostedZones because no identity-based policy allows the route53:ListHostedZones action