AWS Secrets Manager 使用 Lambda 輪替 API 金鑰

Secrets Manager rotation is my answer to an uncomfortable question: how old is the API key your Lambda function has been sending to a partner for months? In many projects the honest answer is “as old as the project”. The key lives in a secret, the secret is read encrypted, and still nobody has ever replaced it. If it slips into a log, a support email or a ticket once, it stays valid until somebody notices.

AWS Secrets Manager can renew such keys automatically. For databases that is almost a single click. For an API key at an external service you need your own rotation function in Lambda, and that is what we build in this article, step by step: with TypeScript, tests and the Serverless Framework. Afterwards you will understand what happens in the four steps, why staging labels matter so much and how your other functions pick up the new key without downtime.

In short

Secrets Manager invokes your rotation function four times: create a new value, set it at the target service, test it, mark it as current. To avoid downtime, the target service has to accept the old and the new key in parallel for a while. The most common pitfall is a stuck AWSPENDING label after a failed rotation.

What you will learn in this article

How Secrets Manager rotation works with versions and staging labels
When you do not need your own rotation function
A rotation function for an API key in TypeScript, including tests
Deployment with the Serverless Framework, schedule and alarm included
How Lambda functions read rotated keys without downtime

What is Secrets Manager rotation?

Rotation means a secret is replaced regularly, and in two places at once: in the secret itself and at the service that checks it, such as the database or the partner API. If you only change one side, you lock yourself out.

Secrets Manager solves this with versions. A secret can have several versions, and each version carries so-called staging labels. These are simple text markers that say which version currently plays which role:

AWSCURRENT is the version your applications read by default.
AWSPENDING is the new version that is being created and has not been released yet.
AWSPREVIOUS is the last valid version. After a successful rotation it stays around as a fallback.

Before, during and after rotation: how AWSCURRENT, AWSPENDING and AWSPREVIOUS move.

In the end, a rotation is nothing more than a carefully choreographed move of these labels. Only when the new value works at the target service does AWSCURRENT move to the new version. Until then, every application keeps reading the old value.

When you do not need your own rotation function

Before you write code, take an honest look at the alternatives. According to the documentation, AWS distinguishes three forms of rotation:

Form	Used for	Lambda needed?
Managed rotation	Secrets that an AWS service manages itself, e.g. the master password of an RDS database	No
Managed external secrets	Credentials at partners that Secrets Manager supports directly	No
Rotation by Lambda function	Everything else: your own databases, internal services, API keys at any provider	Yes

For Amazon RDS I would always start with the master password managed by the service. For many database engines AWS also provides ready-made rotation templates. A custom function is therefore mostly worth it where no standard applies. An API key at a service provider is the classic case.

The four steps of Secrets Manager rotation

When a rotation starts, Secrets Manager invokes your Lambda function four times in a row. Every invocation gets the same secret ID, the same ClientRequestToken (it becomes the version ID of the new version) and a different value in the Step field.

🎬 The four steps of a rotation – this animated diagram only runs on my site: watch it on sascha-ms.de

createSecret creates the new value

First, the function checks whether a value already exists for the token. If so, this was a retry, and it stops. Otherwise it creates a new key with GetRandomPassword and stores it via PutSecretValue with the label AWSPENDING. This check makes the step idempotent, so you can repeat it as often as you like with the same result.

setSecret registers it with the target service

Now the target service learns about the new key. AWS describes the rotation function as a privileged deputy here: it may change both the secret and the credentials at the service. That is why safety checks belong in this step. Do AWSCURRENT and AWSPENDING point to the same target? Does the current key still work at all? If not, the function aborts. This prevents the so-called confused deputy attack, where someone abuses an authorized function for their own purposes.

testSecret checks that it works

The function calls a harmless endpoint with the new key. If the service rejects it, the function throws an error and the rotation ends here. Your applications notice nothing, because they keep reading AWSCURRENT.

finishSecret switches over

Finally, UpdateSecretVersionStage moves the label AWSCURRENT to the new version. Secrets Manager attaches AWSPREVIOUS to the old version automatically. Done.

The example: a bike rental and its lock API

To make this tangible, let us take a fictional bike rental. Its bikes have smart locks, operated by a partner. When a customer wants to unlock a bike, a Lambda function calls the partner lock API and sends an API key in the x-api-key header.

The good news: the (equally made-up) partner API allows two active keys per account. New keys are registered with POST, old ones revoked with DELETE. That is the most important prerequisite for a rotation without downtime, and you should check it with every real provider first. If a service only supports exactly one key, there will inevitably be a moment when running functions fail with the old value.

The flow looks like this: the key from the rotation before last (AWSPREVIOUS) is revoked, the new one is registered, tested and then becomes current. The previously current key stays valid until the next rotation. That gives every function that still has it in its cache plenty of time to switch.

Step by step: rotation with Lambda and the Serverless Framework

Prerequisites

Node.js 22 or newer and npm
An AWS account with a configured AWS CLI (region in the example: eu-central-1)
Serverless Framework v4 (npm i -g serverless), which builds TypeScript handlers automatically
Basic knowledge of Lambda and IAM; I explain the rest along the way

Step 1: Create the project

Create a folder and install the AWS SDK for Secrets Manager plus the development tools:

mkdir key-rotation && cd key-rotation
npm init -y
npm install @aws-sdk/client-secrets-manager
npm install -D typescript @types/node @types/aws-lambda vitest@3 aws-sdk-client-mock

The tsconfig.json stays simple and strict:

// tsconfig.json
{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src", "test"]
}

In package.json, add "type": "module" and the scripts "typecheck": "tsc --noEmit" and "test": "vitest run".

Step 2: Client for the partner API

The client wraps the three calls we need. It is deliberately built as an interface plus a class, so the test can replace it with a fake.

// src/lock-partner-client.ts
// Client for the (fictional) lock API of the partner that runs the smart locks
// on the rental bikes. The API allows two active keys at the same time,
// which is exactly what we need for a rotation without downtime.

export interface LockApiSecret {
  baseUrl: string; // e.g. https://api.lockpartner.example
  accountId: string; // the bike rental's customer account at the partner
  keyId: string; // name of the key at the partner
  apiKey: string; // the actual secret value
}

export interface LockPartnerClient {
  /** Checks whether a key works (true = valid). */
  ping(secret: LockApiSecret): Promise<boolean>;
  /** Registers a new key. Authorized with the currently valid key. */
  registerKey(auth: LockApiSecret, newKey: LockApiSecret): Promise<void>;
  /** Revokes an old key. Authorized with the currently valid key. */
  revokeKey(auth: LockApiSecret, keyId: string): Promise<void>;
}

export class HttpLockPartnerClient implements LockPartnerClient {
  constructor(private readonly timeoutMs = 5000) {}

  async ping(secret: LockApiSecret): Promise<boolean> {
    const res = await this.call(secret, 'GET', `/accounts/${secret.accountId}/ping`);
    return res.status === 200;
  }

  async registerKey(auth: LockApiSecret, newKey: LockApiSecret): Promise<void> {
    const res = await this.call(auth, 'POST', `/accounts/${auth.accountId}/api-keys`, {
      keyId: newKey.keyId,
      apiKey: newKey.apiKey,
    });
    // 409 means the key is already registered (e.g. on a retry), which is fine.
    if (res.status !== 201 && res.status !== 409) {
      throw new Error(`registerKey failed: HTTP ${res.status}`);
    }
  }

  async revokeKey(auth: LockApiSecret, keyId: string): Promise<void> {
    const res = await this.call(auth, 'DELETE', `/accounts/${auth.accountId}/api-keys/${keyId}`);
    // 404 means the key no longer exists, which is fine as well.
    if (res.status !== 204 && res.status !== 404) {
      throw new Error(`revokeKey failed: HTTP ${res.status}`);
    }
  }

  private call(auth: LockApiSecret, method: string, path: string, body?: unknown): Promise<Response> {
    return fetch(new URL(path, auth.baseUrl), {
      method,
      headers: { 'x-api-key': auth.apiKey, 'content-type': 'application/json' },
      body: body === undefined ? undefined : JSON.stringify(body),
      signal: AbortSignal.timeout(this.timeoutMs),
    });
  }
}

Note the status codes 409 and 404: when a step is repeated, the key may already be registered or already gone. Neither is an error; both are exactly the result we want.

Step 3: The rotation function

Now the core. The handler first checks that rotation is enabled and that the token belongs to an AWSPENDING version. Then it branches into the four steps.

// src/rotate.ts
// Rotation function for AWS Secrets Manager: rotates the API key
// for the lock API in the four steps createSecret, setSecret,
// testSecret and finishSecret.
import {
  SecretsManagerClient,
  DescribeSecretCommand,
  GetSecretValueCommand,
  PutSecretValueCommand,
  GetRandomPasswordCommand,
  UpdateSecretVersionStageCommand,
  ResourceNotFoundException,
} from '@aws-sdk/client-secrets-manager';
import { HttpLockPartnerClient, type LockApiSecret, type LockPartnerClient } from './lock-partner-client.js';

export interface RotationEvent {
  SecretId: string;
  ClientRequestToken: string;
  Step: 'createSecret' | 'setSecret' | 'testSecret' | 'finishSecret';
  RotationToken?: string; // pass it on, never log it
}

const CURRENT = 'AWSCURRENT';
const PENDING = 'AWSPENDING';
const PREVIOUS = 'AWSPREVIOUS';

export function createRotationHandler(sm: SecretsManagerClient, partner: LockPartnerClient) {
  async function readSecret(secretId: string, stage: string, versionId?: string): Promise<LockApiSecret> {
    const res = await sm.send(
      new GetSecretValueCommand({ SecretId: secretId, VersionStage: stage, VersionId: versionId }),
    );
    const value = JSON.parse(res.SecretString ?? '{}') as Partial<LockApiSecret>;
    for (const field of ['baseUrl', 'accountId', 'keyId', 'apiKey'] as const) {
      if (!value[field]) throw new Error(`Field ${field} missing in ${stage}`);
    }
    return value as LockApiSecret;
  }

  async function createSecret(secretId: string, token: string, rotationToken?: string): Promise<void> {
    const current = await readSecret(secretId, CURRENT);
    try {
      // If the version already exists (retry), there is nothing to do.
      await readSecret(secretId, PENDING, token);
      console.log('createSecret: AWSPENDING already exists');
      return;
    } catch (err) {
      if (!(err instanceof ResourceNotFoundException)) throw err;
    }
    const random = await sm.send(
      new GetRandomPasswordCommand({ PasswordLength: 48, ExcludePunctuation: true }),
    );
    const pending: LockApiSecret = {
      ...current,
      keyId: `rotation-${token.slice(0, 8)}`,
      apiKey: random.RandomPassword!,
    };
    await sm.send(
      new PutSecretValueCommand({
        SecretId: secretId,
        ClientRequestToken: token,
        SecretString: JSON.stringify(pending),
        VersionStages: [PENDING],
        RotationToken: rotationToken,
      }),
    );
    console.log(`createSecret: stored new version ${token} as AWSPENDING`);
  }

  async function setSecret(secretId: string, token: string, previousVersionId?: string): Promise<void> {
    const current = await readSecret(secretId, CURRENT);
    const pending = await readSecret(secretId, PENDING, token);

    // Confused deputy protection: both versions must belong to the same target.
    if (current.baseUrl !== pending.baseUrl || current.accountId !== pending.accountId) {
      throw new Error('setSecret: AWSCURRENT and AWSPENDING point to different targets');
    }
    // Only rotate if the current key still works.
    if (!(await partner.ping(current))) {
      throw new Error('setSecret: AWSCURRENT is invalid, rotation aborted');
    }

    // The partner allows two keys. We remove the one before last (AWSPREVIOUS)
    // now to make room for the new one. It has not been in use since the
    // last rotation.
    if (previousVersionId) {
      const previous = await readSecret(secretId, PREVIOUS, previousVersionId);
      if (previous.keyId !== current.keyId && previous.keyId !== pending.keyId) {
        await partner.revokeKey(current, previous.keyId);
        console.log(`setSecret: revoked old key ${previous.keyId}`);
      }
    }

    await partner.registerKey(current, pending);
    console.log(`setSecret: registered new key ${pending.keyId}`);
  }

  async function testSecret(secretId: string, token: string): Promise<void> {
    const pending = await readSecret(secretId, PENDING, token);
    if (!(await partner.ping(pending))) {
      throw new Error('testSecret: partner rejects the new key');
    }
    console.log('testSecret: new key works');
  }

  async function finishSecret(secretId: string, token: string, currentVersionId?: string): Promise<void> {
    if (currentVersionId === token) {
      console.log('finishSecret: version is already AWSCURRENT');
      return;
    }
    await sm.send(
      new UpdateSecretVersionStageCommand({
        SecretId: secretId,
        VersionStage: CURRENT,
        MoveToVersionId: token,
        RemoveFromVersionId: currentVersionId,
      }),
    );
    console.log(`finishSecret: AWSCURRENT now points to ${token}`);
  }

  return async function handler(event: RotationEvent): Promise<void> {
    const { SecretId: secretId, ClientRequestToken: token, Step: step } = event;
    // Important: never log the secret value, only step and version.
    console.log(`Rotation ${step} for version ${token}`);

    const meta = await sm.send(new DescribeSecretCommand({ SecretId: secretId }));
    if (!meta.RotationEnabled) throw new Error('Rotation is not enabled for this secret');

    const versions = meta.VersionIdsToStages ?? {};
    const stages = versions[token];
    if (!stages) throw new Error(`Version ${token} does not belong to this secret`);
    if (stages.includes(CURRENT)) {
      console.log('Version is already AWSCURRENT, nothing to do');
      return;
    }
    if (!stages.includes(PENDING)) throw new Error(`Version ${token} is not AWSPENDING`);

    const findVersion = (stage: string) =>
      Object.entries(versions).find(([, s]) => s.includes(stage))?.[0];

    switch (step) {
      case 'createSecret':
        return createSecret(secretId, token, event.RotationToken);
      case 'setSecret':
        return setSecret(secretId, token, findVersion(PREVIOUS));
      case 'testSecret':
        return testSecret(secretId, token);
      case 'finishSecret':
        return finishSecret(secretId, token, findVersion(CURRENT));
      default:
        throw new Error(`Unknown step: ${String(step)}`);
    }
  };
}

// Lambda entry point
export const handler = createRotationHandler(new SecretsManagerClient({}), new HttpLockPartnerClient());

Three details matter to me here. First, the function never logs the value of the secret, only step and version. AWS explicitly warns that debug output would otherwise end up in CloudWatch Logs. Second, it passes the RotationToken field on to PutSecretValue. That is mandatory as soon as you rotate across accounts or with an assumed IAM role, and it does no harm otherwise. Third, setSecret removes the key from AWSPREVIOUS before registering the new one, because the partner only allows two keys.

Step 4: Write tests

I always test rotation functions locally first, because a failure in production leaves a stuck label behind. With aws-sdk-client-mock you simulate Secrets Manager, and a small in-memory fake replaces the partner API:

// test/rotate.test.ts
import { describe, it, expect, beforeEach } from 'vitest';
import { mockClient } from 'aws-sdk-client-mock';
import {
  SecretsManagerClient,
  DescribeSecretCommand,
  GetSecretValueCommand,
  PutSecretValueCommand,
  GetRandomPasswordCommand,
  UpdateSecretVersionStageCommand,
  ResourceNotFoundException,
} from '@aws-sdk/client-secrets-manager';
import { createRotationHandler } from '../src/rotate.js';
import type { LockApiSecret, LockPartnerClient } from '../src/lock-partner-client.js';

const SECRET_ID = 'bike-rental/lock-api';
const TOKEN = 'b7c1d2e3-0000-4000-8000-000000000001';
const base = { baseUrl: 'https://api.lockpartner.example', accountId: 'rental-42' };
const current: LockApiSecret = { ...base, keyId: 'start', apiKey: 'old-123' };
const pending: LockApiSecret = { ...base, keyId: 'rotation-b7c1d2e3', apiKey: 'new-456' };

const sm = mockClient(SecretsManagerClient);

// Fake partner: remembers which keys are valid.
class FakePartner implements LockPartnerClient {
  valid = new Set<string>(['old-123']);
  revoked: string[] = [];
  async ping(s: LockApiSecret) { return this.valid.has(s.apiKey); }
  async registerKey(_auth: LockApiSecret, k: LockApiSecret) { this.valid.add(k.apiKey); }
  async revokeKey(_auth: LockApiSecret, keyId: string) { this.revoked.push(keyId); }
}

function describeWith(stages: Record<string, string[]>) {
  sm.on(DescribeSecretCommand).resolves({ RotationEnabled: true, VersionIdsToStages: stages });
}

describe('Rotation of the lock API key', () => {
  let partner: FakePartner;
  beforeEach(() => {
    sm.reset();
    partner = new FakePartner();
  });

  it('createSecret stores a new AWSPENDING version', async () => {
    describeWith({ v1: ['AWSCURRENT'], [TOKEN]: ['AWSPENDING'] });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSCURRENT' }).resolves({ SecretString: JSON.stringify(current) });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSPENDING' }).rejects(
      new ResourceNotFoundException({ message: 'not found', $metadata: {} }),
    );
    sm.on(GetRandomPasswordCommand).resolves({ RandomPassword: 'new-456' });
    sm.on(PutSecretValueCommand).resolves({});

    await createRotationHandler(sm as unknown as SecretsManagerClient, partner)({
      SecretId: SECRET_ID, ClientRequestToken: TOKEN, Step: 'createSecret',
    });

    const put = sm.commandCalls(PutSecretValueCommand)[0].args[0].input;
    expect(put.VersionStages).toEqual(['AWSPENDING']);
    expect(JSON.parse(put.SecretString!)).toMatchObject({ keyId: 'rotation-b7c1d2e3', apiKey: 'new-456' });
  });

  it('setSecret revokes AWSPREVIOUS and registers the new key', async () => {
    describeWith({ v0: ['AWSPREVIOUS'], v1: ['AWSCURRENT'], [TOKEN]: ['AWSPENDING'] });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSCURRENT' }).resolves({ SecretString: JSON.stringify(current) });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSPENDING' }).resolves({ SecretString: JSON.stringify(pending) });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSPREVIOUS' }).resolves({
      SecretString: JSON.stringify({ ...base, keyId: 'very-old', apiKey: 'xyz' }),
    });

    await createRotationHandler(sm as unknown as SecretsManagerClient, partner)({
      SecretId: SECRET_ID, ClientRequestToken: TOKEN, Step: 'setSecret',
    });

    expect(partner.revoked).toEqual(['very-old']);
    expect(partner.valid.has('new-456')).toBe(true);
  });

  it('setSecret aborts when AWSPENDING points to another account', async () => {
    describeWith({ v1: ['AWSCURRENT'], [TOKEN]: ['AWSPENDING'] });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSCURRENT' }).resolves({ SecretString: JSON.stringify(current) });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSPENDING' }).resolves({
      SecretString: JSON.stringify({ ...pending, accountId: 'foreign-account' }),
    });

    await expect(
      createRotationHandler(sm as unknown as SecretsManagerClient, partner)({
        SecretId: SECRET_ID, ClientRequestToken: TOKEN, Step: 'setSecret',
      }),
    ).rejects.toThrow(/different targets/);
  });

  it('testSecret fails when the partner rejects the new key', async () => {
    describeWith({ v1: ['AWSCURRENT'], [TOKEN]: ['AWSPENDING'] });
    sm.on(GetSecretValueCommand, { VersionStage: 'AWSPENDING' }).resolves({ SecretString: JSON.stringify(pending) });

    await expect(
      createRotationHandler(sm as unknown as SecretsManagerClient, partner)({
        SecretId: SECRET_ID, ClientRequestToken: TOKEN, Step: 'testSecret',
      }),
    ).rejects.toThrow(/rejects the new key/);
  });

  it('finishSecret moves AWSCURRENT to the new version', async () => {
    describeWith({ v1: ['AWSCURRENT'], [TOKEN]: ['AWSPENDING'] });
    sm.on(UpdateSecretVersionStageCommand).resolves({});

    await createRotationHandler(sm as unknown as SecretsManagerClient, partner)({
      SecretId: SECRET_ID, ClientRequestToken: TOKEN, Step: 'finishSecret',
    });

    const input = sm.commandCalls(UpdateSecretVersionStageCommand)[0].args[0].input;
    expect(input).toMatchObject({ VersionStage: 'AWSCURRENT', MoveToVersionId: TOKEN, RemoveFromVersionId: 'v1' });
  });

  it('does nothing when the version is already AWSCURRENT (retry)', async () => {
    describeWith({ [TOKEN]: ['AWSCURRENT', 'AWSPENDING'] });

    await createRotationHandler(sm as unknown as SecretsManagerClient, partner)({
      SecretId: SECRET_ID, ClientRequestToken: TOKEN, Step: 'finishSecret',
    });

    expect(sm.commandCalls(UpdateSecretVersionStageCommand)).toHaveLength(0);
  });
});

npm run typecheck
npm test

Expected result: Tests 6 passed (6). If a test fails, the output shows the step and the version, but never the key itself.

Step 5: serverless.yml with schedule and alarm

The Serverless Framework describes the infrastructure. Besides the function you need three things: a permission that allows Secrets Manager to invoke the function, the rotation schedule and an alarm for failed rotations.

# serverless.yml
service: bike-rental-key-rotation
frameworkVersion: '4'

params:
  default:
    # Create the secret beforehand with the CLI (see step 6) so the
    # initial key never ends up in the template or in CloudFormation.
    secretName: bike-rental/lock-api

provider:
  name: aws
  runtime: nodejs24.x
  architecture: arm64
  region: eu-central-1
  logRetentionInDays: 30
  iam:
    role:
      statements:
        - Effect: Allow
          Action:
            - secretsmanager:DescribeSecret
            - secretsmanager:GetSecretValue
            - secretsmanager:PutSecretValue
            - secretsmanager:UpdateSecretVersionStage
          Resource: arn:aws:secretsmanager:${aws:region}:${aws:accountId}:secret:${param:secretName}-*
        - Effect: Allow
          Action: secretsmanager:GetRandomPassword
          Resource: '*'

functions:
  rotateLockApiKey:
    handler: src/rotate.handler
    description: Rotates the API key for the lock API
    timeout: 30
    memorySize: 256

resources:
  Resources:
    # Secrets Manager may invoke the function, but only from this account.
    RotationInvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !GetAtt RotateLockApiKeyLambdaFunction.Arn
        Action: lambda:InvokeFunction
        Principal: secretsmanager.amazonaws.com
        SourceAccount: !Ref AWS::AccountId

    LockApiKeyRotation:
      Type: AWS::SecretsManager::RotationSchedule
      DependsOn: RotationInvokePermission
      Properties:
        SecretId: ${param:secretName}
        RotationLambdaARN: !GetAtt RotateLockApiKeyLambdaFunction.Arn
        RotateImmediatelyOnUpdate: true
        RotationRules:
          # Every first Tuesday of the month between 02:00 and 04:00 UTC
          ScheduleExpression: cron(0 2 ? * TUE#1 *)
          Duration: 2h

    # Notification when a rotation fails
    RotationAlarmTopic:
      Type: AWS::SNS::Topic

    RotationFailedRule:
      Type: AWS::Events::Rule
      Properties:
        EventPattern:
          source: [aws.secretsmanager]
          detail-type: [AWS Service Event via CloudTrail]
          detail:
            eventName: [RotationFailed, RotationAbandoned]
        Targets:
          - Arn: !Ref RotationAlarmTopic
            Id: rotation-alarm

    RotationAlarmTopicPolicy:
      Type: AWS::SNS::TopicPolicy
      Properties:
        Topics: [!Ref RotationAlarmTopic]
        PolicyDocument:
          Statement:
            - Effect: Allow
              Principal: { Service: events.amazonaws.com }
              Action: sns:Publish
              Resource: !Ref RotationAlarmTopic
              Condition:
                ArnEquals:
                  aws:SourceArn: !GetAtt RotationFailedRule.Arn

A few notes on the settings. The IAM permissions follow the example policy from the AWS documentation and are limited to exactly one secret. AWS recommends the SourceAccount condition in the Lambda permission against confused deputy attacks. ScheduleExpression uses a six-field cron syntax and always runs in UTC; Duration sets the window in which the rotation may happen. If you like to try out cron expressions, the EventBridge cron explainer helps, the syntax is very similar.

RotateImmediatelyOnUpdate: true makes the first rotation run right at deployment. That is also the default, but I write it down on purpose. This way the deployment doubles as the first real test. The EventBridge rule reacts to the events RotationFailed and RotationAbandoned, which Secrets Manager reports through CloudTrail, and forwards them to an SNS topic. Subscribe an email address or your chat channel there.

Step 6: Create the secret, deploy, watch the rotation

You deliberately create the secret beforehand with the CLI. That way the initial key you got from the partner never appears in the template or in the CloudFormation history:

aws secretsmanager create-secret \
  --name bike-rental/lock-api \
  --secret-string '{"baseUrl":"https://api.lockpartner.example","accountId":"rental-42","keyId":"start","apiKey":"<initial key from the partner>"}'

serverless deploy

aws secretsmanager describe-secret \
  --secret-id bike-rental/lock-api \
  --query '{Rotation:RotationEnabled,Versions:VersionIdsToStages}'

After the deployment you should see two versions: the new one with AWSCURRENT and the initial one with AWSPREVIOUS. You can trigger a rotation by hand at any time with aws secretsmanager rotate-secret --secret-id bike-rental/lock-api. The function’s CloudWatch Logs then show four invocations, one per step.

If you do not have a real partner at hand, the tests from step 4 are enough. A deployment against api.lockpartner.example fails at setSecret as expected, because the address does not exist. That is instructive too: afterwards, use describe-secret to see what a stuck AWSPENDING label looks like.

How Lambda functions read the rotated key

Rotation is only half the job. The function that unlocks bikes also has to use the new key at some point. Secrets Manager charges per API call, so you do not read secrets on every request but keep them in memory for a short time. The AWS Parameters and Secrets Lambda Extension, for example, caches for five minutes by default.

This is exactly why the old key has to stay valid after the rotation: a warm Lambda instance may still use it minutes after the switch. In our example it stays valid until the next rotation, so roughly a month. A small safety belt is worth adding as well: if the partner answers with 401, the function reloads the key once and tries again.

// src/lock-key-cache.ts
// How a regular Lambda function (e.g. "unlock bike") reads the rotated
// key: cache it briefly and reload it once on HTTP 401.
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import type { LockApiSecret } from './lock-partner-client.js';

const sm = new SecretsManagerClient({});
const TTL_MS = 5 * 60 * 1000;
let cache: { value: LockApiSecret; expiresAt: number } | undefined;

export async function getLockKey(secretId: string, forceRefresh = false): Promise<LockApiSecret> {
  if (!forceRefresh && cache && cache.expiresAt > Date.now()) return cache.value;
  const res = await sm.send(new GetSecretValueCommand({ SecretId: secretId }));
  const value = JSON.parse(res.SecretString!) as LockApiSecret;
  cache = { value, expiresAt: Date.now() + TTL_MS };
  return value;
}

/** Runs a call and retries exactly once with a fresh key on 401. */
export async function withLockKey(
  secretId: string,
  call: (key: LockApiSecret) => Promise<Response>,
): Promise<Response> {
  const first = await call(await getLockKey(secretId));
  if (first.status !== 401) return first;
  return call(await getLockKey(secretId, true));
}

I described how to load configuration and secrets efficiently in Lambda in detail in Parameter Store in Lambda. The ideas apply to Secrets Manager just as well.

Common mistakes with Secrets Manager rotation

AWSPENDING is stuck after a failure

When a rotation fails, AWSPENDING often stays attached to a version that is not AWSCURRENT. According to AWS, every later rotation then assumes one is still in progress and fails with an error. The fix: resolve the cause, remove the label from the stuck version with update-secret-version-stage and --remove-from-version-id, then rotate again.

The target service only knows one key

Then there is no rotation without a short gap. In that case, plan a maintenance window via ScheduleExpression and make sure your functions reload immediately on 401, as shown above.

Secrets in the log

A quick console.log(event) while debugging is harmless, a console.log(secret) is not. Log steps, versions and status codes, never values.

Function in a VPC without a route to Secrets Manager

If the rotation function runs in a VPC, for example because the database lives there, it needs a route to the Secrets Manager endpoint: a VPC endpoint or a NAT gateway. Otherwise the calls run into a timeout. For a public partner API like this one you do not need a VPC at all.

Forgetting idempotency

Secrets Manager may repeat steps. Every step therefore has to recognize whether its work is already done. How to solve this for regular Lambda handlers is covered in Idempotency in AWS Lambda.

Cost and security

According to the pricing page, a secret costs USD 0.40 per month, plus USD 0.05 for every 10,000 API calls. The rotation function itself runs a few seconds per month and barely registers on your Lambda bill. Things get more expensive when hundreds of functions read the secret on every request, hence the cache. A customer managed KMS key costs extra; the AWS managed key aws/secretsmanager is free.

On the security side: the rotation function is powerful. Grant it permissions on that one secret only, restrict the invoke permission to your account and do not let anyone else invoke the function directly. By the way, according to AWS the rotation function must not delegate the actual rotation to another Lambda function; it has to change the secret itself.

My take

I think automatic rotation is one of those topics that come too late in Serverless projects. Secrets Manager is usually introduced quickly, rotation gets pushed to “later”, and later rarely comes. Yet the effort is manageable: the four steps are always the same, and the real work sits in setSecret and testSecret for your target service.

I would push back on a common attitude: “We rotate every 30 days, so we are safe.” Rotation shortens the time in which a leaked key is useful. It replaces neither tight permissions at the partner nor clean logs. And the most valuable side effect is a different one: a team that rotates automatically every month knows that the swap works when it matters. Whoever replaces a key for the first time during a security incident is testing under pressure.

FAQ

How often can Secrets Manager rotate a secret?

According to the AWS documentation, at most every four hours. For API keys, a cycle of 30 to 90 days is usually enough in practice.

Do I need a custom rotation function for RDS?

Usually not. RDS can manage and rotate the secret for the master password itself, and AWS offers ready-made templates for additional database users.

What happens to the old value after rotation?

It remains as a version with the AWSPREVIOUS label. Whether it is still valid at the target service is up to your rotation function.

Can I trigger a rotation manually?

Yes, with aws secretsmanager rotate-secret --secret-id <name> or in the console via “Rotate secret immediately”.

Is Parameter Store an alternative?

For configuration yes, for automatic rotation no. Parameter Store has no built-in rotation with staging labels.

Your next step

Find the oldest API key in your project and check with the provider whether it allows two active keys. If it does, the example from this article is a good template: swap the client, adapt the tests, deploy. I am curious: which keys do you already rotate automatically today, and what stops you with the others? Let me know in the comments.

All code from this article, ready to run: smirowstanitzok/sascha-ms-examples

Sources

Rotate AWS Secrets Manager secrets (AWS Secrets Manager User Guide)
Lambda rotation functions (AWS Secrets Manager User Guide)
Lambda rotation function execution role permissions (AWS Secrets Manager User Guide)
Set up automatic rotation for non-database secrets (AWS Secrets Manager User Guide)
Rotation schedules (AWS Secrets Manager User Guide)
AWS CloudTrail entries for Secrets Manager (AWS Secrets Manager User Guide)
AWS::SecretsManager::RotationSchedule (AWS CloudFormation Template Reference)
Use Secrets Manager secrets in Lambda functions (AWS Lambda Developer Guide)
Why did my Secrets Manager Lambda rotation fail? (AWS re:Post Knowledge Center)
AWS Secrets Manager Pricing (AWS)

This post was created with AI assistance.

👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en