Hidden costs of Lambda-backed custom resources

CloudFormation does not have native resource types for every AWS service or operation teams need. Lambda-backed custom resources fill that gap. They let you run arbitrary code during stack create, update, and delete operations, and they have been a practical workaround for missing primitives for years.

But custom resources carry operational overhead that is easy to underestimate when you first add one. The timeout failure mode is non-obvious. Rollback semantics interact badly with failure. Long-term, the Lambda function becomes a separate piece of infrastructure that the template depends on. Understanding these costs before you add a custom resource helps you design more resilient handlers and evaluate when alternatives are worth considering.

How the response protocol works

When CloudFormation encounters a custom resource, it invokes the Lambda function asynchronously and passes it an event payload containing the resource properties, the stack ID, a physical resource ID, and a pre-signed S3 URL.

The function must PUT a JSON response to that signed URL before it times out. The response must include a Status field set to SUCCESS or FAILED, an optional Data object with output values, and the physical resource ID.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import cfnresponse

def handler(event, context):
    try:
        result = do_work(event['ResourceProperties'])
        cfnresponse.send(
            event, context, cfnresponse.SUCCESS,
            {'Value': result},
            physicalResourceId=event.get('PhysicalResourceId', 'my-resource')
        )
    except Exception as e:
        cfnresponse.send(
            event, context, cfnresponse.FAILED,
            {'Error': str(e)},
            physicalResourceId=event.get('PhysicalResourceId', 'my-resource')
        )

The signed URL has a fixed expiry. CloudFormation waits up to the ServiceTimeout value on the custom resource definition — the default is 60 minutes — for a response before declaring the resource failed.

The custom resource protocol is documented in the CloudFormation custom resources guide.

The timeout failure mode

Lambda functions have a maximum execution time of 15 minutes. If the function throws an exception after sending a FAILED response, CloudFormation receives it and handles the failure. If the function throws an exception before it reaches the response code — because of a missing import, an unhandled error early in the handler, or an uncaught exception inside the except branch itself — CloudFormation receives nothing.

CloudFormation then waits the full ServiceTimeout window before failing the resource. A stack creation that fails in the first few seconds from the Lambda perspective can appear frozen for up to an hour from the operator’s perspective.

The most common trigger is incomplete error handling. The Python cfnresponse module makes it easy to send a response on success, but sending a FAILED response on every failure path requires explicit wrapping around the entire handler body. A common mistake is an except block that logs but then re-raises, or a handler that only wraps part of the work.

An additional risk is the pre-signed URL expiry. Lambda functions invoked asynchronously can be throttled and queued. If the function sits in the invocation queue long enough for the pre-signed URL to expire, the function can complete successfully but be unable to deliver the response. CloudFormation receives nothing and waits the full timeout.

Rollback and the failure cascade

CloudFormation calls the function with RequestType: Create when a resource is first created. If that call results in FAILED or times out, CloudFormation rolls back the stack and calls the function again with RequestType: Delete.

If the Delete handler also fails or times out, the stack enters UPDATE_ROLLBACK_FAILED state. A stack in this state cannot be updated or deleted through normal means. Recovery requires either calling ContinueUpdateRollback with the failing resource skipped, or importing and manually re-registering the physical resource. Both options require manual intervention that interrupts normal operations.

This failure cascade is why custom resource delete handlers must be idempotent. A delete handler that fails when the underlying resource no longer exists will block rollback on every subsequent update that touches that resource. Writing a safe delete handler means checking whether the resource exists before attempting deletion and treating a missing resource as a successful delete.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
def delete_handler(event, context):
    resource_id = event['PhysicalResourceId']
    try:
        # Check existence before deleting
        client.describe_resource(ResourceId=resource_id)
    except client.exceptions.ResourceNotFoundException:
        # Already gone — treat as success
        cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, resource_id)
        return
    client.delete_resource(ResourceId=resource_id)
    cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, resource_id)

Getting idempotency right requires knowing exactly what failure modes the underlying API can return and handling each one explicitly.

Long-term maintenance

A Lambda function that backs a custom resource is infrastructure that the template depends on. It needs its own operational lifecycle:

Deployment. The function must exist and be accessible before the template that references it can be deployed. If the function is deleted, renamed, or moved to a different account, every stack referencing it breaks on the next update. The template and the Lambda function must be kept in sync through their entire lifetime.

IAM scope. The function’s execution role needs permissions to do its work. Roles tend to accumulate permissions over time as requirements evolve. An overly broad execution role is a security risk that is easy to introduce and harder to clean up once stacks depend on the function.

Runtime deprecation. Lambda runtimes have end-of-life dates. A function written for a runtime that is subsequently deprecated will continue to work until AWS removes support, at which point updates to any stack that uses the function will fail. Runtime upgrades require testing, deployment, and coordination with every team that owns a stack referencing the function.

Testing. Testing a custom resource handler requires constructing CloudFormation event payloads and verifying the response structure. Testing the delete-on-failure path requires simulating a rollback, which typically means deploying to a real stack and deliberately triggering a failure. Integration tests that cover the full protocol are harder to set up than unit tests that cover only the business logic.

When custom resources remain the right choice

Custom resources are appropriate when:

There is no native CloudFormation resource type and no CloudFormation extension available for the operation.
The operation is infrequent enough that the maintenance overhead is acceptable relative to how often it runs.
The team that owns the template also owns the Lambda function and has a plan for keeping both synchronized through their operational lifecycles.

For recurring patterns — SSM parameter resolution, secret generation, ACM certificate requests that require DNS validation — the question is whether the custom resource overhead is proportionate to the problem. A one-off operation in a rarely-updated stack is a different trade-off from the same pattern replicated across dozens of stacks in a shared platform.

The full operational picture

The total cost of a Lambda-backed custom resource is not just the Lambda function code. It includes the potential for hour-long failure windows when error handling is incomplete, the manual recovery path when rollback cascades, and the maintenance burden of a separate deployable tied to stack lifecycle events.

Understanding these costs before adding a custom resource leads to more robust implementations: complete error handling, idempotent delete handlers, well-scoped execution roles, and a clear ownership model for the function. For operations where a native extension exists that covers the same requirement, the trade-off is worth evaluating directly.