Declarative Stream Processing: EventBridge Pipes vs. Custom Lambda Glue

In event-driven architectures on AWS, connecting real-time streaming sources—such as Amazon DynamoDB Streams, Amazon Kinesis Data Streams, Amazon SQS, and Amazon Managed Streaming for Apache Kafka (Amazon MSK)—to downstream event buses, queues, and analytical stores has historically required writing custom integration code.

For years, the universal solution was “Lambda Glue”: a dedicated AWS Lambda function sitting behind an Event Source Mapping (ESM), responsible for deserializing stream batches, applying custom filtering logic in code, enriching records with external database lookups, transforming JSON payloads, and fanning out to downstream targets.

While flexible, bespoke Lambda glue introduces significant operational baggage: account concurrency exhaustion, runtime maintenance debt, brittle partial-batch error handling, and billing for compute that does nothing more than discard uninteresting events.

With Amazon EventBridge Pipes, AWS introduced a managed, declarative point-to-point integration engine designed to eliminate bespoke integration code. But is EventBridge Pipes an automatic replacement for Lambda consumers in high-throughput enterprise pipelines?

This guide benchmarks EventBridge Pipes against custom Lambda glue across throughput dynamics, concurrency overhead, partial batch failure recovery, and real-world cost economics at scale.

1. The Serverless Integration Antipattern: The Lambda Glue Tax

To understand why declarative pipelines exist, examine the operational reality of running traditional Lambda glue at enterprise scale.

Consider an e-commerce order management service processing 50 million DynamoDB stream events per month across 64 Kinesis-backed partitions:

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'actorBkg': '#FFF3E0',
    'actorBorder': '#FF9900',
    'actorTextColor': '#232F3E',
    'actorLineColor': '#FF9900',
    'signalColor': '#0073BB',
    'signalTextColor': '#0F172A',
    'labelBoxBkgColor': '#FFF3E0',
    'labelBoxBorderColor': '#FF9900',
    'labelTextColor': '#232F3E',
    'noteBkgColor': '#FFF8E7',
    'noteBorderColor': '#FF9900',
    'noteTextColor': '#232F3E',
    'sequenceNumberColor': '#FFFFFF'
  }
}}%%
sequenceDiagram
    autonumber
    participant Stream as DynamoDB Streams / Kinesis
    participant ESM as Lambda Event Source Mapping (Poller)
    participant Lambda as Custom Lambda Glue Consumer
    participant Target as Amazon EventBridge Bus / SQS

    Stream->>ESM: Read Shard Batch (e.g., 100 records)
    ESM->>Lambda: Invoke Synchronously (1 Concurrency Slot Allocated)
    Note over Lambda: Custom Python/Node Code Deserializes Batch
    
    Note over Lambda: Filtering in Code: 85 records have status != "SHIPPED"
    Note over Lambda: 85% of compute time spent discarding noise!
    
    Lambda->>Target: Forward 15 Matched Records (PutEvents API)
    Target-->>Lambda: 200 OK Ack
    Lambda-->>ESM: Return Batch Execution Status
    Note over Lambda: Concurrency slot released after 350ms

The Three Inefficiencies of Lambda Glue

1. The Filtered-Event Tax: In many production pipelines, only a fraction of stream records require downstream routing (e.g., only orders transitioning to STATUS: SHIPPED). When filtering in application code, your Lambda function must boot, deserialize the entire JSON batch, iterate over elements, evaluate if statements, discard 85% of records, and bill you for the execution duration of discarded data.
2. Concurrency Exhaustion: Stream-based event source mappings allocate one concurrent Lambda execution per active shard (or more if ParallelizationFactor is enabled). A Kinesis stream with 128 shards processing continuous traffic permanently claims 128 concurrency slots from your regional account pool (default: 1,000)—even during idle periods where functions wake up on empty batches.
3. Poison Pill Brittleness: When a corrupted payload arrives in a stream batch, handling partial failure requires writing complex logic using ReportBatchItemFailures and managing BisectBatchOnFunctionError. A single uncaught exception blocks shard processing, introduces consumer lag, or requires writing manual Dead-Letter Queue (DLQ) routing logic.

2. The Declarative Paradigm: EventBridge Pipes Mechanics

Amazon EventBridge Pipes replaces bespoke glue code with a four-stage declarative pipeline managed entirely at the AWS infrastructure level:

$$\text{Source} \longrightarrow \text{Filter (Free)} \longrightarrow \text{Enrichment (Optional)} \longrightarrow \text{Target}$$

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#FFF3E0',
    'primaryTextColor': '#232F3E',
    'primaryBorderColor': '#FF9900',
    'lineColor': '#232F3E',
    'clusterBkg': '#F8FAFC',
    'clusterBorder': '#CBD5E1',
    'edgeLabelBackground': '#FFFFFF',
    'nodeTextColor': '#232F3E'
  }
}}%%
flowchart TD
    subgraph InputStage ["1. Ingestion & Filtering"]
        Source[("Streaming Source
(DynamoDB / Kinesis / SQS)")] --> Filter{"Declarative Event Filter
(JSON Pattern Matching)"}
        Filter -->|"Matches Filter Criteria"| Enrich{"Enrichment Required?"}
        Filter -->|"Unmatched Noise (85%)"| Discard["Discarded at Source
(Zero Compute Cost!)"]
    end

    subgraph ProcessingStage ["2. Optional Enrichment & Transform"]
        Enrich -->|"Yes"| LambdaEnrich["Lightweight Enrichment
(Lambda / API Destination)"]
        Enrich -->|"No"| Transform["Input Transformer
(Declarative JSONPath)"]
        LambdaEnrich --> Transform
    end

    subgraph OutputStage ["3. Target Delivery & Resilience"]
        Transform --> Target["Downstream Target
(EventBridge Bus / SQS / Step Functions)"]
        Target -.->|"On Permanent Failure"| DLQ[("Dead-Letter Queue
(Amazon SQS)")]
    end

    classDef awsCompute fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E;
    classDef awsDatabase fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74;
    classDef awsSuccess fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202;
    classDef awsDanger fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B;

    class Source,DLQ awsDatabase;
    class Filter,Enrich,Transform awsCompute;
    class Target,LambdaEnrich awsSuccess;
    class Discard awsDanger;

Key Architectural Differentiators

• Pre-Compute Filtering: Filtering occurs at the stream reader layer using standard EventBridge JSON patterns. Events that fail the filter are dropped immediately without triggering downstream compute, charging zero enrichment fees, and consuming zero Lambda concurrency slots.
• Input Transformers Without Code: Using declarative input templates (such as {"order_id": <$.dynamodb.NewImage.OrderId.S>, "amount": <$.dynamodb.NewImage.Total.N>}), Pipes reshapes the payload into standard business schemas natively, eliminating serialization code.
• Native Dead-Letter Queuing: If target delivery fails after configured retry attempts (MaximumRetryAttempts) or exceeds record age limits (MaximumRecordAgeInSeconds), Pipes splits the batch and routes poisoned records to an SQS DLQ natively without blocking the shard.

3. Concurrency & Throughput Benchmarking

How does EventBridge Pipes handle heavy stream throughput compared to traditional Lambda Event Source Mapping (ESM)?

The Concurrency Multiplier in Practice

Consider a bursty telemetry pipeline with 200 Kinesis shards:

• With Lambda Glue: At 200 shards, a sudden traffic spike immediately demands 200 concurrent executions. If the processing duration is 500ms, your Lambda function consumes 20% of a standard 1,000-concurrency account baseline.
• With EventBridge Pipes: The 200 shards are polled within the managed service fleet. If 90% of events are telemetry noise filtered out at the pipe boundary, and only 10% invoke an optional enrichment Lambda in small micro-batches, your customer account concurrency consumption drops from 200 slots down to under 15 slots.

4. Production Code: Bespoke Lambda vs. Declarative Infrastructure

Approach A: The Traditional Bespoke Lambda Glue (Python)

Below is the typical boilerplate required to safely consume a DynamoDB Stream, filter records, handle partial batch failures, and publish to an EventBridge bus:

"""
legacy_stream_consumer.py
Bespoke Lambda glue: Manual filtering, partial failure reporting, and event routing.
"""
import json
import os
import logging
import boto3
from botocore.exceptions import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

eventbridge = boto3.client("eventbridge")
EVENT_BUS_NAME = os.environ["EVENT_BUS_NAME"]

def lambda_handler(event, context):
    records = event.get("Records", [])
    batch_item_failures = []
    matched_events = []

    for record in records:
        record_id = record.get("eventID")
        try:
            # 1. Boilerplate deserialization
            event_name = record.get("eventName")
            if event_name not in ("INSERT", "MODIFY"):
                continue

            new_image = record.get("dynamodb", {}).get("NewImage", {})
            status = new_image.get("Status", {}).get("S")

            # 2. In-code filtering (Consumes billed compute!)
            if status != "SHIPPED":
                continue

            order_id = new_image.get("OrderId", {}).get("S")
            customer_id = new_image.get("CustomerId", {}).get("S")
            amount = float(new_image.get("TotalAmount", {}).get("N", 0))

            # 3. Payload transformation
            matched_events.append({
                "Source": "ecommerce.orders",
                "DetailType": "OrderShipped",
                "Detail": json.dumps({
                    "orderId": order_id,
                    "customerId": customer_id,
                    "totalAmount": amount,
                    "timestamp": record.get("dynamodb", {}).get("ApproximateCreationDateTime")
                }),
                "EventBusName": EVENT_BUS_NAME
            })

        except Exception as e:
            logger.error(f"Error parsing record {record_id}: {str(e)}")
            # Must track partial failures to avoid reprocessing entire batch
            batch_item_failures.append({"itemIdentifier": record_id})

    # 4. Downstream dispatch with chunking (PutEvents limit: 10 per request)
    if matched_events:
        for i in range(0, len(matched_events), 10):
            chunk = matched_events[i:i + 10]
            try:
                eventbridge.put_events(Entries=chunk)
            except ClientError as e:
                logger.error(f"Failed to publish to EventBridge: {str(e)}")
                raise

    # 5. Return partial failure contract
    return {"batchItemFailures": batch_item_failures}

Approach B: Declarative EventBridge Pipe via AWS CDK (TypeScript)

The identical functionality configured declaratively with zero runtime maintenance, zero boilerplate, and native dead-letter queuing:

import * as cdk from 'aws-cdk-lib';
import * as pipes from 'aws-cdk-lib/aws-pipes';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as events from 'aws-cdk-lib/aws-events';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';

export class OrderStreamPipeStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // 1. Existing DynamoDB Table & EventBridge Bus
    const ordersTable = dynamodb.Table.fromTableName(this, 'OrdersTable', 'Orders');
    const targetBus = events.EventBus.fromEventBusName(this, 'TargetBus', 'EnterpriseEventBus');

    // 2. Dead Letter Queue for Poison Pills
    const pipeDlq = new sqs.Queue(this, 'PipeDLQ', {
      queueName: 'orders-stream-pipe-dlq',
      retentionPeriod: cdk.Duration.days(14),
    });

    // 3. IAM Role for Pipe Execution
    const pipeRole = new iam.Role(this, 'OrderPipeRole', {
      assumedBy: new iam.ServicePrincipal('pipes.amazonaws.com'),
    });

    // Grant read on DynamoDB stream, write to target bus, and send to DLQ
    pipeDlq.grantSendMessages(pipeRole);
    targetBus.grantPutEventsTo(pipeRole);

    // 4. Declarative EventBridge Pipe Definition
    new pipes.CfnPipe(this, 'OrderShippedPipe', {
      name: 'orders-shipped-declarative-pipe',
      roleArn: pipeRole.roleArn,
      source: ordersTable.tableStreamArn!,
      sourceParameters: {
        dynamoDbStreamParameters: {
          startingPosition: 'LATEST',
          batchSize: 100,
          maximumBatchingWindowInSeconds: 5,
          maximumRecordAgeInSeconds: 3600,
          maximumRetryAttempts: 3,
          deadLetterConfig: {
            arn: pipeDlq.queueArn,
          },
        },
        // Native Declarative Filter: Discard noise at zero cost!
        filterCriteria: {
          filters: [
            {
              pattern: JSON.stringify({
                eventName: ['INSERT', 'MODIFY'],
                dynamodb: {
                  NewImage: {
                    Status: {
                      S: ['SHIPPED'],
                    },
                  },
                },
              }),
            },
          ],
        },
      },
      target: targetBus.eventBusArn,
      targetParameters: {
        eventBridgeEventBusParameters: {
          detailType: 'OrderShipped',
          source: 'ecommerce.orders',
        },
        // Native Declarative Transformation without Lambda!
        inputTemplate: JSON.stringify({
          orderId: '<$.dynamodb.NewImage.OrderId.S>',
          customerId: '<$.dynamodb.NewImage.CustomerId.S>',
          totalAmount: '<$.dynamodb.NewImage.TotalAmount.N>',
          timestamp: '<$.dynamodb.ApproximateCreationDateTime>',
        }),
      },
    });
  }
}

5. Cost Economics: The Real-World TCO

Pricing structures differ fundamentally between declarative pipes and compute-backed functions.

Cost Parameters (us-east-1)

• EventBridge Pipes: $0.40 per 1 million requests processed. Filtered-out requests are evaluated as part of the source polling at no extra cost.
• AWS Lambda: $0.20 per 1 million requests + $0.0000166667 per GB-second.
• CloudWatch Logs: $0.50 per GB ingested (frequently overlooked in high-volume Lambda glue!).

Monthly TCO Benchmark: 100 Million Stream Events

Consider an enterprise stream generating 100,000,000 events/month, evaluated across three filter selectivity scenarios:

The CloudWatch Logging Trap

In high-volume stream consumers, engineering teams routinely emit standard logs (INFO: Received batch of size 100). At 100 million events per month, logs quickly add tens of gigabytes of CloudWatch ingestion and retention fees.

Because EventBridge Pipes operates declaratively, logging is optional and execution metrics are delivered natively to CloudWatch Metrics at zero extra charge.

6. The 2026 Production Decision Matrix

While EventBridge Pipes dramatically simplifies architecture and lowers costs, it is not a 100% replacement for every stream processing scenario.

Summary: Production Rules for Stream Processing

1. Default to EventBridge Pipes for point-to-point flows: If you are reading from DynamoDB Streams, Kinesis, or SQS simply to filter, transform, and forward to an Event Bus, queue, or Step Functions workflow, writing a custom Lambda is now an architectural antipattern.
2. Push filtering to the infrastructure boundary: Never invoke code to evaluate static conditions. Use EventBridge pattern filtering to drop uninteresting stream records before compute boundaries.
3. Protect your Lambda concurrency pool: Replace shard-polling Lambda functions with Pipes to reclaim concurrency quotas for business-critical synchronous APIs.
4. Use Enrichment Lambdas strictly for external data joins: When extra business context is needed (e.g., querying Redis or calling an external partner API), use the Pipe’s optional Enrichment step. The Lambda runs only on pre-filtered records, drastically cutting compute duration.
5. Always configure an automated SQS Dead-Letter Queue: Take advantage of native deadLetterConfig on Pipes to capture poisoned payloads without blocking stream shard iterators.

References & Further Reading

• → Amazon EventBridge Pipes Developer Guide— Official architectural reference for point-to-point declarative integrations.
• → EventBridge Pipes Filtering Patterns— Reference guide for JSON filtering rules, prefix matching, and numeric comparisons.
• → AWS Compute Blog: Streamlining Event-Driven Architectures with EventBridge Pipes— Deep dive into cost modeling, batching windows, and enrichment patterns.
• → AWS Lambda Event Source Mapping Best Practices— Concurrency allocations, error handling, and bisect-on-error mechanics.