Serverless looks delightfully simple on a slide: drop in a file, fire an event, let Lambda handle the rest. The interesting part starts when retries, failures, observability, and the bill join the diagram.

The Architecture

Our pipeline consists of three main components:

  1. S3 as the data lake - Raw data lands here
  2. EventBridge for routing - Triggers processing based on events
  3. Lambda for compute - Stateless functions that process data

Key Considerations

Cost Optimization

Lambda pricing is based on invocations and duration. To optimize:

  • Use ARM-based Lambda functions (Graviton2) for up to 34% better price-performance
  • Right-size memory allocation
  • Implement batching where possible

Error Handling

Always implement dead letter queues (DLQs) and set up CloudWatch alarms for failures.

def lambda_handler(event, context):
    try:
        # Process event
        process_records(event['Records'])
    except Exception as e:
        # Log and let Lambda retry or send to DLQ
        logger.error(f"Processing failed: {e}")
        raise

Conclusion

Serverless data pipelines are powerful but require careful design around error handling, cost optimization, and observability.