Serverless in Production: Lessons Learned After Two Years

Two Years of Serverless in the Real World

The first production serverless system is often described as a success story: a small team ships a feature quickly, infrastructure shrinks to a configuration file, and the system scales automatically. That version is true, but incomplete.

After two years of operating event-driven functions, queues, scheduled jobs, and serverless containers, the most important lessons are much less glamorous. They concern duplicate messages, exhausted concurrency, mysterious latency, runaway logs, dependency failures, and the operational cost of every design decision.

The central lesson is simple: serverless removes much of the server management, but it does not remove distributed-systems engineering. Your application becomes a collection of independently deployed, intermittently connected components. Production readiness therefore depends on explicit boundaries, idempotent processing, careful backpressure, useful telemetry, and a cost model that reflects real traffic rather than a pricing-page estimate.

This article shares the practical patterns we learned while taking serverless workloads from prototypes to business-critical systems. The examples use AWS Lambda, API Gateway, SQS, and DynamoDB, but the principles also apply to Azure Functions, Google Cloud Run, and other event-driven platforms.

Lesson 1: Serverless Is an Architecture, Not a Hosting Model

Serverless is commonly treated as a deployment option: package a function, connect it to an event source, and let the cloud scale it. That framing hides the architectural shift.

A traditional service may own its entire request lifecycle. A serverless design usually divides that lifecycle into small, event-driven steps:

• An API Gateway accepts a request and invokes a function.
• The function writes a durable event to a queue.
• A worker consumes the event and performs the business operation.
• Another function reacts to the result.
• A scheduler starts recurring work.

Each boundary introduces new questions. What happens if the producer succeeds but the event is never published? What if the worker completes the operation but crashes before acknowledging the message? What if one downstream service is slow and hundreds of functions wait on it?

Start with workload shape

Serverless works best when the workload has at least some of these characteristics:

• Events arrive independently rather than in predictable batches.
• Traffic is spiky, seasonal, or difficult to forecast.
• Work is naturally divided into small units.
• Individual tasks can fail and retry without corrupting shared state.
• The team values rapid iteration and managed operations.

It is less attractive when a service requires continuous CPU utilization, very long-running jobs, tightly coupled in-memory state, or a specialized hardware profile. A hybrid architecture is often the right answer; committing to serverless everywhere is not a maturity model.

Design around explicit contracts

Every event should have a versioned schema, an immutable identifier, a creation time, and a correlation identifier. Treat the event as a public API even when both producer and consumer are owned by the same team. Producers change independently, messages survive deployments, and old events may be processed after new code is released.

A useful event envelope looks like this:

event_id: evt_01J7Y8M3Q2
type: order.created
schema_version: 1
occurred_at: '2025-02-14T10:32:18.441Z'
correlation_id: corr_72a90d1
source: checkout-api
data:
  order_id: ord_10482
  customer_id: cus_8831

This small investment makes idempotency, tracing, debugging, and future schema migration dramatically easier.

Lesson 2: Cold Starts Are a Tail-Latency Problem

Cold starts receive too much attention because they are easy to measure but easy to misunderstand. A cold start is not simply the time between two requests. It can include runtime initialization, extension loading, deployment initialization code, dependency loading, network setup, and the first execution of application code.

The correct question is not whether a function has a cold start. It is whether its cold-start distribution threatens the service-level objective for the endpoint or event path.

Measure the distribution that customers experience

Track at least p50, p95, and p99 latency, separated by cold and warm execution where the platform exposes that information. Include the complete path: API Gateway or queue wait, invocation duration, downstream calls, retries, and response serialization.

Useful measurements include:

• Initialization duration separately from handler duration.
• Error and throttle rates by function version.
• Queue age and event-source concurrency.
• Dependency latency and timeout counts.
• Deployment frequency compared with cold-start frequency.
• p99 latency during low traffic, when cold starts are most likely to dominate.

Do not rely on a laboratory number from a different runtime, package size, memory configuration, or network topology. Benchmark the exact configuration you plan to deploy.

Reduce avoidable initialization work

AWS Lambda currently supports memory settings from 128 MB to 10 GB, ephemeral storage up to 10 GB, and a maximum execution timeout of 15 minutes. More memory can improve CPU performance, but it also changes the cost of every millisecond. Choose memory using measured latency and cost, not intuition alone.

Keep deployment packages small, load only required dependencies, avoid expensive work in global initialization, and initialize reusable SDK clients outside the handler. The following example avoids constructing an S3 client for every request:

const { S3Client, GetObjectCommand } = require('@aws-sdk/client-s3');

const s3 = new S3Client({});

exports.handler = async (event) => {
  const result = await s3.send(new GetObjectCommand({
    Bucket: process.env.BUCKET_NAME,
    Key: event.objectKey
  }));

  return { bytes: result.Body.length };
};

Database connections require more care. A connection opened during initialization may survive across warm invocations, but it can also become stale or leak resources. Use a lazy connection manager, validate connections, and cap reuse time where appropriate.

Use provisioned concurrency selectively

Provisioned concurrency can reduce cold-start latency, but it introduces reserved capacity and additional cost. It makes sense for a small, latency-critical endpoint with predictable baseline traffic. It is usually a poor substitute for fixing oversized packages, inefficient initialization, or slow dependencies.

Lesson 3: Concurrency Requires Backpressure

Automatic scaling does not mean unlimited throughput. Every serverless platform, account, function, and dependency has limits.

For example, AWS documents a regional default concurrency quota of 1,000 concurrent executions for Lambda, although customers can request increases. Reserved concurrency can reserve part of that pool for a critical function and prevent noisy neighbors from exhausting it. Many Lambda event sources also use a default concurrency scaling rate of 100 requests per second per function, but the behavior varies by source and can be configured.

These details matter when a sudden traffic spike turns into a queue backlog instead of an outage.

Put a durable buffer in front of the work

SQS, SNS, EventBridge, Pub/Sub, Service Bus, and similar services absorb bursts and decouple producers from consumers. A queue is not merely a messaging tool; it is a backpressure mechanism.

Do not invoke a large number of downstream operations directly from an HTTP request. Persist the intent to do work, acknowledge the request, and process asynchronously. This improves customer-perceived latency and gives the platform time to scale.

Bound consumer concurrency

An event-source mapping with a controlled maximum concurrency prevents one consumer from monopolizing a shared concurrency pool:

resource 'aws_lambda_event_source_mapping' 'orders' {
  function_name       = var.function_arn
  event_source_arn    = var.queue_arn
  batch_size         = 10
  maximum_batching_window_in_seconds = 5

  scaling_config {
    maximum_concurrency = 50
  }
}

The right value is not a universal number. Derive it from downstream capacity, message processing time, latency requirements, and the number of other functions competing for concurrency. Then test it under load.

Watch queue age, not only queue length

A queue can contain thousands of harmless messages while remaining healthy, or only a few urgent messages while breaching a deadline. Track the age of the oldest message and the age of the 95th or 99th percentile message. Add alarms for:

• Oldest-message age.
• Number of visible and in-flight messages.
• Receive and delete rates.
• Consumer invocation errors and throttles.
• Dead-letter queue depth.
• Messages successfully processed per second.

Queue age is usually a better user-facing signal than request count.

Lesson 4: Assume Every Event Will Be Delivered More Than Once

Most production queues provide at-least-once delivery, not exactly-once processing. A worker can complete a database update and then lose its connection before acknowledging the message. The message becomes visible again and is delivered again.

This is not an edge case. Duplicate delivery is a normal condition that the application must tolerate.

Make the business operation idempotent

Give every event a stable event_id, record processed identifiers, and make duplicate checks part of the same transaction as the business change. If the identifier already exists, return success without repeating the side effect.

INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, NOW())
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;

If the insert returns no row, the event has already been processed. Skip the operation and acknowledge the message.

The implementation can use DynamoDB with a conditional write, PostgreSQL as shown above, Redis with carefully chosen expiration rules, or a dedicated idempotency service. The mechanism matters less than the invariant: one logical event must not create two financial charges, two shipments, or two customer notifications.

Prefer natural idempotency keys

Business identifiers often work better than infrastructure event identifiers. For payment processing, use the payment or order ID. For a webhook, use the provider event ID. A natural key also allows reconciliation across systems.

Use an outbox for dual writes

The hardest distributed write is writing to a database and publishing a message at the same time. If the database commit succeeds and message publication fails, the event is lost. If publication succeeds and the transaction rolls back, an event describes work that never happened.

Use a transactional outbox: write the business record and an outbox row in one database transaction, then publish the outbox row asynchronously. Consumers remain idempotent because publication can still repeat.

Lesson 5: Failure Paths Deserve More Design Than the Happy Path

A production system is defined by what happens when a dependency fails, a timeout occurs, or a message cannot be processed.

Classify errors before retrying

Not every error should be retried. A malformed payload, a schema-validation failure, or a missing required field will usually fail again. Retry transient errors such as connection resets, temporary throttling, and selected 5xx responses.

Use bounded exponential backoff with jitter, and set a maximum attempt count. Coordinate retry budgets across layers so that the function, queue, and downstream SDK do not each add repeated attempts and turn a brief incident into a prolonged one.

Configure dead-letter queues deliberately

A dead-letter queue, or platform equivalent, prevents poison messages from blocking a stream indefinitely. But moving a message to a dead-letter queue is not a resolution. Build an operational process around it:

• Alert when the dead-letter queue first becomes non-empty.
• Store enough context to diagnose the failure.
• Define who owns remediation.
• Provide a safe redrive mechanism.
• Test the redrive process.
• Sample or redact sensitive payloads according to policy.

For SQS, a queue can be configured to move messages to a dead-letter queue after a maximum receive count. For event sources that support partial batch failure, return only failed records so successful records are not processed again.

Set timeouts deliberately

A function timeout should be shorter than the event source's retry or visibility timeout when possible. Otherwise, the platform may deliver the same work while the first invocation is still running. Use connection and request timeouts on every downstream call, propagate deadlines, and cancel work that can no longer meet the request's deadline.

Design for degraded operation

Ask what the business can still do when one dependency is unavailable. Sometimes the correct response is to accept the request and complete it later. Sometimes it is to serve a cached result. Sometimes it is to fail quickly with a clear error. Silent fallback is dangerous unless the business explicitly approves it.

Lesson 6: Observability Must Follow the Event

The biggest operational mistake in early serverless systems was treating each function as an isolated service. A request might pass through an API, two functions, a queue, a worker, and a third-party API, but its logs lived in separate systems with no shared identity.

Use structured logs

Log JSON objects rather than concatenated strings. Include stable fields such as:

• request_id or correlation_id.
• event_id.
• event_type and schema_version.
• customer_id or another safe domain identifier.
• Function version and deployment identifier.
• Downstream operation and result.
• Retry count and error classification.

Never place access tokens, passwords, full payment details, or unnecessary personal data in logs.

Carry trace context through queues

Instrument inbound and outbound calls with OpenTelemetry or the platform's tracing service. When a message is published, inject the trace context into message metadata. Extract it when the message is consumed so the asynchronous workflow forms one distributed trace.

const { trace, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('order-processor');

exports.handler = async (event) => {
  return tracer.startActiveSpan('process-order', async (span) => {
    try {
      span.setAttribute('order.id', event.orderId);
      const result = await processOrder(event.orderId);
      span.setAttribute('order.result', result.status);
      return result;
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
};

Measure user-visible SLOs

Infrastructure metrics are necessary but insufficient. Define service-level indicators around successful business outcomes: accepted requests, processed orders, delivered notifications, or completed settlements. Pair them with error budgets and a burn-rate alert so responders know when to act.

A practical dashboard should include:

• Request and event success rate.
• End-to-end latency distribution.
• Function duration, errors, and throttles.
• Cold-start indicators.
• Queue age and backlog.
• Dependency latency, timeouts, and error rate.
• Dead-letter volume.
• Deployment markers and version comparisons.
• Cost per successful event or transaction.

Lesson 7: Security Must Be Enforced at Every Boundary

Serverless can improve security because workloads are short-lived and permissions can be narrowly scoped. It can also create a false sense of safety if every function receives broad credentials.

Apply least-privilege IAM

Give each function only the actions and resources required for its task. A queue consumer normally needs to receive, delete, and inspect specific queue messages—not administrative access to all SQS resources.

Version: '2012-10-17'
Statement:
  - Effect: Allow
    Action:
      - sqs:ReceiveMessage
      - sqs:DeleteMessage
      - sqs:GetQueueAttributes
    Resource: arn:aws:sqs:us-east-1:123456789012:order-processing

Use separate roles for separate functions. Review policies with automated tools such as IAM Access Analyzer, and remove permissions that are no longer required.

Authenticate at the edge and validate everywhere

API Gateway authorizers, load balancers, or service-to-service identity should establish who is calling the system. The function must still validate the request schema, content type, size, and authorization rules. Authentication is not the same as input validation.

Manage secrets carefully

Retrieve secrets from a managed secret store or workload identity mechanism rather than committing them to code. Rotate credentials, scope them narrowly, and avoid placing long-lived secrets in environment variables or deployment artifacts. Environment-variable encryption protects data at rest but does not eliminate the risk of exposure to the running process.

Control network paths deliberately

Put a function inside a VPC only when it needs private network access. VPC attachment can add initialization latency and creates a dependency on networking, routing, and security-group availability. For functions that only call public managed services over AWS endpoints, a private path may add complexity without meaningful benefit.

Use restrictive egress controls, encryption in transit, protected event-source mappings, and dependency scanning. Serverless reduces patching work for the runtime, but application dependencies and infrastructure definitions still require regular updates.

Lesson 8: Cost Is an Architectural Signal

The phrase pay per use is accurate but incomplete. Serverless billing commonly combines:

• Request or invocation charges.
• Duration charges based on memory and execution time.
• Provisioned concurrency or minimum capacity.
• Logs, traces, metrics, and storage.
• Network transfer and service-specific charges.
• Idle orchestration and supporting infrastructure.

A useful first model is:

monthly compute = invocations × request price + GB-seconds × duration price

where:

GB-seconds = invocations × duration in seconds × memory in gigabytes

As a concrete illustration, consider 10 million Lambda invocations per month at 256 MB and an average duration of 100 ms. At US East published rates around $0.20 per million requests and $0.0000166667 per GB-second, the approximate cost is:

Component	Calculation	Approximate monthly cost
Requests	10 million requests	$2.00
Duration	10M × 0.1s × 0.256GB = 256,000 GB-seconds	$4.27
Compute total	Requests plus duration	$6.27

The same workload at an average duration of one second would cost roughly $44.67 in compute. A 900 ms increase multiplied by 10 million invocations adds about $38.40. Logging, tracing, network transfer, and supporting services are additional.

Pricing and free tiers change, so use current regional rates for financial planning. The important lesson is that small duration changes have large aggregate effects.

Optimize for cost per business outcome

Raw invocation cost is not the most useful metric. Track cost per accepted request, processed order, sent notification, or successful background job. A cheap function that retries aggressively or triggers expensive downstream work may be expensive overall.

Practical optimizations include:

• Right-sizing memory with latency and cost tests.
• Removing unnecessary serialization and network calls.
• Batching when the downstream service supports it.
• Filtering events before invoking a function.
• Moving predictable high-volume work to more suitable compute.
• Sampling verbose logs while retaining errors and audit events.
• Setting log retention by usefulness and compliance needs.
• Using cost-allocation tags and per-service dashboards.
• Budget alerts based on both absolute spend and anomaly detection.

Do not optimize by removing observability or resilience. The cheapest system is not the one with the smallest cloud bill; it is the one with the lowest total cost of reliable operation.

Lesson 9: Test the Failure Modes, Not Just the Handler

A function that passes unit tests can still fail in production because its event is duplicated, its dependency times out, or the queue delivers an old schema.

Use a layered test strategy:

• Unit tests: Cover business logic, validation, error classification, and idempotency without cloud resources.
• Contract tests: Verify that producer events match consumer schemas and that API definitions remain backward compatible.
• Integration tests: Exercise queues, databases, secrets, permissions, and event-source mappings in an isolated environment.
• Resilience tests: Inject timeouts, malformed messages, duplicate deliveries, expired credentials, partial batch failures, and dependency outages.
• Load tests: Measure concurrency, queue drain rate, and downstream saturation under realistic traffic shapes.
• Production validation: Use canary deployments and synthetic transactions before broad traffic shifts.

Local emulators are useful for fast feedback, but they do not reproduce every networking, permission, or concurrency behavior of a cloud platform. Keep a small set of tests running against managed sandbox resources.

A particularly valuable test is the duplicate-delivery test. Process the same event twice and assert that only one business effect occurs. Another is the out-of-order test: deliver order.cancelled before order.created and verify that the consumer reaches the correct final state.

Lesson 10: Deploy Gradually and Roll Back Independently

Serverless deployments are fast, but speed makes accidental changes easier to distribute. Combine infrastructure as code, immutable function versions, automated tests, and progressive traffic shifts.

Maintain infrastructure in Terraform, CloudFormation, the Cloud Development Kit, or an equivalent declarative system. A deployment should create the code, configuration, permissions, event-source mapping, alarms, and dashboards as one reviewable change.

For AWS Lambda, an alias can send a small percentage of traffic to a new version:

Type: AWS::Lambda::Alias
Properties:
  FunctionName: !Ref OrdersFunction
  FunctionVersion: '1'
  RoutingConfig:
    AdditionalVersionWeights:
      - FunctionVersion: '2'
        FunctionWeight: 0.1

During the canary, compare error rate, p95 and p99 latency, downstream load, and business success metrics between versions. Roll back by changing the alias weight; do not rely on rebuilding old code.

Event schemas require their own compatibility strategy. Prefer additive changes, publish new field versions carefully, and use an expand-and-contract migration when old and new consumers overlap. A rollback of function code does not automatically roll back data already written with a new schema.

Feature flags can separate code deployment from business activation. They are especially useful when a serverless release changes pricing, permissions, data handling, or a downstream workflow.

Lesson 11: Build a Platform, but Avoid a Custom Framework Too Early

As more teams adopt serverless, repeated guardrails become necessary. A useful internal platform can provide:

• Function and project templates.
• Standard CI/CD pipelines.
• Preconfigured observability and correlation IDs.
• Least-privilege policy templates.
• Deployment canaries and rollback controls.
• Cost-allocation tags and dashboards.
• Security scanning and dependency updates.
• Approved messaging, tracing, and error-reporting libraries.

The platform should provide a paved road without hiding critical concepts from application teams. Developers should still understand retries, idempotency, concurrency, and event schemas. Excessive abstraction can produce a function framework that is difficult to debug and impossible to upgrade.

Start with templates and policies, observe where teams repeatedly make mistakes, and then automate the recurring tasks. Do not build a bespoke developer platform before several workloads have demonstrated a common need.

Lesson 12: Know When Serverless Is the Wrong Choice

Serverless is an excellent fit for many event-driven workloads, but it is not a universal optimization.

Workload	Serverless fit	Key consideration
Sporadic API traffic	Strong	Idle savings and burst scaling are valuable.
Queue-based background jobs	Strong	Add idempotency, concurrency limits, and DLQ handling.
Scheduled automation	Strong	Keep tasks bounded and prevent overlapping executions.
Serverless containers	Often good	Useful when dependencies or custom runtimes outweigh cold-start sensitivity.
Steady, predictable CPU utilization	Usually weaker	Compare measured unit costs with containers or dedicated compute.
Long-running workflows	Conditional	Lambda's maximum timeout is 15 minutes; use a workflow or batch service for longer work.
Stateful servers	Weak	Externalize state or choose a stateful platform.
Low-latency, always-hot endpoints	Conditional	Provisioned capacity or containers may be more predictable.
GPU-intensive inference	Conditional	Compare accelerator availability, warmup, and utilization directly.

The comparison must use real workload data. Include request shape, memory, duration, utilization, engineering labor, and outage cost. A serverless service that runs continuously at high CPU utilization may cost more than a container platform. Conversely, a highly variable workload often benefits from serverless even if its per-request compute rate is higher.

Avoid a mandatory platform decision driven by fashion. Standardize on mechanisms such as events, observability, and deployment, while allowing different compute choices for different workloads.

A Serverless Production Checklist

Before moving a workload to production, verify that the team can answer these questions:

• Is every event schema documented, versioned, and backward compatible?
• Can every consumer tolerate duplicate and out-of-order delivery?
• Are business operations protected by idempotency keys or conditional writes?
• Are concurrency limits, queue capacity, and downstream quotas known?
• Are transient and permanent errors classified separately?
• Do timeouts and retry budgets prevent cascading failures?
• Is there a dead-letter queue and a tested remediation process?
• Do logs and traces preserve correlation across asynchronous boundaries?
• Are SLOs, burn-rate alerts, and queue-age dashboards in place?
• Are IAM roles narrowly scoped and secrets rotated?
• Are costs allocated and monitored per workload?
• Can the previous version be restored without rebuilding it?
• Are duplicate, malformed, delayed, and out-of-order messages tested?
• Can one noisy function be isolated from critical workloads?
• Does the team know when serverless is no longer the economical or reliable choice?

A checklist is not a substitute for judgment, but it exposes missing operational decisions before customers do.

A Practical 90-Day Adoption Plan

Days 1-30: Establish the baseline

Choose one bounded, event-driven workload with a measurable business outcome. Document its traffic pattern, dependencies, latency objective, failure modes, and current cost. Define success criteria such as deployment lead time, recovery time, error rate, and cost per transaction before changing infrastructure.

Days 31-60: Build the production foundation

Implement infrastructure as code, structured logging, distributed tracing, idempotent processing, bounded concurrency, dead-letter handling, least-privilege permissions, and cost allocation. Run failure-injection and load tests against the same event paths you expect to use in production.

Days 61-90: Release safely and measure

Deploy through a canary, compare technical and business metrics, and rehearse rollback and dead-letter redrive. Document the operational runbook and feed real cost and latency data into the next platform decision.

The goal is not to prove that serverless is superior. The goal is to understand whether it fits the workload and to leave the team with a system whose behavior is observable and explainable.

Conclusion

Two years of production serverless changed our view of scalability. The platform can absorb traffic spikes, but only if the application has durable buffers, controlled concurrency, idempotent operations, realistic timeouts, and useful telemetry. It can reduce infrastructure overhead, but only if teams measure duration and business outcomes instead of assuming that pay-per-use automatically means lower cost.

The most successful serverless teams are not the ones that use the most functions. They are the ones that make the boundaries between functions explicit, design for retries and duplicates, treat observability as part of the architecture, and know when a different compute model is better.

Tanok Tech helps engineering teams design, migrate, and operate production-grade serverless systems without sacrificing reliability, security, or cost control. If your team is evaluating serverless or struggling with an existing workload, contact us for a practical architecture review and a focused production-readiness plan.

References and Further Reading

• AWS Lambda quotas and service limits
• AWS Lambda pricing
• AWS Lambda dead-letter queues
• AWS Well-Architected Serverless Lens
• OpenTelemetry documentation
• Google Cloud Architecture Framework: Serverless