Go on AWS Lambda After go1.x: Moving to provided.al2023

On this page

Go has not had a Lambda runtime of its own for some time. The go1.x runtime is built on the original Amazon Linux AMI, and Amazon Linux 1 support ended on 31 December 2023. AWS scheduled go1.x, java8 and the original provided runtime to be deprecated with it, and pointed Go users at the OS-only runtimes: provided.al2, and since November 2023 provided.al2023.

The migration guides make this sound like a one-line change, and for the handler code it is: nothing in a Go handler needs to change. Everything around the code does. The executable has to be called bootstrap, the runtime setting changes, the build flags change, and a build that silently links against the CI machine’s C library can start failing for reasons that have nothing to do with Lambda. This article runs one ordinary Go function through the builds a team is likely to produce on the way, in the Lambda base images dated 5 January 2024, and records which ones start and which ones do not.

It also covers finding every function and published version still on go1.x, a pipeline gate that refuses a package before it is deployed, and the Terraform and alias change that keeps the move reversible.

Deprecation stages

Deprecation does not stop a function from running. Lambda deprecates a runtime in stages: first the runtime stops receiving security patches and support, then creating new functions on it is blocked, and later updating existing ones is blocked. Invocations carry on throughout. The date that matters for an engineering team is the last one, because after it a go1.x function can be invoked but not changed, including when it needs an emergency fix. Check the current dates in the Lambda runtimes table for your account; AWS has moved them before.

There is a second, quieter consequence. Once updates are blocked, a function that has been moved to provided.al2023 cannot be moved back to go1.x. Rollback has to come from published versions and aliases, not from redeploying the old configuration.

What changes

go1.xprovided.al2023
Operating systemAmazon Linux 1, glibc 2.17Amazon Linux 2023, glibc 2.34
ExecutableAny name, given as the handlerMust be bootstrap in a zip package
Handler settingName of the executableRequired by the API, not used
How the function is startedA shim starts your binary and calls it over net/rpcYour binary is the runtime and calls the Runtime API itself
Architecturesx86_64 onlyx86_64 and arm64
Initialisation billingInit time not billedInit time included in billed duration

The billing row comes from AWS’s own migration post: the go1.x runtime does not bill for initialisation, and the OS-only runtimes include initialisation in the billed duration. For a Go function that does little in main before calling lambda.Start, that is a few milliseconds per cold start. For one that opens database connections or loads configuration in main, it is worth measuring before and after.

Finding every function

The first job is an inventory, and it has to include published versions. An alias such as live can point at version 14 long after $LATEST has moved on, and the version behind the alias is the one serving traffic. The finder lists every function version on an Amazon Linux 1 runtime, with the aliases that point at it, using only lambda:ListFunctions and lambda:ListAliases:

Python19 lines
def scan_region(session: boto3.Session, region: str) -> list[Finding]:
    client = session.client("lambda", region_name=region, config=RETRY)
    findings: list[Finding] = []
    alias_cache: dict[str, dict[str, list[str]]] = {}
    for page in client.get_paginator("list_functions").paginate(FunctionVersion="ALL"):
        for fn in page["Functions"]:
            runtime = fn.get("Runtime", "")          # container image functions have no Runtime
            if runtime not in AL1_RUNTIMES:
                continue
            name = fn["FunctionName"]
            if name not in alias_cache:
                alias_cache[name] = aliases_by_version(client, name)
            findings.append(Finding(
                region=region, function=name, version=fn["Version"], runtime=runtime,
                handler=fn.get("Handler", ""), architectures=",".join(fn.get("Architectures", ["x86_64"])),
                aliases=",".join(sorted(alias_cache[name].get(fn["Version"], []))),
                replacement=AL1_RUNTIMES[runtime],
            ))
    return findings

It exits with 1 when it finds anything, so it can run as a scheduled check per account and Region. It is tested against moto’s in-memory Lambda with a go1.x function that has a published version behind an alias, a java8 function, and functions already on provided.al2023 and python3.12. Functions deployed as container images have no runtime field and are not listed; if they were built FROM public.ecr.aws/lambda/go:1, they are on Amazon Linux 1 too and need a new base image.

Test setup

The function is an API Gateway HTTP API handler that returns an order’s status, written with Go 1.21 and its new log/slog package for JSON logs. It is built several ways and each build is run in the Lambda base image it would meet, under the Runtime Interface Emulator that AWS ships in those images. Each run gets the same three requests: a known order (200), a malformed id (400) and an unknown order (404).

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FFF3E0', 'primaryTextColor': '#232F3E', 'primaryBorderColor': '#FF9900', 'lineColor': '#232F3E', 'clusterBkg': '#F8FAFC', 'clusterBorder': '#CBD5E1', 'edgeLabelBackground': '#FFFFFF', 'nodeTextColor': '#232F3E'}}}%%
flowchart TD
    S["Same handler source<br/>Go 1.21.5, aws-lambda-go 1.43.0"] --> B1["go1.x build<br/>main, RPC compiled in"]
    S --> B2["norpc build<br/>bootstrap, stripped"]
    S --> B3["cgo build<br/>CGO_ENABLED=1 on the CI runner"]
    B1 --> G["go1.x base image<br/>Amazon Linux 1, glibc 2.17"]
    B2 --> P["provided.al2023 base image<br/>glibc 2.34, x86_64 and arm64"]
    B3 --> G
    B3 --> P
    B1 --> P
    G --> R["Three API Gateway requests<br/>200, 400, 404 expected"]
    P --> R
    classDef compute fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E
    classDef data fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
    classDef store fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
    classDef alert fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
    class S store
    class B1,B2 compute
    class B3 alert
    class G,P data
    class R store

The base images are pinned to the dated tags published on 5 January 2024: lambda/go:1.2024.01.05.16 and lambda/provided:al2023.2024.01.05.14. The builds come from one script:

Bash12 lines
build() {  # build <variant> <file name> <GOARCH> <CGO_ENABLED> <tags> <extra ldflags>
  local variant=$1 name=$2 arch=$3 cgo=$4 tags=$5 ldflags=$6
  mkdir -p "$dist/$variant"
  (cd "$src" && GOOS=linux GOARCH="$arch" CGO_ENABLED="$cgo" \
     go build -trimpath -tags "$tags" -ldflags "$ldflags" -o "$dist/$variant/$name" .)
  local bin="$dist/$variant/$name" linking glibc
  if file "$bin" | grep -q 'statically linked'; then linking=static; else linking=dynamic; fi
  glibc=$( (objdump -T "$bin" 2>/dev/null || true) | { grep -o 'GLIBC_[0-9.]*' || true; } | sort -uV | tail -1)
  printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$variant" "$name" "$arch" "$cgo" "${tags:--}" \
    "$(stat -c %s "$bin")" "$linking" "${glibc:--}" >> "$dist/build.tsv"
  go version -m "$bin" | { grep -E 'build[[:space:]]+(CGO_ENABLED|-tags|GOARCH|-ldflags)' || true; } | sed "s|^|$variant: |" >> "$dist/buildinfo.txt"
}
Bash15 lines
# What a go1.x pipeline produced: an executable called main, with the RPC entry point compiled in.
build go1x             main      amd64 0 ""            ""
# The same build renamed: does provided.al2023 accept a binary that still carries the RPC code?
build rpc-bootstrap    bootstrap amd64 0 ""            ""
# The recommended build for provided.al2023: RPC code left out, symbol tables stripped.
build al2023-amd64     bootstrap amd64 0 lambda.norpc  "-s -w"
build al2023-arm64     bootstrap arm64 0 lambda.norpc  "-s -w"
# The two size levers taken one at a time.
build norpc-unstripped bootstrap amd64 0 lambda.norpc  ""
build rpc-stripped     bootstrap amd64 0 ""            "-s -w"
# Built on the CI runner with cgo left on, which is Go's default when a C compiler is present.
build cgo-bootstrap    bootstrap amd64 1 lambda.norpc  "-s -w"
build cgo-main         main      amd64 1 ""            ""
# A norpc build given to the old go1.x runtime.
build norpc-main       main      amd64 0 lambda.norpc  "-s -w"

Test results

Text8 lines
go1.x image, go1.x build                         worked=True  same=True  [known 200, malformed 400, unknown 404] as expected
provided.al2023, norpc build, x86_64             worked=True  same=True  [known 200, malformed 400, unknown 404] as expected
provided.al2023, norpc build, arm64 (QEMU)       worked=True  same=True  [known 200, malformed 400, unknown 404] as expected
provided.al2023, go1.x build renamed bootstrap   worked=True  same=True  [known 200, malformed 400, unknown 404] as expected
provided.al2023, go1.x package unchanged         worked=False same=False [known error, malformed error, unknown error] as expected
provided.al2023, cgo build from the CI runner    worked=True  same=True  [known 200, malformed 400, unknown 404] recorded
go1.x image, cgo build from the CI runner        worked=False same=False [known error, malformed error, unknown error] recorded
go1.x image, norpc build                         worked=True  same=True  [known 200, malformed 400, unknown 404] recorded
Matrix of eight runs: which build was run in which Lambda base image and whether it answered
Eight runs, three requests each. Every build that started returned the same responses.

Four results matter. The recommended build works on both architectures with identical responses. The old go1.x package, unchanged, fails on every request with Runtime.InvalidEntrypoint, because the OS-only runtime looks for bootstrap and finds nothing. The go1.x build simply renamed to bootstrap also works, so the RPC code compiled into it does no harm on provided.al2023. And the cgo build behaves differently depending on which image it meets, which is the trap described below.

How each runtime starts the function

The results make more sense once you look inside the go1.x image. Its /var/runtime/bootstrap is a link to a Go executable called aws-lambda-go, and that executable contains the string _LAMBDA_SERVER_PORT. The go1.x runtime starts your binary with that variable set and calls it over net/rpc:

%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E'}}}%%
sequenceDiagram
    participant L as Lambda (RAPID)
    participant S as go1.x shim /var/runtime/bootstrap
    participant F as Your binary (main)
    L->>S: start runtime
    S->>F: exec with _LAMBDA_SERVER_PORT set
    F-->>S: net/rpc server listening
    L->>S: invocation event
    S->>F: Function.Invoke over RPC
    F-->>S: response
    S-->>L: response

On provided.al2023 there is no shim. Your binary is started as the runtime and polls the Runtime API itself:

%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E'}}}%%
sequenceDiagram
    participant L as Lambda (RAPID)
    participant F as Your binary (bootstrap)
    L->>F: exec bootstrap with AWS_LAMBDA_RUNTIME_API set
    F->>L: GET /runtime/invocation/next
    L-->>F: event
    F->>L: POST /runtime/invocation/{id}/response

The aws-lambda-go library supports both. In version 1.43.0, lambda.Start picks the RPC path when _LAMBDA_SERVER_PORT is set and the Runtime API path when AWS_LAMBDA_RUNTIME_API is set, and the build tag lambda.norpc leaves the RPC path out of the binary. One emulator result needs care: the norpc build also answered in the go1.x image, because the emulator provides the Runtime API there too. That is an emulator result, not evidence that a norpc build runs on the real go1.x runtime, and it does not matter for a migration that is moving away from go1.x anyway.

Binary size

Build (amd64)BytesChange
RPC compiled in, not stripped (go1.x default)9,969,194–
-tags lambda.norpc7,786,381-22%
-ldflags "-s -w"6,832,128-31%
Both5,324,800-47%
Both, arm645,046,272-49%

The two flags are independent and both are free. Stripping removes the symbol table and DWARF data, so a panic still prints a stack trace with function names but a debugger has less to work with; keep an unstripped build as a pipeline artefact if your team debugs core dumps. A smaller package uploads faster and unpacks faster on a cold start, though for a 5 to 10 MB Go binary the difference in start-up time is small next to whatever main does.

cgo and glibc

Go only links against the system C library when cgo is enabled, and it is enabled by default when a C compiler is present, which it is on most Linux build machines and CI runners. The function in this sample uses nothing that needs C, but the net package, which the Lambda client uses, can call the C resolver when cgo is on, and that is enough to link the binary against glibc. The cgo build from the CI runner is dynamically linked and needs GLIBC_2.32 and GLIBC_2.34. In the go1.x image it does not start:

Text2 lines
/var/task/main: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/main)
/var/task/main: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/main)

In the provided.al2023 image, which ships glibc 2.34, the same binary works. That is luck rather than design: a newer runner image or a dependency that calls a newer libc function raises the requirement, and the function fails on its first cold start after a routine CI upgrade. A go1.x package built on macOS or Windows never had the problem, because cross-compiling to Linux turns cgo off by default. Moving the build into a Linux pipeline as part of the migration turns it back on.

⚠️
CGO DEFAULT IN CI

Set CGO_ENABLED=0 explicitly in every Lambda build. It is the default when cross-compiling and the opposite of the default on a Linux runner with gcc installed, so the same Makefile produces a static binary on a laptop and a glibc-dependent one in CI.

Build gate

All three failures above are visible in the package before it is deployed. The gate unzips the package, checks that bootstrap exists and is executable, and reads the build settings that the Go toolchain records in every binary with go version -m:

Bash13 lines
info="$(go version -m "$bin" 2>/dev/null)" || { fail "bootstrap is not a Go executable"; exit 1; }
setting() { awk -v k="$1" '$1=="build" && index($2, k"=")==1 {sub(k"=", "", $2); print $2}' <<< "$info"; }

[ "$(setting GOOS)" = linux ] || fail "GOOS is $(setting GOOS), not linux"
case "$want_arch:$(setting GOARCH)" in
  x86_64:amd64|arm64:arm64) ;;
  *) fail "GOARCH $(setting GOARCH) does not match the function architecture $want_arch" ;;
esac
[ "$(setting CGO_ENABLED)" = 0 ] || fail "built with CGO_ENABLED=$(setting CGO_ENABLED); the binary depends on the build machine's glibc"
case ",$(setting -tags)," in
  *,lambda.norpc,*) ;;
  *) echo "WARN $zip: built without -tags lambda.norpc (works, but carries the unused RPC code)" ;;
esac

Run against six packages in CI, it accepted the three that work and refused the three that would fail:

Text7 lines
OK   al2023-amd64.zip: bootstrap, linux/amd64, static
OK   al2023-arm64.zip: bootstrap, linux/arm64, static
WARN rpc-bootstrap.zip: built without -tags lambda.norpc (works, but carries the unused RPC code)
OK   rpc-bootstrap.zip: bootstrap, linux/amd64, static
FAIL go1x.zip: no file named bootstrap at the root of the zip (found: main )
FAIL cgo-bootstrap.zip: built with CGO_ENABLED=1; the binary depends on the build machine's glibc
FAIL al2023-arm64.zip: GOARCH arm64 does not match the function architecture x86_64
💡
GATE BEFORE DEPLOY

Put the gate between the build and the upload, and pass it the architecture from the same variable that sets the function’s architectures. Most arm64 migrations that fail do so because the build and the function configuration read the architecture from two different places.

Infrastructure change

In Terraform the function definition changes in four places: runtime, handler, architecture and the package. Publishing a version and moving an alias makes the switch reversible without needing go1.x again:

HCL20 lines
resource "aws_lambda_function" "order_status" {
  function_name    = "order-status"
  role             = var.execution_role_arn
  filename         = var.package_path
  source_code_hash = filebase64sha256(var.package_path)

  runtime       = "provided.al2023"
  handler       = "bootstrap" # ignored by the OS-only runtime, but required by the API
  architectures = ["arm64"]
  memory_size   = 256
  timeout       = 10
  publish       = true

  environment {
    variables = {
      # Not needed by Lambda; makes the build mode visible in the console during the migration.
      BUILD_FLAVOUR = "norpc-al2023"
    }
  }
}

This passes terraform validate with Terraform 1.6.6 and AWS provider 5.31.0. Callers should invoke the alias ARN, not the unqualified function name, so that rollback is a single alias update. If API Gateway or an event source mapping points at the unqualified ARN today, move it to the alias first, while still on go1.x, as a separate change.

The move from go1.x to provided.al2023 in four steps: build, package, function, release, with the three failures seen in the emulator
The handler stays the same; the build, the package, the runtime setting and the release change.

Cost

The runtime change itself has no price. Two things move the bill. Initialisation becomes billed time on the OS-only runtimes, as noted above. And arm64 becomes available: AWS’s migration post cites up to 34% better price-performance for Graviton2 against x86_64. For a Go function the arm64 build is a cross-compile with no code change, so it is worth doing in the same release if every dependency is pure Go. If anything in the build needs cgo, the cross-compile needs an arm64 C toolchain, which is one more reason to keep cgo off.

Production considerations

  • Order of work. Inventory with the finder, move callers to aliases, change the build and add the gate, then switch functions one at a time with a published version behind an alias.
  • Logs. The function’s own log lines do not change. Lambda’s platform lines do: there is no shim process on provided.al2023, so any log filter that matched shim output needs checking.
  • Timeouts and memory. Keep them as they were for the first release. Change one thing at a time, so a regression can be attributed.
  • Layers. Layers built for Amazon Linux 1 that contain native libraries need rebuilding against Amazon Linux 2023.
  • Monitoring. Watch the error rate and Init Duration in the REPORT lines for the first day on each function. A function that fails to start shows Runtime.InvalidEntrypoint or Runtime.ExitError rather than a handler error.

Decision framework

SituationChoice
Pure Go dependenciesprovided.al2023, arm64, CGO_ENABLED=0, norpc, stripped
A dependency needs cgoprovided.al2023, x86_64, built inside an Amazon Linux 2023 container so glibc matches
Native libraries in a layerRebuild the layer for Amazon Linux 2023 before switching
Container image function on lambda/go:1Rebuild FROM public.ecr.aws/lambda/provided:al2023
Organisation still standardised on Amazon Linux 2provided.al2 works the same way, but AL2023 is the longer-lived target

The first row should be the default. The second is the fallback when cgo cannot be avoided, and building inside the target image is the only reliable way to match the C library.

📝
HOW THIS WAS CHECKED

Everything ran in GitHub Actions (run) with Go 1.21.5, aws-lambda-go 1.43.0, Terraform 1.6.6 with AWS provider 5.31.0, and Python 3.12 with boto3 1.34.14, moto 4.2.12 and pytest 7.4.4. The base images were the dated tags of 5 January 2024, run under the Runtime Interface Emulator; arm64 ran under QEMU, so no timings are reported. Nothing was deployed to AWS, so billing, real start-up behaviour and the real go1.x environment were not tested. The three API Gateway events are hand-written. The finder ran against moto, not an AWS account.

Key takeaways

  • The handler code does not change; the executable name, runtime setting, build flags and release process do.
  • A go1.x package deployed unchanged to provided.al2023 fails every request with Runtime.InvalidEntrypoint.
  • Set CGO_ENABLED=0: the cgo build from a Linux CI runner needed GLIBC_2.34 and failed in the Amazon Linux 1 image.
  • -tags lambda.norpc and -ldflags "-s -w" together cut the binary by 47%.
  • Inventory published versions and aliases, not just $LATEST, and roll back by moving an alias.

If your migration turned up something this list misses, let me know in the comments.

Glossary

go1.x
The managed Lambda runtime for Go, based on Amazon Linux 1, which starts the function through an RPC shim.
provided.al2023
The OS-only Lambda runtime on Amazon Linux 2023, launched in November 2023, which runs an executable named bootstrap.
Runtime API
The HTTP interface a custom runtime polls for the next event and posts responses to; its address is in AWS_LAMBDA_RUNTIME_API.
Runtime Interface Emulator
The local stand-in for the Lambda service included in AWS’s base images, used to invoke a function over HTTP.
lambda.norpc
An aws-lambda-go build tag that leaves out the net/rpc start-up path used only by the go1.x runtime.
cgo
Go’s mechanism for calling C code; when enabled on Linux, the binary is dynamically linked to the system C library.
glibc
The GNU C library; a binary built against a newer version can fail to load where an older version is installed.
Symbol versioning
glibc labels functions with versions such as GLIBC_2.34; the loader refuses a binary that needs a version the system lacks.
Published version
An immutable snapshot of a Lambda function’s code and configuration, numbered from 1.
Alias
A named pointer to a published version, used so that callers can be switched or rolled back without redeploying.
Graviton2
AWS’s Arm-based processor, used by Lambda functions with the arm64 architecture.
RAPID
The process inside the Lambda execution environment that manages the runtime and serves the Runtime API.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.