Skip to content

JMESPath Functions

Tip

JMESPath is a query language for JSON used by AWS CLI, AWS Python SDK, and AWS Lambda Powertools for Python.

Built-in JMESPath Functions to easily deserialize common encoded JSON payloads in Lambda functions.

Key features

  • Deserialize JSON from JSON strings, base64, and compressed data
  • Use JMESPath to extract and combine data recursively

Getting started

You might have events that contains encoded JSON payloads as string, base64, or even in compressed format. It is a common use case to decode and extract them partially or fully as part of your Lambda function invocation.

Lambda Powertools also have utilities like validation, idempotency, or feature flags where you might need to extract a portion of your data before using them.

Info

Envelope is the terminology we use for the JMESPath expression to extract your JSON object from your data input.

Extracting data

You can use the extract_data_from_envelope function along with any JMESPath expression.

1
2
3
4
5
6
7
8
9
from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope

from aws_lambda_powertools.utilities.typing import LambdaContext


def handler(event: dict, context: LambdaContext):
    payload = extract_data_from_envelope(data=event, envelope="powertools_json(body)")
    customer = payload.get("customerId")  # now deserialized
    ...

1
2
3
{
    "body": "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\"}"
}

Built-in envelopes

We provide built-in envelopes for popular JMESPath expressions used when looking to decode/deserialize JSON objects within AWS Lambda Event Sources.

1
2
3
4
5
6
7
8
9
from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope, envelopes

from aws_lambda_powertools.utilities.typing import LambdaContext


def handler(event: dict, context: LambdaContext):
    payload = extract_data_from_envelope(data=event, envelope=envelopes.SNS)
    customer = payload.get("customerId")  # now deserialized
    ...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
    "Records": [
        {
            "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
            "receiptHandle": "MessageReceiptHandle",
            "body": "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\",\"booking\":{\"id\":\"5b2c4803-330b-42b7-811a-c68689425de1\",\"reference\":\"ySz7oA\",\"outboundFlightId\":\"20c0d2f2-56a3-4068-bf20-ff7703db552d\"},\"payment\":{\"receipt\":\"https:\/\/pay.stripe.com\/receipts\/acct_1Dvn7pF4aIiftV70\/ch_3JTC14F4aIiftV700iFq2CHB\/rcpt_K7QsrFln9FgFnzUuBIiNdkkRYGxUL0X\",\"amount\":100}}",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1523232000000",
                "SenderId": "123456789012",
                "ApproximateFirstReceiveTimestamp": "1523232000001"
            },
            "messageAttributes": {},
            "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
            "awsRegion": "us-east-1"
        }
    ]
}

These are all built-in envelopes you can use along with their expression as a reference:

Envelope JMESPath expression
API_GATEWAY_REST powertools_json(body)
API_GATEWAY_HTTP API_GATEWAY_REST
SQS Records[*].powertools_json(body)
SNS Records[0].Sns.Message | powertools_json(@)
EVENTBRIDGE detail
CLOUDWATCH_EVENTS_SCHEDULED EVENTBRIDGE
KINESIS_DATA_STREAM Records[*].kinesis.powertools_json(powertools_base64(data))
CLOUDWATCH_LOGS awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]

Advanced

Built-in JMESPath functions

You can use our built-in JMESPath functions within your expressions to do exactly that to decode JSON Strings, base64, and uncompress gzip data.

Info

We use these for built-in envelopes to easily decode and unwrap events from sources like API Gateway, Kinesis, CloudWatch Logs, etc.

powertools_json function

Use powertools_json function to decode any JSON String anywhere a JMESPath expression is allowed.

Validation scenario

This sample will decode the value within the data key into a valid JSON before we can validate it.

1
2
3
4
5
6
7
8
9
from aws_lambda_powertools.utilities.validation import validate

import schemas

sample_event = {
    'data': '{"payload": {"message": "hello hello", "username": "blah blah"}}'
}

validate(event=sample_event, schema=schemas.INPUT, envelope="powertools_json(data)")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
INPUT = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "Sample schema",
    "description": "The root schema comprises the entire JSON document.",
    "examples": [{"message": "hello world", "username": "lessa"}],
    "required": ["message", "username"],
    "properties": {
        "message": {
            "$id": "#/properties/message",
            "type": "string",
            "title": "The message",
            "examples": ["hello world"],
            "maxLength": 100,
        },
        "username": {
            "$id": "#/properties/username",
            "type": "string",
            "title": "The username",
            "examples": ["lessa"],
            "maxLength": 30,
        },
    },
}

OUTPUT = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "Sample outgoing schema",
    "description": "The root schema comprises the entire JSON document.",
    "examples": [{"statusCode": 200, "body": "response"}],
    "required": ["statusCode", "body"],
    "properties": {
        "statusCode": {"$id": "#/properties/statusCode", "type": "integer", "title": "The statusCode"},
        "body": {"$id": "#/properties/body", "type": "string", "title": "The response"},
    },
}

Idempotency scenario

This sample will decode the value within the body key of an API Gateway event into a valid JSON object to ensure the Idempotency utility processes a JSON object instead of a string.

Deserializing JSON before using as idempotency key
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import json
from aws_lambda_powertools.utilities.idempotency import (
    IdempotencyConfig, DynamoDBPersistenceLayer, idempotent
)

persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")
config = IdempotencyConfig(event_key_jmespath="powertools_json(body)")

@idempotent(config=config, persistence_store=persistence_layer)
def handler(event:APIGatewayProxyEvent, context):
    body = json.loads(event['body'])
    payment = create_subscription_payment(
        user=body['user'],
        product=body['product_id']
    )
    ...
    return {
        "payment_id": payment.id,
        "message": "success",
        "statusCode": 200
    }

powertools_base64 function

Use powertools_base64 function to decode any base64 data.

This sample will decode the base64 value within the data key, and decode the JSON string into a valid JSON before we can validate it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from aws_lambda_powertools.utilities.validation import validate

import schemas

sample_event = {
    "data": "eyJtZXNzYWdlIjogImhlbGxvIGhlbGxvIiwgInVzZXJuYW1lIjogImJsYWggYmxhaCJ9="
}

validate(
      event=sample_event,
      schema=schemas.INPUT,
      envelope="powertools_json(powertools_base64(data))"
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
INPUT = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "Sample schema",
    "description": "The root schema comprises the entire JSON document.",
    "examples": [{"message": "hello world", "username": "lessa"}],
    "required": ["message", "username"],
    "properties": {
        "message": {
            "$id": "#/properties/message",
            "type": "string",
            "title": "The message",
            "examples": ["hello world"],
            "maxLength": 100,
        },
        "username": {
            "$id": "#/properties/username",
            "type": "string",
            "title": "The username",
            "examples": ["lessa"],
            "maxLength": 30,
        },
    },
}

OUTPUT = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "Sample outgoing schema",
    "description": "The root schema comprises the entire JSON document.",
    "examples": [{"statusCode": 200, "body": "response"}],
    "required": ["statusCode", "body"],
    "properties": {
        "statusCode": {"$id": "#/properties/statusCode", "type": "integer", "title": "The statusCode"},
        "body": {"$id": "#/properties/body", "type": "string", "title": "The response"},
    },
}

powertools_base64_gzip function

Use powertools_base64_gzip function to decompress and decode base64 data.

This sample will decompress and decode base64 data, then use JMESPath pipeline expression to pass the result for decoding its JSON string.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from aws_lambda_powertools.utilities.validation import validate

import schemas

sample_event = {
    "data": "H4sIACZAXl8C/52PzUrEMBhFX2UILpX8tPbHXWHqIOiq3Q1F0ubrWEiakqTWofTdTYYB0YWL2d5zvnuTFellBIOedoiyKH5M0iwnlKH7HZL6dDB6ngLDfLFYctUKjie9gHFaS/sAX1xNEq525QxwFXRGGMEkx4Th491rUZdV3YiIZ6Ljfd+lfSyAtZloacQgAkqSJCGhxM6t7cwwuUGPz4N0YKyvO6I9WDeMPMSo8Z4Ca/kJ6vMEYW5f1MX7W1lVxaG8vqX8hNFdjlc0iCBBSF4ERT/3Pl7RbMGMXF2KZMh/C+gDpNS7RRsp0OaRGzx0/t8e0jgmcczyLCWEePhni/23JWalzjdu0a3ZvgEaNLXeugEAAA=="
}

validate(
    event=sample_event,
    schema=schemas.INPUT,
    envelope="powertools_base64_gzip(data) | powertools_json(@)"
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
INPUT = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "Sample schema",
    "description": "The root schema comprises the entire JSON document.",
    "examples": [{"message": "hello world", "username": "lessa"}],
    "required": ["message", "username"],
    "properties": {
        "message": {
            "$id": "#/properties/message",
            "type": "string",
            "title": "The message",
            "examples": ["hello world"],
            "maxLength": 100,
        },
        "username": {
            "$id": "#/properties/username",
            "type": "string",
            "title": "The username",
            "examples": ["lessa"],
            "maxLength": 30,
        },
    },
}

OUTPUT = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "$id": "http://example.com/example.json",
    "type": "object",
    "title": "Sample outgoing schema",
    "description": "The root schema comprises the entire JSON document.",
    "examples": [{"statusCode": 200, "body": "response"}],
    "required": ["statusCode", "body"],
    "properties": {
        "statusCode": {"$id": "#/properties/statusCode", "type": "integer", "title": "The statusCode"},
        "body": {"$id": "#/properties/body", "type": "string", "title": "The response"},
    },
}

Bring your own JMESPath function

Warning

This should only be used for advanced use cases where you have special formats not covered by the built-in functions.

For special binary formats that you want to decode before applying JSON Schema validation, you can bring your own JMESPath function and any additional option via jmespath_options param.

In order to keep the built-in functions from Powertools, you can subclass from PowertoolsFunctions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from aws_lambda_powertools.utilities.jmespath_utils import (
    PowertoolsFunctions, extract_data_from_envelope)
from jmespath.functions import signature


class CustomFunctions(PowertoolsFunctions):
    @signature({'types': ['string']})  # Only decode if value is a string
    def _func_special_decoder(self, s):
        return my_custom_decoder_logic(s)

custom_jmespath_options = {"custom_functions": CustomFunctions()}

def handler(event, context):
    # use the custom name after `_func_`
    extract_data_from_envelope(data=event,
                              envelope="special_decoder(body)",
                              jmespath_options=**custom_jmespath_options)
    ...

1
    {"body": "custom_encoded_data"}
Last update: 2021-12-30
Back to top