Blob container polling vs. Event Grid events

The first choice in our journey is how to react to the payloads being stored. The choice is between a polling-based approach and an event-based approach.

The polling approach is covered in the Microsoft article Azure Blob storage trigger for Azure Functions.

Below is an abbreviated version of the example given. The code uses the BlobTrigger attribute with the path to match on specified in the constructor:

public static class BlobFunction
{
    [Function(nameof(BlobFunction))]
    public static void Run(
        [BlobTrigger("test-samples-trigger/{name}")] string myTriggerItem)
    {
        // <snip>
    }
}

However, I noted the following 'Tip':

There are several ways to execute your function code based on changes to blobs in a storage container. If you choose to use the Blob storage trigger, note that there are two implementations offered: a polling-based one (referenced in this article) and an event-based one. It is recommended that you use the event-based implementation as it has lower latency than the other.

This tip also included a link to the Trigger Azure Functions on blob containers using an event subscription Microsoft tutorial.

After reading up on both, I decided to go down the Event Grid route. Using events felt more decoupled than having functions polling the blob container. Plus the lower latency would be an advantage when delivering the webhook payloads.

Creating an Event Grid function

The next step was to use the Visual Studio wizard to create the Azure function. This function will be triggered on accepted messages and do the following two steps:

1. Deduplicate messages based on the payload
2. Forward messages to the appropriate endpoint

In this post, I won't be covering the first step. We will just look at triggering the function and using the trigger to retrieve the payload and forward it on.

With this functionality in mind, I used the Visual Studio wizard to create a new Azure function called DedupeAndForwardFunction and selected the option for 'Event Grid trigger'. The resulting code is shown below.

public class DedupeAndForwardFunction
{
    private readonly ILogger<DedupeAndForwardFunction> _logger;

    public DedupeAndForwardFunction(ILogger<DedupeAndForwardFunction> logger)
    {
        _logger = logger;
    }

    [Function(nameof(DedupeAndForwardFunction))]
    public void Run([EventGridTrigger] CloudEvent cloudEvent) // <-- CloudEvent?
    {
        _logger.LogInformation(
            "Event type: {type}, Event subject: {subject}",
            cloudEvent.Type, cloudEvent.Subject);
    }
}

What caught my eye was the type of event parameter. What is a CloudEvent?

Cloud Events vs Event Grid Events

A bit of searching led me to the CloudEvents v1.0 schema with Azure Event Grid Microsoft article that states:

Azure Event Grid natively supports events in the JSON implementation of CloudEvents v1.0 and HTTP protocol binding. CloudEvents is an open specification for describing event data. CloudEvents simplifies interoperability by providing a common event schema for publishing, and consuming cloud based events. This schema allows for uniform tooling, standard ways of routing & handling events, and universal ways of deserializing the outer event schema. With a common schema, you can more easily integrate work across platforms.

Looking a bit deeper, the key differences appear to be:

1. Schema Structure:
   • Event Grid: Uses Azure-specific field names (topic, eventType)
   • Cloud Events: Uses standardised field names (source, type)
2. Version Handling:
   • Event Grid: Has separate dataVersion and metadataVersion fields
   • Cloud Events: Uses single specversion field

Although Visual Studio pushed me down the CloudEvent route, I decided to revert to using EventGridEvent. This was because I would be working within the Azure ecosystem and the examples and documentation also favour the use of EventGridEvent.

So, the function code became the following, with an extra statement added to output the event as JSON.

[Function(nameof(DedupeAndForwardFunction))]
public void Run([EventGridTrigger] EventGridEvent eventGridEvent)
{
    _logger.LogInformation(
        "Event type: {eventType}, Event subject: {subject}",
        eventGridEvent.EventType, eventGridEvent.Subject); // <-- EventType

    _logger.LogDebug(
        "EVENT: {event}",
        JsonSerializer.Serialize(eventGridEvent));
}

For more information on Cloud Events, see cloudevents.io.

Hooking up the function to events

The next step was to get events from Blob Storage triggering the Azure Function. To do this, I navigated to the storage account in the Azure Portal and clicked the 'Event Subscription' button.

I was then prompted to create an event subscription, specifying the name of the subscription and the schema type. It is here that you choose whether the event will be in the Event Grid schema or Cloud Event schema.

I was also prompted for the name to the topic to which the storage account will publish events. It turns out that a storage account can only publish to one topic, so given this I chose a name to reflect the storage account. If you create further event subscriptions, then you are not prompted for the topic name again.

Below the topic name, you can select which events are to be published. In my case, I only wanted 'Blob Created' events. Finally, you select the endpoint for the event, which is the DedupeAndForwardFunction Azure Function I created and published earlier.

With the event subscription in place, I sent a valid request to the ValidateAndStoreFunction Azure Function. The resulting Blob Storage event trigged the DedupeAndForwardFunction and the event below was logged.

{
  "id": "0d666914-901e-0082-26b5-3ac1de067d1c",
  "topic": "<snip>",
  "subject": "/blobServices/default/containers/webhook-payloads-accepted/blobs/LovelyLoans/QuickValuationCo/2024-11-19/2024-11-19T18:58:31UTC-be2f0960-ff56-4cab-8465-9ebd0f6b1d5c.json",
  "data": {
    "api": "PutBlob",
    "clientRequestId": "d630505b-bffd-4419-b2a7-f8f109eb275f",
    "requestId": "0d666914-901e-0082-26b5-3ac1de000000",
    "eTag": "0x8DD08CC29D75EE3",
    "contentType": "application/octet-stream",
    "contentLength": 2266,
    "blobType": "BlockBlob",
    "accessTier": "Default",
    "url": "<snip>/webhook-payloads-accepted/LovelyLoans/QuickValuationCo/2024-11-19/2024-11-19T18:58:31UTC-be2f0960-ff56-4cab-8465-9ebd0f6b1d5c.json",
    "sequencer": "0000000000000000000000000003C5ED00000000002bc82a",
    "storageDiagnostics": {
      "batchId": "a7d80882-4006-005a-00b5-3ae687000000"
    }
  },
  "eventType": "Microsoft.Storage.BlobCreated",
  "eventTime": "2024-11-19T18:58:32.4819404Z",
  "dataVersion": ""
}

This was all good, but I then sent an invalid request. As I expected, a Blob Storage event trigged the DedupeAndForwardFunction Azure Function even though the design requires that only valid requests are forwarded. We can see this by the webhook-payloads-rejected in the subject.

{
  "id": "b9eb1363-a01e-001f-2bb5-3a3364069d50",
  "topic": "<snip>",
  "subject": "/blobServices/default/containers/webhook-payloads-rejected/blobs/LovelyLoans/QuickValuationCo/2024-11-19/2024-11-19T19:02:58UTC-4ea825da-ac01-4cf4-bcca-86f8d1d36d6d.json",
  "data": {
    "api": "PutBlob",
    "<snip>": "..."
  },
  "eventType": "Microsoft.Storage.BlobCreated",
  "eventTime": "2024-11-19T19:02:58.8654233Z",
  "dataVersion": ""
}

One possible solution would be to amend the DedupeAndForwardFunction Azure Function to introspect the event and ignore anything not from the webhook-payloads-accepted container. However, this would have cost each time it ran. A better way is to add an event filter.

Adding an event filter

I went back into the Azure Portal and edited the subscription. Selecting the 'Filters' tab, I enabled subject filtering and added a prefix to match on /blobServices/default/containers/webhook-payloads-accepted/blobs. This is where having an example event from Azure proved very useful.

With the subscription updated, I ran my tests. First checking that a valid request triggered the function, then a second test checking that only the first Azure Function was triggered. Sure enough, all I saw in the logs was the following.

2024-11-19T19:12:32Z   [Information]   Executing 'Functions.ValidateAndStoreFunction' (Reason='This function was programmatically called via the host APIs.', Id=ab31e19f-eb17-42f1-891b-ae2bdb80bdab)
2024-11-19T19:12:32Z   [Information]   Executed 'Functions.ValidateAndStoreFunction' (Succeeded, Id=ab31e19f-eb17-42f1-891b-ae2bdb80bdab, Duration=16ms)

It would still be worth putting a check in the DedupeAndForwardFunction Azure Function. It should protect itself from misconfiguration, but log an error to indicate that there is a misconfiguration.

The Microsoft Event subscription filter object article covers the various options available for filtering. One thing I did note, was that there is no support for wildcards.

Debugging locally

Now I had my Azure Function triggering, I could start on the actual functionality. Developing this would be much easier if I could run my code locally. A bit of searching turned up the article Debugging Azure Function Event Grid Triggers Locally.

This did point me in the right direction, but I could not get it to work. More searching then turned up the following two articles:

• Unable to debug Event Grid Trigger Azure function locally
• Azure Event Grid Trigger function is not working locally

These both highlighted that this comment, added by Visual Studio to the boilerplate Azure Function code, is misleading.

// Default URL for triggering event grid function in the local environment.
// http://localhost:7071/runtime/webhooks/EventGrid?functionName={functionname}

The port number is not always 7071. To find out what it is, you need to look in the launchSettings.json file.

Here I could see that, in my case, the port was 7089.

{
  "profiles": {
    "WebhookFunctionApp": {
      "commandName": "Project",
      "commandLineArgs": "--port 7089",
      "launchBrowser": false
    }
  }
}

With this knowledge, I was able to hit F5 and trigger my Azure Function locally with the following request. Note that the aeg-event-type: Notification header is required for this to work.

POST http://localhost:7089/runtime/webhooks/EventGrid?functionName=EventGridFunction
content-type: application/json
aeg-event-type: Notification

{
  "id": "0d666914-901e-0082-26b5-3ac1de067d1c",
  "subject": "/blobServices/default/containers/webhook-payloads-accepted/blobs/LovelyLoans/QuickValuationCo/2024-11-19/2024-11-19T18:58:31UTC-be2f0960-ff56-4cab-8465-9ebd0f6b1d5c.json",
  "<snip>": "...",
  "eventType": "Microsoft.Storage.BlobCreated",
  "eventTime": "2024-11-19T19:02:58.8654233Z",
  "dataVersion": ""
}

See also, the Microsoft article Test your Event Grid handler locally for more details local testing.

Processing system events

The final step for this blog post was to process the events and forward the request on to a downstream endpoint. Guided by the Microsoft Azure Event Grid client library article, I was pointed to the TryGetSystemEventData() method and the following code.

foreach (EventGridEvent egEvent in egEvents)
{
    // If the event is a system event, TryGetSystemEventData will return the deserialized system event
    if (egEvent.TryGetSystemEventData(out object systemEvent))
    {
        switch (systemEvent)
        {
            case SubscriptionValidationEventData subscriptionValidated:
                Console.WriteLine(subscriptionValidated.ValidationCode);
                break;
            case StorageBlobCreatedEventData blobCreated:
                Console.WriteLine(blobCreated.BlobType);
                break;
            // Handle any other system event type
            default:
                Console.WriteLine(egEvent.EventType);
                // we can get the raw Json for the event using Data
                Console.WriteLine(egEvent.Data.ToString());
                break;
        }
    }
    else
    {
        // <snip>
    }
}

Note here the handling of SubscriptionValidationEventData. When Azure Event Grid creates a webhook subscription, it sends a subscription validation event to verify the endpoint. The endpoint needs to respond with a specific validation response to confirm the webhook endpoint. However, it turns out you don't need to manually return the validation response for an Azure Function EventGridTrigger, as the Azure Functions runtime handles this automatically. We do still have to handle the event.

Now that I knew how to handle the incoming events, I was able to extract the Blob Storage URL from the event, load the payload from Blob Storage, then use the details to resolve to the downstream endpoint and invoke the endpoint with the body of the original request.

switch (systemEvent)
{
    case SubscriptionValidationEventData subscriptionValidated:
        _logger.LogInformation(
            "subscriptionValidated.ValidationCode: {code}",
            subscriptionValidated.ValidationCode);
        break;

    case StorageBlobCreatedEventData blobCreated:
        _logger.LogDebug(
            "blobCreated.Url: {url}",
            blobCreated.Url);

        var acceptedPayload =
            await _payloadStore.GetAcceptedPayloadAsync(blobCreated.Url);

        var endpointProxy =
            _endpointProxyFactory.GetEndpointProxy(
                acceptedPayload.TenantId, acceptedPayload.ContractId);

        await endpointProxy.InvokeAsync(acceptedPayload.Body);

        break;

    default:
        _logger.LogError(
            "Unhandled event type: {eventType}",
            eventGridEvent.EventType);
        break;
}

I was able to take advantage of the work I did in my Connecting to cloud Azure Blob Storage post, which introduced an abstraction over Blob Storage that decouples the code from whether using local or cloud storage and avoids any connection strings.

This allowed me to test my function locally, by amending the url in the event to point to a local storage URL.

{
  "<snip>": "...",
  "subject": "/devstoreaccount1/webhook-payloads-accepted/LovelyLoans/QuickValuationCo/2024-04-30/2024-04-30T18:13:03UTC-e2a8e023-079c-4ce5-ac0f-2021264f92fe.json",
  "data": {
    "<snip>": "...",
    "url": "http://127.0.0.1:10000/devstoreaccount1/webhook-payloads-accepted/LovelyLoans/QuickValuationCo/2024-04-30/2024-04-30T18:13:03UTC-e2a8e023-079c-4ce5-ac0f-2021264f92fe.json"
  }
}

Summary and next steps

In this post, I looked at the options available for Azure Functions to react to Blob Storage events. I chose the event-driven option over polling. I also had to choose between Event Grid events and Cloud Events. Both of these choices highlights how often in the evolving cloud world, the software architect is faced with such dilemmas. What was considered best practice one day, may well not be the next.

I was able to use to the Azure Portal to set up a subscription with a filter, have my Azure Function handling the events, and then forwarding on the original webhook calls. I was also able to test my function locally, after working out that the Visual Studio boilerplate code had mislead me.

To simplify the post, I did skip over was error handling and the deduplication of resent events. Both of these are critical to any code approaching production-level quality. I intend to return to these aspects in later posts.

With this in mind, I wondered if I could create a client class that is a hybrid of the two. An example usage is shown below.

var client =
    await PetstoreHybridOpenApiClient.CreateAsync(
        new Uri("http://petstore.swagger.io"));

await client.AddPetAsync(new Pet { Name = "Luna" });

Pet pet = await client.GetPetByIdAsync(1);

ICollection<Pet> pets = await client.FindPetsByStatusAsync([Anonymous.Sold]);

As an additional challenge, I decided to try implementing it all without resorting to inheritance. As someone who has used object orientation for a long time, my first thought is often to start creating subclasses. However, I am aware of the option to use composition over inheritance. So I thought it would be interesting to try that and see how it felt.

Creating the constructors and factory methods

The dynamic client class that I developed has a factory method that is used as follows.

var client = await OpenApiClientV2.CreateAsync(openApiJson, domainUri);

This relies on the calling code reading the OpenAPI document into a JSON string and passing it to the method. For the hybrid client, I decided that I would use a convention instead. The OpenAPI document would be an embedded resource, with the same name as the client class, but with a .OpenAPI.json extension.

This meant that the factory method calling code would be simplified, with just the domain URI being required.

var client =
    await PetstoreHybridOpenApiClient.CreateAsync(
        new Uri("http://petstore.swagger.io"));

With this usage in mind, I created the skeleton for the PetstoreHybridClient class, wrapping an OpenApiClientV2 instance in a 'has-a' relationship.

public class PetstoreHybridClient
{
    private readonly OpenApiClientV2 _client;

    private PetstoreHybridClient(OpenApiClientV2 client)
    {
        _client = client;

    public static async Task<PetstoreHybridClient> CreateAsync(Uri domainUri)
    {
        // Instantiate an instance of PetstoreHybridClient...
    }
}

The next task was for the factory method to create and return an instance. To do this, it delegates to a method on a new static class HybridOpenApiClient. The method takes the domain and a function that receives an OpenApiClientV2 instance and instantiates a new class. In this case, the PetstoreHybridClient class.

public static async Task<PetstoreHybridClient> CreateAsync(Uri domainUri)
{
    return
        await HybridOpenApiClient.CreateAsync(
            domainUri, (client) => new PetstoreHybridClient(client));
}

The HybridOpenApiClient class was then implemented as follows. Note that, the createHybridClient function would not be necessary if C# type constraints could specify anything other that a default constructor.

public static class HybridOpenApiClient
{
    public static async Task<T> CreateAsync<T>(
        Uri domainUri, Func<OpenApiClientV2, T> createHybridClient) where T : class
    {
        var openApiJson = LoadOpenApiJsonForType<T>();

        var client = await OpenApiClientV2.CreateAsync(openApiJson, domainUri);

        return createHybridClient(client);
    }
}

I have omitted the details for the LoadOpenApiJsonForType, but the full code can be found on the accompanying GitHub repo.

Adding the API methods

public async Task AddPetAsync(
    Pet body)
{
    await _client.PerformAsync(
        "addPet",
        [
            ("body", JsonConvert.Serialize(body)),
        ]);
}

In the previous post in the series, I used Visual Studio to generate a client and a set of model classes. These model classes are annotated with attributes that implement the validation in the OpenAPI document. For example, the Name property on the Pet class is annotated as follows.

public partial class Pet
{
    // <snip>

    [Newtonsoft.Json.JsonProperty("name", Required = Newtonsoft.Json.Required.Always)]
    [System.ComponentModel.DataAnnotations.Required(AllowEmptyStrings = true)]
    public string Name { get; set; }

    // <snip>
}

The problem with this was that the JsonConvert.Serialize(body) results in a JsonSerializationException, which I didn't want. I wanted all failures to be handled the same. However, on the other hand, I wanted to use the generated classes.

The solution was to create a pair of serialization helpers as follows:

private static readonly JsonSerializerSettings _serializerSettings =
    new()
    {
        MissingMemberHandling = MissingMemberHandling.Ignore,
        NullValueHandling = NullValueHandling.Ignore,
        Converters = [new StringEnumConverter()],
    };

public static string Serialize(object value)
{
    if (value == null) return null;

    var valueJson = JsonConvert.SerializeObject(value, _serializerSettings);

    var isJsonString = valueJson.StartsWith("\"");
    if (isJsonString) return valueJson.Trim('"');

    return valueJson;
}

public static T Deserialize<T>(JsonResponse response)
{
    return JsonConvert.DeserializeObject<T>(response.Payload, _serializerSettings);
}

These could then be used as follows in the hybrid client.

  public async Task<Pet> GetPetByIdAsync(
      long petId)
  {
      var response =
          await _client.PerformAsync(
              "getPetById",
              [
                  ("petId", HybridOpenApiClient.Serialize(petId)),
              ]);

      return HybridOpenApiClient.Deserialize<Pet>(response);
  }

Adding and overriding default behaviour

The OpenApiClientV2 class allows custom error handling. This is done by supplying an OnFailure function. For the hybrid client, I wanted the base implementation to throw an exception for any failure. To do this, I extended HybridOpenApiClient with the following code.

public static class HybridOpenApiClient
{
    public static async Task<T> CreateAsync<T>(
        Uri domainUri, Func<OpenApiClientV2, T> createHybridClient) where T : class
    {
        // <snip>

        client.OnFailure = OnFailure;

        return createHybridClient(client);
    }

    public static void OnFailure(
        string operationId,
        IEnumerable<(string, string)> parameters,
        JsonResponse response)
    {
        if (response.HttpStatusCode.HasValue)
        {
            throw new OpenApiException(
                $"{operationId} received {(int)response.HttpStatusCode}: " +
                    $"{string.Join(", ", response.FailureReasons)}",
                response.HttpStatusCode.Value);
        }

        throw new OpenApiException(
            $"{operationId} failed: " +
            $"{string.Join(", ", response.FailureReasons)}",
            response.Exception);
    }
}

So now, any failure results in a OpenApiException being thrown with a description of the failure reason. As an exercise, I wanted the specific implementation, PetstoreHybridClient, to also log the failure details.

This was possible by supplying a custom OnFailure implementation as follows.

private PetstoreHybridClient(OpenApiClientV2 client)
{
    _client = client;

    _client.OnFailure =
        (o, p, r) =>
            {
                LogFailure(o, r);
                HybridOpenApiClient.OnFailure(o, p, r);
            };
}

public static void LogFailure(string operationId, JsonResponse response)
{
    Console.WriteLine(
        $"{operationId} failed in {response.ElapsedMilliseconds}ms");
}

With this, the PetstoreHybridClient logs the failure, but retains the behaviour of throwing exceptions.

Comparison with a base client

For comparison, I also created an abstract class HybridOpenApiClientBase. Below is an abbreviated version of the resulting subclass.

public class PetstoreHybridClientSubclass : HybridOpenApiClientBase
{
    // Base overrides

    protected override void OnFailure(
        string operationId,
        IEnumerable<(string, string)> parameters,
        JsonResponse response)
    {
        LogFailure(operationId, response);
        base.OnFailure(operationId, parameters, response);
    }

    private static void LogFailure(string operationId, JsonResponse response)
    {
        // ... as before ...
    }

    // API methods as before...

    public async Task AddPetAsync(
        Pet body)
    {
        await Client.PerformAsync(
            "addPet",
            [
                ("body", Serialize(body)),
            ]);
    }
}

The result was less code than the composition-based version, as there was no need to implement a factory method. Another nice feature was that Visual Studio prompts for overrides, which provides guidance as to what can be customised.

Summary

It was an educational experience to explore creating the client class without using inheritance. In this case, I actually favoured the inheritance-based result. I guess, in this case, the relationship was more of an 'is-a' than a 'has-a'. What was interesting, was that the base class was built using the composition-based implementation. This does show the flexibility of composition.

What I definitely did learn, was that I should keep an open mind when designing classes and not immediately start thinking of class hierarchies. To start with composition, and then wait for the need for inheritance to emerge. As the Thoughtworks Composition vs. Inheritance: How to Choose? article says:

If you find that you are using a component to provide the vast majority of your functionality, creating forwarding methods on your class to call the component's methods, exposing the component's fields, etc., consider whether inheritance - for some or all of the desired behavior - might be more appropriate.

Which is echoed by the Code Maze article Composition vs Inheritance in C#:

So, we should use inheritance if:

• There is an "is-a" relationship between classes (X is a Y)
• The derived class can have all the functionality of the base class

For all other instances, the composition is the preferred choice.

Links and further reading

• Wikipedia - Composition over inheritance

  • Composition over inheritance (or composite reuse principle) in object-oriented programming (OOP) is the principle that classes should favor polymorphic behavior and code reuse by their composition (by containing instances of other classes that implement the desired functionality) over inheritance from a base or parent class.

  • To favor composition over inheritance is a design principle that gives the design higher flexibility. It is more natural to build business-domain classes out of various components than trying to find commonality between them and creating a family tree. In other words, it is better to compose what an object can do (has-a) than extend what it is (is-a).

  • One common drawback of using composition instead of inheritance is that methods being provided by individual components may have to be implemented in the derived type, even if they are only forwarding methods. In contrast, inheritance does not require all of the base class's methods to be re-implemented within the derived class.

    • C# provides default interface methods since version 8.0 which allows to define body to interface member.

• Thoughtworks - Composition vs. Inheritance: How to Choose?

  • Composition is fairly easy to understand - we can see composition in everyday life: a chair has legs, a wall is composed of bricks and mortar, and so on. While the definition of inheritance is simple, it can become a complicated, tangled thing when used unwisely. Inheritance is more of an abstraction that we can only talk about, not touch directly. Though it is possible to mimic inheritance using composition in many situations, it is often unwieldy to do so. The purpose of composition is obvious: make wholes out of parts. The purpose of inheritance is a bit more complex because inheritance serves two purposes, semantics and mechanics.

  • Inheritance captures semantics (meaning) in a classification hierarchy (a taxonomy), arranging concepts from generalized to specialized, grouping related concepts in subtrees, and so on. Inheritance captures mechanics by encoding the representation of the data (fields) and behavior (methods) of a class and making it available for reuse and augmentation in subclasses. Mechanically, the subclass will inherit the implementation of the superclass and thus also its interface.

• Code Maze - Composition vs Inheritance in C#

At the end of that post, I pondered if it would be possible to build on my experience to create a client that could be used as follows.

var petStoreClient =
    await OpenApiClient.CreateAsync(
        File.ReadAllText("petstore.swagger.json"), "https://petstore.swagger.io");

var getPetByIdResponse =
    await petStoreClient.PerformAsync("getPetById", [("petId", "0")]);

Now, there are many good options for creating static clients. According to Claude.ai, these include:

1. Swagger Codegen

   • An open-source tool that can generate client SDKs in various languages, including C#.

2. OpenAPI Generator

   • A fork of Swagger Codegen with additional features and improvements.

3. NSwag

   • A .NET/TypeScript toolchain for OpenAPI.

4. Kiota

   • Microsoft's latest API client generator, designed to be lightweight and adaptable.

5. SwaggerHub

   • Offers code generation capabilities, including C# clients.

Claude.ai also reminded me that Visual Studio Connected Services is built into Visual Studio, and it can generate C# clients from OpenAPI specifications.

Although I would probably go down the static route for a production system, I was still intrigued by the idea of having a single class that I could configure dynamically configure with just the OpenAPI document. Given this I decided to press on.

How did I get on?

In short, I succeeded. The result is the OpenApiClientV2 class that can be found in GitHub here.

Example usage can be seen below:

var petStoreClient =
    await OpenApiClientV2.CreateAsync(
        File.ReadAllText("petstore.swagger.json"),
        new Uri("https://petstore.swagger.io"));

var getPetByIdResponse =
    await petStoreClient.PerformAsync("getPetById", [("petId", "2")]);

if (getPetByIdResponse.IsSuccessful)
    Console.WriteLine(getPetByIdResponse.Payload);

In addition to using NJsonSchema to validate the request bodies, I used the popular RestSharp package to make the HTTP calls. The main routine panned out as follows.

private async Task<JsonResponse> PerformClientOperationAsync(
    ClientOperation clientOperation,
    IEnumerable<(string, string)> parameters)
{
    var restRequest =
        new RestRequest(
            clientOperation.Path, GetMethod(clientOperation.OperationType));

    var parameterErrors = new List<string>();

    SetNonBodyParameters(clientOperation, parameters, restRequest, parameterErrors);

    SetBodyParameter(clientOperation, parameters, restRequest, parameterErrors);

    if (parameterErrors.Count > 0)
    {
        return GetJsonResponse(parameterErrors);
    }

    var restResponse = await _restClient.ExecuteAsync(restRequest);

    var jsonResponse = GetJsonResponse(clientOperation, restResponse);

    return jsonResponse;
}

By the time this method is called, the OpenAPI document had been pre-processed. In the factory method, the OpenAPI document is read, checked for errors, then turned into a dictionary of ClientOperation instances. The ClientOperation class encapsulates the details for a particular operation, as defined in the OpenAPI document.

public static async Task<OpenApiClientV2> CreateAsync(string openApiJson, Uri domainUri)
{
    using var openApiJsonStream =
        new MemoryStream(Encoding.UTF8.GetBytes(openApiJson));

    var openApiDocument =
        new OpenApiStreamReader().Read(openApiJsonStream, out var openApiDiagnostic);

    AssertNoOpenApiErrors(openApiDiagnostic);

    var clientOperations = await BuildClientOperationsAsync(openApiDocument);

    return new OpenApiClientV2(clientOperations, baseUri);
}

This processing allows an efficient use of the operation details to validate the body and non-body parameters. For example, the JSON schemas for request bodies are generated at the point, to be reused as long as the client instance is held.

The RestSharp package greatly simplified the client development. In particular, the AddUrlSegment method allowed me to set the request parameters without having to worry about any string parsing or encoding.

private static void AddPathParameter(
    OpenApiParameter openApiParameter,
    IEnumerable<string> parameterValues,
    RestRequest restRequest,
    List<string> parameterErrors)
{
    if (parameterValues.Count() > 1)
    {
        parameterErrors.Add(
            $"{openApiParameter.Name} path parameter has multiple values");
        return;
    }

    restRequest.AddUrlSegment(openApiParameter.Name, parameterValues.First());
}

It was a similar situation for the other types of parameters. I could write my code without worrying about encodings, so keeping it nice and clean.

restRequest.AddQueryParameter(openApiParameter.Name, parameterValue);
restRequest.AddHeader(openApiParameter.Name, parameterValues.First());

I do confess to only going so far with validating the non-body parameters. My code does check string lengths and apply the supplied regular expression, if available. However, I did not implement numerical limit checks or support for mixed types as mentioned in the Swagger data types specification.

I also left placeholders for extension points. These would allow customisation of the headers supplied for each call. The idea being that this would allow the appropriate authorisation headers to be set for each call.

One thing that did become apparent during development, was that the OpenApiDocument implementation does not contain all possible OpenAPI properties. For example, the basePath property is specified in the Petstore example:

However, when inspecting the OpenApiDocument instance, it was nowhere to be seen:

As a result, I had to add a SelectBasePath method that parsed the OpenAPI document JSON and extracted the value.

public static async Task<OpenApiClientV2> CreateAsync(string openApiJson, Uri domainUri)
{
    // <snip>

    var basePath = SelectBasePath(openApiJson); // basePath not in OpenApiDocument
    var baseUri = new Uri(domainUri, basePath);

    return new OpenApiClientV2(clientOperations, baseUri);
}

This wasn't a big deal, but is something to be aware of if you are using OpenApiDocument. Another example is collectionFormat, which specifies how a collection of parameters is packaged.

Overall, I was quite pleased with the final result and felt it had quite a bit of promise.

Comparing with a statically-generated client

I thought it would be interesting to compare my dynamic client with a statically-generated client. To do this, I thought I would use the built in functionality in Visual Studio.

This is done by right-clicking on a project and adding a connected service.

The next step is to select the type of connected service. This will depending on the type of your Visual Studio project. I was working with a .NET Framework project, so I only got the option for OpenAPI.

For .NET Core projects, I believe you get the option of gRPC and perhaps others.

Adding an OpenAPI service is as simple as pointing the wizard to the OpenAPI document and providing the namespace, class name, and language of your choice.

After the wizard runs, we see a single .cs file containing multiple classes for the API client, the API models, and other sundries.

The wizard adds a number of NuGet packages, but still doesn't compile. For some reason, it fails to add the System.ComponentModel.DataAnnotations package.

Adding this package was all that was required to get the code compiling and ready for use, an example of which is shown below.

HttpClient httpClient = new();

PetstoreClient petstoreClient = new(httpClient);

Pet getPetByIdResponse = await petstoreClient.GetPetByIdAsync(2);

I am generally a fan of strong-typing, so this usage does appeal to me.

Comparing and contrasting the two approaches

Success Flag vs Exceptions

The dynamic client catches all failures and returns an envelope class with a IsSuccessful flag. This includes all parameter validation errors, non-success HTTP status codes, and any exceptions. This provides consistency for the calling code, making the code cleaner.

Depending of the failure mode, the static client throws a variety of exceptions. For example, if you supply an invalid request body then you get a JsonSerializationException thrown. If the domain is incorrect, you get a WebException, and if you get a non-success HTTP status code then an ApiException is thrown. There may be others that I did not find. This does mean that the calling code has to be aware of all of these, if it wants to make the most of them when handling them.

I do like consistency, so here I favoured the approach taken by my dynamic client.

Runtime checking vs compile-time parameter checking

One clear difference between the two approaches is in the type checking. The dynamic client fits the scenario I had, where the calling code was generating JSON. However, in general, I would favour leaning on the compiler to verify types wherever I can. With this in mind, as a user, I would prefer the static client.

Code ownership

It was interesting to have a look at the generated code. Below is a snippet from one of the implemented operations. As you can see, there is quite a bit of code and this is largely repeated in each method.

I have underlined in green the handling of non-success HTTP status codes, which - as mentioned above - results in ApiException instances being thrown.

I have also underlined, this time in red, some of the extension points that are available to you. The client is generated as a partial class. This allows you to create your own partial class to provides your own custom implementations for these methods. This allows the generated client class to be regenerated at any time and also avoids using inheritance to provide the extension points.

What struck me about the generate code, was that there was quite a bit of it, and I would have to own it all if it was part of my project. I am not sure whether I would be overly comfortable with that. On the other hand, the dynamic client has much less code. Once the single class has been tested thoroughly, I would feel happier to use that rather than lots of generated code.

Summary

The combination of the Microsoft.OpenApi, NJsonSchema, and RestSharp packages made is straightforward to implement my vision of a dynamic client that could be used against any REST endpoint with a supporting OpenAPI document. Using this client would provide a consistent way for a codebase to interact with these services.

The comparison with a static client highlighted that you have to own the generated code and its inconsistencies. In the case shown, the exception throwing. However, there is definitely advantages to having strong typing for compile-time checking. With this in mind, I wondered if I could make a hybrid client. Something that could use the automatically-generated models, but would use the dynamic client internally and would have a usage as follows:

var client =
    await PetstoreHybridOpenApiClient.CreateAsync(
        new Uri("http://petstore.swagger.io"));

await client.AddPetAsync(new Pet { Name = "Luna" });

Pet pet = await client.GetPetByIdAsync(2999);

ICollection<Pet> pets = await client.FindPetsByStatusAsync([Anonymous.Sold]);

One for another post perhaps.

The specification includes definitions of the objects used by the API. In the Petstore example, the method to add a pet has a payload that takes a Pet object as a its payload.

Pet:
  required:
    - name
    - photoUrls
  type: object
  properties:
    id:
      type: integer
      format: int64
    name:
      type: string
    category:
      $ref: "#/components/schemas/Category"
    photoUrls:
      type: array
      items:
        type: string

What I wanted to do, was use this information to validate the JSON being generated by the internal system before it was sent to the external API. The question was how?

The road to the Microsoft.OpenApi library

My first thought was to manually parse the OpenAPI specification. So I asked Claude.ai:

Please generate a C# method that takes a Swagger definition in JSON and an operation id and returns a JSON schema for the operation request body.

Claude obliged, but the resulting code was far from promising.

using JsonDocument doc = JsonDocument.Parse(swaggerJson);
  JsonElement root = doc.RootElement;

  // Iterate through paths
  foreach (JsonProperty pathProp in root.GetProperty("paths").EnumerateObject())
  {
      foreach (JsonProperty methodProp in pathProp.Value.EnumerateObject())
      {
          if (methodProp.Value.TryGetProperty("operationId", out JsonElement opIdElement)
              && opIdElement.GetString() == operationId)
          {
              // Found the operation, now extract the request body schema
              if (methodProp.Value.TryGetProperty("requestBody", out JsonElement requestBody)
                  && requestBody.TryGetProperty("content", out JsonElement content))
              {
                  foreach (JsonProperty contentType in content.EnumerateObject())
                  {
                      if (contentType.Value.TryGetProperty("schema", out JsonElement schema))
                      {

It occurred to me that my prompt was not high-level enough. I was assuming a certain solution. So instead, I tried the following request.

I am using .NET Framework and C#. I have a Swagger definition for an API. How can I extract the JSON Schema for the request and response body for each operation in the definition?

And the response led me to the Microsoft.OpenApi library.

To extract the JSON Schema for request and response bodies from a Swagger definition in a .NET Framework and C# environment, you can use the Microsoft.OpenApi library. This library provides tools to parse and manipulate OpenAPI (formerly known as Swagger) documents.

As I could see from the generated code, this looked far more like it.

var openApiDocument =
    new OpenApiStreamReader().Read(
        File.OpenRead(swaggerFilePath), out var diagnostic);

foreach (var path in openApiDocument.Paths)
{
    foreach (var operation in path.Value.Operations)
    {
        var operationType = operation.Key.ToString();
        var operationId = operation.Value.OperationId;

        // Extract request body schema
        if (operation.Value.RequestBody != null &&
            operation.Value.RequestBody.Content.TryGetValue(
                "application/json", out var requestMediaType))
        {
            var requestSchema = requestMediaType.Schema;

Getting Claude.ai's code to work

This was great, but it only had one flaw. It didn't work. When I tried running the code I got the following exception.

Newtonsoft.Json.JsonSerializationException
  HResult=0x80131500
  Message=Self referencing loop detected for property 'HostDocument' with type 'Microsoft.OpenApi.Models.OpenApiDocument'. Path 'Properties.category.Reference.HostDocument.Paths['/pet/{petId}/uploadImage'].Operations.Post.Tags[0].Reference'.
  Source=Newtonsoft.Json

The exceptioin occurred on the following line.

var schemaData = JsonConvert.SerializeObject(openApiSchema);

Claude.ai had predicted that something called OpenApiSchema was a serializable JSON schema. It turns out that it isn't, so I had to go back to some old-fashioned searching on StackOverflow. My search turned up the following question, Get the JSON Schema's from a large OpenAPI Document OR using NewtonSoft and resolve refs. The accepted answer had the following code (slightly abbreviated for this post).

var reader = new OpenApiStreamReader();
var result =
    await reader.ReadAsync(new FileStream(file.FullName, FileMode.Open));

foreach (var schemaEntry in result.OpenApiDocument.Components.Schemas)
{
    var schemaFileName = schemaEntry.Key + ".json";
    var outputPath =
        Path.Combine(outputDir, schemaFileName + "-Schema.json");

    using var fileStream = new FileStream(outputPath, FileMode.CreateNew);
    using var writer = new StreamWriter(fileStream);

    var writerSettings =
        new OpenApiWriterSettings()
        {
            InlineLocalReferences = true,
            InlineExternalReferences = true
        };

    schemaEntry.Value
        .SerializeAsV2WithoutReference(
            new OpenApiJsonWriter(writer, writerSettings));
}

Sure enough, when I ran this code against the Petstore OpenAPI document, it successfully created the following schema files:

Opening up User.json-Schema.json, I could see that the contents looked to my eye like a valid JSON schema.

{
  "type": "object",
  "properties": {
    "id": {
      "format": "int64",
      "type": "integer"
    },
    "petId": {
      "format": "int64",
      "type": "integer"
    },
    "quantity": {
      "format": "int32",
      "type": "integer"
    },
    "shipDate": {
      "format": "date-time",
      "type": "string"
    },
    "status": {
      "description": "Order Status",
      "enum": ["placed", "approved", "delivered"],
      "type": "string"
    },
    "complete": {
      "type": "boolean"
    }
  },
  "xml": {
    "name": "Order"
  }
}

The key here was the use of the SerializeAsV2WithoutReference method with the OpenApiJsonWriter and OpenApiWriterSettings. Together they control how an OpenApiSchema instance is serialized. In this case, we want to inline all references to get a self-contained schema.

var writerSettings =
    new OpenApiWriterSettings()
    {
        InlineLocalReferences = true,
        InlineExternalReferences = true
    };

schemaEntry.Value
    .SerializeAsV2WithoutReference(
        new OpenApiJsonWriter(writer, writerSettings));

Now I had a way of extracting the schemas, I could move on to my ultimate aim of using the schemas to validate my dynamically-created requests.

Creating my OpenApiSchemaValidator

As I mentioned in my Designing a CDK State Machine Builder post, I like to build software using an API-first approach. That is, before jumping into the component implementation, I imagine it exists and write the code the calling code. This allows me to quickly iterate over the external API until it feels it has the right level of expression and abstraction.

With this in mind, I imagined a OpenApiSchemaValidator class and then iterated on the calling code until I got the following.

var validator =
    new OpenApiSchemaValidator(
        new FileStream("petstore.swagger.json", FileMode.Open));

var validationResult =
    validator.ValidateRequestBodyJson(
        operationId: "addPet", bodyJson: petJson);

if (validationResult.IsValid)
    Console.WriteLine("Request JSON is valid");
else
    Console.WriteLine(
        $"Request JSON had the following errors: \n- " +
        $"{string.Join("\n- ", validationResult.Errors)}");

I decided to split the creation of OpenApiSchemaValidator from the method call. This would allow the creation of the instance to process the OpenAPI document and avoid this static overhead on each validation call.

However, when I dived int the implementation, I had to make a small change. I was using the NJsonSchema for .NET NuGet package to validate the JSON requests against the JSON schemas. It turned out that the method to parse the JSON schema was async. Since I could not have this in a constructor, I had to have an async factory method instead.

var validator =
    await OpenApiSchemaValidator.CreateAsync(
        new FileStream("petstore.swagger.json", FileMode.Open));

The implementation is shown below.

private readonly IReadOnlyDictionary<string, JsonSchema> _requestBodyJsonSchemas;

private OpenApiSchemaValidator(IDictionary<string, JsonSchema> jsonSchemas) =>
    _requestBodyJsonSchemas = new Dictionary<string, JsonSchema>(jsonSchemas);

public static async Task<OpenApiSchemaValidator> CreateAsync(Stream openApiStream)
{
    // Get the operations with JSON request bodies

    var openApiDocument =
        new OpenApiStreamReader().Read(openApiStream, out _);

    var requestBodyOperations =
        openApiDocument.Paths
            .SelectMany(p => p.Value.Operations)
            .Where(o =>
                o.Value.RequestBody != null &&
                o.Value.RequestBody.Content.ContainsKey("application/json"));

    // Convert the OpenAPI schemas to JSON schemas and index by Operation Id

    var jsonSchemas = new Dictionary<string, JsonSchema>();

    foreach (var operation in requestBodyOperations)
    {
        var openApiSchema =
            operation.Value.RequestBody.Content["application/json"].Schema;
        var jsonSchema =
            await JsonSchema.FromJsonAsync(SerializeToJsonSchema(openApiSchema));
        jsonSchemas.Add(operation.Value.OperationId, jsonSchema);
    }

    return new(jsonSchemas);
}

The SerializeToJsonSchema method used the SerializeAsV2WithoutReference method as discussed earlier.

With the dictionary of schemas in place, applying them was straightforward.

public OpenApiSchemaValidationResult ValidateRequestBodyJson(
    string operationId,
    string bodyJson)
{
    if (_requestBodyJsonSchemas.TryGetValue(operationId, out var jsonSchema))
    {
        // Validate the JSON against the schema

        var errors = jsonSchema.Validate(JToken.Parse(bodyJson));

        return
            new OpenApiSchemaValidationResult
            {
                IsValid = errors.Count == 0,
                Errors = errors.Select(e => $"{e.Path}: {e.Kind}")
            };
    }

    throw new ArgumentException(
        $"Operation does not have a JSON request body: {operationId}",
        nameof(operationId));
}

And with that in place, I could test the validation with an empty JSON object.

var validationResult =
    validator.ValidateRequestBodyJson(operationId: "addPet", bodyJson: "{}");

Which resulted in the following output to be written to the console, successfully reporting that the JSON request had two missing properties.

Request JSON had the following errors:
- #/name: PropertyRequired
- #/photoUrls: PropertyRequired

Summary

Thanks to Claude.ai and StackOverflow, I was able to implement the functionality I wanted, in a class only 95 lines long. The key piece of information was the existence of the Microsoft.OpenApi namespace. This did all the heavy lifting of reading the OpenApi document and outputting the request schemas as JSON schemas. From there, it was straightforward to use NJsonSchema to do the validation. The resulting OpenApiSchemaValidator class can be found on GitHub here.

Now that I had a way of reading the OpenAPI document, I realised that I could do more with it than just validating request bodies. An OpenAPI document includes the paths, parameters, and more for each endpoint. I asked myself, would it be possible to build a dynamic OpenAPI client that I could use as follows?

var petStoreClient =
    await OpenApiClient.CreateAsync(
        new FileStream("petstore.swagger.json", FileMode.Open),
        "https://petstore.swagger.io/v2");

var getPetByIdResponse =
    await petStoreClient.PerformAsync("getPetById", [("petId", "0")]);

One for another post I think.

Connection strings and account keys are evil

'Evil' is perhaps a strong word, but both are risky as they can easily be leaked. I have seen too many credentials shared in plain text in emails or chats. There have also been too many stories of credentials being accidentally checked into source control. So, what is the best way to handle them? The answer is, not to have them in the first place.

I am familiar with the AWS serverless offerings, Lambda functions, DynamoDB, SQS, and so forth. In all these cases, I have never had to use a connection string or an account key. In AWS, every component needs to be granted access to the resources that it needs interact with. This is done through the AWS IAM (Identity and Access Management). This avoids the use of explicit credentials and the risk associated with them. It turns out that Azure is going the same way, but first we need to create our Blob Storage containers.

Creating the storage account

In future, I want to deploy all the cloud resources via infrastructure as code (IaC) using Bicep. However, for the first pass I used click-ops and the Azure portal. From within my project resource group, I clicked to add a service and selected the 'Storage account' service from Microsoft.

Next, I specified the basic properties of the account. I provided the account name and changed the 'Redundancy' level to the cheapest option, but otherwise left the properties at their default values.

For the networking options, I left the access level at 'Enable public access from all networks'. I experimented with a more restrictive access level, as I wasn't comfortable with the public access. However, I was not able to access the account unless this level was selected. At this point in my journey, I did not want to start creating private networks. However, for a production system, this is the route I would go.

Here was a significant difference to my experience with AWS, where the default for S3 buckets is to prevent public access. You can then use AWS IAM to grant bucket access to the appropriate resources, such as Lambda functions, without having to create any private networking.

Although I had no intention of using them, I took a look at the 'Access keys' blade. Here, as you would expect, you can find the access keys and connection strings to access the account. You can also manually rotate the keys from this blade, which highlights another good reason to avoid using them.

Creating and configuring the containers

With the storage account in place, I clicked the option to add a container.

Then specified the name of the container and accepted the other defaults.

I then repeated this for the other container that I needed. The resulting list showed the two containers that I had created and one that seemed to have been created by Azure ($logs).

As these container are going to container personally identifiable information (PII), it is important not to hold on to the data for longer than is strictly necessary. This is where time-to-live (TTL) functionality comes in very handy. This functionality allows us to defer the responsibility of deleting old data to the cloud provider.

For Blob Storage, this is done by selecting 'Lifecycle management' blade, under 'Data management' in the Azure portal.

The first step was to give the rule a name and specify the blobs to which the rule applies. In my case, I wanted to delete all request blobs after 30 days. So I selected the rule scope to filter the blobs to which the rule applies.

The next section allowed me to specify an action to take and the condition under which to take it. In my case, I wanted to delete all blobs 30 days after creation.

The final step was to specify the filter set. In my case, I specified webhook-payload- to filter the rule to the request containers.

Now I had my containers in place, with rules to keep them managed and, although there was public access via keys, I had no intention of using them. This is because I intended to use managed identities.

Managed identities

The Microsoft Learn article What are managed identities for Azure resources? give a good overview of what managed identities are.

A common challenge for developers is the management of secrets, credentials, certificates, and keys used to secure communication between services. Managed identities eliminate the need for developers to manage these credentials.

While developers can securely store the secrets in Azure Key Vault, services need a way to access Azure Key Vault. Managed identities provide an automatically managed identity in Microsoft Entra ID for applications to use when connecting to resources that support Microsoft Entra authentication. Applications can use managed identities to obtain Microsoft Entra tokens without having to manage any credentials.

It goes on to list some of echo some the benefits of using managed identities that I mentioned earlier.

• You don't need to manage credentials. Credentials aren’t even accessible to you.
• You can use managed identities to authenticate to any resource that supports Microsoft Entra authentication, including your own applications.
• Managed identities can be used at no extra cost.

As I also mentioned, in AWS this is the default. In fact, it is the only option in order to access some services. It is good to see Azure following down this path.

The article goes on to mention that there are two types of managed identity, system-assigned and user-assigned. In my case, I want a system-managed identity for my Azure Function.

Assigning a managed identity to a function app using the Azure portal was a simple task. I selected the function app and the 'Identity' blade. Then all that was required was to switch the status to 'On'.

To grant access to the storage account, I needed to open the storage account and select the 'Access Control' blade. I then selected 'Add role assignment' from the 'Add' menu.

The next step was to select the role that I wished to give my function app. I filtered the list to those with 'blob' in them and was given the list shown below. As the function app needs to write to the storage account, I chose 'Storage Blob Data Contributor' and to assign it to a managed identity.

The next step was to select my function app as a member of this role. The portal gives a dropdown list of managed identity types, so I selected 'Function App' and my function app from the resulting list.

I then took the defaults and completed the wizard. The resulting list of role assignment confirmed that the function app identity now had access to the storage account.

Now all that was left was to update the code to use the managed identity to connect to the storage account.

Using token credentials to connect

The current code was connecting to the local Azurite storage emulator using a hardcoded connection string. I wanted to keep this behaviour for local testing, but also wanted to be able to connect to cloud storage when running in the cloud.

I tried in vain to use the DefaultAzureCredential class to seamlessly change the access mechanism depending on environment. However, despite my best efforts, I was not able to get it to work. Instead, I decided to hide the logic behind a new IBlobServiceClientFactory interface implementation.

This simple interface had a single method for creating a BlobServiceClient.

public interface IBlobServiceClientFactory
{
    BlobServiceClient CreateClient();
}

The implementation is also quite simple. It checks an environment variable to see if the code is being run in an environment with a storage emulator. If so, the hardcoded connection string is used. If not, then a storage URI and ManagedIdentityCredential instance are used.

public class BlobServiceClientFactory : IBlobServiceClientFactory
{
    private readonly TokenCredential _tokenCredential;

    public BlobServiceClientFactory()
    {
        _tokenCredential = new ManagedIdentityCredential();
    }

    public BlobServiceClient CreateClient()
    {
        BlobServiceClient blobServiceClient;

        if (Environment.GetEnvironmentVariable(
            "AZURE_STORAGE_EMULATOR_RUNNING") == "true")
        {
            // Use connection string for Azurite
            string connectionString = "UseDevelopmentStorage=true";
            blobServiceClient = new BlobServiceClient(connectionString);
        }
        else
        {
            // Use TokenCredential for Azure Storage
            var webhookStorageAccount =
                Environment.GetEnvironmentVariable("WEBHOOK_STORAGE_ACCOUNT");
            var blobServiceUri =
                new Uri($"https://{webhookStorageAccount}.blob.core.windows.net");
            blobServiceClient =
                new BlobServiceClient(blobServiceUri, _tokenCredential);
        }

        return blobServiceClient;
    }
}

I then added this to the dependency injection configuration.

.ConfigureServices(services =>
{
   // <snip>
   services.AddSingleton<IBlobServiceClientFactory, BlobServiceClientFactory>();
})

Finally, I updated the BlobPayloadStore to have an instance injected and to use this instance to create an appropriate client.

private readonly BlobServiceClient _blobServiceClient;

public BlobPayloadStore(IBlobServiceClientFactory blobServiceClientFactory)
{
   _blobServiceClient = blobServiceClientFactory.CreateClient();
}

A special mention for DefaultAzureCredential

As mentioned in the previous section. I tried to use the DefaultAzureCredential class. This was because the documentation states:

The DefaultAzureCredential class provided by the Azure SDK allows apps to use different authentication methods depending on the environment they're run in. This allows apps to be promoted from local development to test environments to production without code changes. You configure the appropriate authentication method for each environment and DefaultAzureCredential will automatically detect and use that authentication method. The use of DefaultAzureCredential should be preferred over manually coding conditional logic or feature flags to use different authentication methods in different environments.

This sounded exactly what I was after, but I could not find a way to get it to connect to the local emulator or the cloud storage. In the end, I had to resort to my own logic and using the ManagedIdentityCredential class explicitly. I am quite happy with this choice, as this means my function app can only use managed identities, which is the intention.

However, you may have different intentions and so it is definitely worth consideration. For more details, see the following excellent blog post: DefaultAzureCredentials Under the Hood

Using Log Stream for quick feedback

When developing and debugging my function in Azure, I found the Log Stream functionality in Azure very useful. As mentioned in an earlier post, you can attach to functions running in Azure and debug them that way. However, I find judicious log statements can be almost as good.

As shown below, the Log Stream blade allows an almost real-time view of the log from your function. You can filter by level, copy the output, and jump into the raw logs if you really need to.

Summary

In this post, I managed (no pun intended) to achieve my desired objective of updating my Azure Function, so that it could access cloud Blob Storage without the use of connection strings or API keys. The way this was done was by assigning my Azure Function a managed identity and ensuring that identity had the appropriate role.

Now that we have data going into the Blob Storage, the next step is to react to the resulting events and take action based on them.

Links

• Authorize access and connect to Blob Storage
• Manage storage account access keys
• Authorize access to data in Azure Storage
• How to authenticate .NET apps to Azure services using the .NET Azure SDK
• Use the Azurite emulator for local Azure Storage development
• DefaultAzureCredentials Under the Hood
• Quickstart: Azure Blob Storage client library for .NET

A quick overview of Azure Blob Storage

Blob Storage is the Azure service that is equivalent to AWS S3, in that it is used for storing large amounts of unstructured data, such as text or binary data.

Blob Storage is composed of the following resource hierarchy:

• Storage Account: The top-level resource, providing a globally-unique namespace in Azure for your storage data.
• Container: Provides a grouping of a set of blobs.
• Blob: The fundamental storage entity in Azure Blob Storage. There are three types of blobs:
  • Block Blobs: Used for storing text or binary files, like documents, media files, etc.
  • Append Blobs: Optimized for append operations, making them ideal for scenarios like logging.
  • Page Blobs: Designed for frequent read/write operations.

Access to blobs and containers is controlled through:

• Access Keys: Storage account keys that give full privileges to the storage account.
• Shared Access Signatures (SAS): Provides restricted access rights to containers and blobs, with a defined start time, expiry time, and permissions.
• Azure Active Directory (Azure AD): For RBAC (role-based access control) to manage and control access.

There are other aspects to Blob Storage, such as:

• Access Tiers: To store data based on the frequency of access, such as 'Hot', 'Cool', and 'Archive'.
• Lifecycle Management: Automating the process of moving blobs to cooler storage tiers or deleting old blobs that are no longer needed.
• Security: Options regarding encryption at rest and in transit.
• Redundancy: Options such as 'Locally Redundant Storage (LRS)', 'Zone-Redundant Storage (ZRS)', and 'Geo-Redundant Storage (GRS)'.

Creating our containers

Visual Studio 2022 ships with the Azurite emulator for local Azure Storage development, so I already had it installed. However, if you're running an earlier version of Visual Studio, you can install Azurite by using either Node Package Manager (npm), DockerHub, or by cloning the Azurite GitHub repository.

Another useful tool is the Azure Storage Explorer. This allows you to upload, download, and manage Azure Storage blobs, files, queues, and tables as well as configuring storage permissions and access controls, tiers, and rules.

One thing I did note was that to start the emulator, I needed to run the function app using F5. This started the Azurite background process, which then showed up in the storage explorer as follows:

If I didn't do this, the storage explorer would tell me to install Azurite.

With the emulator up and running, the next step was to create some containers for the payloads. I had decided to split the payloads into two containers, one for payloads that had passed validation (webhook-payloads-accepted) and were accepted for further processing and another for payloads that were rejected (webhook-payloads-rejected).

Storing the payloads

Before storing the payloads in the new containers, I needed to decide how they should be stored. Blob Storage is essentially a flat namespace, which means it doesn't have real directories or folders. However, it does support a folder-like structure using naming conventions and delimiters, typically the forward slash (/), within blob names.

Putting myself in the place of someone fielding a support query, I imagined being told that a tenant was expecting a payload from a particular third party on a specific day. With this hierarchy in mind, I decided upon the following 'folder' structure.

private static string GetBlobName(
    string tenantId,
    string senderId,
    string messageId)
{
    var blobName =
        $"{tenantId}/{senderId}/{DateTime.UtcNow:yyyy-MM-dd}/{messageId}.json";
    return blobName;
}

The value of messageId is globally unique and will be passed back to the caller in a custom header. This adds another possible route for debugging calls between the systems.

Now we know how we are going to store the payloads, we need to use the SDK to store them. First of all we need a BlobServiceClient instance. It can be good practice to avoid over-instantiation of SDK clients. ChatGPT seemed to think that reusing the same instance is recommended in official Azure SDK documentation to improve performance and resource utilization. For production, I would double-check this, but for now that is good enough for me and so I stored the client at a module level.

private readonly BlobServiceClient _blobServiceClient;

public BlobPayloadStore(ILoggerFactory loggerFactory)
{
    _logger = loggerFactory.CreateLogger<BlobPayloadStore>();
    _blobServiceClient = new BlobServiceClient("UseDevelopmentStorage=true");
}

BlobPayloadStore was registered as a singleton with the dependency injection, so there was no need to have any statics involved. I.e., in Program.cs:

.ConfigureServices(services =>
{
    // <snip>
    services.AddSingleton<IPayloadStore, BlobPayloadStore>();
})

This just left me the task of writing the code to upload the payloads to the appropriate containers. The result is as follows.

private async Task UploadPayloadAsync<T>(
    string containerName,
    string tenantId,
    string senderId,
    string contractId,
    string messageId,
    T payload) where T : PayloadBase
{
    string payloadJsonString = JsonConvert.SerializeObject(payload, Formatting.Indented);

    var blobServiceClient = GetBlobServiceClient();
    var containerClient = blobServiceClient.GetBlobContainerClient(containerName);

    var blobName = GetBlobName(tenantId, senderId, contractId, messageId);
    var blobClient = containerClient.GetBlobClient(blobName);

    var byteArray = Encoding.UTF8.GetBytes(payloadJsonString);
    using var stream = new MemoryStream(byteArray);

    await blobClient.UploadAsync(stream, overwrite: true);
}

Local testing

After hitting F5 to run the function, I submitted a valid request to the local endpoint. Opening up Azure Storage Explorer, I could see a blob had been added as expected to the 'accepted' container.

The Azure Storage Explorer has a handy feature to preview the contents. Using this, I inspected the contents and could see that they contained the expected details. I did note that the API key value was not present, so must have been very sensibly filtered out.

I then ran a test with an invalid payload and, sure enough, a blob was added to the rejected container.

Previewing this, I could see that the errors had been passed through as expected.

Cloud considerations

The next step is to deploy to the cloud and test there. However, this raises a number of questions.

• How should the Blob containers be exposed?
  • Public vs. Private endpoints
• How should the Azure Function connect to the containers?
  • Connection string?
  • Managed identity?
• If using a connection string, how should it be obtained?
  • Environment variable?
  • Key vault?

Given these considerations, this feels like a post in itself. So I will leave it till next time.

Summary

In this post, I showed how I was able to use the local Azure development tools to implement and test storing the request payloads. However, to get the functionality deployed and working in the cloud will require some more thought and experimentation.

Why use API Management?

As we saw in the previous post, the Azure function is accessible from the public internet provided you know the appropriate API key. So why would you need a service such as API Management? A few reasons are listed below.

• Rate limiting, ensuring fair usage among consumers
• Subscription-level control, such as key rotation
• Advanced security, such as OAuth 2.0

API Management has many more features that we won't explore in this post, but include the following:

• Centralized management of APIs
• Customizable API facades
• API Documentation and Developer Portals
• API Analytics and Insights
• Caching Mechanisms
• Versioning and Revision Control

I find so much in software design is down to positioning. By using a service such as API Management, we will position ourselves to provide production-level API.

Are there any alternatives?

As with other cloud providers, Azure offers services that somewhat overlap in what they offer. In this case, ChatGPT was able to offer the following options amongst some others:

Azure Application Gateway with Web Application Firewall (WAF)

Use Case: If you're primarily looking for API gateway capabilities with security features like a Web Application Firewall, SSL termination, and URL-based routing.

• Why Use Over APIM: Offers Layer 7 load balancing with built-in WAF for security-focused scenarios, especially where protection against common web vulnerabilities and exploits is a priority.

Azure Functions Proxies

Use Case: For lightweight API orchestration or when you need a simple facade in front of multiple Azure Functions.

• Why Use Over APIM: It's a simple solution to create a single API surface for multiple microservices, particularly when these services are implemented using Azure Functions. However, it's less feature-rich compared to APIM.

Azure Front Door

Use Case: For global routing and load balancing needs, offering capabilities like URL-based routing, SSL termination, and global load balancing.

• Why Use Over APIM: It is more focused on content delivery, global routing, and ensuring high availability and performance for your web applications and APIs.

Given the API-focused nature of the application I am building, and the cost, API Management seems to be a good fit.

Creating the API Management instance

Using the Expose serverless APIs from HTTP endpoints using Azure API Management article as a guide, I opened up the Azure portal and navigated to my function app and selected the API Management blade.

This brought up the following option to create a new API Management instance.

Clicking on 'Create new' brought up the following UI. The Region and Resource name were defaulted and I filled in the other details. To save money, I chose the 'Developer' pricing tier noting that there is no Service Level Agreement (SLA) for this tier.

The next step gave the option to link the new instance with Application Insights and utilise 'Defender for Could' (sic). I enabled the former, but there wasn't an option to enable the latter. This may have been due to using the cheapest tier.

The final step was to choose the network connectivity. As this is to be a public endpoint, I chose 'None'.

After clicking 'Create' and waiting a while, the new API Management instance was ready to be connected to the function and expose the API.

Exposing the Azure function as an API

Now when I selected the API Management blade from the function app, it reported that it was now linked with the new instance. However, it had not imported anything automatically and presented me with the option to create a new API.

There was nothing promising in the API dropdown, so I selected 'Create New' and clicked on 'Link API'. I was then presented with the following list of Azure Functions.

Now this was looking more promising. The wizard appears to have recognised my Azure Function. I selected it and tried to progress. However, rather confusingly, I was then prompted to 'Define a new API' by selecting from a list of potential sources. This wasn't what I was expecting, but I selected 'Function App' and clicked to progress.

When I was presented with the next screen, I felt that I had somewhat gone round in a circle. It looked very similar to one earlier when my Azure Function was listed. However, I decided to click 'Select' and continue.

I was then presented with a list of Function Apps from which to import functions. I selected my Function App and clicked to progress.

The next step defaulted in a set of values, which all seemed reasonable to me. So I simply clicked 'Create'.

Once the creation was complete, I was able to view the 'Design' page for the new API and see the integration with the Azure Function in the backend. I could see how the process had recognised the request parameters and I also noticed the 'Inbound processing' box mentioned modifying the request. Also in the 'Inbound processing' box was the option to add policies. These, it turns out, are where you can do things such as filtering by IP address or rate limiting by key.

My eye was also caught by the 'Test' tab, so I clicked on it and gave it a go.

The response below showed that my Azure Function had been successfully called through the new API Management instance. So after quite a bit of clicking, it looks like I had managed to achieve my first aim.

I was a bit curious as how API Management was authorised to access my Azure Function. A bit of searching found this in the Authorization section of a Microsoft article:

Import of an Azure Function App automatically generates:

• Host key inside the Function App with the name apim-{your Azure API Management service instance name},
• Named value inside the Azure API Management instance with the name {your Azure Function App instance name}-key, which contains the created host key.

Sure enough, when I looked in the portal, I could see the following App key in my Function App.

And in the API Management instance, I found the same value as a 'Named value' with the expected name.

Ideally, I would like to used a Managed Identity and grant this identity access to the Azure Function. This method is more secure and manageable than function keys, but is only available to Premium Tier instances. So, for now, we will go with function keys.

Creating tenant-specific subscriptions

The Azure portal test had shown that the function was successfully integrated with API Management. The next test was to invoke it from outside of the Azure portal. As it happens, the Azure portal test UI has a feature that makes this easy. There is an option to copy the test HTTP request, which is highlighted below.

Using VS Code and the REST Client extension, you can paste the copied value into an .http file and you get the following.

So all I needed to do was find a suitable subscription key. I went into the API Management UI and the Subscriptions blade. This listed three subscription keys that appeared to have been created by default.

I tried the first two primary keys, but neither worked. So, with some reluctance, I tried to select the 'all-access' key. The result was this warning.

I decided to persevere and was able to successfully use the all-access key to make calls from VS Code. However, this was far from ideal and I wanted to work out how I would be able to issue and maintain separate keys for separate callers. So I clicked the option to create a new subscription.

This was more like it. I could restrict the access of the subscription to a specific API. There was also an option to trace, but for now I did not explore it. Once created, the new subscription appeared in the list as shown below.

I copied the primary key locally and re-ran my test from VS Code. As hoped, this was successful and now I had a way of issuing and rotating keys that were scoped to just a specific API.

There was one more thing that was bugging me though. It was still possible to call the Azure Function directly from the public internet. Although this would require the function key to be leaked, it was far from ideal. So I started to look into how I could restrict access and remove this possibility.

Disabling public access to the function

I went into the Networking blade for the Function App and clicked on the link next to 'Public network access'. This brought up the options below with the 'Enabled from all networks' selected. My first thought was to disable all public access. Maybe access from API Management doesn't count as public.

However, a quick test showed that this wasn't the case. Sure, direct access had been stopped, but so had access through API Management.

Engaging with ChatGPT, there was mention of virtual networks and managed identities. The former I didn't want to get into at that point in time and the latter wasn't available at my pricing tier.

I asked ChatGPT "Is it possible to add an access rule to an Azure Function that restricts access only from Azure?" The response came back:

Yes, it is possible to restrict access to an Azure Function so that it's accessible only from within Azure services. This can be done by configuring network access restrictions in your Azure Function settings. However, it's important to note that "from Azure" can encompass a wide range of sources, including other Azure services, VMs hosted in Azure, or services running in Azure regions.

The suggestion from ChatGPT was to use an Azure Service tag in the network access restrictions. As it put it:

Azure Service Tags represent a group of IP address prefixes from a given Azure service, which are used to help minimize complexity for security rule creation. You can leverage these in your network security rules.

Given this, I selected the option to enable from selected IP address.

This brought up a list of rules and I clicked on the option to add a new one. Following ChatGPT's advice, I added a rule for the AzureCloud service tag.

Once added, my new rule took pride of place at the top of the list. The portal defaults the unmatched rule action to 'Allow'. This isn't what I wanted, so I changed it to 'Deny' which resulted in the rule at the bottom.

Again I tested. This time I was able to access the function through API Management, but not directly. So I now had the behaviour that I wanted. However, ChatGPT did highlight these valid considerations for using AzureCloud Service Tag:

• Broad Access: The AzureCloud service tag is quite broad and includes all of Azure's public IP addresses. It doesn't restrict access to only your Azure services but allows access from any Azure-hosted service, which can include Azure services used by others.

• Other Azure Services: If your intention is to allow access only from specific Azure services (like Azure Logic Apps, Azure VMs, etc.), you might need a more granular approach. You can specify the IP addresses or ranges of those specific services or use relevant service tags if available.

Summary

API Management appears to be a powerful tool to expose and manage external APIs. I barely scratched the surface of its capabilities, as I was satisfied in just knowing that my Azure Function was now behind a suitable service. There was an awful lot of clicking and it makes me wonder about how all this would be done through infrastructure as code.

Comparing this experience to the one I have had with AWS is interesting. With Azure, I had to find ways to stop my Azure Function from being exposed. With AWS, you have to find ways to expose your Lambda functions. With Azure, you have to pay for managed identity functionality to integrate API Management with Azure Functions. With AWS, you have to use Identity and Access Management (IAM) for everything and it is completely free to use.

However, I now have the front of my application in a place I want. So the next step is to look at extending the back-end functionality, which will mean integrating with Blob storage.

Links

• What is Azure API Management?

• Expose serverless APIs from HTTP endpoints using Azure API Management

  • Create serverless APIs in Visual Studio using Azure Functions and API Management integration

• Import an Azure Function App as an API in Azure API Management

• Quickstart: Create a new Azure API Management instance by using the Azure portal

  • Quickstart: Create a new Azure API Management service instance using Bicep

• Observability in Azure API Management

  • How to integrate Azure API Management with Azure Application Insights

• Authentication and authorization to APIs in Azure API Management

• Enable advanced API security features using Microsoft Defender for Cloud

It's all gone south

Ultimately, I want to deploy the final application using infrastructure as code. However, first I thought I would try the ClickOps approach. This is done by right-clicking on the project and selecting 'Publish'.

The first time through it did successfully publish the function to Azure. However, I did notice something odd in the wizard. At one point it asks you to create a new Functions instance. The odd thing is that it only gave me one option for the storage, and that option was on the other side of the world.

I thought that perhaps it was perhaps a user interface issue. Surely, it wouldn't create resources 12,000 miles apart. I clicked to continue and then looked at the created resources.

Unfortunately, it was very much the case that resources could not have been more geographically disparate if you tried.

Back to the Portal

Try as I might, I could not get the Visual Studio wizard to create a set of geographically-sensible resources. Instead, I went into the Azure portal and selected the option to create a Function App. This did allow me full control over the location of any created resources. In particular, the Application Insights could now be located in the UK.

Once the Function App had been created via the portal, I could select it in the Visual Studio wizard.

Once this wizard had completed, and generated a few interesting files, I was presented by the 'Publish' button below.

Pressing this started the deployment and, before long, I had my function deployed into the cloud and ready to be tested.

Debugging the deployment

Full of excitement, I fired off a request to the function and got the following response.

HTTP/1.1 500 Internal Server Error

Now, I could test the function locally to recreate and debug the issue. However, it struck me that it would be an opportunity to look at what diagnostics are available in the cloud. With this in mind, I opened the Azure portal and went into the function app overview. Here, I was presented with a list of functions.

After selecting the function, I was then given the following developer options.

Out of curiosity, I selected the 'Integration' option and got a diagram showing how the triggers and inputs for the function and the outputs from the function. There was also a warning me that I could not edit my function in the portal, as I had chosen to use the isolated worker model. If such editing is important to you, then this is perhaps a reason not to choose that model. For me, I would rather not have the option.

Selecting 'Monitor' resulted in a promising list of function invocations. At the top of the list was my failure.

Clicking on the hyperlinked date brought up the details that I was looking for.

Here was the exception stack trace that clearly showed that the error lay in the code I had written. Well, the code I had copied from ChatGPT. Clearly, I had not been as diligent as I should have been with my testing.

There was also an option to run a query in Application Insights. Clicking this caused the following query to run and return all the relevant entries. Note how the query uses the 'union' operator to combine data from both the traces and exceptions tables.

For completeness, I open the 'Logs' page. This appeared to be some sort of realtime view of the logs. I ran my faulty function again and saw the following entries appear.

By default, it doesn't appear to log the actual exception. This seems to rather limit its usage, as the other views capture the full details. However, it might have some uses that are not apparent to me at the moment. It is good to know it is there though.

Now I knew what the problem was, I could go and fix it. But before that, there was one more thing I wanted to try and that was remote debugging.

Remote debugging (eventually)

To tell the truth, I could have saved myself quite a bit of frustration if I had read the remote debugging section of the Microsoft Develop Azure Functions using Visual Studio guide. However, here is the tale of my more circuitous route to success.

My first attempt was the most obvious option. That is, to use the option in the Publish page to attach a debugger.

This then indicated some activity, but ultimately no attachment occurred. Undeterred, I searched the internet and found the blog post How to remote debug an HTTP trigger Azure Functions in Visual Studio 2022. Amongst the steps mentioned, was to enable remote debugging in the Azure portal. However, when I looked I found this was already enabled.

With hindsight, what I suspect had happened was that the 'Attach Debugger' operation had enabled this. The Microsoft article advises the following:

After you're done remote debugging your code, you should disable remote debugging in the Azure portal. Remote debugging is automatically disabled after 48 hours, in case you forget.

The next thing I tried was to update the publish settings. The configuration was set to 'Release', so I changed it to 'Debug'.

Following the instructions in the blog post, I tried manually attaching to the remote process and was prompted for credentials to connect.

The credentials required had to be downloaded from the Azure portal, via a publish profile.

In the downloaded file, I found the details required in the 'Zip Deploy' element.

After a few tries, I was finally able to see the processes. So I followed the blog post and tried attaching to the w3wp.exe process.

However, Visual Studio still reported that my breakpoints were not active.

I wondered for a short while if remote debugging was not supported for the isolated worker model. Then it dawned on me that it wasn't the w3wp.exe process that I should be attaching to, it was the isolated dotnet.exe process instead.

Once I had done this, everything fell into place. My breakpoint was hit and I could step through my function remotely.

As mentioned, I could have avoided this, as the Microsoft article clearly states:

Check Show process from all users and then choose dotnet.exe and select Attach.

Summary

My experience highlighted the upsides and downsides of high-level wizards. When they work, they can be very productive. However, when they don't, it can be very difficult to understand what is going on an how to fix it. I also fell foul of the changing technology, as I was reading articles for the older function model whilst using the newer model.

However, I got there in the end and the option of remote debugging is an interesting one. It is one that I have never really felt I needed with AWS Lambda functions, but perhaps I will find a use now it is available. The option does expose the internals of the isolated worker model, as you can see how there is a separate process being called from the usual w3wp.exe process. If you step through an unhandled exception, you can even see how gRPC is used to communicate. All quite different from the black box of AWS Lambda functions.

The Webhook Proxy Application

The ultimate goal is to create a serverless application that can be placed in front of internal systems to robustly handle webhook callbacks. Its functionality will cover:

• Validating the content of the callback
• Storing the content of the callback
• Forwarding the content to a downstream system
• Automatically retrying if the downstream system is offline
• Forwarding to a dead letter queue if not able to forward

In AWS, I would probably build the application using API Gateway calling a Lambda function, that stored the request in S3. Then handle the S3 events raised, using a combination of Lambda functions and Event Bridge to deliver the request.

The appealing aspect of this task is that it is a real-world need, and that it covers API management, functions as a service, serverless storage, and events. This means I will need to tangle with the following Azure technologies:

• Functions
• Blob Storage
• API Management
• Service Bus

My first step on this journey is to create, test, and deploy an Azure Function that validates the contents of an HTTP according to a schema specified as part of the path.

Choosing an Azure Function model

Things rarely turn out to be as straightforward as you think, and straight away I was forced to make a choice as to the Azure Function model to use. Since I last explored them, there is a new 'isolated worker model' for running Azure Functions. The Microsoft article on the differences between the isolated worker model and the in-process model explains that there are two execution models for .NET functions:

Execution model	Description
Isolated worker model	Your function code runs in a separate .NET worker process. Use with supported versions of .NET and .NET Framework.
In-process model	Your function code runs in the same process as the Functions host process. Supports only Long Term Support (LTS) versions of .NET.

The Guide for running C# Azure Functions in an isolated worker process explains the benefits for the newer model:

• Fewer conflicts: Because your functions run in a separate process, assemblies used in your app don't conflict with different versions of the same assemblies used by the host process.
• Full control of the process: You control the start-up of the app, which means that you can manage the configurations used and the middleware started.
• Standard dependency injection: Because you have full control of the process, you can use current .NET behaviors for dependency injection and incorporating middleware into your function app.

In my experience, when Microsoft develop a new model then it is better to adopt it if you can. The older models do not seem to get quite the same love. So given that, it was the isolated worker model for me.

The 'Out of the Box' Experience

The Microsoft article Create your first C# function in Azure using Visual Studio was my guide to getting started. The experience compared to Lambda functions in AWS is quite stark. Visual Studio is a one-stop development environment for .NET, whereas AWS and VS Code feels like an 'assemble your own' adventure.

As long as you remember to select the Azure development workload during installation of Visual Studio, then it only took a few steps after selecting the 'Azure Functions` template before I could hit F5 and have the following code executing:

[Function("HttpExample")]
public IActionResult Run(
  [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req)
{
    return new OkObjectResult("Welcome to Azure Functions!");
}

Visual Studio had seamlessly started a local hosting process at http://localhost:7166/api/HttpExample, as can be seen below:

So straight away, I could set break points and benefit from a super-quick inner development loop. However, it turned out that the generated code was not the only way to go.

The Built-in HTTP model

As the HTTP trigger documentation states:

HTTP triggers allow a function to be invoked by an HTTP request. There are two different approaches that can be used:

• An ASP.NET Core integration model that uses concepts familiar to ASP.NET Core developers
• A built-in model, which doesn't require extra dependencies and uses custom types for HTTP requests and responses.

I consulted with ChatGPT, which had the opinion:

The choice between the built-in Azure Functions HTTP trigger model and ASP.NET Core integration depends on the complexity of your application, the need for control over the HTTP pipeline, and your familiarity with ASP.NET Core. For simpler, serverless applications, the built-in model is often the best choice due to its simplicity and tight integration with Azure. For more complex applications requiring the full feature set of ASP.NET Core, integrating with ASP.NET Core would be more appropriate.

Probing ChatGPT a bit more as to the limitations of the Built-in HTTP model, I got the following scenario where the more complex model might be needed:

While the built-in model of Azure Functions is powerful for many use cases, especially those that fit well within a serverless paradigm, it has its limitations in scenarios requiring advanced control over HTTP pipeline processing, complex authentication and authorization, and other sophisticated web API features. In these cases, integrating with a more feature-rich framework like ASP.NET Core would be more appropriate.

Given that I intend to front the function with Azure API Management to handle the authorisation aspects, I decided to go with the simplicity of the built-in model. This meant that the original boilerplate code became the following:

[Function("HttpExample")]
public HttpResponseData Run(
  [HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequestData req)
{
    var response = req.CreateResponse(HttpStatusCode.OK);
    response.WriteString("Welcome to Azure Functions!");
    return response;
}

So a little more verbose, but I could get rid of a dependency which was nice. I did have to amend Program.cs to use the defaults for a worker process:

var host = new HostBuilder()
    // Was ConfigureFunctionsWebApplication()
    .ConfigureFunctionsWorkerDefaults()
    .Build();

Dependency Injection and Logging

One significant difference that struck me between AWS Lambda functions and Azure Functions, was that the latter starts you off down the dependency injection (DI) route. The boilerplate code uses DI out of the box to inject an ILoggerFactory implementation that can then be used to obtain a logger:

public class SimpleHttpFunction(ILoggerFactory loggerFactory)
{
    private readonly ILogger _logger =
      loggerFactory.CreateLogger<SimpleHttpFunction>();

    [Function("HttpExample")]
    public HttpResponseData Run(
      [HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequestData req)
    {
        _logger.LogInformation("C# HTTP trigger function processed a request.");

As with a 'normal' application, the logging level is controlled within the Program.cs file. Here I want to have a very verbose output in development, but leave the level when in higher environments.

var host = new HostBuilder()
    // ...
    .ConfigureLogging((context, loggingConfig) =>
    {
        var env = context.HostingEnvironment;
        if (env.IsDevelopment())
            loggingConfig.SetMinimumLevel(LogLevel.Trace);
    })
    // ...
    .Build();

I decided to take advantage of the dependency injection and adopt a little hexagonal architecture. I created a couple of services to isolate my function from the details of how a request is validated and how the request content is stored.

var host = new HostBuilder()
    // ...
    .ConfigureServices(services =>
    {
        services.AddSingleton<IRequestValidator, RequestValidator>();
        services.AddSingleton<IRequestStore, BlobRequestStore>();
    })
    .Build();

With this in place, I could then use a primary constructor to have the implementations injected at runtime. This would also set me nicely to do some unit testing later on.

public class ValidateAndStoreFunction(
    ILoggerFactory loggerFactory,
    IRequestValidator requestValidator,
    IRequestStore requestStore)
{
    private readonly ILogger _logger =
        loggerFactory.CreateLogger<ValidateAndStoreFunction>();
    private readonly IRequestValidator _requestValidator = requestValidator;
    private readonly IRequestStore _requestStore = requestStore;

As mentioned earlier, this all feels quite different to working with AWS Lambda functions. In AWS, it felt like the onus was on keeping everything as light as possible. I have never used any dependency injection frameworks with AWS, so this felt much more like 'normal' application development. My concern, as with all serverless functions, was that this would add to any cold start times. However, for my learning application, I am not too concerned. For those that are, Mikhail Shilkov has written this excellent article on Cold Starts in Azure Functions.

Unit testing my function

Whilst it was great to be able to run my function and invoke it from cURL or Postman, I like to write pure unit tests that can be run from anywhere. This turned out to be quite straightforward, at least in the case of my function.

The first task was to instantiate and instance of ValidateAndStoreFunction. The primary constructor for this is as follows:

ValidateAndStoreFunction(
    ILoggerFactory loggerFactory,
    IRequestValidator requestValidator,
    IRequestStore requestStore)

Using the Moq mocking framework, I was able to supply mocks for these and setup appropriate return values.

_mockLoggerFactory = new Mock<ILoggerFactory>();
_mockRequestValidator = new Mock<IRequestValidator>();
_mockRequestStore = new Mock<IRequestStore>();

The first real snag was the signature of the Run method. It requires a HttpRequestData as input:

HttpResponseData Run(
    HttpRequestData req,
    string contractId,
    string senderId,
    string tenantId)

It turns out that HttpRequestData is an abstract class, so I tried to subclass it. This hit a couple of issues. The first was that HttpRequestData has a constructor that requires a FunctionContext instance. FunctionContext is itself abstract, so I tried using Moq to provide one.

class MockHttpRequestData()
    : HttpRequestData(new Mock<FunctionContext>().Object)

HttpRequestData also need to be able to create an HttpResponseData instance. The solution, again I subclassed HttpResponseData and return an instance of my new class.

public override HttpResponseData CreateResponse() => new MockHttpResponseData();

The final step was to be able to pass in an object to be returned as a JSON stream.

class MockHttpRequestData(object bodyObject)
    : HttpRequestData(new Mock<FunctionContext>().Object)
{
    private readonly string _bodyJson = JsonConvert.SerializeObject(bodyObject);

    public override Stream Body => GetStringAsStream(_bodyJson);

Now with my mocks in place, I could create some nice simple unit tests that would run anywhere.

// Arrange
var validateAndStoreSUT = new ValidateAndStoreFunction(
    _mockLoggerFactory.Object, _mockRequestValidator.Object, mockRequestStore.Object);

// Act
var response =
    validateAndStoreSUT.Run(
        new MockHttpRequestData(
            new { }), ExpectedContractId, ExpectedSenderId, ExpectedTenantId);

// Assert
response.Should().NotBeNull();
response.StatusCode.Should().Be(HttpStatusCode.Created);

I don't know if other trigger would be more difficult to mock out, but the ease of mocking HttpRequestData gives me cause for optimism.

Deploying to Azure

My original intention was to finish this post with deploying to Azure from Visual Studio. However, this did not prove as straightforward as thought and I will defer the trials and tribulations to the next post.

Summary

Here are my observations on my first real experience with Azure Functions:

• Azure Functions feel more like a traditional application than AWS Lambda functions. With dependency injection and a local development experience.

• The two available models, in-process and isolated, complicates initial decisions.

• Strong support for the local (F5) experience. This makes it easy to get going and there is support for remote debugging too that I didn't have time to investigate.

• The middleware limitations stopped me in my tracks. I was hoping to be able to extend my functions with middleware that would short-circuit the pipeline, but this turned out not to be possible.

• Unit testing was straightforward, with support for dependency injection and the framework classes making it easy to mock the inputs to the function.

Every February, under the guise of Andy Balham, I take part in FAWM (February is Album Writing Month). This is an online challenge to write and record 14 songs in 28 days (14.5 in a leap year!). The problem I had was that the resulting MP3 files need to be hosted, but my usual option was is not available this year.

So I set myself the challenge of finding a hosting alternative and, to provide a bit more of a challenge, develop a simple HTML interface on which to view and play the resulting songs.

What are my options?

I first started asking ChatGPT about hosting static websites in AWS, as I have experience in that area. However, when I asked ChatGPT the following question I got an interesting suggestion:

How can I create a simple static website that looks good?

The part that caught my eye was 'GitHub Pages' (emphasis mine):

Deploy Your Website:

Once you're satisfied with the look and feel:

• Host it on platforms like GitHub Pages, Netlify, Vercel, or any web hosting provider.

I hadn't heard of this, so I prompted ChatGPT for more information:

Can you tell me more about GitHub pages please?

ChatGPT duly responded:

GitHub Pages is a free web hosting service offered by GitHub that allows you to easily publish static websites directly from your GitHub repositories. It's a convenient way to showcase your projects, blogs, documentation, or personal websites to the world. Here are some key points about GitHub Pages:

1. Free Hosting: GitHub Pages provides free hosting for static websites, which includes HTML, CSS, JavaScript, and other static files. This makes it a cost-effective solution for hosting personal or project websites.

2. GitHub Integration: GitHub Pages is tightly integrated with GitHub repositories. You can create a dedicated branch (usually named gh-pages or main for user/organization pages) or use the docs folder in your repository to store your website files. When you push changes to this branch or folder, your website is automatically updated.

This sounded ideal, as I already had a GitHub account and the price was right in my sweet spot (free 🙂).

The original chat can be found here.

How can I test it?

Like all good software engineers, my first thought was how to test my static website as I developed it. So I lent on my assistant again:

I want to develop a static website. Whilst I am doing development on my PC, how can I host the website so that I can see instant results when I save changes?

As a regular user of VS Code, the following suggestion jumped out:

Integrated Development Environment (IDE) Servers:

1. Most modern Integrated Development Environments (IDEs) or code editors come with built-in tools to preview websites. For instance:

   • Visual Studio Code: Has an extension called Live Server which provides a live preview of your site with hot-reload functionality.

A quick search for the Live Server extension in VS Code and I soon had a way to instantly view rendered HTML. Now I was all set to go into the unfamiliar territory of website development.

The original chat can be found here.

But I'm not an HTML expert

Although I have been in the IT business for quite a while, my focus has pretty much always been on the back end. I have a rudimentary knowledge of HTML, but not much beyond that. What I needed what an expert to help me, so using SudoLang I prompted ChatGPT with the following:

# An expert in writing HTML

Role-play as a world-class HTML writer who is proficient in using the Bootstrap framework to produce attractive and responsive pages.

HTMLExpert {
  /write - Take a description of a page and output HTML to match it
}

With the role set, I asked the following:

/write a page that contains links to an album of MP3 files. The page should contain a placeholder for a description of the album. Along with links to download the MP3 files, there should be the name of each song and a link to a page containing the lyrics for that song.

The result was an HTML page (source) that admirably met my specification:

ChatGPT also added the following, providing helpful assistance in how to use the result:

To use this template:

• Replace "Album Name" with the name of your album.
• Replace the placeholder text under the ID albumDescription with your album's description.
• For each song in your album:
  • Duplicate the <li> element.
  • Replace "Song 1" with the song name.
  • Replace "path/to/song1.mp3" with the correct path or URL to the MP3 file.
  • Replace "path/to/song1_lyrics.html" with the correct path or URL to the song's lyrics page.

This looked like a great start, so I wondered if it could be improved:

That is great. What would be even better would be a music player so that each song could be played directly from the page.

To which I got the response:

Of course! Let's integrate an HTML5 <audio> element with controls for each song so that users can play them directly from the page.

Here's an enhanced version of your album page with the music player functionality:

And sure enough, the page (source) now had a music player on it. I supplied an MP3 in the appropriate location, and sure enough I could play it from my browser.

I then decided to see if the process would benefit from a more precise specification. So, as an experiment, I tried using Markdown as a specification format and prompted ChatGPT as follows:

/write

# Album Name

A link to an image for the album cover

A list of songs on the album and for each song:

- The title of the song
- A link to download the MP3
- A toggle button to show and hide the notes for the song, the text for the button changes on toggle
- A toggle button to show and hide the lyrics for the song, the text for the button changes on toggle
- An audio player to allow the MP3 to played from the page

The songs are in a sub-folder call mp3s
The images are in a sub-folder called images

The footer should say that all material is copyrighted to Andy Balham

The result, once again, was pretty good (source):

However, testing the page I found that the toggling didn't quite work. Using my programming knowledge I prompted ChatGPT:

The toggling doesn't work. I think targetElement.classList.contains('show') should be btn.textContent.includes('Hide')).

To which ChatGPT replied:

You're right; the toggle behaviour I provided previously was based on Bootstrap's show class state on the target element, which might not reflect the exact moment when the toggle happens. Let's adjust the toggle behaviour by updating the script section.

Here's the updated script:

And sure enough, an amended script was provided which fixed the issue (source).

I then spent some time experimenting with refining the SudoLang program to style the page in different ways, before I decided that including the lyrics in the main page would be a bit onerous.

So I asked ChatGPT to regenerate the page with the lyrics details of the page definition updated as follows:

For each song:

- A toggle button 'Lyrics' to show and hide the lyrics for the song

Folder information:

- The lyrics are in a sub-folder called lyrics, one file per song

The result was an amendment to the collapsible lyrics section:

<!-- Lyrics Collapsible Section -->
<div class="collapse mt-3" id="lyricsSample">
  <div class="card card-body">
    <a href="lyrics/song-title-lyrics.txt">View Lyrics</a>
  </div>
</div>

A link wasn't what I wanted, so I asked for alternatives:

Is there any way the text from the lyrics files could be included without using an <a> ref?

Sure enough ChatGPT responded with some options:

To include the lyrics text directly on the page without using an <a> link, you have a couple of options, but each comes with its own considerations and limitations:

1. Embedding Lyrics Directly in HTML: <snip>
2. Using JavaScript to Load Lyrics: <snip>
3. Server-Side Inclusion: <snip>

The JavaScript option looked like what I wanted, so asked for that:

Please update the HTML with the JavaScript version

As you can see from below, the result (source) was a partial success. The lyrics were loaded on demand, but the rendering left a little to be desired:

So I asked ChatGPT to sort that out.

If the lyrics are in plain text format, can you amend the JavaScript to insert suitable HTML markup to make the lyrics break across lines

The response was an amended JavaScript function, which I then copy and pasted into the page (source) and the results are shown below:

The result is not perfect, blank lines have not been preserved, but it will suffice for my needs. Overall, I was impressed with how I was able to go from nothing to a page that contains functionality that I wanted. It won't win any design awards, but it would have taken me significantly longer to develop with the AI assistance.

The whole chat can be found via this link.

How could I repeat it?

As a final step, I thought I would lean on ChatGPT once more and ask it to write a JavaScript program to merge the HTML template it had created with a YAML file in the following format:

albumName: Aye-Aye?
songs:
  - title: Guinness Greed
    mp3Filename: 01_guinness_greed
    notes: "Prompt: /genre greed /subject Guinness /write must include reference to John"
  - title: Flood's Domain
    mp3Filename: 02_floods_domain
    notes: "Prompt: /genre fear /subject floods /write"

My prompt for this was simply the following:

You are an expert in write Node.js programs. I would like you to write a program that receives a YAML file as input, merges the details of that file with the an HTML template, and then outputs an HTML file containing the merged result.

Here is an example YAML file: <snip>

Here is the HTML template: <snip>

The resulting output from the resulting program was almost perfect:

The only thing that wasn't right was that the H1 element hadn't been replaced with the album name. ChatGPT had used the replace method and that seemed to only replace the first match, so I asked the following:

How can the program be changed to make replace('Album Name', yamlData.albumName) replace all instances of 'Album Name' and not just the first instance?

To which I got the following response and an updated mergeYAMLWithTemplate function:

To replace all instances of a specific string in JavaScript, you can use a regular expression with the global (g) flag. This way, the replace method will replace all occurrences of the string in the text, not just the first one.

I pasted in the updated function, ran the program and was presented with the following expected result:

The whole chat can be found via this link and the JavaScript program here.

Summary

It was a very interesting to try a new process for developing a solution to a problem. At each stage, it was a significant advantage to be able to ask for options on how to solve the next step. The interaction felt very natural, asking for further details or asking for small changes. The process allowed me to explore very quickly and, although I could have written the final program, it was much quicker to get it generated for me. Interestingly, my solution probably would have been at risk of gold-plating. Taking the simple solution has its advantages.

There is no doubt to me that these tools are a significant productivity boost. In the software realm, we can test the results (you do test don't you? 😉). This means we can easily check the output that we are given, which is not the case in others.

The resulting website can be found here.

See the accompanying GitHub repo for working code examples.

TL;DR

• Be wary when using step functions errors for flow control in step functions
• After throwing and catching an error, only the inputs and the basic error details are available

Our inheritance

We inherited a codebase that contained a step function which was asynchronously invoked from an API Gateway request. The step function performed validation on the request, before proceeding to processing the request if the validation was successful. When the validation failed, either due to schema errors or due to the content of the request, an error was thrown by the Lambda function. These errors were caught by the step function and, depending on the name of the error, the step function invoked a DynamoDB integration task to update a table with the result. Users of the API could then use another API call to query the state of the request.

A simplified version is shown below:

Improving things for our users

Users don't always get things right, so it was common for requests to fail. In particular, they would fail for one of the validation reasons. Whilst the underlying reasons for the failures were logged to CloudWatch, it quickly became a pain to keep looking them up whenever a user of the API suffered from a failure.

To provide better feedback for our users, we decided to add the validation errors to the result table. The users could then query this table via the API to understand why their request had failed, and all without bothering us.

The only problem was, at the point the table was being updated, we had no way of accessing the validation failures.

Post-error step function context

The problem was that after an error has been handled by a step function, only the following is available:

• The inputs to the step function
• The error type

This makes sense, as there can be no guarantee of the state once an error occurs. The inputs never change, so it is safe to access their values, but it is not safe to access anything else apart from the type of error.

This means that it isn't possible to pass structured information into the error handlers. Which happened to be exactly what we needed to do if we were to extend the DynamoDB integrations to store the extra details.

Are these really errors?

It is somewhat a matter of opinion, but I would argue that neither validation failure is a true error. I say this, as we would expect both scenarios to occur in normal operations. The use of throwing and catching errors in this case looked to me like it was done as a convenience.

To resolve the issue, the validation Lambda function was updated to add the errors directly to the table. This took advantage of the fact that the relevant item was already present and that the DynamoDB integration would not overwrite any added errors. This allowed us to avoid changing the flow of the step function, but came at the cost of increasing the responsibility of the validation function.

A better approach

I would argue that the validation step should produce a validation result.

interface ValidationResult {  
  user?: User;
  formatErrors?: ZodIssue[]; // The errors from using the zod npm package
  contentErrors?: string[];
}

function validateUser(event as any): ValidationResult {
  // <snip>
}

export const handler = async (
  event: Record<string, any>
): Promise<ValidationResult> => {
  return validateUser(event);
};

With this, we can then rework the step function with an additional choice step, as shown below:

Now the DynamoDB integrations can access the relevant details. For example, for format errors.

{
  "Key": {
    "key": {
      "S.$": "$$.Execution.Input.requestId"
    }
  },
  "TableName": "<snip>",
  "ExpressionAttributeNames": {
    "#status": "status",
    "#formatErrors": "formatErrors"
  },
  "ExpressionAttributeValues": {
    ":status": {
      "S": "InvalidFormat"
    },
    ":formatErrors": {
      "S.$": "States.JsonToString($.validationResult.formatErrors)"
    }
  },
  "UpdateExpression": "SET #status = :status, #formatErrors = :formatErrors"
}

With this approach, the validation step is pure validation and not given a secondary responsibility of storing any validation errors.

General discussion

Consider the following C# method:

MyEntity FindMyEntity(string id);

What should the method return if there is no such instance with a matching id? Should it return null or throw an exception?

I would say it depends on the caller. If the caller cannot continue without the entity, throw an exception. If it can continue, return null. This could be catered for with the following:

MyEntity FindMyEntity(string id, bool throwExceptionIfNotFound = false);

Now the caller can control the behaviour as they need.

You could go further and return a result object.

FindResult<MyEntity> FindMyEntity(string id);

You can then explicitly check for the 'not found' scenario.

if (entityFindResult.NotFound) // ...
else // Use entityFindResult.Value

Summary

I would advise against using errors as a replacement for step function flow control using choice steps. Once the error is thrown, the current state is lost and only the error type is then available downstream.

In general, I would advise against throwing errors/exceptions/<insert your favourite language version here> as part of 'normal' processing. That is, unless the idioms of your preferred language recommend it. I have found that there are usually alternatives that make your code cleaner and more extensible.

40 years ago

It was 1982 and I was attending the school computer club. We were taking turns to program the 1K ZX81. One member took their turn at the 'keyboard' and attempted to type something along the lines of...

10 IF THE PLAYER PRESSES LEFT THEN MOVE THE SPACESHIP LEFT

The result was the following...

10 INPUT F THE PLAYER PRESSES LEFT THEN MOVE THE SPACESHIP LEFT

Apart from the interesting fact that variables on the ZX81 could have spaces in them, the computer was clearly not going to do what was being asked. How foolish they were, my 13yo self thought smugly to himself.

Fast-forward 40 years and this sort of programming is now possible, thanks to LLMs such as ChatGPT and the insight of Eric Elliot and his work developing SudoLang.

Encountering SudoLang

I was listening to the excellent .Net Rocks! podcast, when they mentioned SudoLang. When I mentioned to my co-workers that SudoLang was being developed by Eric Elliot, I was told in no uncertain terms what a very smart fellow he was. So, with my interest sufficiently piqued, I got a ChatGPT Plus subscription, followed the steps in the AI Programming for Absolute Beginners article, and my mind was blown.

To quote from the SudoLang GitHub repo:

SudoLang is a programming language designed to collaborate with AI language models including ChatGPT, Bing Chat, Anthropic Claude, and Google Bard. It is designed to be easy to learn and use. It is also very expressive and powerful.

All sufficiently advanced language models understand it without any special prompting. You do not need to paste the SudoLang specification before using SudoLang with your favorite AI.

Programming with SudoLang

The question was what sort of program should I write. How about one that generates some comedy names in the spirit of Key and Peele and their East vs West sketches.

SudoLang programs are written in Markdown and start with the role that provides the context. Think of it like briefing an actor. In our case, an actor that does the following.

# East vs West Player Generator

Generate a plausible, but comedic sounding name for an American football player.

We then specify the state of the application. The LLM understands that the curly braces delimit the various components, giving them scope. In SudoLang you lean heavily on what the LLM can infer, so We don't need to specify types were they can be inferred. Where they can't, we provide a hint in a comment such as East|West.

NameGenerator {
  State {
    Name
    CollegeOrSchoolOrPrison
    Team // East|West
    Position
  }
}

A key concept in SudoLang is programming by constraints, so now we add some to our program.

NameGenerator {
  State {
    ...
  }
  Constraints {
    The name must consist of at least a first name and a surname.
    The name must not be the name of a real player.
    The name must sound amusing, but must be able to be pronounced.
  }
}

Like the pupil 40 years ago, all we do is express our intention in natural language. In SudoLang, natural language is preferable, as that is what LLMs understand.

Finally, we add a couple of commands and a function to greet the user. Again, we use a convention for commands that the LLM will be familiar with (i.e. /command).

NameGenerator {
  State {
    ...
  }
  Constraints {
    ...
  }
  /next Generate the next name
  /help Provide help
}

welcome("Welcome to the name generator. Type /next to get the next name")

And now our program ready to run or, if you prefer, our actor is ready to perform.

Pasting it into ChatGPT GPT-4, we get the following response:

Welcome to the name generator. Type /next to get the next name. If you need assistance or further information, type /help. Enjoy!

Note here that it has given us the greeting we programmed, but our actor has gone a little further and added that assistance is also available.

Let's try a name, so typing /name returns:

I'm sorry, the provided command "/name" is not recognized. Please use one of the supported commands such as "/next" or "/help". Would you like to proceed with "/next" to generate a name?

Oops, I forgot my own commands. However, the program gave a helpful response without us having to explicitly program for that eventuality. Typing /next returns:

I apologize for the confusion. Let me generate a name for you now.

NameGenerator:
  State:
    Name: "Dexter 'Dizzy Dukes' Duckworth"
    CollegeOrSchoolOrPrison: "Puddle Jumpers Academy for Overachievers"
    Team: "East"
    Position: "Running Back"

Our actor is almost British here, apologising for other's mistakes. However, it has successfully done what we wanted and generated a mildly amusing name, plus a manufactured college of sorts. Not exactly comedy gold, but not too bad.

Refining the program

To refine our generator, let's add another line to the context to try to get names more like Key and Peele.

# East vs West Player Generator
...
The name can have a middle name or initial and can have a suffix, such as III.

We'll also add a height and weight to the state. We will leave it to the LLM to work out suitable units.

State {
  ...
  Weight
  Height
}

The colleges in the Key and Peele sketch are real ones, so let's add a constraint to ensure that ours are too. While we are at it, we add a constraint to make sure our offensive linemen are suitably hefty.

Constraints {
  ...
  The college, school, or prison must be real.
  The weight and height must be consistent with their position in the team
}

Finally, we add a /refine command and amend the /next to output YAML.

/next Generate the next name and output the result in YAML
...
/refine Generate another name based on the last, but with more exaggeration

OK. When we paste in the program we get:

Welcome to the name generator. Type /next to get the next name. For assistance or further options, type /help. Let's create some memorable players!

Once more, our actor has taken some liberties with the script, but the gist is spot on. Let's try out the /next command.

NameGenerator:
  State:
    Name: "Thaddeus T. 'Thunder Thighs' Thompson III"
    CollegeOrSchoolOrPrison: "University of Alabama"
    Team: "West"
    Position: "Defensive Lineman"
    Weight: "310 lbs"
    Height: "6'4''"

Success. We have our output in YAML, the college is real, and the height and weight are position-appropriate. Let's try /refine.

NameGenerator:
  State:
    Name: "Thaddeus T. 'Titanium Tackle' Thompson IV"
    CollegeOrSchoolOrPrison: "University of Alabama"
    Team: "West"
    Position: "Defensive Lineman"
    Weight: "325 lbs"
    Height: "6'5''"

OK. Not the greatest exaggeration perhaps, but the program did do what we asked.

Let's try one final revision of our program. First let's give each player a catchphrase.

# East vs West Player Generator
...
Each player has a catchphrase based on their background, position,
  and physical stature.

NameGenerator {
  State {
    ...
    Catchphrase
  }
  Constraints {
    ...
    The weight and height must be consistent with their position in the team,
      but can be exaggerated for comic purposes.
    The catchphrase must not be based on any stereotypes.
  }
  ...
  /opposite Generate another name based on the last,
    but with opposite characteristics
}

overTheTopWelcomeInTheStyleOfAFootballCoach()

We have also loosened the height and weight constraint to try to get more extreme values and added a new command to generate an opposite character. The welcome was good before, but it could be more in keeping with the subject matter. So let's see if the amended version can gee us up a bit.

Alright, team! Gather 'round! 🏈 This ain't just any old name generator! We're here to scout out the funniest, quirkiest, and downright unforgettable players! You want comedy? We got it! You want star power? We got it! Strap on those helmets, get in formation, and type "/next" to discover the next legendary player! Remember, we play hard, we play fair, but most importantly, we play for laughs! LET'S GO TEAM! 📣🎉

Yes, that certainly had an effect. Let's do what coach says and type /next.

NameGenerator:
  State:
    Name: "Fitzgerald F. 'Flashy Footwork' Fiddlesticks"
    CollegeOrSchoolOrPrison: "Harvard University"
    Team: "East"
    Position: "Wide Receiver"
    Weight: "185 lbs"
    Height: "6'2''"
    Catchphrase: "You can't catch what you can't see!"

Not bad, let's try the /opposite.

NameGenerator:
  State:
    Name: "Barnaby B. 'Boulder Boot' Bumblebee"
    CollegeOrSchoolOrPrison: "Stanford University"
    Team: "West"
    Position: "Defensive Tackle"
    Weight: "320 lbs"
    Height: "5'10''"
    Catchphrase: "I'm the mountain you can't move!"

Now let's go off-piste and just ask Please generate a punter from a prison.

NameGenerator:
  State:
    Name: "Percival P. 'Penitentiary Punt' Peterson"
    CollegeOrSchoolOrPrison: "San Quentin State Prison"
    Team: "West"
    Position: "Punter"
    Weight: "195 lbs"
    Height: "6'0''"
    Catchphrase: "From the yard to the 50-yard line!"

This highlights that our programs can do more than we explicitly specify. So let's push that a bit further.

Using the LLM to improve our program

Now that the LLM knows is very familiar with the context we are dealing with, let's see if it can suggest ways we can improve the program. Typing Please suggest some possible commands to help refine the comedy potential of the characters gets us the following.

Certainly! Here are some potential commands to help refine the comedy potential:

1. /exaggerate: Amp up the comedy by making a character's feature or trait more extreme.
2. /mashup: Combine elements of two or more previously generated characters.
3. /nickname: Generate a unique and funny nickname based on the character's existing traits.
4. /backstory: Provide a short, humorous backstory for the character that ties into their attributes or catchphrase.
5. /random_trait: Add a random and unexpected trait or hobby that contrasts with the character's football position or personality for comedic effect.
6. /swap: Swap one trait or attribute for its comedic opposite.
7. ...and another six

These commands would help add layers of comedy and depth to each character, making them more memorable and entertaining.

Not bad suggestions, let's add them to our program and try out /backstory for 'Bubba "Biscuit" McGillicuddy IV'.

Bubba "Biscuit" McGillicuddy IV hails from the small town of Butterville, a place famous for its annual Butter Festival. Born into the McGillicuddy family, owners of the renowned "McGillicuddy's Marvelous Muffins & More" bakery, Bubba was always the center of attention.

Growing up, while other kids played with toys, young Bubba played with dough. In fact, his nickname "Biscuit" was coined when, at the age of seven, he mistakenly wore a biscuit mold as a helmet and ran head-first into a wall, leaving a biscuit-shaped indentation. The name stuck, and so did his love for the sport.

...and so on.

This got some genuine laughs when I read it out, so I consider our quest for humour successful.

Summary

SudoLang has changed the way I look at interacting with LLMs such as ChatGPT. The genius of SudoLang is that it takes advantage of the conventions that the LLMs have learnt, such a /commands, interfaces, and { scoping }. I now approach each interaction with ChatGPT as if I am approaching an expert who I need to brief on what I want, what I don't want, and how I want it. SudoLang gives me a way to express this in a structured way. Once the expert has some experience of what I am after, I can even ask the expert how I can improve on what I am asking for.

I feel the same excitement of possibilities as I did when I got my own ZX81 40 years ago. But now, of course, I can use natural language.

Links

• AI Programming for Absolute Beginners: Your First Steps with SudoLang
• SudoLang on GitHub
• SudoLang: A Powerful Pseudocode Programming Language for LLMs
• Anatomy of a SudoLang Program: Prompt Engineering by Example
• Learn SudoLang on ChatGPT
• The Art of Effortless Programming: Why Every Developer Should Learn ChatGPT and SudoLang
• AI Driven Development with SudoLang - Autodux (YouTube video)

You can download the accompanying code and try it yourself from the GitHub repo.

The example application

The case study we looked at in the series on implementing Enterprise Integration patterns is an application that acts as a loan broker. The application in question receives a request containing the details of the loan required via an API, and then returns the best rate to a webhook.

The following diagram shows how we use a central EventBridge event bus to implement this.

View the service map

In the last post, I walked through how I added X-Ray to the whole application. Now when I run some requests through the API, we see the following service map.

What is quite clear from this picture, is that events are at the heart of this application.

Now, by clicking on the client, we can trace a request all the way through the application and out to the webhook.

However, when I tried this, I found something that was hindering the observability. Ironically, it was observability that I added in the post on Domain Observability.

Removing observability from tracing

In that post, I added business-level observability by hooking a Lambda function up to the domain events being raised.

However, as I had enabled tracing for this Lambda function, the trace included numerous entries for the observers which clouded the picture of the process.

When viewing the log, this was further apparent.

The solution is to specify Tracing.DISABLED for the observability Lambda functions. However, as I still wanted the traces when testing the Lambda functions, I added a isTestMode property the observability CDK stack as follows.

const loggerFunction = new NodejsFunction(
  this,
  'Logger',
  getNodejsFunctionProps({
    // We don't want the observing in the trace for production
    tracing: props.isTestMode ? Tracing.ACTIVE : Tracing.DISABLED,
    // <snip>
  })
);

Now production traces are clean, but we can also take advantage of X-Ray when testing the functionality.

Adding custom subsegments

The article Generating custom subsegments with the X-Ray SDK for Node.js describes subsegments as follows.

Subsegments extend a trace's segment with details about work done in order to serve a request. Each time you make a call with an instrumented client, the X-Ray SDK records the information generated in a subsegment. You can create additional subsegments to group other subsegments, to measure the performance of a section of code, or to record annotations and metadata.

In our application, we have Lambda functions that simulate response from lender systems. At the moment, this is just an algorithm, but in practise would be a call that would take time and be prone to error. This would be an ideal call to surround with a custom subsegment.

With this in mind, I added the following code to allow the lender configuration to control the delay in responding and whether an error occurred.

const simulateExternalCallAsync = async (
  lenderConfig: LenderConfig
): Promise<void> => {
  const randomPercentage = randomInt(100);
  const errorPercentage = lenderConfig.errorPercentage ?? 0;
  const throwError = randomPercentage <= errorPercentage;

  const delayMillis = lenderConfig.minDelayMillis ?? 1000 + randomInt(1000);
  await new Promise((resolve) => setTimeout(resolve, delayMillis));

  if (throwError) {
    const errorMessage = `Simulated error (${randomPercentage} <= ${errorPercentage})`;
    throw new Error(errorMessage);
  }
};

With this in place, I added the code below around the call to simulateExternalCallAsync to add and close the subsegment.

import * as AWSXRay from 'aws-xray-sdk';

// <snip>

const segment = AWSXRay.getSegment();
const subsegment = segment?.addNewSubsegment('External Call');

try {
  // Simple values that are indexed for filter expressions
  subsegment?.addAnnotation('callType', 'Lender');
  subsegment?.addAnnotation('lenderId', lenderConfig.lenderId);
  // Related data for debugging purposes
  subsegment?.addMetadata('lenderDetails', {
    lenderId: lenderConfig.lenderId,
    lenderName: lenderConfig.lenderName,
    lenderUrl: `https://${lenderConfig.lenderId}.com`,
  });

  await simulateExternalCallAsync(lenderConfig);

} catch (error) {
  if (error instanceof Error) {
    // Add error to the subsegment
    subsegment?.addError(error);
  }
  throw error;
} finally {
  // Ensure the subsegment is closed
  subsegment?.close();
}

I redeployed the lenders, with the configuration set to introduce delays but not throw any errors. After running a request, I could see the following in the trace.

So we can now see the time taken for our 'external call'. What we can also see is the annotation and metadata that we added to the subsegment. Annotations are key-value pairs with simple data (strings, numbers, or booleans) that are indexed for use with filter expressions, whilst metadata can be any related data you'd like to store that's not indexed.

Clicking on the 'Annotations' tab of the segment details, we can see what type of call it was and which lender was called.

For more data, clicking on 'Metadata' shows the lender name and the URL being 'called'.

Forcing some errors

After having a look at what a happy system looks like, I decided to introduce some errors. I updated the configuration for one of the lenders so that it would always throw an error and redeployed.

After running a request through the system, the service map clearly showed that the Lambda function for the lender has errors.

Looking at the trace, I could see that the asynchronous request from the step function had timed out, as the timeout was set to 12 seconds.

The trace also contained the custom subsegment, which clearly shows that there was an error making the external call.

As before, the 'Annotations' and 'Metadata' tabs showed the call details. However, as we added the error to the subsegment, the 'Exceptions' tab also shows us the error details, including the stack trace.

This shows the power of using subsegments to instrument key parts of the application.

Running a workload

The final part of my adventure with X-Ray involved configuring the Lambda functions for the lenders to randomly error and then to put multiple requests through the application.

Looking at the resulting service map, I could see some errors I was expecting and some I didn't.

In particular, the step function was showing that it had errored. So I selected it and filtered the traces to see what was going on. On inspection, the Lambda function that sends the response to the webhook was erroring. Clicking on the 'Exceptions' tab clearly indicated to me that the issue was that there was no response to send in this case, but the code didn't cater for it.

Looking at another error, I saw that the Lambda function that looks up the lenders from the parameter store is throwing an error. Again, the 'Exceptions' tab shows the underlying reason. In this case, there is a rate limit on accessing the parameters in the parameter store. This indicates to me that perhaps we need the application to implement some sort of cache in front of the raw access.

Summary

In this post, I documented my experience using X-Ray with an application. I found the ability to view individual traces through the application, along with the associated logs, very valuable indeed. The ability to see errors and then drill down to the causes was also very valuable and allowed me to see some behaviour that otherwise would have been tricky to spot. Overall, I was very impressed as to what X-Ray has to offer.

Of course, there are also numerous excellent third-party offerings, such as the following:

• New Relic
• Datadog
• Lumigo
• Thundra

So, although you get X-Ray out of the box with AWS, please consider these as well.

Links

• Accompanying GitHub repo
• Integrating AWS X-Ray with other AWS services
• How to do Distributed tracing in AWS? | AWS X-ray and Cloudwatch Service Lens
• aws-cdk-lib.aws_xray module
• Tracing AWS SDK calls with the X-Ray SDK for Node.js
• How to do Distributed tracing in AWS? | AWS X-ray and Cloudwatch Service Lens
• Using AWS Lambda with AWS X-Ray
• X-Ray Tracing Modes
• Setting up AWS X-Ray with API Gateway REST APIs

The example application

The case study we looked at in the series on implementing Enterprise Integration patterns is an application that acts as a loan broker. The application in question receives a request containing the details of the loan required via an API, and then returns the best rate to a webhook.

The following diagram shows how we use a central EventBridge event bus to implement this.

Adding X-Ray

Adding X-Ray to the application involved the following steps.

1. Enable tracing via CDK
   • Lambda functions
   • Step functions
2. Wrap SDK clients
   • EventBridge

Enabling tracing on the Lambda functions was as straightforward as adding the following line to the default properties for all Lambda functions in the application.

export const NODE_DEFAULT_PROPS = {
  // <snip>
  tracing: Tracing.ACTIVE,
};

The only place I chose to override this default behaviour was in the API handler. Here I used Tracing.PASS_THROUGH, so that it would adhere to the upstream sampling set in the API. See the following StackOverflow post What is Active tracing mean in lambda with Xray? for a good explanation on what the tracing levels mean.

The application only uses one step function and so it was amended directly as follows.

this.stateMachine = new StateMachine(this, 'StateMachine', {
  tracingEnabled: true,
  // <snip>
});

The final step was to wrap all the SDK clients that make requests via passive services, such as EventBridge or SQS.

As was shown in the diagram above, all communication in the application is done through EventBridge. In fact, all Lambda functions use the same putDomainEventAsync method to send domain events.

export const putDomainEventAsync = async <T extends Record<string, any>>({
  eventBusName,
  domainEvent,
}: {
  eventBusName?: string;
  domainEvent: DomainEvent<T>;
}): Promise<AWS_EventBridge.PutEventsCommandOutput> => {
  // <snip>
};

The upshot of this is that there was only one place to wrap the EventBridge SDK client:

const eventBridge = AWSXRay.captureAWSv3Client(new EventBridge({}));

And with this, I had added X-Ray to the whole application.

Step Functions and EventBridge

One thing that I was aware of, was that the step function uses EventBridgePutEvents direct integration, as highlighted below.

I was asking myself if this call would be traced by X-Ray. To test if this is the case, I decided to run one of the the unit tests that executes the step function.

The unit test

The unit test differs from a more traditional unit test in that it exercises the step function in the cloud as part of an ephemeral, test-specific CDK stack. This approach allows the Lambda functions used by the step function to be swapped for test-specific implementations. This allows us to script responses for those function and so test all the routes through the step function. For an in-depth look at the approach, please see my post Step Function integration testing with CDK.

The first test I tried initiates the step function by publishing a quoteSubmitted event. It then waits for a Lambda function, acting as a test observer, to receive a quoteProcessedEvent.

// Act

await putDomainEventAsync({
  eventBusName: loanBrokerEventBus.eventBusArn,
  domainEvent: quoteSubmitted,
});

// Await

const { observations: quoteProcessedObservations, timedOut } =
  await testClient.pollTestAsync({
    filterById: LoanBrokerTestStack.QuoteProcessedObserverId,
    until: async (o) => o.length > 0,
    timeoutSeconds: 30,
  });

// Assert

expect(timedOut).toBeFalsy();

However, when I ran the test, I found that it was timing out waiting for the expected event.

Why is my test failing?

As I was now using X-Ray, I had a look at the service map.

On the up side, I could see that the EventBridgePutEvents step function task does allow events to be traced through EventBridge. On the down side, I could see that there were failures in three places, the step function and two Lambda functions.

Looking at the trace for the step function, I could see that the 'RequestCreditReport' task was failing just after 6 seconds. The timeout for this task was set to 6 seconds, so this looked like the task was probably timing out. That would certainly explain why the expected event was not being published.

I could also see that the Lambda function that provides mock credit references was failing. This would explain why the step function task was timing out, as the step function task was never receiving an event back.

The invocation duration of 2.99s also looked like a timeout, as the timeout for Lambda functions was set to the default of 3 seconds. The console allowed me to quickly dive into the logs and confirm that that was the case.

Looking at the trace for the other failure, I could see that three attempts were made to the Lambda function that handles callbacks to the step function.

Again, I was easily able to navigate to the logs and see the reason.

I could see that the Lambda function is being invoked as the result of an EventBridge rule. The Lambda function is then trying to restart the step function, but the step function to be restarted has already finished due to the task timeout. The result is an error, which then causes EventBridge to retry.

The solution

The solution to the timeouts was quite simply to double the memory of the highlighted Lambda functions to 256mb and double the timeout to 6 seconds. With these changes in place and deployed, the unit test ran successfully and the resulting service map reflect the clean run.

This service map clearly shows how EventBridge is at the heart of our application. When I selected a successful trace, I could see all the relevant logs in one place.

Summary

Although I started this post with the intention of diving into the traces of a full application, that will have to wait until the next post.

This short exercise with the unit test shows how you can use the service map and traces to see where issues are. The integration with the logs then allows you to drill down and see the underlying reasons. In particular, X-Ray gave visibility to the asynchronous event-driven behaviour and the behaviour under failure conditions.

I am looking forward to seeing what tracing through the application will bring.

Addendum

The unit tests used a wrapped SDK client to publish events. This cause the following error to appear in the console.

console.error
  2023-08-19 14:28:41.709 +01:00 [ERROR] Error: Failed to get the current sub/segment from the context.
      at Object.contextMissingLogError [as contextMissing] (D:\Users\andyb\Documents\github\blog-enterprise-integration\node_modules\aws-xray-sdk-core\dist\lib\context_utils.js:22:27)
      <snip>
      at Object.<anonymous> (D:\Users\andyb\Documents\github\blog-enterprise-integration\tests\loan-broker\loan-broker.test.ts:178:5)

As mention in Configuring the X-Ray SDK for Node.js, the solution was to add AWS_XRAY_CONTEXT_MISSING=IGNORE_ERROR in my .env file.

Links

• Integrating AWS X-Ray with other AWS services
• How to do Distributed tracing in AWS? | AWS X-ray and Cloudwatch Service Lens
• Tracing AWS SDK calls with the X-Ray SDK for Node.js
• How to do Distributed tracing in AWS? | AWS X-ray and Cloudwatch Service Lens
• For Lambda
  • Using AWS Lambda with AWS X-Ray
  • X-Ray Tracing Modes

Please be aware there are numerous excellent third-party offerings, such as the following:

• New Relic
• Datadog
• Lumigo
• Thundra

In order to dip my toe into the world of X-Ray, I decided to instrument the examples I created for my CDK Cloud Test Kit npm package. These example use a variety of services, e.g. SNS, SQS, EventBridge, etc, so provide a good starting point to learn.

API Gateway, Lambda, and EventBridge

This example consists of an API Gateway backed by a Lambda function. The Lambda function does the following:

• Generates a unique reference
• Stores the body of the request in S3, under the reference generated
• Creates a pre-signed URL, allowing access to the S3 object created
• Publishes an event to EventBridge containing the reference and the pre-signed URL

This gives us a nice example to see if we can trace requests from API Gateway, through a Lambda function, through EventBridge, to another Lambda function.

Adding X-Ray to our CDK code

The RequestApi construct contains the RestApi instance and the Lambda function it calls.

For the RestApi, we set the deployOptions property tracingEnabled to be true:

this.api = new RestApi(this, 'RequestApi', {
  // <snip>
  deployOptions: {
    tracingEnabled: true,
    // <snip>
  },
});

In the corresponding Lambda function, we set tracing to be PASS_THROUGH.

const eventPublisherFunction = new NodejsFunction(this, 'EventPublisher', {
  // <snip>
  tracing: Tracing.PASS_THROUGH,
});

The options for tracing are:

• ACTIVE: Lambda will respect any tracing header it receives from an upstream service. If no tracing header is received, Lambda will sample the request based on a fixed rate.
• PASS_THROUGH: Lambda will only trace the request from an upstream service if it contains a tracing header with "sampled=1"
• DISABLED

To quote the StackOverflow post What is Active tracing mean in lambda with Xray?:

AWS Lambda supports both active and passive instrumentation. So basically you use passive instrumentation if your function handles requests that have been sampled by some other service (e.g. API gateway). In contrast, if your function gets "raw" un-sampled requests, you should use active instrumentation, so that the sampling takes place.

In our case, as the Lambda function is called from API Gateway, we can set the value to PASS_THROUGH.

However, in the RequestApiTestStack stack, we have a Lambda function that is triggered by EventBridge. Although EventBridge integrates with X-Ray, it does so passively. This means that the Lambda function receiving the event needs to have its tracing set to ACTIVE.

this.addTestFunction(
  new NodejsFunction(this, RequestApiTestStack.EventObserverId, {
    // <snip>
    tracing: Tracing.ACTIVE,
  })
);

After deploying to AWS and running a few request, the X-Ray service map view shows the following:

We can see API Gateway invoking the associated Lambda function, and we can also see observer Lambda function being invoked. However, clearly we are missing something that links them all together.

Adding X-Ray to our Lambda code

To get our trace linked up, we need to wrap the EventBridgeClient instance with the appropriate middleware to inject the context into the calls to publish events. This is done in the RequestApi.EventPublisher function as follows:

import * as AWSXRay from 'aws-xray-sdk';

const eventBridgeClient = AWSXRay.captureAWSv3Client(new EventBridgeClient({}));

Now when we view X-Ray, we can see the trace from API Gateway to the handle Lambda function, through EventBridge, and finally to our observer Lambda function.

You may notice that there are two circles in the map for each Lambda function. In the video How to do Distributed tracing in AWS? | AWS X-ray and Cloudwatch Service Lens, Marcia Villalba explains that there is one for the Lambda runtime and one for the handler code.

Viewing the traces

Now we have everything joined up, we can start looking at some traces to give ourself an idea of what X-Ray can do for us. Looking at the trace list, one entry jumps out as being considerably slower than the rest.

Clicking on this, we can see the following trace. We can also see the reason for why it is slower, as what we can see is a cold start in action.

If we look at a quicker trace, we can see a warmed-up Lambda function.

This small example gives a flavour of the insight that these traces can provide.

SNS and SQS

The CDK Cloud Test Kit also contains a couple of examples using SNS and SQS. This gives us a chance to add X-Ray to those and see what happens.

For the SNS example, we wrap the SNSClient in the X-Ray middleware.

const sns = AWSXRay.captureAWSv3Client(new SNSClient({}));

Running the tests, we see the following service map. This clearly shows the structure of the application, where one Lambda function publishes events to one of two topics and two Lambda functions subscribe as observers.

What I also noted, was that the relative sizes indicates the weight of traffic through the system. This could be useful to see if the flow within your application is as your would expect.

For SQS, we again wrap the client as we have done before.

const sqs = AWSXRay.captureAWSv3Client(new SQSClient({}));

Now when we run the tests, we see the following in X-Ray.

The interesting thing here is that the view includes metrics along with the service map. Two of the circles indicate that error metrics were recorded. Now if we select them, we get the option to view filtered traces.

Clicking on this we get a list of traces where errors occurred.

Selecting one, we go straight to the logs and we can see the error.

Hopefully, this gives you some idea of how X-Ray can help bring together traces, metrics, and logs. Allowing you to identify errors and get to the relevant logs, in order to debug issues quickly.

Step Functions

The final example that we will instrument with X-Ray is one that contains a step function. The step function implements a process that obtains a credit rating and then decides whether to accept or decline a loan. Where errors occur, a message is placed on an SQS queue. The step function definition is shown below. We can see that it includes integrations with SNS and SQS.

As well as setting the appropriate property on each Lambda function, we also need to set the tracingEnabled property to true on our step function.

export interface StateMachineProps {
    // <snip>
    /**
     * Specifies whether Amazon X-Ray tracing is enabled for this state machine.
     *
     * @default false
     */
    readonly tracingEnabled?: boolean;
    // <snip>
}

With this in place, we can deploy the updated example and run our tests that exercise all the routes through the step function. The result in X-Ray is shown below.

Here we can see the step function integrations with Lambda, SNS, SQS, and DynamoDB. We can see the traces go through Lambda, SNS, and SQS, but stop at DynamoDB. Although our example observes DynamoDB events, and we can see the Lambda function elsewhere in the service map, the context is lost as soon as the record is written.

Again, we can see the metrics overlaid on the service map. The tests include some error scenarios, and the resulting metrics are reflected in the highlighting of the services. Drilling into the step function service, we can view the error trace and we see the following.

Here we can clearly see the retry behaviour occurring, before the step function errors.

Summary

In this post, we looked at how we can use X-Ray to instrument a variety of services. Whilst it was simple to do, it was invasive. As in, we had to change both the CDK code and the Lambda function code. Although I haven't tried them, I believe that some of the third-party offerings avoid such changes. We also need to be aware of the cost of using X-Ray, the pricing of which can be found at AWS X-Ray pricing.

Although I feel I have only just scratched the surface, I can see how powerful X-Ray can be in the way it combines the service map, traces, metrics, alarms, and logs. In the second part of the series, I will take what I have found so far and try instrumenting my Loan Broker example application.

What is encapsulation

The following links provide an admirable overview of the concept:

• Stackify: What is Encapsulation
• Wikipedia: Encapsulation

The following in particular sums up the idea:

Under the definition that encapsulation "can be used to hide data members and member functions", the internal representation of an object is generally hidden from view outside of the object's definition. Typically, only the object's own methods can directly inspect or manipulate its fields. Hiding the internals of the object protects its integrity by preventing users from setting the internal data of the component into an invalid or inconsistent state. A supposed benefit of encapsulation is that it can reduce system complexity, and thus increase robustness, by allowing the developer to limit the interdependencies between software components

Encapsulation is the default

The majority of my object-oriented programming experience is in Java and C#. In both these languages, the default visibility for class members is non-public. That is, they cannot be access by any class without a special relationship with that class. Such relationships are a sub-class or a class in the same packaging unit. This implicitly pushes the developer to conceal as much as possible.

When I encountered TypeScript, I was surprised to find that the opposite was true. I had to explicitly make any members private. I had to consciously make the decision to hide them from potential users of the class.

When encapsulation gets in the way

This subtle change made me question my default thinking. This was coupled with my experience of using a third-party component in a way similar to the following hypothetical code:

const myClient = new ServiceClient({ region: 'eu-west-2'});

// ...

myClient.region; // Not accessible now, although it was provided earlier

The component was hiding information that I had provided. What was the point of doing this? How was I going to misuse this information? Now I had to pass around the information that existed inside another parameter that was passed with it.

I also had the experience of trying to subclass a class I had published as part of an npm package. I had diligently hidden everything deemed 'not essential', but now had cut off the ability to extend it.

All this caused me to further question how I thought about member visibility.

Don't throw the baby out with the bathwater

Should we just make everything public? Of course not. Anything public forms part of the contract of your class and you should be committed to honouring that as best you can. Once you have published, then breaking that contract could result in very unhappy clients. If you are using Domain-driven Design, then you will also need to implement some business rules to keep your domain objects consistent. Again, encapsulation plays a vary valuable role here.

What I am thinking about here are primarily read-only properties that have been hidden without thought. Perhaps because the language made that the default.

Sometimes privacy is just a facade

As it turns out, sometimes privacy is just a facade. Take C# for example, where you can define this seemingly well-encapsulated class.

class MyEncapsulatedClass
{
    private int PrivateProperty { get; set; }

    public MyEncapsulatedClass(int myPropertyValue)
    {
        PrivateProperty = myPropertyValue;
    }
}

However, using reflection you can still access the supposedly-private value.

var myEncapsulatedInstance = new MyEncapsulatedClass(666);

var myEncapsulatedClass = typeof(MyEncapsulatedClass);
var privateProperty =
    myEncapsulatedClass.GetProperty(
        "PrivateProperty", BindingFlags.NonPublic | BindingFlags.Instance);
var privatePropertyValue =
    privateProperty.GetValue(myEncapsulatedInstance, null);

Console.WriteLine("privatePropertyValue=" + privatePropertyValue);

I concede that you have to do some work here to get the value. The point I am trying to make is that you might want to check your language before relying on encapsulation for anything security-related.

Summary

My experience with TypeScript's public-by-default approach led me to question my default position of hiding everything possible in a class. This was combined with my experience of being frustrated with using libraries that hid information unnecessarily.

Now for utility libraries or packages, I try to keep things as open as possible. Where properties are passed in on construction, I try to make them public and read-only. Internal structures are made private by conscious choice, only if exposing them would make for a fragile contract.

When implementing Domain-driven Design business rules, encapsulation becomes necessary in order to enforce those rules and keep the domain model consistent. Again, this is a conscious choice.

What I also learned, is that trying different programming languages can make you think differently and challenge your own assumptions and habits. This can only be a good thing.

DynamoDB clients

As we saw in the first part of this series, converting from V2 to V3 can be as straightforward as changing the type of service client, then using that client to send a command rather than invoking a method.

So the following:

readonly sns: AWS.SNS;
...
const publishInput: PublishInput = {
  Message: JSON.stringify(message),
  TopicArn: this.topicArn,
  MessageAttributes: messageAttributes,
};

await this.sns.publish(publishInput).promise();

Becomes:

readonly sns: SNSClient;
...
const publishInput: PublishInput = {
  Message: JSON.stringify(message),
  TopicArn: this.topicArn,
  MessageAttributes: messageAttributes,
};

await this.sns.send(new PublishCommand(publishInput));

The same applies for DynamoDB, there is a DynamoDBClient class in the @aws-sdk/client-dynamodb package. As expected, this can be used to send commands to put and get items. However, unlike the AWS.DynamoDB.DocumentClient class in the V2 SDK, the DynamoDBClient class only deals with objects in the low-level DynamoDB format. As you can see from the example below, this is a JSON format that uses typed objects for properties:

{
  "Item": {
    "Age": { "N": "8" },
    "Name": { "S": "Rover" },
    "Breed": { "S": "Beagle" },
    "AnimalType": { "S": "Dog" }
  }
}

I decided to do the modern thing and ask ChatGPT if there was a way to convert the output of DynamoDBClient. To its credit it did point me in one of the possible directions.

If you are using the query method of the DynamoDBClient object to query a DynamoDB table in JavaScript, you will need to unmarshall the output to convert the raw DynamoDB item format into a more usable JavaScript format. The output of the query method is returned in the native DynamoDB item format, which is a complex JSON structure that includes data types and other metadata. To work with the data in a more convenient way, you will need to unmarshall the output using the unmarshall function provided by the @aws-sdk/util-dynamodb module. Here's an example code snippet that includes unmarshalling of the query results:

// Import the required modules from the AWS SDK for JavaScript V3
import { DynamoDBClient, QueryCommand } from "@aws-sdk/client-dynamodb";
import { unmarshall } from "@aws-sdk/util-dynamodb";

// Set the AWS region and create a new DynamoDB client object
const REGION = "us-east-1";
const dynamodbClient = new DynamoDBClient({ region: REGION });

// Set the parameters for the query
const params = {
  TableName: "my-table",
  KeyConditionExpression: "partitionKey = :pk",
  ExpressionAttributeValues: {
    ":pk": { S: "my-partition-key" },
  },
};

// Create a new QueryCommand object and execute the query
const command = new QueryCommand(params);
const response = await dynamodbClient.send(command);

// Unmarshall the results
const items = response.Items.map((item) => unmarshall(item));

// Output the results to the console
console.log(items);

Although this is correct, and the unmarshall function will come in useful later, this isn't the route I ended up taking. The V3 SDK uses a middleware-based approach and the DynamoDBClient can be wrapped with middleware to do the marshalling as part of the pipeline.

This approach is explained in Using the DynamoDB Document Client. This shows how the @aws-sdk/lib-dynamodb package can be used as follows.

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb";
...
// Wrap a DynamoDBClient instance
const documentClient = DynamoDBDocumentClient.from(new DynamoDBClient({ region }));
...
const queryOutput = await this.documentClient.send(
  new QueryCommand(queryParams)
);

// Return unmarshalled objects
return queryOutput.Items;

One thing to be aware of is that DynamoDBDocumentClient does not support all the same commands as DynamoDBClient. So you might need to have an instance of the latter available as well as the wrapped version.

In part of the codebase, a DynamoDB stream event is used to retrieve the corresponding item from the table. One thing I found was that the stream AttributeValue appears to no longer be compatible with DynamoDB version. To get round this, I had to add an explicit cast. It was here that the unmarshall function came in useful, as the key is returned in the low-level JSON format.

import { AttributeValue, DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { AttributeValue as StreamAttributeValue } from 'aws-lambda/trigger/dynamodb-stream';

async getItemByEventKeyAsync<T>(
    eventKey: { [key: string]: StreamAttributeValue } | undefined
  ): Promise<T | undefined> {
    //
    if (eventKey === undefined) {
      return undefined;
    }

    // Cast to prevent: 'AWSLambda.AttributeValue' is not assignable to type 'DynamoDB.AttributeValue'
    const key = unmarshall(eventKey as Record<string, AttributeValue>);

    return getItem(this.region, this.tableName, key) as unknown as T;
  }

S3

One of the examples in the codebase being converted used pre-signed URLs to pass data. It turns out that pre-signing has changed with the V3 SDK. There is now a separate package (s3-request-presigner) that you need to reference to produce a URL for a V3 command.

import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
...
const s3Params = {
  Bucket: bucketName,
  Key: key,
};
...
const signedCommand = new GetObjectCommand(s3Params);
const signedUrl = await getSignedUrl(s3, signedCommand, {
  expiresIn: expirySeconds ?? 60,
});

Lists now can return undefined

Another thing that I noticed as part of the conversion process was that lists returned by the APIs can now be undefined. Below is an example where step function executions are being listed.

const { executions } = await stepFunctions.listExecutions(opts).promise();
if (executions.length > 0) {
  const newestRunning = executions[0];

When converting, I had to add an extra test to cater for the possibility of undefined.

const { executions } = await sfnClient.send(new ListExecutionsCommand(opts)); // Can be undefined
if (executions && executions.length > 0) {
  const newestRunning = executions[0];

Invoking Lambda functions

Another small quirk that emerged from my conversion was that I needed to encode/decode the payloads when invoking a Lambda function. The Payload is now returned as a Uint8Array, so we need to use a TextEncoder to convert from and to JSON objects.

Here we encode the stringify-ed JSON object:

const encoder = new TextEncoder();
const lambdaPayload = request ? { Payload: encoder.encode(JSON.stringify(request)) } : {};

And here we decode it before parsing:

const decoder = new TextDecoder();
return JSON.parse(decoder.decode(Payload));

Discoverability thoughts

As part of the conversion process, I encountered the following code that I had in place to reuse connections in Node.js.

const documentClient = new DocumentClient({
  httpOptions: {
    agent,
  },
});

My thought was to navigate to the definition of the new options and look for something similar. However, I quickly found myself lost.

constructor(configuration: DynamoDBClientConfig);

Led to...

export interface DynamoDBClientConfig extends DynamoDBClientConfigType {
}

Which led to...

type DynamoDBClientConfigType = Partial<__SmithyConfiguration<__HttpHandlerOptions>> & ClientDefaults & RegionInputConfig & EndpointInputConfig<EndpointParameters> & RetryInputConfig & HostHeaderInputConfig & AwsAuthInputConfig & UserAgentInputConfig & EndpointDiscoveryInputConfig & ClientInputEndpointParameters;

At which point I stopped and searched for 'aws sdk V3 keep-alive' and found Reusing connections with keep-alive in Node.js

This allowed me to rewrite the original as follows:

const documentClient = DynamoDBDocumentClient.from(
  new DynamoDBClient({
    requestHandler: new NodeHttpHandler({
      httpAgent: agent,
    }),
  })
);

I appreciate there is a good reason for how the options are now defined, but I do feel it has affected discoverability via the definition. I just need to remember to fall back on search and AI chatbots.

As it turns out, this 'keep alive' code is not needed any more. See HTTP keep-alive is on by default in modular AWS SDK for JavaScript

The middleware-based approach

As touched on in the DynamoDB section, the V3 SDK uses a middleware-based approach. We saw it when we wrapped a DynamoDBClient instance in a DynamoDBDocumentClient instance.

const documentClient = DynamoDBDocumentClient.from(new DynamoDBClient({ region }));

The article What's the AWS SDK for JavaScript? explains how you can create your own customisations.

In V3, you can use a new middleware stack to control the lifecycle of an operation call. Each middleware stage in the stack calls the next middleware stage after making any changes to the request object.

It goes on to give the following example of adding a custom header to a Amazon DynamoDB client.

dbClient.middlewareStack.add(
  (next, context) => args => {
    args.request.headers["Custom-Header"] = "value";
    return next(args);
  },
  {
    step: "build"
  }
);

dbClient.send(new PutObjectCommand(params));

This approach, coupled with the ability to have a smaller bundle size, helped me understand the change in approach in the V3 SDK. On the surface, the changes looked a bit like unnecessary complication.

Summary

In this post, we looked at the challenges that I had when converting code for DynamoDB, Step Functions, S3, and Lambda functions and how I solved them. Hopefully, my experience can help others. In the main, the process was quite painless. However, my codebase was small and I had integration tests to verify the changes in the cloud.

If you have many unit tests that mock the older SDK, then your challenges may be greater than mine. Personally, I would try to avoid mocking at that level in the first place, but that might be a subject for another post.

It is well worth knowing that there is much improved documentation in Developer Preview. This documentation is searchable and goes beyond the original auto-generated version and includes code samples.

For those that like to look at code, here are the links to the resulting pull requests from my upgrading:

• Cloud test kit PR
• Loan Broker example PR

Links

• AWS SDK for JavaScript V3
• AWS SDK for JavaScript - Developer Preview
• Migrating your code to SDK for JavaScript V3
• aws-sdk-js-codemod
• Setting up the SDK for JavaScript
• DynamoDB marshalling
• Using the DynamoDB Document Client

Up until Node.js 16, Lambda’s Node.js runtimes have included the AWS SDK for JavaScript version 2. This has since been superseded by the AWS SDK for JavaScript version 3, which was released in December 2020. With this release, Lambda has upgraded the version of the AWS SDK for JavaScript included with the runtime from v2 to v3.

I also noted that when doing development, I was being nagged as follows.

(node:16616) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy

The announcement and the nag sufficiently motivated myself to look at my CDK Cloud Test Kit and make the leap from SDK v2 to v3, whilst documenting my experience along the way.

TL;DR

• Use the service clients for smaller bundle size and future-proofing
• aws-sdk-js-codemod works OK, but the result may be deprecated
• CDK hotswap doesn't update the Node.js version

Upgrade approach

My first thought was to question how should I approach the process of upgrading. Should I uninstall the aws-sdk package, see what breaks, then fix it all up? Or should I take a more step-by-step approach? Ultimately, I will need to uninstall the aws-sdk package to be sure I have amended all references, but to keep things manageable I decided to tackle the functionality service by service.

When identifying what needed to change, I noted that my codebase was not consistently explicit in the Node.js version being used. The reason for this was that the code used the NodejsFunction CDK construct and the default value for runtime is NODEJS_14_X.

export interface NodejsFunctionProps extends FunctionOptions {
    /**
     * @default Runtime.NODEJS_14_X
     */
    readonly runtime?: lambda.Runtime;
}

With hindsight, in future I would favour being explicit with the runtime version. I think defaults have their place, but I feel such a key dependency deserves to have full visibility.

SNS first

As CodeWhisperer has just been released for personal use, I decided to give it a try with the prompt 'Publish a message to the selected output topic using the javascript sdk v3' and got the following result.

// Publish a message to the selected output topic using the javascript sdk v3
const command = new PublishCommand({
    TopicArn: outputTopicArn,
    Message: JSON.stringify(numbersEvent),
}
);

const result = await sns.send(command);

console.log(JSON.stringify({ result }, null, 2));

However, when I ran my cloud-based tests, I got the failure below:

  SimpleEventRouter Test Suite
spec.js:54
    1) Routes positive sums
    <snip>
  0 passing (1m)
base.js:379
  5 failing

Looking in CloudWatch, I could see that the @aws-sdk/client-sns package could not be found.

2023-05-07T07:23:27.329Z  undefined ERROR Uncaught Exception
{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '@aws-sdk/client-sns'\nRequire stack:\n- /var/task/index.js\n- /var/runtime/UserFunction.js\n- /var/runtime/Runtime.js\n- /var/runtime/index.js",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module '@aws-sdk/client-sns'",
        "Require stack:",
        <snip>
    ]
}

The reason for this turned out to be that I was using the --hotswap option with cdk deploy. This updates the Lambda function code, but not the runtime. As @aws-sdk/client-sns is not bundled with the Node.js 14 runtime, we get the error above.

When a full deployment was done, we got the happy sight of all the tests passing. As these are cloud-based, I have a high-confidence in a successful migration.

  SimpleEventRouter Test Suite
spec.js:54
    √ Routes positive sums (4175ms)
    <snip>
  5 passing (13s)

SQS next with aws-sdk-js-codemod

The AWS documentation on upgrading mentions a package called aws-sdk-js-codemod. To quote the README, 'This repository contains a collection of codemod scripts for use with JSCodeshift that help update AWS SDK for JavaScript APIs.' This sounded promising, so I decided to give it a go.

I followed the instructions and ran the following, pointing at TypeScript file with SQS references.

npx aws-sdk-js-codemod -t v2-to-v3 PATH...

The results can be seen below.

This all look reasonable. I needed to install the @aws-sdk/client-sqs package and to add an empty configuration to the SQS constructor (new SQS({});), but after that I was able to deploy and test.

  SimpleMessageRouter Test Suite
spec.js:54
    √ Routes as expected: {"values":[],"isExpectedPositive":true} (4178ms)
    <snip>
  6 passing (28s)

So success, but the style looks a little different from the SNS.

Why does codemod SQS code differ from the SNS code?

I decided to ask CodeWhisperer how to send an SQS message using the v3 SDK and got another way.

// Send an SQS message using v3 sdk
const sendMessageRequest: AWS_SQS.SendMessageRequest = {
    QueueUrl: outputQueueUrl,
    MessageBody: JSON.stringify(numbersMessage),
};
await sqs.sendMessage(sendMessageRequest);

I was getting, if not confused, a little intrigued by these alternatives. Reading the AWS documentation, I could see that it points you down the SQSClient route as shown below.

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs"; // ES Modules import
const client = new SQSClient(config);
const input = {
  // SendMessageRequest
  QueueUrl: "STRING_VALUE", // required
  MessageBody: "STRING_VALUE", // required
};
const command = new SendMessageCommand(input);
const response = await client.send(command);

I tried this approach and, unsurprisingly, this worked as well. So now, we have three possible ways:

• Use SQS.sendMessage()
  • With SendMessageCommandInput
  • With SendMessageRequest
• Use SQSClient.send() with SendMessageCommand

SendMessageCommandInput turns out to be a subclass of SendMessageRequest as documented here.

So, which to use?

I dug a little further into the documentation and found this on the 'v2 compatible style' (highlighting my own).

The client can also send requests using v2 compatible style. However, it results in a bigger bundle size and may be dropped in next major version. More details in the blog post on modular packages in AWS SDK for JavaScript

The key takeaway for me here is that if you take the easiest approach now with your codebase, then you may face another round of updates if your application sees out the support lifetime of SDK v3. I noted that this is the approach that the aws-sdk-js-codemod defaults to, so that is something to bear in mind.

Summary

So far, the process of updating has been pretty painless. Admittedly, I have been tackling a small codebase and only two AWS services. On a larger codebase that has not wrapped the AWS services in more domain-level abstractions, then this could be quite a task. Especially if it is not easy to exercise the code thoroughly in the cloud.

In the next post, I will move on to updating the rest of the AWS services being used, including DynamoDB and the marshalling challenge.

After posting, it was brought to my attention that there is another way. I have added an update at the end to cover this approach.

The Problem

I wanted to create a class to build queries for DynamoDB. DynamoDB is a NoSQL database that indexes each item by two keys. A partition key and a sort key. When querying a DynamoDB table you always supply a partition key, and you optionally supply a sort key along with an operator such as 'greater than'. Another option is to supply two sort key values to provide a range.

The C# Solution

In C#, we could define as follows:

enum SortKeyOperator
{
    EQUALS,
    LESS_THAN,
    LESS_THAN_OR_EQUAL,
    GREATER_THAN_OR_EQUAL,
    GREATER_THAN,
    BEGINS_WITH,
}

class QueryBuilder
{
    public void Build(string partitionKeyValue)
    {
    }

    public void Build(
        string partitionKeyValue,
        string sortKeyValue)
    {
    }

    public void Build(
        string partitionKeyValue,
        SortKeyOperator sortKeyOperator,
        string sortKeyValue)
    {
    }

    public void Build(
        string partitionKeyValue,
        string sortKeyFromValue,
        string sortKeyToValue)
    {
    }
}

This is allowed, as the combination of parameters means that each method signature is unique, even though the name of the method is not. When using the Visual Studio IDE the intellisense prompts as follows.

This allows the user to scroll through the various overloaded versions of the method.

By adding documentation to the methods, you can clearly communication the intended use of each overload.

TypeScript Attempt No.1 - Separate Methods

The simplest way I could think of to try to replicate method overloading is to have separate methods that share a common prefix. In this case, buildWith. The result is as follows:

class QueryBuilder {
  buildWithPartitionKeyOnly(partitionKeyValue: string) {...}

  buildWithSortKey(partitionKeyValue: string, sortKeyValue: string) {...}

  buildWithComparison(
    partitionKeyValue: string,
    sortKeyOperator: SortKeyOperator,
    sortKeyValue: string
  ) {...}

  buildWithRange(
    partitionKeyValue: string,
    sortKeyFromValue: string,
    sortKeyToValue: string
  ) {...}
}

This would result in the following prompt when using VS Code:

I actually think this approach has some merit. The explicit naming provides some level of self-documentation. A downside is that the underlying implementation might need either some duplication in the separate methods, or some common code outside them.

TypeScript Attempt No.2 - Optional Parameters

Another approach is to use optional parameters and deconstructed parameters. We can define a single method with a single object parameter, and we can make the sort key parameters all optional. The result is as follows:

build({
  partitionKeyValue,
  sortKeyValue,
  sortKeyComparison,
  sortKeyRange,
}: {
  partitionKeyValue: string;
  sortKeyValue?: string;
  sortKeyComparison?: {
    operator: SortKeyOperator;
    value: string;
  };
  sortKeyRange?: {
    fromValue: string;
    toValue: string;
  };
}) {
  if (sortKeyValue) {
    // Handle case where we match by value equality
  } else if (sortKeyComparison) {
    // Handle case where we match by comparison
  } else if (sortKeyRange) {
    // Handle case where we match by range
  } else {
    // Handle case where we match just by primary key
  }
}

Whilst this works, it isn't obvious to the caller what combination of parameters should be used to get the various outcomes. For example, can sortKeyRange be used with sortKeyValue? The only way to know this, is to look inside the method. Not ideal. Can we do better?

TypeScript Attempt No.3 - Naive Discriminated Types

TypeScript allows you to define that a value can be one of a set of types, for example:

var v: number | string;

Can we take advantage of this to give the callers of the method a set of exclusive choices, so that they do not need to look inside the method to work out how to use it?

Below was my first effort:

build({
  partitionKeyValue,
  sortKeyCriteria,
}: {
  partitionKeyValue: string;
  sortKeyCriteria?:
    | {
        value: string;
      }
    | {
        comparison: {
          operator: SortKeyOperator;
          value: string;
        };
      }
    | {
        range: {
          fromValue: string;
          toValue: string;
        };
      };
}) {
  if (sortKeyCriteria) {
    if ('value' in sortKeyCriteria) {
      // Handle case where we match by value equality
    } else if ('comparison' in sortKeyCriteria) {
      // Handle case where we match by comparison
    } else if ('range' in sortKeyCriteria) {
      // Handle case where we match by range
    } else {
    }
  } else {
    // Handle case where we match just by primary key
  }
}

Here we use the in operator to work out which of the types has been specified. This all seemed to be working as I expected until I tried the following:

queryBuilder.build({
  partitionKeyValue: "pk",
  sortKeyCriteria: {
    value: "sortKeyValue",
    range: {
      fromValue: "sortKeyValue1",
      toValue: "sortKeyValue2",
    },
    comparison: {
      operator: SortKeyOperator.GREATER_THAN,
      value: "sortKeyValue",
    },
  },
});

I was expecting a compiler error, as I had specified all three options. However, clearly TypeScript does not work that way. What I had I done wrong?

TypeScript Attempt No.4 - Discriminated Types Done Properly

The solution came from an example in the TypeScript playground. What I needed to do was define a value that would discriminate the types. The result is as follows:

build({
  partitionKeyValue,
  sortKeyCriteria,
}: {
  partitionKeyValue: string;
  sortKeyCriteria?:
    | {
        type: 'value';
        value: string;
      }
    | {
        type: 'comparison';
        operator: SortKeyOperator;
        value: string;
      }
    | {
        type: 'range';
        fromValue: string;
        toValue: string;
      };
}) {
  if (sortKeyCriteria?.type === 'value') {
    // Handle case where we match by value equality
  } else if (sortKeyCriteria?.type === 'comparison') {
    // Handle case where we match by comparison
  } else if (sortKeyCriteria?.type === 'range') {
    // Handle case where we match by range
  } else {
    // Handle case where we match just by primary key
  }
}

Now when using the class in VS Code, when you select the type you get the corresponding options. For example, when using range you get prompted for the relevant fromValue and toValue values.

Now we have a single method that presents a set of mutually exclusive choices. The caller cannot get the parameters wrong and doesn't need to look inside the method.

Documentation

One downside to using deconstructed parameters is that I could not find a way to document them well using JSDoc. The best I could come up with was the following.

/**
 * Builds a query based on the key criteria supplied.
 * @param param0 Key criteria
 */
build({ partitionKeyValue, sortKeyCriteria }: {...}) {...}

This resulted in the following prompt in VS Code, which does give some indication of the options via the type values.

I may have been missing something, but for this reason I found myself quite liking the solution with separate methods. That approach was easy to document and also somewhat documented itself with the verbose names.

Here we can see that the individual parameters can be documented in the same way that they can be in the C# example, as shown below.

Summary

In this post we looked at various ways that we can implement some form of method overloading in TypeScript and compared these with an equivalent in C#. My feeling is that for internal libraries, I would favour the discriminated type approach. However, for external libraries, I feel that the ability to fully document means that the simple, multi-named approach would be better. Behind the scenes, these methods may map onto a single discriminated type method, in order to keep functionality together.

Update

After posting this, it was pointed out to me that there is another way that we can implement method overloading in TypeScript.

The way we can do it is by defining an args parameter with the JavaScript spread syntax and a list of possible parameters. In our example this would be the following:

build(
  ...args:
    | [partitionKeyValue: string]
    | [partitionKeyValue: string, sortKeyValue: string]
    | [
        partitionKeyValue: string,
        sortKeyOperator: NumericSortKeyOperator,
        sortKeyValue: string
      ]
    | [
        partitionKeyValue: string,
        sortKeyFromValue: string,
        sortKeyToValue: string
      ]
) {...}

In VS Code, this gives an experience very similar to that we had for C#:

We still have the problem of how to know which overload is being called. I solved this by building up a signature string from the arguments types and switching on the result.

const signature = args
  .map((arg) => typeof arg)
  .reduce((accumulator, argType) => `${accumulator}${argType}|`, '|');

switch (signature) {
  case '|string|':
    // Handle case where we match by partition key only
    break;
  case '|string|string|':
    // Handle case where we match by compound key
    break;
  case '|string|string|string|':
    // Handle case where we match by range
    break;
  case '|string|number|string|':
    // Handle case where we match by comparison
    break;
  default:
    throw new Error(`Unhandled signature`);
}

One thing I did have to change was the type of enum. Originally, it was a set of strings, but this would cause a clash of signatures. I changed it for a set of integers and this avoided the issue.

I used array destructuring to access the values as follows:

case '|string|number|string|':
  // Handle case where we match by comparison
  {
    const [partitionKeyValue, sortKeyOperator, sortKeyValue] = args;
    // Call the method implementation
  }
  break;

Whilst TypeScript does infer the types, it does not discriminate. So it can only assert that some values are one of a set:

This approach suffers from the documentation issue that other advanced approaches do. My feeling overall is that, although it gives a similar intellisense experience, it falls down when implementing the underlying functionality and I would still be tempted to go down the explicit naming route with discriminated types underneath.

The application in question acts as a loan broker, it receives a request containing the details of the loan required via an API, and then returns the best rate to a webhook.

The following diagram shows how we use a central EventBridge event bus to implement this application using an event-driven design.

The code for this post can be found in the accompanying GitHub repo.

Business metrics vs. System metrics

By default, AWS outputs a large number of metrics that you can use to visualise and monitor the health of your application. For example, the Working with Lambda function metrics AWS documentation page goes into the detail of what is outputted by default by Lambda functions. You get invocation metrics, such as the number of times that your function code is invoked, performance metrics, such as the amount of time that your function code spends processing an event, and concurrency metrics, such as the number of function instances that are processing events.

These provide an invaluable insight into the health of your application, but don't necessarily answer business-level questions such as, "How many quotes are we receiving per hour?", "How fast are we processing them?", or "How many rates are we receiving from lender X?". Whilst AWS provides system metrics, to answer these business-level questions we need business metrics.

This is where we can take advantage of our event-driven architecture. The application is already producing events such as the following:

• QuoteSubmitted
• CreditReportFailed
• LenderRateReceived

What we can do is subscribe to these events and translate them into custom business metrics. We can then build dashboards, alarms, and whatever else we want on top of those metrics.

Decoupling observability

In the first part of the series, we considered using SQS as part of our messaging technology. One limitation of SQS is that each message can only be processed by one consumer. Our use of EventBridge has the advantage that we can subscribe to any business events without affecting any existing processing. This means we can add an observability stack entirely independently of the existing application. This demonstrates the high-level of decoupling that can be achieved and the extensibility you get with an event-driven architecture.

The diagram below shows how we are going add business observability to our architecture. As you can see, it simply plugs into the event bus.

Simple logging and Logs Insights

One simple way to turn the business events into a searchable resource is to log our business events in a structured way. This can be done via the following single-line Lambda function.

export const handler = async (
  event: EventBridgeEvent<'DomainEventBase', DomainEventBase>
): Promise<void> => {
  console.log({ ...event.detail.metadata, data: event.detail.data });
};

This will result in log entries such as the following.

{
  "eventType": "QuoteSubmitted",
  "eventVersion": "1.0",
  "correlationId": "ac702216-ac35-48cb-be3c-26beb523897e",
  "requestId": "35fb604e-c56b-457b-97a8-fbdae9fd0644",
  "eventId": "8e71aecb-839f-4477-a9db-5353e1f23d04",
  "domain": "LoanBroker",
  "service": "RequestApi",
  "timestamp": "2023-01-28T15:05:16.237Z",
  "data": {
    "quoteReference": "2023-01-28-RHJ9YUL71",
    "quoteRequestDataUrl": "https://requestapistack-requestapibucket...-Amz-SignedHeaders=host"
  }
}

We have flattened the metadata about the event, such as the correlation id and event type, and also included the actual data for the event, such as the quote reference. In the second part of the series, we designed our domain events to be self-contained and to be made up from metadata about the event and the event data itself. We are taking advantage of the consistency here to output an easily searchable entry.

We also made another choice when designing out domain events. That was to pass large or sensitive data as time-limited presigned URLs. This choice means that we are free to log the events without the risk of logging sensitive information by accident.

Now we have our structured logging Lambda function, we can look at how we hook it up to monitor our application. To do this we create a CDK construct as shown below.

export interface ObserverProps {
  loanBrokerEventBus: EventBus;
}

export default class Observer extends Construct {
  constructor(scope: Construct, id: string, props: ObserverProps) {
    super(scope, id);

    const loggerFunction = new NodejsFunction(this, 'Logger');

    const domainEventRule = new Rule(this, id, {
      eventBus: props.loanBrokerEventBus,
      eventPattern: {
        detail: {
          metadata: {
            domain: [EventDomain.LoanBroker],
          },
        },
      },
    });

    domainEventRule.addTarget(new LambdaFunction(loggerFunction));
}

The construct properties allow us to pass in the event bus to subscribe to. We create our Lambda function, along with a rule that will listen to all domain events in the application. Finally, we add our Lambda function as the target for the rule.

The final step is to define a stack and add it to the application.

export interface ObservabilityStackProps extends StackProps {
  loanBrokerEventBus: EventBus;
}

export default class ObservabilityStack extends Stack {
  constructor(scope: Construct, id: string, props: ObservabilityStackProps) {
    super(scope, id, props);

    new Observer(this, 'Observer', {
      loanBrokerEventBus: props.loanBrokerEventBus,
    });
  }
}

new ObservabilityStack(app, 'ObservabilityStack', {
  loanBrokerEventBus: messagingStack.loanBrokerEventBus,
});

With this Lambda function in place, we can now use CloudWatch logs to see all the domain events in a single place.

We can take advantage of log filtering to provide a more focussed view of the events.

However, we can do even better by using CloudWatch Logs Insights. To quote the article:

CloudWatch Logs Insights enables you to interactively search and analyze your log data in Amazon CloudWatch Logs. You can perform queries to help you more efficiently and effectively respond to operational issues. If an issue occurs, you can use CloudWatch Logs Insights to identify potential causes and validate deployed fixes.

This allows us to run a query such as the following and get a picture of the domain events flowing through the system.

As we log correlation ids, we can use these when we want to focus in on a particular request by adding criteria to our queries. The query below show how this is done.

Logging business metrics

As well as outputting structured logs, we can also use this subscription to publish custom Amazon CloudWatch metrics. By publishing custom metrics we can create dashboards and alarms.

To make our lives easier, we are going to use the AWS Lambda Powertools for TypeScript Metrics npm package. This will allow us to create custom metrics asynchronously by logging metrics to standard output following Amazon CloudWatch Embedded Metric Format (EMF).

The metric we are going to capture is a count of how many times the call to the credit report service has failed.

We start by declaring a Metrics instance outside of the Lambda function handler. We assign it a namespace and service name using exported constants, which we will use later for setting up an alarm.

import { Metrics, MetricUnits } from '@aws-lambda-powertools/metrics';

export const METRICS_NAMESPACE = 'LoanBroker';
export const METRICS_SERVICE_NAME = 'observer';

const metrics = new Metrics({
  namespace: METRICS_NAMESPACE,
  serviceName: METRICS_SERVICE_NAME,
});

The next step is to create the handler to publish the metric. First, we create a function to publish a count of all the failure events. Along with adding one to the overall count, we also publish metadata about the metric. This includes as much contextual information as possible, so that the resulting log entry can aid us in debugging what failed.

export const CREDIT_REPORT_FAILED_METRIC = 'creditReportFailed';

const publishCreditReportFailedMetrics = (
  creditReportFailed: CreditReportFailedV1
): void => {
  metrics.addMetric(CREDIT_REPORT_FAILED_METRIC, MetricUnits.Count, 1);

  addMetadata(creditReportFailed, {
    quoteReference: creditReportFailed.data.quoteReference,
    error: creditReportFailed.data.error,
    executionId: creditReportFailed.data.executionId,
    executionStartTime: creditReportFailed.data.executionStartTime,
    stateMachineId: creditReportFailed.data.stateMachineId,
  });

  metrics.publishStoredMetrics();
};

Now we have our function to publish the metric, we create a handler with a switch statement to route the event to our function.

export const handler = async (
  event: EventBridgeEvent<'DomainEventBase', DomainEventBase>
): Promise<void> => {

  switch (event.detail.metadata.eventType) {
    case EventType.CreditReportFailed:
      publishCreditReportFailedMetrics(event.detail as CreditReportFailedV1);
      break;
    default:
      break;
  }
};

Finally, we add a subscription using the domainEventRule we created earlier.

const measurerFunction = new NodejsFunction(
  this, 'Measurer', getNodejsFunctionProps()
);

domainEventRule.addTarget(new LambdaFunction(measurerFunction));

Now that we have a metric for our failures, we can create an alarm to notify ourselves when the failures are occurring. For the purpose of this post, we create a simple alarm that triggers if there is at least one failure in a five minute period.

const creditReportFailedCount = new Metric({
  namespace: OBSERVER_NAMESPACE,
  metricName: CREDIT_REPORT_FAILED_METRIC,
  dimensionsMap: {
    service: OBSERVER_SERVICE_NAME,
  },
}).with({
  statistic: 'sum',
  period: Duration.minutes(5),
});

creditReportFailedCount.createAlarm(this, 'CreditReportFailedAlarm', {
  evaluationPeriods: 1,
  comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
  threshold: 0,
  treatMissingData: TreatMissingData.NOT_BREACHING,
});

We can repeat this process for any of our domain events, creating metrics and alarms that allow us to observe the business performance of the application.

Deriving business metrics

Some business metrics directly correlate with individual business events, such as CreditReportFailed as we saw previously. However, there are some business metrics that do not. One example is the length of time is takes to process a quote. This process is asynchronous and, as a result, there is no single place within the application that could measure this duration.

One solution is to derive such a metric by a combination of events, in this case QuoteSubmitted and QuoteProcessed. By storing the events in a DynamoDB table, indexed by the request id, we can use this table to retrieve the corresponding event and derive the metric.

The first step is to extend the Lambda function that logs the event. In addition to the logging, we now write the event to a DynamoDB table.

export const handler = async (
  event: EventBridgeEvent<'DomainEventBase', DomainEventBase>
): Promise<void> => {
  console.log({ ...event.detail.metadata, data: event.detail.data });

  // Record the event in DynamoDB
  await requestEventTableClient.putEventAsync(event.detail);
};

An example for for a single request is shown below.

The primary key is the requestId and the sort key is a combination of the time the event was received by the logger and the eventId. This allows us to have a chronological view of the events for a particular request. In this example, we can see the requests for rates and their corresponding responses.

Now that we have a searchable log of the events, we can create a Lambda function that will use this to derive our metric. The code for this is shown below.

const publishQuoteProcessedMetricsAsync = async (
  quoteProcessed: QuoteProcessedV1
): Promise<void> => {

  // Retrieve the corresponding event for when the quote was submitted

  const [quoteSubmitted] = await requestEventTableClient.getEventsByType(
    quoteProcessed.metadata.requestId,
    EventType.QuoteSubmitted
  );

  // Calculate the duration

  const quoteSubmittedMillis = DateTime.fromISO(
    quoteSubmitted.metadata.timestamp
  ).toMillis();
  const quoteProcessedMillis = DateTime.fromISO(
    quoteProcessed.metadata.timestamp
  ).toMillis();

  const durationMillis = quoteProcessedMillis - quoteSubmittedMillis;

  // Publish the metric

  metrics.addMetric(
    QUOTE_PROCESSED_DURATION_METRIC,
    MetricUnits.Milliseconds,
    durationMillis
  );

  addMetadata(quoteProcessed, {
    quoteReference: quoteProcessed.data.quoteReference,
  });

  metrics.publishStoredMetrics();
};

By hooking this Lambda function up to the quoteProcessed event, it will start emitting metrics for the duration of processing quotes. We could extend this to derive and emit metrics for how long each lender takes to respond, or any other metric that requires correlating multiple events together.

Summary

In this post, we have seen how we can add business-level observability to our event-driven application. By taking advantage of the decoupled nature of an event-driven architecture, we were able to do this without touching the core application code at all.