Mysterious Pants

Reactive Rocket on Lambda

2021-07-21T00:00:00-07:00

This walkthrough shows how to use Rust on an AWS Lambda handling requests from AWS API Gateway V2, ostensibly to serve as the backend for a client application served through AWS S3 or AWS CloudFront. The AWS API Gateway V2 and static site in AWS S3 are connected to real domain names managed by AWS Route53. The overall architecture provides an inexpensive, highly-scalable infrastructure for web services, costing a few cents a month for the seldom used to hundreds for the heavily trafficked. This is all constructed and deployed with the AWS Cloud Development Toolkit, or CDK.

Cloud resources represent an enormous opportunity for hackers, as “free” data storage, compute, and simply mining your customer’s data. Rust’s strong consistency and security guarantees make it a challenging target for criminals to exploit as a vector into your cloud. In addition to security, Rust presents a particularly strong presence as the language of implementation for a Lambda, where execution speed translates directly into cost savings as billing is calculated by the millisecond, prorated to the nearest ten.

This guide requires the following already be installed and configured:

The Rust Programming Language.
A NodeJS interpreter. You need to be able to install NodeJS packages globally (npm i -g must not fail).
A Docker runtime, and you must be privileged to run containers on it (docker run --rm -it amazonlinux:latest bash -c 'yum install -y cowsay && cowsay Moo!' should not fail).

If you don’t want to follow along manually you can get started quickly with the end-product of this guide at this template - just create a new repository from the template, fill in some of the variables, and you’re good to go! You might still scan through this guide to see some of the rationale behind the decisions made if you’re not familiar with the technologies used.

Get Started with AWS

Get an AWS Account. Do the sensible thing, put MFA on the root account, create a new IAM user for yourself and use that instead (with MFA). Configure your local AWS CLI. Remember to rotate your keys regularly, I recommend putting a recurring appointment on your personal calendar to do so!

xpm@mysteriousnotebook ~/C> sudo apt-get install -y awscli
xpm@mysteriousnotebook ~/C> mkdir ~/.aws
xpm@mysteriousnotebook ~/C> cat << 'EOF' >> ~/.aws/credentials
[default]
aws_access_key_id=...
aws_secret_access_key=...
EOF
xpm@mysteriousnotebook ~/C> chmod go-rwx ~/.aws/credentials
xpm@mysteriousnotebook ~/C> cat << 'EOF' >> ~/.aws/config
[default]
region=us-west-1
output=json
EOF

Test that it works by hitting SSM Parameter Store, which requires credentials.

xpm@mysteriousnotebook ~/C> aws ec2 describe-images \
  --owners 137112412989 \
  --image-ids $(aws ssm get-parameters \
    --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-arm64-gp2 \
    | jq -r '.Parameters[0].Value') \
  | jq -r '.Images[0].Name'
amzn2-ami-hvm-2.0.20210701.0-arm64-gp2

Your AWS CLI is now configured. On to CDK.

Install CDK

xpm@mysteriousnotebook ~> npm i -g typescript aws-cdk@1.116.0
xpm@mysteriousnotebook ~> asdf reshim # if using asdf
xpm@mysteriousnotebook ~> cdk --version

Bootstrap CDK in your account. This will create some buckets and roles so that CDK can work with CloudFormation in your account.

AWS CloudFormation is a way of managing groups of related resources, called “Stacks.” CDK is a programmatic way of creating CloudFormation Stack definitions, uploading them to AWS CloudFormation, and having that create the resources described in the Stack.

xpm@mysteriousnotebook ~> cdk bootstrap aws://${AWS_ACCOUNT}/us-west-1
 ⏳  Bootstrapping environment aws://_/us-west-1...
CDKToolkit: creating CloudFormation changeset...
 ✅  Environment aws://_/us-west-1 bootstrapped.
xpm@mysteriousnotebook ~> aws cloudformation \
  describe-stack-resources --stack-name CDKToolkit
{
    "StackResources": [
        {
            "StackName": "CDKToolkit",
            "StackId": "arn:aws:cloudformation:us-west-1:_:stack/CDKToolkit/_",
            "LogicalResourceId": "StagingBucket",
            "PhysicalResourceId": "cdktoolkit-stagingbucket-_",
            "ResourceType": "AWS::S3::Bucket",
            "Timestamp": "2021-07-14T05:09:11.999Z",
            "ResourceStatus": "CREATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        },
        {
            "StackName": "CDKToolkit",
            "StackId": "arn:aws:cloudformation:us-west-1:_:stack/CDKToolkit/_",
            "LogicalResourceId": "StagingBucketPolicy",
            "PhysicalResourceId": "CDKToolkit-StagingBucketPolicy-_",
            "ResourceType": "AWS::S3::BucketPolicy",
            "Timestamp": "2021-07-14T05:09:14.829Z",
            "ResourceStatus": "CREATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

Now create a project. I’m calling mine Pax Imperia, after a domain name that I bought some time ago and have lying around unused, perfect for experimenting with!

xpm@mysteriousnotebook ~> mkdir Code/pax-imperia.com
xpm@mysteriosunodebook ~/C> cd Code/pax-imperia.com
xpm@mysteriousnotebook ~/C/pax-imperia.com> git init
xpm@mysteriousnotebook ~/C/pax-imperia.com> git branch -m mainline
xpm@mysteriousnotebook ~/C/pax-imperia.com> mkdir cdk && pushd cdk
xpm@mysteriousnotebook ~/C/p/cdk> cdk init app --language typescript
Applying project template app for typescript
# Welcome to your CDK TypeScript project!

This is a blank project for TypeScript development with CDK.

The `cdk.json` file tells the CDK Toolkit how to execute your app.

## Useful commands

 * `npm run build`   compile typescript to js
 * `npm run watch`   watch for changes and compile
 * `npm run test`    perform the jest unit tests
 * `cdk deploy`      deploy this stack to your default AWS account/region
 * `cdk diff`        compare deployed stack with current state
 * `cdk synth`       emits the synthesized CloudFormation template

Executing npm install...
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated sane@4.1.0: some dependency vulnerabilities fixed, support for node < 10 dropped, and newer ECMAScript syntax/features added
✅ All done!

Next we’ll install some dependencies, which are the pieces of the CDK library describing the tools we’ll be using.

aws-lambda is a “serverless” code execution engine. When your code needs to be executed it creates a small micro-virtual machine to run it; when there are no more requests, this micro-vm can be reaped. As a result you only pay for the time that it is actually in use - idle lambdas are completely free.
aws-api-gatewayv2 is a configurable proxy which can route HTTP traffic, in our case to a Lambda via the Lambda integration.
aws-logs allows us to configure the way the logs generated by the Lambda are retained. Lambda by default sends logs and metrics to CloudWatch, which gives you rich insights into how your service is running. CloudWatch logs cost money to retain, so let’s reduce that down to a more sensible retention period.

xpm@mysteriousnotebook ~/C/p/cdk> npm i \
  @aws-cdk/aws-{lambda,api-gatewayv2,api-gatewayv2-integrations,aws-logs}@1.116.0
xpm@mysteriousnotebook ~/C/p/cdk> popd

Now you have an empty CDK Stack at cdk/lib/cdk-stack.ts.

import * as cdk from '@aws-cdk/core';

export class CdkStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here
  }
}

Let’s customize it a little bit and add some custom properties to define the domain names that our website will use. If you’re not familiar with typescript, what we’re saying here is that the props property must be supplied and must have the attributes of CustomStackProps and the cdk.StackProps it originally had. We remove the ? to indicate that this can no longer be undefined. By working with TypeScript we can catch many kinds of misconfiguration right in the editor with the language server (or at build time), saving time.

import * as cdk from '@aws-cdk/core';

export interface CustomStackProps {
  domainName: string,
  wwwRecordName: string,
  apiRecordName: string,
}

export class CdkStack extends cdk.Stack {
  constructor(
      scope: cdk.Construct,
       id: string,
       props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);
    
    // The cloud is your oyster
  }
}

This will cause an error in cdk/bin/cdk.ts which can be remedied by specifying the attributes I just mentioned, like so, although I’m using a domain name that I have control over. You can substitute one of your own.

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '@aws-cdk/core';
import { CdkStack } from '../lib/cdk-stack';

const app = new cdk.App();
const [wwwRecordName, apiRecordName, domainName] = ['www', 'api', 'pax-imperia.com'];

new CdkStack(app, 'PaxImperiaInfaStack', {
  domainName, wwwRecordName, apiRecordName,
});

Add Some Rust

Next up we need some Rust to put in a Lambda. Let’s create that now.

xpm@mysteriousnotebook ~/C/pax-imperia.com> cargo new --bin api-lambda
     Created binary (application) `api-lambda` package
xpm@mysteriousnotebook ~/C/pax-imperia.com> pushd api-lambda

Immediately we need some new dependencies, and we’ll name the output program bootstrap, which is the name of the program that Lambda will always run.

Crates you should know!

Rocket is a Rust web framework.
Lambda Web is an adapter that takes AWS API Gateway shapes and dispatches them to local request handlers implemented by Rust web frameworks, such as Rocket, Actix, or Warp. It’s using the Lambda Runtime crate under the hood.

xpm@mysteriousnotebook ~/C/p/api-lambda> cat << 'EOF' > Cargo.toml
[package]
name = "api-lambda"
version = "0.1.0"
edition = "2018"
publish = false

[[bin]]
name = "bootstrap"
path = "src/main.rs"

[dependencies]
lambda-web = { version = "0.1", features=[ "rocket05" ] }
rocket = { version = "0.5.0-rc.1", features = [ "json" ] }
EOF

Let’s put a stubbed-out program in main.rs.

xpm@mysteriousnotebook ~/C/p/api-lambda> cat << 'EOF' > src/main.rs
use lambda_web::{is_running_on_lambda, launch_rocket_on_lambda, LambdaError};
use rocket::{self, get, routes};

#[get("/hello/<name>")]
fn hello(name: &str) -> String {
    format!("Hello, {}!", name)
}

#[rocket::main]
async fn main() -> Result<(), LambdaError> {
    let rocket = rocket::build().mount("/", routes![hello]);
    if is_running_on_lambda() {
        // Launch on AWS Lambda
        launch_rocket_on_lambda(rocket).await?;
    } else {
        // Launch local server
        rocket.launch().await?;
    }
    Ok(())
}
EOF
xpm@mysteriousnotebook ~/C/p/api-lambda> popd

Deploying a Rust Lambda

CDK can build and deploy our Lambda (yes, our Rust lambda!) for us. Update your infrastructure stack in CDK at cdk/lib/cdk-stack.ts.

Step by step, we specify our Rust version. CDK will build the binary for the lambda in a Docker container, so we have to specify the Rust version to pick the correct container. Next we specify the target toolchain, which for us is the musl variation of the Linux toolchain. MUSL breaks our dependency on a local glibc, as the Amazon Linux 2 microvm the Lambda is running will likely not have the same version of glibc that the Debian-derived Docker container has. Programs do strange things when run on versions of glibc they aren’t expecting.

If you do need to back out of using MUSL for some reason, such as a dependency on OpenSSL you can’t get out of, you can configure an Amazon Linux 2 Docker image (amazonlinux:latest) with the Rust toolchain and use that as a builder instead - it’ll give you the same userspace as the microvm variant we select, so it’ll work just fine.

Further down we create our API Gateway, backed by the Lambda. Specifying the V2 model version is important for the lambda_web crate, which doesn’t work with the other V1 type.

import * as cdk from '@aws-cdk/core';
import * as lambda from '@aws-cdk/aws-lambda';
import * as awslogs from '@aws-cdk/aws-logs';
import * as apigw from '@aws-cdk/aws-apigatewayv2';
import * as apigwi from '@aws-cdk/aws-apigatewayv2-integrations';

export interface CustomStackProps { /* ... */ }
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName } = props;
    const rustVersion = '1.53';
    const target = 'x86_64-unknown-linux-musl';
    const apiLambda = new lambda.Function(this, 'ApiHandler', {
      code: lambda.Code.fromAsset('../api-lambda', {
        bundling: {
          command: [
            'bash', '-c',
            `rustup target add ${target} && \
             cargo build --release --target ${target} && \
             cp target/${target}/release/bootstrap /asset-output/bootstrap`
          ],
          image: cdk.DockerImage.fromRegistry(`rust:${rustVersion}-slim`),
        }
      }),
      handler: 'main',
      runtime: lambda.Runtime.PROVIDED_AL2,
      logRetention: awslogs.RetentionDays.ONE_MONTH,
    });

    const apiGateway = new apigw.HttpApi(this, 'ApiEndpoint', {
      defaultIntegration: new apigwi.LambdaProxyIntegration({
        handler: apiLambda,
        payloadFormatVersion: apigw.PayloadFormatVersion.VERSION_2_0,
      }),
    });
  }
}

Run cdk deploy to put it into the cloud! CDK will build your lambda using Docker and then show you a table. This contains all the IAM Role grants you’re about the deploy. It’s a good idea to give this a study and make sure there isn’t anything overly-broad or unexpected. For example, a general rule is that you should never have an * in your IAM Resource position - always narrow it to a specific service. Security is job zero!

xpm@mysteriousnotebook ~/C/pax-imperia.com> pushd cdk
xpm@mysteriousnotebook ~/C/p/cdk> cdk deploy
Bundling asset PaxImperiaInfaStack/ApiHandler/Code/Stage...
info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'
    Updating crates.io index
 Downloading crates ...
  Downloaded parking_lot v0.11.1
  ...
  Downloaded encoding_rs v0.8.28
    Finished release [optimized] target(s) in 20.95s
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes
┌───┬───────────────────┬────────┬───────────────────┬───────────────────┬─────────────────────┐
│   │ Resource          │ Effect │ Action            │ Principal         │ Condition           │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼─────────────────────┤
│ + │ ${ApiHandler.Arn} │ Allow  │ lambda:InvokeFunc │ Service:apigatewa │ "ArnLike": {        │
│   │                   │        │ tion              │ y.amazonaws.com   │   "AWS:SourceArn":  │
│   │                   │        │                   │                   │ "arn:${AWS::Partiti │
│   │                   │        │                   │                   │ on}:execute-api:${A │
│   │                   │        │                   │                   │ WS::Region}:${AWS:: │
│   │                   │        │                   │                   │ AccountId}:${ApiEnd │
│   │                   │        │                   │                   │ pointA9C748C3}/*/*" │
│   │                   │        │                   │                   │ }                   │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼─────────────────────┤
│ + │ ${ApiHandler/Serv │ Allow  │ sts:AssumeRole    │ Service:lambda.am │                     │
│   │ iceRole.Arn}      │        │                   │ azonaws.com       │                     │
└───┴───────────────────┴────────┴───────────────────┴───────────────────┴─────────────────────┘
IAM Policy Changes
┌───┬───────────────────────────┬──────────────────────────────────────────────────────────────┐
│   │ Resource                  │ Managed Policy ARN                                           │
├───┼───────────────────────────┼──────────────────────────────────────────────────────────────┤
│ + │ ${ApiHandler/ServiceRole} │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambda │
│   │                           │ BasicExecutionRole                                           │
└───┴───────────────────────────┴──────────────────────────────────────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Do you wish to deploy these changes (y/n)? y
PaxImperiaInfaStack: deploying...
[0%] start: Publishing b40974ec3d9d3ea078e23b4f5bc7d3804bef22ef24234f3af8306d7be9d114be:current
[100%] success: Published b40974ec3d9d3ea078e23b4f5bc7d3804bef22ef24234f3af8306d7be9d114be:current
PaxImperiaInfaStack: creating CloudFormation changeset...

 ✅  PaxImperiaInfaStack

Stack ARN:
arn:aws:cloudformation:us-west-1:_:stack/PaxImperiaInfaStack/c42c5f90-e5f6-11eb-af32-_
xpm@mysteriousnotebook ~/C/p/cdk> popd

Finally we can use Curl to test that our Lambda is working.

xpm@mysteriousnotebook ~/C/pax-imperia.com> curl \
  https://1cmn8nk2jh.execute-api.us-west-1.amazonaws.com/hello/xpm
Hello, xpm!%      

A Static Website

While the main focus of this article is on Rust and AWS CDK, a critical component of the product stack is the static website that is vended to customers which uses the API Lambda. Here we shall use ReactJS, though you should feel free to substitute any other web stack you desire.

xpm@mysteriousnotebook ~/C/pax-imperia.com> npx create-react-app \
  react-website --template typescript

This gives us a basic React website to work with. The most interesting thing for the purpose of this walkthrough is that it can be built with the command npm run build and it places the built products into react-website/build.

xpm@mysteriousnotebook ~/C/p/react-website> npm run build

> react-website@0.1.0 build
> react-scripts build

Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  41.34 KB  build/static/js/2.d06a0c4f.chunk.js
  1.63 KB   build/static/js/3.44599fc6.chunk.js
  1.17 KB   build/static/js/runtime-main.c7bb1896.js
  596 B     build/static/js/main.71b8fcf2.chunk.js
  556 B     build/static/css/main.a617e044.chunk.css

The project was built assuming it is hosted at /.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.
You may serve it with a static server:

  npm install -g serve
  serve -s build

Find out more about deployment here:

  https://cra.link/deployment
xpm@mysteriousnotebook ~/C/p/react-website> ls build
asset-manifest.json  index.html   logo512.png    robots.txt
favicon.ico          logo192.png  manifest.json  static
xpm@mysteriousnotebook ~/C/p/react-website> popd

Deploying a Static Website

We can have CDK build and deploy the static website much the same way we had it build and deploy the Lambda. We’ll first need some additional CDK features.

xpm@mysteriousnotebook > pushd cdk
xpm@mysteriousnotebook > npm i @aws-cdk/aws-s3{,-deployment}@1.116.0

Now back to our infrastructure stack, let’s create a new open S3 bucket and a deployment task, that just like the Lambda earlier will take a Docker image and run a command in it to generate the website.

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as s3deploy from '@aws-cdk/aws-s3-deployment';

export interface CustomStackProps { /* ... */ }
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName } = props;

    // previous lambda code
      
    const websiteBucket = new s3.Bucket(this, 'ReactWebsiteBucket', {
      websiteIndexDocument: 'index.html',
      publicReadAccess: true,
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    new s3deploy.BucketDeployment(this, 'DeployWebsite', {
      sources: [
        s3deploy.Source.asset('../react-website', {
          bundling: {
            command: [
              'bash', '-c',
              'npm i && npm run build && cp -r build/. /asset-output/'
            ],
            image: cdk.DockerImage.fromRegistry('node:latest'),
          }
        })
      ],
      destinationBucket: websiteBucket,
    });
  }
}

Running cdk deploy will place the static files in a new S3 bucket. We can find and inspect the bucket:

xpm@mysteriousnotebook ~/C/p/cdk> aws cloudformation \
  describe-stack-resources --stack-name PaxImperiaInfaStack \
  | jq '.StackResources[] | select(.ResourceType == "AWS::S3::Bucket")'
{
  "StackName": "PaxImperiaInfaStack",
  "StackId": "arn:aws:cloudformation:us-west-1:_:stack/PaxImperiaInfaStack/c42c5f90-e5f6-11eb-af32-060036321973",
  "LogicalResourceId": "reactwebsitebucketEC162B96",
  "PhysicalResourceId": "paximperiainfastack-reactwebsitebucketec162b96-ljnqedr1ncwm",
  "ResourceType": "AWS::S3::Bucket",
  "Timestamp": "2021-07-21T05:39:14.832Z",
  "ResourceStatus": "CREATE_COMPLETE",
  "DriftInformation": {
    "StackResourceDriftStatus": "NOT_CHECKED"
  }
}
xpm@mysteriousnotebook ~/C/p/cdk> aws s3 ls \
  s3://paximperiainfastack-reactwebsitebucketec162b96-ljnqedr1ncwm/     
                           PRE static/
2021-07-20 22:50:53       1092 asset-manifest.json
2021-07-20 22:50:56       3870 favicon.ico
2021-07-20 22:50:56       3034 index.html
2021-07-20 22:50:56       5347 logo192.png
2021-07-20 22:50:56       9664 logo512.png
2021-07-20 22:50:57        492 manifest.json
2021-07-20 22:50:56         67 robots.txt

Further, I can view the app in the web browser. For the example bucket just created, that would be at http://paximperiainfastack-reactwebsitebucketec162b96-ljnqedr1ncwm.s3-website-us-west-1.amazonaws.com/.

Nice URLs with Route53

AWS Route53 is a DNS service. I’m going to connect a domain name I have lying around not doing anything, pax-imperia.com, to my AWS Account by adding it as a hosted zone. I keep my domains registered externally, so all that has to be done is to configure the nameservers for the domain to the AWS nameservers listed in the AWS Console when viewing the hosted zone.

To use the domain from our CDK Stack, we’ll need some more dependencies.

xpm@mysteriousnotebook ~/C/p/cdk > npm i \
  @aws-cdk/aws-{route53,route53-targets,certificatemanager}

Lookup the Hosted Zone, Certificate Management

Let’s create a new CDK Stack to hold our DNS constructs in cdk/lib/dns.ts.

import * as acm from '@aws-cdk/aws-certificatemanager';
import * as cdk from '@aws-cdk/core';
import * as route53 from '@aws-cdk/aws-route53';

export interface CustomStackProps {
  domainName: string,
}

export class DnsStack extends cdk.Stack {
  readonly zone: route53.IHostedZone;
  readonly cert: acm.ICertificate;

  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);
    const { domainName } = props;

    this.zone = route53.HostedZone.fromLookup(
      this, 'HostedZone', { domainName }
    );
    this.cert = new acm.Certificate(this, 'DomainCertificate', {
      domainName,
      validation: acm.CertificateValidation.fromDns(this.zone),
      subjectAlternativeNames: [`*.${domainName}`],
    });
  }
}

This stack grabs an existing hosted zone that you should create manually and exports it and an automatically generated certificate we can use in other stacks. I believe you should create the hosted zone manually because DNS is a very big foot-gun to leave up to cdk destroy which I guarantee you’ll do at least once while experimenting. Public SSL Certificates are free from AWS, and at time of writing Route53 is $0.50 for each hosted zone, so this all keeps in the spirit of building cool web services on a shoestring budget.

Now we should provide the hosted zone and certificate as properties in our main application stack.

import * as route53 from '@aws-cdk/aws-route53';
import * as route53Targets from '@aws-cdk/aws-route53-targets';
import * as acm from '@aws-cdk/aws-certificatemanager';

export interface CustomStackProps {
    zone: route53.IHostedZone,
    cert: acm.ICertificate,
    redirects?: boolean,
    /* ... */
}
    
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName, zone, cert } = props;
    
    // other constructs previously created in this walkthrough
  }
}

Now we should create the DNS Stack and wire in the constructs to the application stack in cdk/bin/cdk.ts. We also need to include an account number and region in the stack properties in order to work with Route53.

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '@aws-cdk/core';
import { CdkStack } from '../lib/cdk-stack';
import { DnsStack } from '../lib/dns-stack';

const app = new cdk.App();
const [
  wwwRecordName, apiRecordName, domainName
] = ['www', 'api', 'pax-imperia.com'];

const dnsStack = new DnsStack(app, 'DnsStack', {
  domainName,
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION
  },
});

new CdkStack(app, 'PaxImperiaInfaStack', {
  domainName, wwwRecordName, apiRecordName, redirects: true,
  zone: dnsStack.zone, cert: dnsStack.cert,
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION
  },
});

API Gateway using a Custom Domain

First up, let’s put the API Gateway behind a custom domain. We do this by creating an API Gateway Domain and then forwarding a subdomain to it using an A record.

import * as apigw from '@aws-cdk/aws-apigatewayv2';
import * as apigwi from '@aws-cdk/aws-apigatewayv2-integrations';
import * as route53 from '@aws-cdk/aws-route53';
import * as route53Targets from '@aws-cdk/aws-route53-targets';

export interface CustomStackProps { /* ... */ }
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName, zone, cert } = props;
    
    // hosted zone and lambda configuration
      
    const apiGwCustomDomain = new apigw.DomainName(this, 'ApiGatewayDomain', {
      domainName: [apiRecordName, domainName].join('.'),
      certificate: cert,
    });

    new apigw.HttpApi(this, 'ApiEndpoint', {
      defaultIntegration: new apigwi.LambdaProxyIntegration({
        handler: apiLambda,
        payloadFormatVersion: apigw.PayloadFormatVersion.VERSION_2_0,
      }),
      defaultDomainMapping: {
        domainName: apiGwCustomDomain,
        mappingKey: 'latest',
      }
    });

    new route53.ARecord(this, apiRecordName, {
      zone, recordName: apiRecordName,
      target: route53.RecordTarget.fromAlias(
        new route53Targets.ApiGatewayv2DomainProperties(
          apiGwCustomDomain.regionalDomainName,
          apiGwCustomDomain.regionalHostedZoneId,
        )
      )
    });
    
    // s3 bucket configuration
  }
}

With this deployed you should be able to access your lambda through API Gateway using a custom domain name.

xpm@mysteriousnotebook ~/C/p/cdk> curl https://api.pax-imperia.com/latest/hello/xpm
Hello, xpm!%

S3 Website Redirects and Custom Domains

Next up the site we have in S3. We want it to be accessible from http://www.pax-imperia.com, but users who access http://pax-imperia.com should get redirected to the www subdomain for convenience.

We’ll start with the redirect, which will require another bucket configured to redirect to to www subdomain. The redirect is conditional so that if you create any more stacks under the same domain only one can “win” the redirect from the top level domain. Otherwise they fight over who gets to create the right bucket, and nobody likes that.

Note that S3 does not support HTTPS/TLS websites! You must use CloudFront for that, although it is more expensive. Here we have the API secured with HTTPS but not the static website parts. Pick a security stance that best matches your use case.

S3 Static Websites with custom domains have a catch: the bucket must be named the name of the domain you want to host from it. So in this case, pax-imperia.com and www.pax-imperia.com.

import * as s3 from '@aws-cdk/aws-s3';
import * as route53 from '@aws-cdk/aws-route53';
import * as route53Targets from '@aws-cdk/aws-route53-targets';

export interface CustomStackProps { /* ... */ }
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName, zone, cert } = props;
    
    // hosted zone, lambda, and api gateway configuration
    
    if (props.redirects === true) {
      const redirectBucket = new s3.Bucket(this, 'RedirectBucket', {
        bucketName: domainName,
        websiteRedirect: { hostName: `${wwwRecordName}.${domainName}` },
        autoDeleteObjects: true,
        removalPolicy: cdk.RemovalPolicy.DESTROY,
      });

      new route53.ARecord(this, 'RedirectARecord', {
        zone, recordName: domainName,
        target: route53.RecordTarget.fromAlias(
          new route53Targets.BucketWebsiteTarget(redirectBucket)
        ),
      });
    }
      
    // static website bucket and deployment
  }
}

Finally we can configure the bucket with the main static website to be served by a custom domain.

import * as s3 from '@aws-cdk/aws-s3';
import * as route53 from '@aws-cdk/aws-route53';

export interface CustomStackProps { /* ... */ }
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName, zone, cert } = props;
    
    // hosted zone, lambda, api gateway, and redirection bucket configuration
    
    const websiteBucket = new s3.Bucket(this, 'ReactWebsiteBucket', {
      bucketName: [wwwRecordName, domainName].join('.'),
      websiteIndexDocument: 'index.html',
      publicReadAccess: true,
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    // deployment task

    new route53.ARecord(this, wwwRecordName, {
      zone, recordName: wwwRecordName,
      target: route53.RecordTarget.fromAlias(
        new route53Targets.BucketWebsiteTarget(websiteBucket)
      ),
    });
  }
}

With this deployed you can see your site on the internet!

xpm@mysteriousnotebook ~/C/p/cdk> curl --head www.pax-imperia.com
HTTP/1.1 200 OK
x-amz-id-2: pb5DXFUS1+HfNrIBVXhJBslcfpqWOUvXGXkIn4tepUfINqe2KmnAssWEEWMpOmyfGyBJmjBhR+Q=
x-amz-request-id: 9BV6DXKF08EF4JR6
Date: Wed, 21 Jul 2021 23:20:09 GMT
Last-Modified: Wed, 21 Jul 2021 07:48:03 GMT
ETag: "52bbf41df80401cf14bdc26b20dc3def"
Content-Type: text/html
Server: AmazonS3
Content-Length: 3034

Security with CORS

By default your web browser will not allow your web app to access your API because they’re on different domains. You have to open it up with a CORS header declaration on the API Gateway first. Before we do that, let’s throw a quick test into our React app to show this is the case. Put this in the component function of App.tsx:

fetch("https://api.pax-imperia.com/latest/hello/xpm", {
  credentials: 'include'
}).then(response => {
  return response.text();
}).then(text => {
  console.log(text);
});

When you deploy it and load the site the console should tell you that you’re a naughty person and cannot do this thing. To fix it we’ll modify the API Gateway construct, just a little.

import * as apigw from '@aws-cdk/aws-apigatewayv2';
import * as apigwi from '@aws-cdk/aws-apigatewayv2-integrations';

export interface CustomStackProps { /* ... */ }
export class CdkStack extends cdk.Stack {
  constructor(
    scope: cdk.Construct, id: string,
    props: cdk.StackProps & CustomStackProps
  ) {
    super(scope, id, props);

    const { wwwRecordName, apiRecordName, domainName } = props;
      
    // ..
    
    new apigw.HttpApi(this, 'ApiEndpoint', {
      defaultIntegration: new apigwi.LambdaProxyIntegration({
        handler: apiLambda,
        payloadFormatVersion: apigw.PayloadFormatVersion.VERSION_2_0,
      }),
      defaultDomainMapping: {
        domainName: apiGwCustomDomain,
        mappingKey: 'latest',
      },
      corsPreflight: {
        allowOrigins: [
          'http://' + [wwwRecordName, domainName].join('.')
        ],
        allowCredentials: true,
        allowMethods: [
          apigw.CorsHttpMethod.ANY,
        ],
      },
    });
    
    // ..
  }
}

When you deploy this change and reload the page you should get properly greeted by the API.

With that you have a functional Rust Lambda with a ReactJS frontend, all deployed through CDK with a single command. From here I suggest adding DynamoDB for an inexpensive database to back that Lambda, should you have data storage requirements.

Happy coding!

Sources

A Beginner’s Guide to Running Rust on AWS Lamba by Nicholas Hiasson. Accessed 21 July 2021.
Cross-origin Resource Sharing (CORS). Accessed 21 July 2021.
The AWS CDK documentation is excellent. These are the documentation modules relevant to the tools used in this walkthrough. All accessed on 21 July 2021.
Crates you should know:
- Rocket 0.5.0-rc.1, an easy to use web framework that handles routing and is otherwise fairly easy to set up.
- lambda_web, which adapts the requests from API Gateway V2 passed through the Lambda runtime to Rocket. It also handles Actix and Warp, so pick the web stack which you like the most!

Developing a Discord Bot Over a (Long) Weekend

2020-07-06T00:00:00-07:00

Over the extended 4 July weekend I spent some time building a chatbot for Discord. I, of course, picked Rust, and wanted to record some thoughts and notes from the experience.

The Goal

The goal of the bot is to automate some community housekeeping at the iDevGames Discord Server. Particularly we want to experiment with jailing political discussion to a #politics channel so it doesn’t disturb those who don’t want to see that discussion. Whenever discussion arises out of that channel the bot should nudge the participants to take it to the appropriate channel.

That channel is controlled by roles - if you have the role political animal then you can see the #politics channel, otherwise it’s totally hidden. The bot should accept requests for that role, both grants and revocation.

The Crates

I picked the following crates, in alphabetical order:

lazy_static, which I use in place of a heap-allocated static.

regex, which is incidentally what I like to pretend is a heap-allocated static.

serenity, a client for Discord’s API. Much of this post is a thinly-disguised propaganda work for Serenity - it was really easy to work with, has a fairly convincing feature set out of the box, and was just generally a really fun tool to work with. I have some wishlist items that I hope to PR, but overall a terrific job well done.

toml, which I used for configuration parsing. TOML is simple, this wasn’t a statement on anything other than it was easy to get going and didn’t impede my progress.

Abstraction

The first major departure from the Serenity example code is that I opted to not use their Framework. It was rather invested in the notion that bot commands ought to start with a specific character, such as a bang or a tilde. It had an escape method which would run on any message, which seemed to me no better than simply using the ordinary, non-Framework handler.

I ended up rolling my own, which works on a command-map pattern with a slight twist. Each item in the command map is given each message and given the opportunity to decide whether it wants to act on that message. While there is not mechanism to stop a handler from performative action in that test, it does help break up the handler into different functions. I consider it a mild success in separation of concerns.

Logic

The logic is broadly separated into three distinct areas of discussion, which are a command for granting/revoking roles, the use of to_lowercase/1 to normalize messages for comparison, and using contains/2 for detecting those pesky political words.

The use of a regex is not a particular point of pride. It’s a little prescriptive and I’m not sure it creates a stellar user experience. But it does work in role_wizard.rs, where it just uses capture groups to structure input text into the “arguments” of the command. Because I prefer to compile the regex as few times as possible, this is where I employed lazy_static.

A repeated thing is calling to_lowercase/1 on messages. Internally everything is lowercase, because Discord can get silly and I don’t want to start punching every possible capitalization of things into my configuration file. I think that having better case insensitivity options in string operations would make this perhaps unnecessary. However, it could be just as possible that this could create many calls to to_lowercase/1 under the hood, so maybe it’s best this way?

Finally, the distinct reason for this entire adventure (and why the bot exists) is really to loop through a list of words and find matching ones, then perform an action on that stimulus. This is WordWatcher. It’s too early in the bot’s lifecycle to say for sure whether this incredibly simplistic approach is sufficient, however, I suspect that it will be as long as the watch words are adequate.

Itches

Throughout development there were a few minor pain points that I think should have been simple. These range from things that I could submit a PR for to things that will likely take a PhD researcher some time to determine if an answer can even be made! In no particular order, these are my itches.

Getting the current (bot) user name

Getting the bot’s user name is surprisingly difficult, to the point that I looked it up on Discord’s site instead of the Serenity API docs to see if it was even supported. It created some bizarre code that looks like this, from AckMessageHandler’s should_handle/3 method.

// don't copy-paste this, i learned a better way described below!
if let Ok(current_user) = ctx.http.get_current_user() {
    let user_id = current_user.id;
    
    if msg.mentions_user_id(user_id) {
        return true;
    }
}

On the surface now it doesn’t look that bad, but determining that the functionality I wanted was on the ctx.http object was a huge leap for me! Serenity has packaged so much functionality into the thoughtfully-laid-out model structs that I honestly took ctx.http as an internal pointer to pass about on ceremony and not something that I should be looking at.

I spend perhaps a few hours on that. I’d like to submit a PR which can add a simple helper to Message itself, just a simple one that adds a mentions_me method. I know it would have saved me some time, but I’m fully willing to admit that I may be on the only person stuck on that.

Update 2020-07-18

This was a relatively easy itch to scratch, so I made a PR to add a simple mentions_me function to Message. A Serenity developer helpfully pointed out that the means I had been using is not cached! To use the cache collapses this down to a very neat one-liner (if and only if you have the cache feature enabled).

msg.mentions_user_id(ctx.cache.read().user.id)

Unpacking this a little, ctx.cache.read() is because that cache is really a CacheRwLock, which itself is a wrapper around an Arc<RwLock<Cache>>, which enables it to be passed around multiple threads.

Calling read/1 on it is just saying “I only want to read from the cache.” This is rather nifty because it returns a wrapper around the cache, and doesn’t lock the cache for any other readers. Because of RAII semantics, when that wrapper goes out of scope it gets dropped and that read lock goes away, so someone else may be able to write to the cache. This is a neat example of how Rust’s ownership model has shepherded us into a safe-by-default idiom when dealing with concurrency!

As it so happens, this cache has a pre-cached CurrentUser, from which we fetch the id we wanted in the first place. No need for another HTTP call and possible error to handle from that!

Getting the current channel name

Getting the current channel id is trivial, but converting that to a name was surprisingly exciting. Again, from AckMessageHandler, I found it was easier to convert a list of channel names from configuration to channel ids.

if let Some(guild_lock) = msg.guild(&ctx.cache) {
    let guild = guild_lock.read();
    let deny_channel_ids: Vec<Option<ChannelId>> =
        self.deny_channels.iter()
        .map(|channel_name|
            guild.channel_id_from_name(&ctx.cache, channel_name)
        ).collect();

    if deny_channel_ids.contains(&Some(msg.channel_id)) {
        return false;
    }
}

Now, these are cached locally, so I don’t feel too bad about it. But now that I write this and research again with a bit more experience I find that there’s also Guild#channel_id_from_name/3. I would have liked to have seen a helper method on Channel to get the name of it. It would have saved a bit of time. While this doesn’t really solve for really big professional applications that are designed for connecting to multiple Discord servers at a time, it would definitely have helped my small hobby project.

Testing

There are no tests, and of that I am not proud. Some of this is inexperience on me. Most of my testing experience is with Ruby and Java, both of which are more dynamic languages and it’s fairly easy to mock out parts of a program. I would really appreciate some way to mock out parts of a program in Rust, as it’s much more relevant for me to test the business logic in my handlers than it is to create some Rube Goldberg contraption that mimics Discord’s official servers!

This is a sticky point that makes me silently kind of wish I had picked Ruby for this project - but Rust definitely redeems itself on deployment, so stay tuned!

Configuration parsing

Configuration parsing is super unwrappy. Look at this horrible code I wrote in main.rs.

// this code is kinda unwrappy but I think that's okay because dying in
// initialization is sorta expected on bad config, right?
fn parse_handlers(raw_toml: String) -> Vec<Box<dyn MysteriousMessageHandler>> {
    let toml = raw_toml.parse::<Value>().unwrap();
    let handlers = toml.as_table().unwrap().get("handlers").unwrap().as_array()
        .unwrap();
    ...

Now, some of this is on me for insisting on having configuration. This could have just as easily been handled as code, however, I thought it was better to have a configuration file that a community member could easily understand and submit a PR for.

By the comment I was feeling guilty about the unwraps, and rationalized that unwrapping on service start (where it’s immediately obvious and not going to blow up a running service based on arbitrary user data from Discord) isn’t really a big deal. In other words, it’s a code smell, but it’s in the kitty litter box where it belongs.

I chalk this up to some of the immaturity in Rust’s error handling. There’s a fair bit of churn between Anyhow and Eyre and the others, and the simple matter is that effectively modelling errors is hard. I understand that. I’m excited to see that conversation progress and for this little bit of a cobweb to get better in the future.

For now, it’s just a todo in the back of my head for a lazy Saturday to beat on this code a little bit to make it less embarrassing.

Speaking of this code, why not Serde? Serde is fantastic, but I rather wanted something that I had more control over. My background is in Objective-C, where I got really good with NSCoder (if I do say so myself). The thing I really appreciated about NSCoder and friends was the control I had over the serialization and deserialization process. I could branch down a completely different path on a different serial version id if I really wanted to. It was all written down and easy to follow. I’d like to see something similar as an option in Rust, and I’d like to write it. I just need to find the time.

Ownership and Threading

Right in the header of the MysteriousMessageHandler is the constraint that implementers of this trait must be both Send and Sync.

pub trait MysteriousMessageHandler : Send + Sync {

I had hoped to avoid that requirement and instead wrap the list of handlers in Arc<RwLock<Vec<dyn MysteriousMessageHandler>>> or otherwise Vec<Arc<RwLock<dyn MysteriousMessageHandler>>> but each incantation I tried simply would not elide the non-Send non-Sync nature of the trait. I read the relevant Rust Book and Rustnomicon pages on the topic, and still couldn’t make it work. I found that frustrating, and would like to correct it in the future so that future handlers can store whatever data they like. Or perhaps that’s a bad decision on the surface of it, as then handlers will block?

Either way, I spend an hour or two on that little behavior which diverged from what I thought I read and what I expected. I’d like to revisit that sometime.

Deployment

The part of this process where I think Rust really shone was in deploying the bot. In the README I wrote as much, that to deploy you just take the built binary, scp it somewhere, and run it. There’s no grand dependency chain from the OS to worry about, even the TLS layer is handled internally thanks to RusTLS. I think that’s really cool!

Getting it set up to run as a service on my personal server was similarly really easy, I just used systemd. For all it’s malignment, this part was delightfully simple compared to writing a service shell script and managing PID files by hand. It really was as simple as saying “here’s an executable, make sure it’s running as this user after boot.”

The deployment instructions are available in the README, and they’re surprisingly robust for something cobbled together in a few hours.

Conclusion

Overall I enjoyed the time spend on this little project. I created something that is useful, and I hope maintainable. Though there were a few sandtraps along the way, the proof really was in the deployment. Rust created a system dependency-free binary that idles at about a megabyte and a half of RAM use. For someone as cost-sensitive as I, that’s a win!

If you’ve got an itch to scratch in Discord that could be served by a Bot, I’d suggest giving Rust and Serenity a spin. Please make use of my notes and code to give yourself a head start, even if that’s identifying dead ends you don’t want to go down.

Happy coding!

Level up your Crate

2019-03-27T00:00:00-07:00

This is adapted from a talk given on 27 March 2019 at Desert Rust, a meetup group of Rust enthusiasts in Mesa, Arizona. If you live in the area or are just visiting during a meetup, please stop by - we’d love to see you there!

You probably already know how to make a Rust crate.

cargo new --lib crate

You’ve probably written some Rust code in it, code that is good and ought to be shared. You’ve published it to crates.io, and nothing happened. Nobody appreciated it, why?

When I was a young boy and still foolish, my brother had a soccer (football) coach who would tell him, “if you look good, you play good.” In the spirit of that council, I believe that even the best software will be ignored if it’s not dressed up to impress.

The following tips can help juice up your crate and help it sell itself to potential users.

The elements of a solid crate, unless you know better ones, are:

README
Documentation
Tests
Continuous Integration (if applicable)

README

The README is your 60-seconds in an elevator pitch to your potential user. This is the part of your craft where you should throw modesty to the wind, because this is advertising, and advertising is war.

I organize my README into four parts. Everything else is extraneous and belongs in separate documents. These parts are:

What is this? What does it do?
How does this work? No, how does it work for me? (Best approached with a short code sample).
How do I get it? (This is Rust, so I assume it’s a crate).
Can I use this? By this I mean licensing - not everybody can use every license of software.

I try to be as customer-centric as possible when writing these. The template for my idea of a perfect README looks like the following:

vim README.md
# Crate Name
Elevator pitch! What does this do, and why is it awesome
at doing it?

```
fn main() {
  println!("Code sample!");
}
```

Elaboration about what this does, with more code samples,
or a link to a wiki or some other documentation.

# License
[2-Clause BSD](https://opensource.org/licenses/BSD-2-Clause)

It’s short, and it frontloads the most pertinent information to someone who’s shopping for crates and may have dozens of tabs open for research.

Remember to add your README to your Cargo.toml file so it gets picked up by cargo build and shows upon your Crates.io listing page as well, such as with throttle!

vim Cargo.toml
[package]
readme = "README.md"

While it may seem trivial, this is another point of contact to make it easy for potential customers to quickly learn about your crate.

Badges

Badges are little images which automatically updated and tell consumers information about your crate. By being on Crates.io, you get the these two badges for free,

a crates.io current version badge, which looks like , and
a docs.rs current version badge, which looks like .

Use these badges to link readers on your Github/Gitlab page to crates.io or to docs.rs.

I like to add these to my README directly below the Crate name, but just before the elevator pitch.

vim README.md
# Crate name

[![Crates.io][cios]][cio] [![Docs.rs][drss]][]drs

Elevator pitch!

...

[cio]: https://crates.io/crates/crate-name
[cios]: https://img.shields.io/crates/v1/crate-name.svg
[drs]: https://docs.rs/crate-name
[drss]: https://docs.rs/crate-name/badge.svg

You can also add these links to your Cargo.toml so they are visible from your Crates.io listing as well.

vim Cargo.toml
[package]
documentation = "https://docs.rs/crate-name"
repository = "https://github.com/..."

Documentation

Documentation is what separates code for me from code for you - and code for future me who has forgotten how everything works.

You’ve probably already worked out that Rust documentation is in CommonMark (or Markdown) format. Docs for ordinary items, such as structs, fields, and functions, are given by three slashes.

/// Control the rate of novertrunnions.
pub mut novertrunnion_choke: f32;

Bigger ideas, such as how an entire crate is to be used or a module within that crate, are given by module docs, which use two slashes and a bang!

//! For a number of years now, work has been proceeding
//! in order to bring perfection to the crudely conceived
//! idea of a transmission that would not only supply
//! ...

Tests

You probably don’t write bugs, but I do, and I try to prove that my code works using tests. Writing good tests is among the harder things to do, and entire books have been written on the subject, so I’ll spare you any incomplete thoughts on the subject here.

The default crate template comes with a stub for you to fill in some tests:

vim src/lib.rs
...
#[cfg(test)]
mod tests {
  #[test]
  fn it_works() {
    assert!(1 + 1 == 2);
  }
}

Cargo comes with a built-in test runner, so running those tests is remarkably easy.

➜  retry git:(master) ✗ cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running target/debug/deps/mysteriouspants_retry-6c4c9e88511db7ca

running 2 tests
test tests::succeeds_after_two_retries ... ok
test tests::fails_after_exhausting_retries ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests mysteriouspants-retry

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Going further, we arrive at my favorite Rust feature, the part that I think takes Rust from a toy language to something capable of building durable software: doctests.

cargo test will try to run code samples in your documentation, so if you change your crate’s implementation, you can easily catch idiosyncrasies in your documentation’s code examples. This encourages you to document using prose and code, creating APIs with rich reference material.

/// The main winding was of the normal lotus-o-delta type
/// placed in pandndermic semi-boloid slots of the stator,
/// ...
///
/// ```rust
/// # extern crate mysteriouspants_rockwell;
/// # use rockwell::TurboEncabulator;
/// # fn main() {
/// // create a new TurboEncabulator
/// let mut encabulator = TurboEncabulator {
///   cardinal_grammeters: 3
/// };
/// assert_eq!(encabulator.cardinal_grammeters, 3);
/// # }

Any code sample line that does not begin with a # (octothorpe) will appear in your documentation, letting you de-noise what shows up on docs.rs.

Travis

The last thing I look for in a well-constructed crate meant for third-party consumption is some kind of Continuous Integration. For Open-Source, the easiest I know of it Travis CI.

Tell Travis how to build your Crate by telling it that you’re writing Rust in a .travis.yml file.

vim .travis.yml
language: rust
rust:
  - stable
  - beta
  - nightly
matrix:
  allow_failures:
    - rust: nightly
fast_finish: true

Using Travis gets you another badge for your README, as well.

vim Cargo.toml
[badges]
travis-ci = { repository = "github-user/repo" }

vim README.md
[other badges from before] [![Build Status][traviss]][travis]

...

[travis]: https://travis-ci.org/github-user/repo
[traviss]: https://travis-ci.org/github-user/repo.svg?branch=master

If you’ve followed these suggestions, I consider your crate as having leveled up! Potential users can

see what your crate is about at a glance in your README, both on your repo page and on crates.io,
read detailed documentation and examples to confirm that what you’ve written is what they’re after, and
see that your code does everything you say that it does with a passing CI build!

These suggestions are unless you know better ones. If you do, please open an issue or PR on github and be clear it’s this post you think ought to be revised.

The Case of the Leaky Pool

2012-12-04T00:00:00-08:00

It always has to happen right before your system administrator leaves on vacation. The game is set, the stakes are high, and nobody is willing to fold. It was at a time like this that the Java Virtual Machine (JVM) on our second production server was discovered to be leaking memory like a gangster leaking blood in the Valentine’s Day Massacre. Unlike the mafiosos in Chicago, we couldn’t figure out why our JVM was bleeding.

Note that this post is best read in a 1920’s-style detective voice. I tried to have fun with the writing style, so just give it a go, OK? You might want to go re-read that first paragraph again, 314, just for practice. Additionally, some of the technical information might be slightly off-base (though I’m pretty certain that it’s not too far from the truth;) if you know better, please consider doing me a favor and letting me know in the comments section! I’d love the correction!

Like any good beat, we got two production servers with a fat load balancer in front. app01.prod is usually the point man, with app02.prod ready to take the heat if app01.prod gets iced. The cool kids call it a “hot failover,” but me? I just call it smart.

And being smart paid off. On Monday we looked at our Munin graphs to check in on the situation. The admin likes to do that before he checks out. Whether for peace of mind, or through grizzled experience doesn’t matter as much. Our monitoring reported all systems normal - except app02.prod.

app02.prod.proxy memory-use-day4

Unlike most smart shops, we were running a different setup. app01.prod was running our code using Apache Passenger and MRI Ruby 1.8.7. It was the first gig we could get in the club of the corporate private cloud. The bouncer wouldn’t let any cool toys inside the party, so we had to dance to the music they played.

But that was a year and a half ago, and the bouncer changed his mind. And we got good at smuggling.

We were running into teething issues with MRI 1.8.7. MRI/YARV 1.9.3 has been roaming the streets for a while now, and we’re not sure we want to be stuck on yesterday’s ride. With MRI/YARV 2.0 just around the corner, it made sense to make a push for a better stack. So for the past four months we’ve been in the garage trying to refit the old car for a new trick: JRuby.

Particularly handy is the JRuby way of deploying to J2EE servers, which we can push into the private cloud like we own the joint. The whole organization is chill with anything Java, so we’re golden, right? Just grab a server, stand it up, and we’re in to the hottest cloud in town. A quick snoop around the scene shows a few promising candidates. I could deploy through GlassFish and cut WAR files using Warbler. I like GlassFish. I used it before at an old job. We go way back, GlassFish and I.

Or I could try this new kid on the block. It supports clustering, queues, and scheduled tasks. Long-running requests (for WebSockets and such) are included. I wryly thought back to a few dirty hacks I had running with delayed_job. Yeah, this TorqueBox kid is the right tool for the job. I tossed out the hacks and spruced up the app, and faster than you could say “UTF-8” it worked.

The sysadmin packaged up TorqueBox into a Debian package so we could install it in our private cloud on our production servers. Because of some legacy decisions in our build servers we had to take the whole .tar.gz file and rename it .png, then un-tar that in a post-install script. Yeah, we got good at smuggling. But it beat the bouncers, and we had our best tools inside the club for the first time.

But it’s not all roses when you’re trying to deploy something as complex as this. You look at a cron-style task schedule the wrong way and you can find yourself in a world of pain. Like I did this past week.

The leak was huge. We weren’t really sure how it could leak 18GB of memory on a 6GB dime, but it had. And the sysadmin was on his way out the door. In order to keep our production systems reliable he tore down app02.prod, which was running our experimental TorqueBox server, and restored it to running the same Apache/Passenger MRI 1.8.7 gig that app01.prod is running. If app01.prod gets iced, app02.prod will be at least as good as it was. And ain’t nobody wanted to think about what would happen if they both went down in a hail of requests. There are some things too horrifying to plan for.

This put my investigation in a pickle. The memory leak was only manifesting on app02.prod, and without the extant bug, it was my job to try and replicate it. I started spinning up VirtualBox VMs and tried to start replicating the bug in an environment as close to prod as was possible. But again, the sysadmin saved me.

He installed Munin on a little-known dame sitting in the other aisle, app01.stage, where we try out our new kicks before promoting them to a full prod showing.

app01.stage.proxy leak-day2

Stage was showing the same kind of problem as prod: an 18GB memory leak. app01.stage had been booted around the same time we last rebooted app02.prod, so the similar velocity in leak proved that we had the same dirt bag in our sights. The sysadmin left for the week, leaving me with app01.stage and a bug to find.

I spun down my VMs and SSH’d into app01.stage to take a look. I didn’t think I’d find anything, but it was worth a try. The system was stable, and it seemed to think that the culprit JVM was only taking 800MB of memory.

A clue!

I looked at the graph again. Munin says it was taking well beyond the memory space, yet there wasn’t any swap activity. Years of tracking down bugs hadn’t prepared me for this. But a crash-course in memory mapping I’d taken when experimenting with libdispatch’s I/O facilities had prepared me for this. Funny how the gigs you take for fun end up saving you in the end!

What I learned from libdispatch is that all programs are incurably greedy.

They allocate a city of memory, but only ever use a few blocks. Maybe they just keep the rest for a rainy day? Back in the old days of DOS and tommy guns this wasn’t a concern. You only ever ran a program at a time, and so nobody cared that the racketeers were running away with the city hall. There was one capo and he set The Rules. Life was simpler back then.

But the world got bigger, and more gangs started vying for the limelight. And they all wanted the same city - all of it. So we give it to them. All of it. And things got more complex.

When a program allocates memory, it runs through a call to either malloc or calloc, and a few others. But those are the big two. malloc returns unformatted memory, while calloc fills the memory with zeroes. malloc and calloc themselves chains down through the local runtime memory manager to see if it has free memory in a held page.

Memory is cordoned off into pages, like the blocks of a city. Unlike a city block, only one program can own a single page at a time - we can’t split ‘em up. So the local runtime has a small memory manager which will try and give you parts of existing pages it holds when you need more memory. If you have a 4KB page, and you have used 1KB of it, and you ask for 2KB of memory, it should give you part of that existing page. There are tools like guard malloc which change this slightly, but for everyone else this is the way it works.

If the local memory manager doesn’t have the memory, it hits the operating system kernel.

Now, kernels are smart. They know that programs are greedy, and so they cheat them. When the program malloc’s a fat load of memory, the kernel just pretends to give it what it wants. But in reality it just gives it pages of memory. Address space. Imaginary property. A deed with no land. Smoke and mirrors. A mirage. It even does this for calloc.

In Linux and Darwin (a FreeBSD fork), the kernel allocates a page of all-zero memory. Whenever a made man demands a leveled (zero-filled) block, the kernel gives him one, but it just forwards to that pre-made zero page. The mafioso can go look at the page and read it all. Everything’s zeroes. And that page is shared between potentially hundreds of gangsters at a time. It’s a city of addresses which all go to the same vacant lot. It’s deception on a grand scale!

The trick happens when they try to build a safe house on that block. The moment that they try to write to that zero-filled page, the kernel gets wise and creates a zero-filled page, then hands that right over to the program. Because of CPU-level magic, the address doesn’t change - everything is still distinct. If a page size is 4KB (which is typical), and a program asks for 2040KB of zero-filled pages, that program will get 510 pages which all map back to a single zero-filled page.

The program has 2040KB of address space allocated, but is only taking 0KB of physical memory.

The kernel beat the system, and over-sold the city.

When that program tries to write to that first page, the kernel fixes the situation and hands the program a real page, so now the program has 509 references to the shared page and one reference to a real, private page. It’s a slick trick called “Copy on Write.”

Now, the linearly increasing line on the Munin graph is showing the committed memory, which is address space given to programs, but which may or may not actually be used. That the real memory graph down below is incredibly low suggests that the program, in this case the JVM, is allocating lots of memory, but never touching it. Because of the fancy footwork on part of the kernel, this prevented the program from crashing due to a memory shortage.

Believe it or not, this narrows down the culprit significantly. But it requires some savvy with the JVM itself.

The JVM, as a virtual machine, allocates a heap space, usually something fairly large, like 64MB. It also enforces a heap space ceiling, usually at 256MB. In my server’s case, the JVM started with 64MB and had a ceiling of 512MB. But it also has a “permanent generation” space, which in this case was 256MB. This is where class definitions live, and other such things.

Anything Java, and by extension JRuby, is going to live in the Java heap space. Because of how the heap works, it won’t be living in mapped pages, so it should always be in physical RAM. To prove this, I used a tool called VisualVM to get a bug in their court. I watched the gang, and they never knew any better.

At first I installed VisualVM on my iMac, and then enabled the remote JMX connector on app01.stage. But then the corporate firewall blocked everything. A few experiments with SSH port forwarding similarly failed. I needed a way in, or I couldn’t see if I was right.

So I cheated. I ran wget on app01.stage to put VisualVM on the staging server, which is running headless. Then I ran SSH with X11 forwarding to put that window onto my local X server (XQuartz for Mac).

ssh -X app01.stage /root/visualvm/bin/visualvm

I had a way in. And it didn’t look bad. These mafia types are much nicer than the drug runners from LA.

app01.stage.proxy visualvm-threadleak-day2

The heap space is normal. So it’s something causing a memory allocation that’s going deeper in the JVM.

Another clue.

This narrows down the list of suspects a lot. Nothing pure-Java or pure-JRuby will be doing this. It has to be a native call. Something is leaking inside the JVM itself.

I called in my friends. TorqueBox and JRuby have friendly, helpful communities. I hopped on IRC, my favorite diner, freenode.net, and poked the folks in #torquebox and #jruby. They confirmed my suspicions throughout. I’m on the right track.

After an APB to Google, I found a tool called pmap, which shows all allocated memory space for a process. Given that I was profiling a JVM with a full enterprise stack, running in a clustered High-Availability mode, there was a lot of stuff. I saw the Java heap space, the permanent generation space, as well as a few memory-mapped files.

The kernel is tricky again. When you read a file into memory, it will pretend to do so. Using the same ideas behind copy-on-write, it will expose the whole file’s length to your address space, but will swap out the actual memory. This partial in-memory caching can save big on space without impacting performance too much. pmap shows you files mapped to memory, which really helps to figure out what memory is doing what. Throw away all the files and you’re left with potentials for leaks.

But most interesting to me were hundreds of 2040KB “anonymous” pages. An anonymous page isn’t mapped to a specific file or network stream. It’s memory the program has allocated because the program needs memory. The Java heap space was a huge 300MB anonymous memory block. The permanent generation space was a similarly large 250MB anonymous memory block.

But what are these 2040KB blocks? And they were all exactly 2040KB.

By piping pmap into grep, I started counting them:

pmap 3248 | grep -c "2040K\\ rwx--\\ \\ \\ \\ \\[\\\ anon\\\ \\]"

I would run that, wait a little while, then run it again. I was getting another 2040KB allocation every minute. Then I noticed the VisualVM window behind that. I noticed the VM thread count.

1,740 threads sounds a little high, and it’s climbing. I know it’s an enterprise application server, but 1,740 is gratuitous. And I have 1,741 2040KB pages.

And the default stack size for a thread is 2040KB. After a few more minutes of watching, I’m sure. There’s a huge thread leak. Something is creating threads and never destroying them when it’s done.

VisualVM has a thread tab, which shows pure chaos: among the normal threads are around 1,550 threads, all helpfully named things like pool-1350-thread-1.

Another clue.

It’s not just leaking threads, it’s leaking thread pools with just one thread in them. That narrows it down a lot. Lots of code creates threads, but less code creates thread pools.

At this point I’m sure as Machine Gun Kelly’s spit that it’s thread leaks. A thread allocates a stack which lives outside the heap space, which fits the M.O. of the leak perfectly. My TorqueBox and JRuby associates on IRC heartily agreed.

My application doesn’t perform any manual threading, so it’s all going to be something from a underlying framework. The list of possibilities was nauseatingly endless, however:

oauth, which shouldn’t be spawning thread pools every minute. But it’s possible. It has to whack an external server and check back, so it could be using a thread.
jdbc-mysql, which could very well be naughty with threads. I wasn’t too sure though. This one’s been through a lot, and he doesn’t seem like the type to simply throw caution to the wind and start doing illegal things.
torquebox, which gets me crazy fun distributed caching and session stores, as well as exposes a logger which chains down to the JBoss AS 7 logger.
torquebox-messaging, used to enqueue a delayed job.

The hunt went on. I started commenting out code and disabling parts of my app to try and shake out the leak, which helpfully reproduced on my iMac without complaint. But the app was too big, too complex to start tearing out suspects. I needed another in.

So I started a new, blank Rails app. I started a TorqueBox server, installed the app, and watched the thread count. The leak wasn’t there - good.

I added a few render-able pages, and checked the session for a user id (to verify that it wasn’t happening somewhere in the Infinispan cache). Still no leak.

I added a scheduled task. I had a scheduled task to run every 15 minutes and another to run every hour on my production app, so I was sure that it wasn’t in Quartz, the system which dispatches the scheduled tasks. Still no leak.

Then I added something innocent. I added a timeout. TorqueBox, via Quartz, will enforce a timeout on your tasks, notifying them when they should stop running. And sure enough, that timeout spawned a thread pool with a single thread to effect that notification, but there was no code to clean up that thread when it finished.

I looked at my production application. I had made a mistake. My hourly task was firing every minute, hence the leaked thread pool every minute. If I had caught that mistake I might have suspected the scheduled task earlier, and thereby caught the bug faster. Or I could have fixed it and never seen the bug again because one hour is far too intermittent to try and track down a single thread leak. My timeout was correctly set at 59 minutes, but even so, after that 59 minutes the timeout should have fired and cleaned itself up.

However, the important thing is that I caught the leaker red-handed. I had the whole police department surrounding the building, so I sent in a SWAT team to rustle out the no-good. I sent in one of the TorqueBox lead developers, Benjamin Browning. (Or rather he decided to help fix this of his own volition, but just work with the writing style a bit, please). He raced through the TorqueBox (and JBoss Polyglot) code base and discovered the culprit.

And Benjamin Browning, being the awesome person he is, fixed it. People using TorqueBox 2.2.0 will never have to worry about their scheduled tasks leaking timeout threads.

As I sat at my desk in the middle of the bullpen that afternoon I leaned back in my old chair and grinned. I was a heck of a chase, but in the end, I found the bad guy, and we brought him to justice.

There were a few other steps I didn’t mention, such as a cryptic spawn-stack trace I saw using a trial version of YourKit analyzer. This is also my first attempt at writing like a pulp-fiction crime novel. I hope you enjoyed it, 314!