Going Serverless

How do I get my API Gateway URL?

Sat, 08 Apr 2017 09:00:00 +1000

Occasionally you need to know the API Gateway URL for your services inside your Lambda. This happened to me recently when one of my Lambda’s needed to provide a callback URL to a third party service that it was using.

From what I’ve read on the Serverless forum a lot of people are deploying their API’s using Serverless, then copying the URL and redeploying again with that URL hard coded into their serverless.yml as an environment variable.

If you’re one of those people then STOP. This is both the hard way and it has two very negative side effects.

It’s difficult to deploy your application to other stages, and
It introduces a risk that you’ll break something by depolying to one stage with the API Gateway URL for the different stage.

So how do I get around this?

Instead of hard coding the API GW URL you can reconstruct it using a combination of Serverless variables and CloudFormation. Here’s how I set the API gateway URL so it’s passed to my Lambda as the GW_URL environment variable.

provider:
  environment:
    GW_URL: { "Fn::Join" : ["", [ "https://", { "Ref" : "ApiGatewayRestApi" }, ".execute-api.${self:custom.region}.amazonaws.com/${self:custom.stage}" ] ]  }

Inside my code I can now reference the current API GW URL using.

process.env.GW_URL

If you’re using the serverless-offline plugin (or similar) then you will probably want to combine this with per stage environment variables. You’ll need to use per stage environment variables because this approach assumes you’re deploying with CloudFormation but running it locally means no CloudFormation to resolve the URL. Instead you’ll need to hard code a URL for local development.

The “cheat sheet” version is below but I strongly recommend you read the post as the full version explains it in more depth and covers topics like having separate files for each environment.

provider:
  environment:
    GW_URL: ${self:provider.${self:custom.stage}.GW_URL}

custom:
  stage: "${opt:stage, self:provider.stage}"
  prod:
    GW_URL: { "Fn::Join" : ["", [ "https://", { "Ref" : "ApiGatewayRestApi" }, ".execute-api.${self:custom.region}.amazonaws.com/${self:custom.stage}" ] ]  }
  dev:
    GW_URL: "http://localhost:3000/"

Keeping Dev Dependencies Out Of Your Serverless Package

Sat, 01 Apr 2017 10:00:00 +1100

NOTE: This has been tested using Node 6.10, NPM 4.5 and Serverless 1.11. I don’t know the first version of those packages that this works with.

Most developers will find themselves with a number of dependencies in their package.json that are only required for development. In the serverless world it’s not uncommon to require a compiler/transpiler (TypeScript/Flow/Babel), type definitions, a unit testing framework, additional plugins, etc.

Even though they’re included in your package.json as devDependencies they’re put into the same node_modules folder as your regular runtime dependencies. This means that when Serverless creates your deployment package you often end up shipping a large number of packages that are only required for development. Obviously this is less than ideal for a number of reasons.

While you could try to using the package include/exclude directives in your serverless.yml it quickly gets out of control.

For a long time I lived with this problem but it appears a solution is now available.

With NPM 4.5 you can now put a package.json/node_modules in the projects root folder with your development dependencies and a package.json/node_modules in the service folder with runtime dependencies. When you run sls from the service folder it has access to both sets of packages, allowing you to use plugins from the root folder, but when it creates the deployment package (the .zip file that is uploaded) it will only includes the node_modules from the service folder. This excludes all of the development dependencies.

Still lost?

Start by creating a root folder for your project. If you’re using git then this is probably going to be the root folder for your repository.
Create a package.json using npm init. You’re going to install all of your development dependencies here using npm install PACKAGE --save-dev from this folder.
Now create a new folder for each service. You can do that by using sls create -t aws-nodejs -p service-name.
In each service create another package.json using npm init. Your runtime dependencies are installed here by using npm install PACKAGE --save from this folder.

If you managed to following this correctly you’ll probably have something that looks like:

root
  + package.json
  + node_modules
  + service-name
      + package.json
      + serverless.yml
      + node_modules

When you run sls deploy from the service folder only the packages from the root/service-name/node_modules folder are included.

Keeping secrets out of Git

Mon, 23 Jan 2017 10:00:00 +1100

In a previous post I wrote about using per stage environment variables with the Serverless Framework. That article showed how you could set different values for environment variables depending on the stage you were deploying to.

There still seems to be a lot of misunderstaning about how powerful the environment variable implementation in Serverless is. Today I want to show you how to keep your secrets out of version control.

Before I show you how it’s important to understand why. Secrets like API keys, database passwords, etc should never be committed into version control. There are two main reasons for this.

Once they’re in version control they’re available to everyone who has access to version control. In many environments you don’t want to provide developers with access to production systems.
It forces everyone and every environment to share the same credentials. Typically you want developers to build in their own environment and your staging/production environments will have their own credentials.

There’s a really simple way to handle this with Serverless.

It all starts by creating a secrets.yml file to hold your secrets for each environment. I’ve included an example file below.

default: &default
  <<: *default
  COMMON_API_KEY: "AN API KEY COMMON TO ALL ENVIRONMENTS"
  COMMON_API_SECRET: "AN API KEY COMMON TO ALL ENVIRONMENTS"

dev:
  <<: *default
  API_KEY: "YOUR DEVELOPMENT API KEY"
  API_SECRET: "YOUR DEVELOPMENT API SECRET"

stage:
  <<: *default
  API_KEY: "YOUR STAGING API KEY"
  API_SECRET: "YOUR STAGING API SECRET"

prod:
  <<: *default
  API_KEY: "YOUR PRODUCTION API KEY"
  API_SECRET: "YOUR PRODUCTION API SECRET"

The default section at the beginning sets default values that will be used by all environment. For example - you may be using an external service to lookup IP addresses and you only have one key that is used by all environments.

Below that is one section for each stage. The name should match the stage you’re deploying to. Each of these environments inherits values from default because of the line <<: *default. You can override any values set in default by setting a different value for that stage.

Now we have a secrets.yml file we don’t want it commited to version control or uploaded to AWS with our code.

Start by adding secrets.yml to your .gitignore file.

echo "secrets.yml" >> .gitignore

You can check that this is working by running git status and if the file doesn’t show up you’re good to go.

Next you want to add secrets.yml to the list of files excluded from your package that is uploaded to Lambda. You do this inside your serverless.yml.

package:
  exclude:
    - secrets.yml

Finally you need to include the secrets.yml file in your serverless.yml file. I’ve been doing this in the custom section by using.

custom:
  stage: ${opt:stage, self:provider.stage}
  secrets: ${file(secrets.yml):${self:custom.stage}}

This allows you to set your environment varilables to use the values imported from the secrets.yml. Below is an example.

provider:
  environment:
    API_KEY: "${self:custom.secrets.API_KEY}"
    API_SECRET: "${self:custom.secrets.API_SECRET}"
    COMMON_API_KEY: "${self:custom.secrets.COMMON_API_KEY}"
    COMMON_API_SECRET: "${self:custom.secrets.COMMON_API_SECRET}"

When you deploy your application using Serverless it will now load your secrets from secrets.yml that won’t be commited to version or uploaded to Lambda.

To make it easy for new developers I tend to maintain a secrets-example.ymlfile that has all of the keys required by secrets.yml with dummy values. This makes it easy for them to create their own secrets.yml and tells them which secrets they need to acquire.

PS: There’s going to be one more part to this series explaining how to combine env.yml and secrets.yml. Join my mailing list today to get updates when that’s released.

Routing API Gateway Traffic Through One Lamda Function

Fri, 30 Dec 2016 10:00:00 +1100

One question that I keep hearing is “Can I have more than one handler in my AWS Lambda function?”. This usually in the context of someone wanting to implement a services pattern using the API Gateway where all traffic for a resource is handled within a single handler.js.

For example: GET /resource goes to handler.list, POST /resource goes to handler.create, GET /resource/{id} goes to handler.find, etc.

AWS only allows you to define one handler per Lambda function. While you can put multiple handlers into a single handler.js each of these needs to be setup as its own Lambda function.

functions:
  hello:
    handler: handler.createResource
    events:
      - http:
          path: resource
          method: post
          cors: true
    handler: handler.listResources
    events:
      - http:
          path: resource
          method: get
          cors: true
    handler: handler.deleteResource
    events:
      - http:
          path: resource/{id}
          method: delete
          cors: true
    handler: handler.findResource
    events:
      - http:
          path: resource/{id}
          method: get
          cors: true
    handler: handler.updateResource
    events:
      - http:
          path: resource/{id}
          method: patch
          method: put
          cors: true

This solution isn’t always ideal.

If you know your events are coming from the API Gateway you can get around this limitation by implementing a handler that routes events internally based on the HTTP method and optional parameters. This allows you to have a single AWS Lambda function that accepts events from the API Gateway instead of one for each HTTP method.

Note: In this example I’m using the API Gateway Lambda proxy integration method. You should be able to do something similar with the older API Gateway Lambda integration method but I’m not going to cover that.

To start I’m going to define a single Lambda function inside my serverless.yml that handles all requests to /resource and /resource/{id}.

functions:
  hello:
    handler: handler.router
    events:
      - http:
          path: resource
          method: any
          cors: true
      - http:
          path: resource/{id}
          method: any
          cors: true

The default response from this handler should be HTTP 405 Invalid HTTP Method. This response tells the client that we don’t implement the HTTP method they requested.

module.exports.router = (event, context, callback) => {
  const response = {
    statusCode: 405,
    body: JSON.stringify({
      message: `Invalid HTTP Method: ${httpMethod}`
    }),
  };

  callback(null, response);
};

Next I’ll create two objects that define the handlers to use when interacting with the collection and items in the collection. The collection handlers will be used for requests to /resource and the item handlers for requests to /resource/{id}.

const collectionHandlers = {
  "GET": listItems,
  "POST": createItem,
}

const itemHandlers = {
  "DELETE": deleteItem,
  "GET": getItem,
  "PATCH": patchItem,
  "POST": postItem,
  "PUT": putItem,
}

You’ve probably guessed that the key is the HTTP method so you could define DELETE or PUT on the collection by adding handlers with the relevant keys.

At the top of my handler I need to determine if I’m using the collection handlers or item handlers.This can be done by looking at event["pathParameters"]. AWS sets this value to null when there are no path parameters or an object if there are any path parameters. Because our Lambda function will only be executed for requests to /resource and /resource/{id} I can cheat a little and only check if event["pathParameters"] is null instead of checking if id is set.

let handlers = (event["pathParameters"] == null) ? collectionHandlers : itemHandlers;

If I wanted to be more cautious I could check that the id path parameter was actually set by using:

let id = (event["pathParameters"] !== null && "id" in event["pathParameters"]) ? event["pathParameters"]["id"] : undefined;
let handlers = (id === undefined) ? collectionHandlers : itemHandlers;

Now I need to check if the HTTP method has a handler configured for it. If it does then I can call that method passing the event, context and callback parameters that the main handler received.

let httpMethod = event["httpMethod"];
if (httpMethod in handlers) {
  return handlers[httpMethod](event, context, callback);
}

Remember our default is to return a 405 so we’re covered if the handler isn’t defined.

Lastly I just need to implement each of the handlers.

The full code for this example is:

"use strict";

const collectionHandlers = {
  "GET": listItems,
  "POST": createItem,
}

const itemHandlers = {
  "DELETE": deleteItem,
  "GET": getItem,
  "PATCH": patchItem,
  "POST": postItem,
  "PUT": putItem,
}

module.exports.router = (event, context, callback) => {
  let handlers = (event["pathParameters"] == null) ? collectionHandlers : itemHandlers;

  let httpMethod = event["httpMethod"];
  if (httpMethod in handlers) {
    return handlers[httpMethod](event, context, callback);
  }

  const response = {
    statusCode: 405,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({
      message: `Invalid HTTP Method: ${httpMethod}`
    }),
  };

  callback(null, response);
};

function listItems(event, context, callback) {
  const response = {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "listItems" })
  };

  callback(null, response);
}

function createItem(event, context, callback) {
  const response = {
    statusCode: 201,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "createItem" })
  };

  callback(null, response);
}

function deleteItem(event, context, callback) {
  const response = {
    statusCode: 204,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "deleteItem", id: event["pathParameters"]["id"] })
  };

  callback(null, response);
}

function getItem(event, context, callback) {
  const response = {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "getItem", id: event["pathParameters"]["id"] })
  };

  callback(null, response);
}

function patchItem(event, context, callback) {
  const response = {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "patchItem", id: event["pathParameters"]["id"] })
  };

  callback(null, response);
}

function postItem(event, context, callback) {
  const response = {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "postItem", id: event["pathParameters"]["id"] })
  };

  callback(null, response);
}

function putItem(event, context, callback) {
  const response = {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Credentials" : true
    },
    body: JSON.stringify({ action: "putItem", id: event["pathParameters"]["id"] })
  };

  callback(null, response);
}

You can also grab this from Github.

Update 12 May 2017: Added the headers and serverless.yml settings to support CORS.

Using Environment Variables with the Serverless Framework

Mon, 28 Nov 2016 10:00:00 +1100

As developers we all know that it’s best practice to keep configuration outside of the application. This serves two purposes:

It makes it easier to deploy the application into different environments (stages) like development, staging and production because only the configuration needs to be modified.
It allows you to keep your production credentials from being committed into version control. This is particularly important in organisations where developers don’t have access to production.

Prior to Serverless 1.2 you needed to use a plugin and the dotenv package as AWS Lambda didn’t include support environment variables. This changed a few weeks ago and Serverless 1.2 quickly followed suit by adding support for environment variables into the core.

Setting environment variables with Serverless 1.2 is as simple an adding an environment key to either your function or the provider section inside your server.yml. For example:

provider:
  name: aws
  environment:
    VAR_1: "something"
    VAR_2: "something"

functions:
  myfunction:
    environment:
      VAR_2: "something"

Environment variables in the provider section are set for all functions. This is useful for settings like API keys or a table names that every function inside your service needs to access. Environment variables set in the function settings only apply to that function.

If you set the same variable in both places the function’s environment variable will override the provider environment variable. This is ideal when you have an environment variable which needs one value for most functions but a slightly different value for one or two functions.

Inside your application environment variables are accessed exactly the same way you normally access your environment variables.

For NodeJS (Javascript) use:

process.env.MY_VAR

For Python use:

os.environ['MY_VAR']

While you can set environment variables directly inside your serverless.yml this doesn’t solve the problem of allowing different values for each stage or keeping your secrets outside of version control. There are two approaches you can use to for this.

Regardless of the approach you use you will want to add a custom variable to your serverless.yml file so that you have an easy way to reference the current stage.

custom:
  stage: "${opt:stage, self:provider.stage}"

For personal projects I prefer to have one environment file. I find it easier than managing multiple environment files and it’s really not an issue if I have production credentials.

Start by creating an env.yml file. In that file you want one key for each environment with all of the environment variables for that environment set below it. Example:

dev:
  MY_VAR: 'the dev value'

prod:
  MY_VAR: 'the prod value'

You can now set your environment to use all of the keys from your env.yml file that are below the key for your current stage.

environment: ${file(env.yml):${self:custom.stage}}

If you don’t understand that last line then read referencing variables in other files from the Serverless documentation.

Now that you have per stage environment variables working you may realize that some environment variables have defaults that don’t change from one stage to the next. Copying these to every stage is laborious and error prone. Most developers would prefer to have a section with common environment variable settings that are only overridden when the stage requires a different value.

The YAML syntax supports this.

Add a new default_env section to your env.yml and place your default environment variables below it. You can then tell YAML to include those in your stage specific environment variables. Here’s an example that demonstrates default values, overriding default values and adding additional stage specific variables.

default_env: &default_env
  SCOPE: 'read'
  TABLE: 'apple'

dev:
  <<: *default_env
  TABLE: 'banana'
  PASSWORD: 'my-password'

The end result would be like typing in

dev:
  SCOPE: 'read'
  TABLE: 'banana'
  PASSWORD: 'my-password'

Lastly you will probably want to add env.yml to your .gitignore file so that it doesn’t accidentally get committed to version control.

You can also use different files for each stage. I’m not going to cover this in as much details as I prefer the single env.yml approach.

Like the single environment file solution you will want to add a custom variable to your serverless.yml file so that you have an easy way to reference the current stage.

custom:
  stage: "${opt:stage, self:provider.stage}"

Now create an env-STAGE.yml file for each stage you have. For example: env-dev.yml or env-prod.yml. Inside those files place the environment variables for that stage.

MY_VAR: 'the value'

Then set your environment variables using all the values in the file for your current stage.

environment: ${file(env-${self:custom.stage}.yml)}

Lastly you will want to add env-*.yml to your .gitignore file so that they don’t accidentally get committed to version control.

When you deploy your service your should now see the environment variables in the AWS console. If you’ve followed these instructions and your environment variables aren’t set then

Check that you’re using Serverless 1.2 or later.
Try deploying the entire service and not just the function
Check that your keys or file names match the current stage you’re deploying to. I’ve found that Serverless will happily deploy my functions without a warning or error if the files or keys didn’t exist.