[Prefinem]

We follow a very similar pattern and we are at over 150 micro services right now (AWS Lambda).

A couple of things different that we do since we are building and then deploying to AWS:

- Build only on dedicated deployment branches (beta, qa, preview, prod)

- Build all functions (transpile, yarn, lint, etc) on every merge into the branch, but only deploy functions with different checksums (saves on api calls to AWS)

- We cache node_modules, but otherwise don't have any special build requirements and babel takes care of targeting node6.10 for Lambda

Total build time is between 8-13 minutes. There are some things we can do to speed up install that we haven't yet because it's not an issue yet but just a short list of things to note.

- Each function has it's own package.json for it's own packages. We maintain a list of npm packages that we download into a single folder first (that doesn't get deployed) to allow yarn to use those files from cache. We will eventually switch to an offline install for each function which essentially just copies the package folder and sets up anything it needs.

- We have a tarball package that includes all of our shared code / config files. Yarn seems to always want to download this file, regardless if we pre-download it.

- We deploy a single api endpoint for all of our micro services through API Gateway which cuts down on the time to deploy since API Gateway has a pretty hard throttle. This means we create a deployment on API Gateway every merge. We have one APIG for each environment

[jrowley]

Just to be clear, all of your functions exist inside one mono-repo, correct? You don't use git submodules at all?

Looks like a pretty solid build process. Thanks for the insight!

[Prefinem]

Yes, we have all of our functions in a single mono repo, broken into projects, and then folders for each function, similar to this:

- src/project1/function1/

- src/project1/function2/

- src/project2/function1/

- src/project3/function1/

- src/project3/function2/

- src/project3/function3/

Deploying the functions is done by project, so we deploy all of one project, then move to the next, and so on and so forth.

[greendata]

That's a great model. Do you use cloudformation each for deployment? If so, have you thought of creating a single cloudformation template for the whole deployment so you can do the entire deployment in one stack update?

Have you encountered any issues to watch out for when only using one APIG for each environment (150 micro-services). Have you encountered any downsides to doing this versus 1 micro-service to 1 APIG? I'm also running into the Gateway throttle limits and I think deploying many micro-services (like you have done) to 1 APIG is the best solution.

[Prefinem]

We don't use cloudformation because honestly, it sucks. It's hard limits are a pain in the ass to get around (with 150 lambda functions, we need hundreds of resources, so that means nested stacks, which just suck) and it managing the api gateway just doesn't do what we want.

We have a custom script to deploy our own API Gateway using the AWS SDK and we generate a swagger file from simple json config files.

For the API Gateway issues, so far, we have a few things that are something we have to watch out for.

- All lambda endpoints through APIG are lambda proxy type. This means we can have a framework handle standard request / response stuff. The downside is that we can't support binary endpoints easily because they haven't fixed that issue yet.

- HTTP proxy pass through endpoints have to be added to the swagger somehow before we deploy. This is a little annoying, but not a huge issue

- Merge vs Override for deployments. We merge in beta, and override in other environments. This allows us to keep endpoints exactly as they are, but allow flexible testing in beta

1 APIG for 1 micro service isn't great IMO at scale since we run all our endpoints under on domain and mapping all of them would be a pain.

[greendata]

That's very valuable advice. Thank you. I've been following the serverless.com model of 1 APIG to 1 lambda, but that quickly puts you over the AWS limits when trying to manage hundreds or thousands of micro-services.

[Prefinem]

Yup yup, I went down that path and converted our then very basic deployment process to use serverless and instantly hit hard limits.