There Was an Error Creating Your Log Stream Please Try Again

Going serverless relieves y'all of setting up servers, updating operating systems, or maintaining concrete infrastructure. But what happens when a office doesn't piece of work and things go wrong? Given the added complexity, working in a serverless surround requires engineers to capture the data flowing from functions and their epithermal containers with a centralized AWS Lambda logging solution.

In this Lambda logging guide:

• Examine what'due south important when designing a monitoring and alerting solution for serverless applications
• Build a test application and await at how monitoring/alerting could exist relevant to it
• Deploy the awarding to AWS Lambda using its serverless framework
• Make certain that the errors logged by our Lambda application are not ignored

Planning, Monitoring, and Alerting of Serverless Applications

When planning out a monitoring and alerting solution for serverless applications, there are several factors to take into consideration. We will discuss each of these in turn.

Scalability

How well does your monitoring solution scale when your awarding grows? One of the primary advantages of deploying your applications on a serverless platform is that they can scale nearly indefinitely. Withal, yous demand to brand sure that your monitoring solution can keep up.

It should be able to handle your growing application landscape without you manually provisioning the scaling solution. Information technology should be configured to handle big, apace-growing amounts of information provided by replicated applications, and still be able to evaluate them in real-fourth dimension.

Integration

Your monitoring solution should be the one place where all monitored information comes together. That's why it should provide easy-to-use integration points to have metrics and logs from different systems and applications.

Independence and Reliability

If your application goes down, yous must have access to any logs and metrics and be alerted. Then you are in a position to analyze what is happening and to react as speedily equally possible. That's why you lot'll desire your monitoring and alerting systems to run reliably and independently of your application.

Ease of Utilize

Let'south face up it, if it is a hassle to read through logs or gear up alerts, you will end up not doing it. A monitoring and alerting system is only useful if it is actually easy to use and doesn't bloat your administrative workload.

Separation of Concerns/Loose Coupling

If you want to reap the benefits of cloud platforms, yous'll desire to exist able to deploy your application to new platforms quickly. Logging shouldn't stand in the style. You don't want to exist forced to change awarding code when yous change your logging provider or your deployment platform. You don't want to implement unlike log-transportation strategies into your application, but rather only write to STDOUT and exist sure that your logs are dealt with.

Also, logging large amounts of data can exist a time- and resource-consuming task. It can fifty-fifty slow downward your core awarding and introduce production problems. That'south a situation you'll want to avoid by loosely coupling your application and your logging platform.

General Best Practices

Stream your CloudWatch apps to Lambda – I way of making certain that your logs proceed their integrity across applications is to gear up streaming of your logs directly to the Amazon Elasticsearch Service.

Machine-subscribe new log groups – Whenever a new Lambda function is created, information technology automatically creates a new grouping in your logs. You can enable CloudTrail to recognize new event patterns and handle the subscription procedure automatically.

Motorcar-set Log Retention policies – Past default Lambda creates new logs for each part with a "never elapse" setting for all events. This is understandably overkill and can fill your log files with a neat deal of unnecessary data, resulting in a reduction in your signal to dissonance ratio. You may wish to change this setting to auto-expire logs after a certain number of days.

A Sensible Architecture for Logging in Cloud Environments

And so, how tin yous set logging to comply with these requirements? A bones, nevertheless scalable compages could exist designed like this:

Your application simply logs out to STDOUT, for example, using panel.log. Writing to STDOUT is all your application needs to know nigh logging. In this style, you have separated your application logic from your logging activities.

A log transporter then fetches the logs from where they are stored temporarily subsequently writing to STDOUT, due east.g., from Syslog. It and so forwards new log entries to the logging platform. Your logging platform: stores the log data, has monitoring capabilities, and tin warning you to noteworthy events.

Using this pattern, yous can ensure that your application remains decoupled from your logging platform. If your system goes down, the relevant logs will even so be available in the logging platform.

Too, you enable scalability. Your applications can calibration equally you delight without the need to think about transporting big amounts of log data. You tin can just increase the number of log transporters, or alter the send strategy altogether, without affecting your application at all.

The ELK Stack

A classical way of implementing an compages similar that is using the ELK stack. Information technology's fabricated upwardly of three main components:

• Elasticsearch–a search engine and indexer for log data.
• Logstash–a log transporter that can accept logs from a wide multifariousness of input channels and formats and can transform (if required) and forrad them to output channels, for example, Elasticsearch.
• Kibana–a data visualization dashboard that reads and presents data from Elasticsearch.

The ELK stack applies the requirements mentioned in a higher place and makes upwardly one of the most widely-used logging platforms. In that location are a number of cloud-based SaaS platforms that provide you with an ELK stack, for example, Coralogix.

Alternative Solutions

Deject platforms such equally AWS usually provide their own logging stack. AWS, for example, has CloudWatch, which tin collect logs from services running on top of AWS, and provides basic stream and search functionality for log data.

When you run single applications that aren't besides complex, using CloudWatch exclusively can be an option. However, when your awarding consists of multiple components that log information, or when information technology combines services that run inside and outside of a cloud platform, you'll chop-chop achieve the limits of that solution. You'll want a central view on your entire application and the chance to not only read, but to clarify and understand your data. Therefore, it will ofttimes make sense to make your logs available in a dedicated logging platform.

Even when doing so, CloudWatch and similar services volition play a role in your compages. They collect your awarding logs inside your deject platform and make them acc log transporter.

Learn More: The complete guide to centralized AWS logging

An Example Awarding

Let's get our sleeves rolled upwardly and our easily dirty. We'll build a demo application using the Serverless framework and deploy it to AWS Lambda. So we'll find a potential error scenario in the application and prepare monitoring and alerting for that.

Prerequisites

For this article, nosotros presume that y'all have both Node.js and npm installed. If not, delight go to Node.js for instructions.

Set up AWS

If you lot already have your AWS CLI configured, you can skip this section.

To go started, you'll need an AWS account. If you don't have i yet, just follow the instructions on the AWS Portal.

As well, y'all'll need to have the AWS CLI (Command Line Interface) installed. You can discover instructions in the AWS Documentation.

To make sure that the CLI can piece of work with your AWS business relationship, go an AWS Access Key ID and an AWS Surreptitious Access Key by following the instructions here.

Then, store your credentials as described here.

Install Serverless

Install the Serverless CLI on your computer by running:

npm i -grand serverless

Create the Example Application

Consequence Handler

Create an empty folder somewhere on your computer. Now create a handler.js file with the following contents:

module.exports.greet = async event => {     const trunk = JSON.parse(issue.body);     if (!trunk.name) {         throw new Error('No proper noun provided');     }     return {         statusCode: 200,         body: JSON.stringify(             {                 greeting: `Hi, ${body.name}`             },             null,             2         ),     }; };        

This script defines an upshot handler called greet. When an HTTP request is sent to our awarding, it will be processed by this handler. It parses the asking trunk and checks if the proper name property is set. If information technology is non, the handler throws an error. If it is gear up, information technology returns a JSON object to the client, greeting the user by this property value (proper name).

Serverless Configuration

At present to deploy the application to AWS Lambda, we'll use Serverless. Serverless volition read its configuration from a serverless.yml file and take care of the setup and deployment for us. Create the YAML file in the same folder as handler.js and populate it with the post-obit:

service: greeter provider:   name: aws   runtime: nodejs12.x  functions:   greet:     handler: handler.greet     events:       - http:           path: /           method: post        

This configuration tells Serverless that nosotros're running a service called greeter on an AWS Lambda stack with Node.js 12. Information technology and then defines one Lambda part called greet, and maps that role to the greet part we divers in our handler.js. Finally, it instructs Lambda to mind to HTTP requests coming in under the path / and, therefore, treat them as events in our greet function.

Deploy the Application

We are set to deploy the application to Lambda. Simply run sls deploy in the same binder that also contains your handler.js and serverless.yml.

You will get an output like the following:

Service Information service: greeter phase: dev region: united states of america-e-1 stack: greeter-dev resources: 10 api keys:   None endpoints:   Post - https://xxxxxxxxxx.execute-api.usa-east-i.amazonaws.com/dev/ functions:   greet: greeter-dev-greet layers:   None        

The most important piece of information hither is the listing of endpoints. In this instance, Serverless tells us that we can reach our awarding under https://xxxxxxxxxx.execute-api.united states-due east-1.amazonaws.com/dev/. This URL will be unique to you, so be sure to note it down.

Test the Application

Now that we have our application deployed, let'southward go alee and test it by shooting information technology an HTTP Post request, letting it know that our name is Maria and waiting for a greeting:

$ curl --data '{"name":"Maria"}' https://xxxxxxxxxx.execute-api.us-east-one.amazonaws.com/dev/  {   "greeting": "How-do-you-do, Maria" }        

As expected, we're warmly greeted.

Now let'due south look at what happens if we modify the request slightly. In the handler.js, we expect the proper name to exist handed over in a request body property called proper noun. What happens if, instead, we commit an error and send it in a holding called firstName?

$ curl --data '{"firstName":"Maria"}' https://xxxxxxxxxx.execute-api.us-due east-ane.amazonaws.com/dev/  {"bulletin": "Internal server error"}        

Oops, that doesn't look so good. It doesn't come up as a surprise in this test application (nosotros explicitly throw an exception). Just with just this 500 error message from Lambda, all we know is that there was some upshot along the fashion. Our fault does not provide information about what the root crusade is.

Monitoring and Alerting

Our test application is primitive, and it's easy to pinpoint the root crusade of an error. In a larger awarding, nevertheless, there are many potential causes for errors. A database connectedness might have failed. A user's input data might accept been poorly formed and not properly handled. A 3rd-party Residuum API service might suddenly have become unavailable.

Nosotros need some way to see what's actually going on, rather than merely receiving a generic "Internal Server Error" message. One thing we can do to analyze the crusade of the fault is to look into the logs. We can do this using the Serverless CLI:

$ sls logs -f greet  START RequestId: b01870f3-b5f0-48dd-9cdc-fccfcade4940 Version: $LATEST  2020-02-01 x:33:24.766 (+01:00)        b01870f3-b5f0-48dd-9cdc-fccfcade4940    ERROR   Invoke Error         {"errorType":"Error","errorMessage":"No name provided","stack":["Error: No proper name provided","    at      Runtime.module.exports.greet [as handler] (/var/task/handler.js:vi:15)","    at Runtime.handleOnce (/var/     runtime/Runtime.js:66:25)"]}  END RequestId: b01870f3-b5f0-48dd-9cdc-fccfcade4940  REPORT RequestId: b01870f3-b5f0-48dd-9cdc-fccfcade4940  Duration: 3.91 ms       Billed Duration: 100 ms      Memory Size: 1024 MB    Max Retentiveness Used: 72 MB        

That'due south already meliorate. Information technology provides the actual exception nosotros throw in our application, so that helps us quickly identify the root cause. But how tin we make certain we actually meet errors when they popular up without manually checking the logs every time?

Check Logs in CloudWatch

The beginning pace is to make utilise of CloudWatch. Lambda logging should be stored in that location. And then, open the AWS console and, from the services bill of fare, open CloudWatch. Now in the left menu, click "Log groups":

You lot should meet one group for your greeter office (called, for example, /aws/lambda/greeter-dev-greet). A log group is a drove of log streams. The log streams in a log group share some mutual properties, amongst others: access control and information retention settings.

Click that log group.

You'll now see a listing of log streams. Each stream contains log messages from ane source. Find that re-deploying a Lambda role tin can atomic number 82 to a new log stream being created. Right now, you lot should run across at least i log stream:

Click it and bank check out the log entries. You'll come across the aforementioned messages returned when yous pulled the logs using the Serverless CLI earlier.

Transport Logs to a Lambda Logging Platform

While you now have a manner of collecting and seeing logs using CloudWatch, it'd surely be great to take your logs available in an ELK stack. While information technology's hands possible to ready your own ELK stack, it can be cumbersome to become started, to operate it, and to secure it properly. That'south why, for this article, we'll apply Coralogix, a cloud-based observability platform.

So let's go ahead and send our AWS logs to Coralogix.

Register with Coralogix

Beginning, annals for a costless Coralogix account. After registration, yous'll receive an email with the subject "Start sending your logs". It contains a individual cardinal, which we'll need to send logs from AWS to Coralogix. Then, be sure to note downwardly that private central.

Fix upwards Lambda Log Transporter

Next, nosotros'll set up AWS to send its logs to Coralogix. While information technology is possible to use Logstash for that, Coralogix provides a elementary log transporter out of the box.

To get started with it, nosotros'll prepare up another Lambda role that simply takes all logs of our test application and forwards them to Coralogix.

Open the AWS Panel, log in and type "Lambda" in the Service menu search field. Then click the "Lambda" entry:

Now click "Writer from scratch" to create a new Lambda function. Give information technology a name (e.g. "CoralogixLogTransporter") and choose "Node.js 12.x" as the runtime. And so click "Create Function":

Under "Function code", paste the following code. This will forward logs from your Lambda applications to Coralogix:

"use strict";  // Import required libraries const https = crave("https"); const zlib = require("zlib"); const affirm = require("assert");  // Check Lambda function parameters assert(process.env.private_key, "No individual primal!"); const newlinePattern = process.env.newline_pattern ? RegExp(process.env.newline_pattern) : /(?:\r\n|\r|\n)/yard; const sampling = procedure.env.sampling ? parseInt(process.env.sampling) : 1; const coralogixUrl = procedure.env.CORALOGIX_URL || "api.coralogix.com";  /** * @clarification Send logs to Coralogix via API * @param {Buffer} logs - GZip compressed logs messages payload * @param {function} callback - Function callback * @param {number} retryNumber - Retry attempt * @param {number} retryLimit - Retry attempts limit */ part postToCoralogix(logs, callback, retryNumber = 0, retryLimit = three) {     let responseBody = "";      try {         const request = https.request({             hostname: coralogixUrl,             port: 443,             path: "/logs/rest/singles",             method: "POST",             headers: {                 "Content-Blazon": "application/json",                 "Content-Encoding": "gzip",                 "Content-Length": logs.length,                 "private_key": process.env.private_key             },             timeout: 10000         });          request.on("response", (response) => {             console.log("Status: %d", response.statusCode);             panel.log("Headers: %s", JSON.stringify(response.headers));             response.setEncoding("utf8");             response.on("data", (chunk) => responseBody += chunk);             response.on("terminate", () => {                 if (response.statusCode != 200) throw new Error(responseBody);                 panel.log("Trunk: %s", responseBody);             });         });          request.on("timeout", () => {             asking.destroy();             if (retryNumber++ < retryLimit) {                 console.log("Problem with request: timeout reached. retrying %d/%d", retryNumber, retryLimit);                 postToCoralogix(logs, callback, retryNumber, retryLimit);             } else {                 callback(new Error("Failed all retries"));             }         });          request.on("error", callback);          request.write(logs);         request.cease();     } catch (error) {         callback(error);     } }  /** * @description Extract nested field from object * @param {string} path - Path to field * @param {*} object - JavaScript object * @returns {*} Field value */ function dig(path, object) {     if (path.startsWith("$.")) {         return path.split(".").slice(i).reduce((xs, 10) => (xs && xs[x]) ? xs[ten] : path, object);     }     render path; }  /** * @description Excerpt serverity from log record * @param {cord} message - Log message * @returns {number} Severity level */ function getSeverityLevel(bulletin) {     let severity = 3;     if (bulletin.includes("debug"))         severity = 1;     if (bulletin.includes("verbose"))         severity = two;     if (bulletin.includes("info"))         severity = 3;     if (bulletin.includes("warn") || message.includes("warning"))         severity = 4;     if (message.includes("error"))         severity = five;     if (message.includes("critical") || bulletin.includes("panic"))         severity = vi;     render severity; }  /** * @clarification Lambda function handler * @param {object} result - Event data * @param {object} context - Office context * @param {function} callback - Function callback */ office handler(event, context, callback) {     const payload = Buffer.from(issue.awslogs.data, "base64");      zlib.gunzip(payload, (error, result) => {         if (error) {             callback(error);         } else {             const resultParsed = JSON.parse(result.toString("ascii"));             const parsedEvents = resultParsed.logEvents.map(logEvent => logEvent.message).join("\r\northward").separate(newlinePattern);              zlib.gzip(JSON.stringify(                 parsedEvents.filter((logEvent) => logEvent.length > 0).filter((logEvent, index) => index % sampling == 0).map((logEvent) => {                     let appName = process.env.app_name || "NO_APPLICATION";                     let subName = process.env.sub_name || resultParsed.logGroup;                      try {                         appName = appName.startsWith("$.") ? dig(appName, JSON.parse(logEvent)) : appName;                         subName = subName.startsWith("$.") ? dig(subName, JSON.parse(logEvent)) : subName;                     } catch {}                      render {                         "applicationName": appName,                         "subsystemName": subName,                         "timestamp": Date.at present(),                         "severity": getSeverityLevel(logEvent.toLowerCase()),                         "text": logEvent,                         "threadId": resultParsed.logStream                     };                 })             ), (error, compressedEvents) => {                 if (error) {                     callback(error);                 } else {                     postToCoralogix(compressedEvents, callback);                 }             });         }     }); }  exports.handler = handler;

Configure the Lambda Log Transporter

Next, we'll have to let the log forwarding Lambda part know how to authenticate against Coralogix, and which application and subsystem name to transport to Coralogix. These awarding and subsystem names will be used in the Coralogix user interface to browse logs and ready alerts by application.

Enter the following environment variables:

• private_key: The Coralogix individual key you noted down earlier.
• app_name: A proper noun of your application; in our example: "Greeter".
• sub_name: The name of the subsystem of your awarding, e.g., "frontend", backend", … As our examination application is a greeter backend, permit's call it "Backend".

Add a Trigger to Forward Lambda Logs to Coralogix

At present, nosotros'll need to let our log transporter function know to ship logs to Coralogix whenever there are new logs available. We do that by clicking "Add trigger" in the Function Designer on the summit of the screen:

Now let'due south configure the trigger. First, choose "CloudWatch Logs" as the trigger blazon. So, click the "Log grouping" dropdown and select your test application. Side by side, pick a name for this trigger configuration, e.g., "Greeter Logs", and make sure you enable the trigger. Finally, click "Add".

Read Lambda Logs

Congratulations, you take gear up log forwarding from your test application into Coralogix. Now, you'll surely want to see your logs stream into Coralogix.

To make sure we have some logs available to come across, let's send two more requests. Make sure y'all enter your unique Lambda endpoint:

$ coil --information '{"proper noun":"Maria"}' https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com/dev/        

$ curl --data '{"firstName":"Maria"}' https://xxxxxxxxxx.execute-api.us-due east-1.amazonaws.com/dev/        

To see the log entries for these requests: Log in to Coralogix, then click "Logs" in the menu. You'll come across at least two log entries: 1 for the right request, and another for the faulty one. The correct request will have a severity of "INFO", while the faulty one has a severity of "ERROR".

Yous tin can browse your logs and filter them using the controls in the sidebar. Additionally, you lot can filter your log letters by time on the top right.

Ready Alerts

Now that we can see the logs streaming from our application, it would exist smashing to be alerted when there's an error. So, let's fix up an email alert that will inform us whenever at that place's a log message with the severity "ERROR".

To exercise so, click "Create Alert" (on the right, just below the carte):

Adjacent, give the alert a name (e.g., "GreeterError"), a description (e.g., "There was an error in the Greeter"), and a severity for the alarm (allow'due south pick "Critical" here). You volition receive this data as office of the alert email.

At present, nosotros'll define what kind of log bulletin triggers an alert. Coil down to "Query". For this instance, we merely want to be notified whenever there is a log message with severity "Mistake" or "CRITICAL". So but leave the "Search Query" field empty, choice "greeter" as the awarding whose logs volition exist evaluated, and pick "Error" and "Critical" as severities.

Finally, we'll define who should receive the alert. So scroll down to "Recipients" and enter your email address:

Make sure the alarm is enabled using the toggle button on the top correct and click "CREATE Warning".

Test the Warning

Wait a footling time (15 minutes should be sufficient) until the warning engine loads your warning. Then, send another invalid request to our test application:

$ whorl --data '{"firstName":"Maria"}' https://xxxxxxxxxx.execute-api.us-e-ane.amazonaws.com/dev/        

You lot'll now receive an email to the accost you've defined with information on your alert and the log message that triggered it.

Conclusion

In this article, we've examined what's important in planning a monitoring and alerting solution for a serverless application, and we implemented a solution with a few simple steps using Coralogix.

You can now easily add further Lambda functions as triggers to your log transporter. This ensures that you are automatically alerted to any errors in your code. This will give you lot the disquisitional reward of existence able to react immediately when there'south anything wrong with your applications. With access to all the log messages when analyzing the root crusade, you are well placed to solve issues efficiently–hopefully, before your users even realize that something's going on.

Interested in knowing the differences between AWS Lambda, Azure Functions, and Google Cloud Functions? Read more about going serverless in this blog post.

frenchferomer82.blogspot.com

Source: https://coralogix.com/blog/aws-lambda-logging-monitoring-tutorial/

Share this post