When and How to Search with Amazon CloudWatch Logs


Amazon CloudWatch Logs provides storage, processing, analysis, and monitoring of a wide variety of log data from multiple sources, such as other AWS services. The service offers search functionality and integration capabilities with more sophisticated mechanisms for more detailed log analysis, at any scale.

Developers can configure their applications to export log data to CloudWatch Logs instead of keeping it on their own servers or EC2 instances. Once log data arrives in CloudWatch Logs, developers have the option to configure a retention period, export it to Amazon S3, or trigger custom actions on incoming log events. There is also the ability to search and analyze log data directly in the CloudWatch Logs service.

Follow this step-by-step CloudWatch Logs tutorial and explore other tools for working with log data.

Log events, groups and flows

There are three main CloudWatch Logs concepts that users should be aware of.

Logging event. When a monitored application or resource records activity, it creates a log event. The event receives a timestamp and the raw event message.

Newspaper groups. A log group is the top-level grouping related to log data.

Log streams. Streams are a way to group incoming log events into a log group.

For example, AWS sends log events related to a Lambda function to a log group dedicated to that particular function. Events are divided into log streams containing log events for specific runs.

For custom logs, such as Apache access logs generated in EC2 instances, it’s common to create a log group for the application and log type, i.e. access and l error, then one log stream per EC2 instance. When developers configure custom application logs, they can specify how to group log events into log groups and log streams. For AWS Integrated Logging, the granularity of log groups and log streams is defined by each AWS service that sends data to CloudWatch Logs.

How to Search with CloudWatch Logs

AWS provides many options for performing searches on log data. The easiest is through the CloudWatch Logs console.

Step 1. Select the applicable log group. There is an option to select a particular log stream or to search all events in the whole log group.

When selecting a particular log group from the Log Groups screen, the console offers the menu shown in Figure 1.

Figure 1

2nd step. After clicking Research Journal Groupthe console displays Figure 2, which is also displayed when users click on a particular log stream.

Search this form on Amazon CloudWatch Logs to find specific text in groups or streams
Figure 2

This gives users the option to choose a relative time range or select a custom time range with specific start and end timestamps. This feature performs a text search on selected log data, making it easy to search for specific text at the log group or log stream level.

This option supports filter patterns in the search field, as in the following examples:

  • STRING OF CHARACTERS. This pattern would look for STRING in the selected log data.
  • ?CHAIN1 -CHAIN2. This pattern would search for log events containing STRING1, but exclude those with STRING2.
  • ?CHANNEL1 ?CHANNEL2. This pattern would search for both STRING1 or STRING2

For space-delimited logs, such as Apache access logs, search filters can analyze and filter specific fields. Here is an example Apache access log record for a failed health check from the Application Load Balancer (ALB): [1/Jun/2022:10:10:10 +0000] "GET / HTTP/1.1" 500 800  "ELB-HealthChecker/2.0"

The fields in this example can be parsed using the following pattern:

[ip, timestamp, request, status_code, bytes, user_agent]

Records that failed ALB health checks can be searched using the following filter, which returns log events with a user_agent “ELB-HealthChecker” and status_code with the value 500:

[ip, timestamp, request, status_code=500, bytes, user_agent="ELB-HealthChecker*"]

AWS Software Development Kit and CLI

For automation purposes, developers can also use the AWS SDK or CLI to find CloudWatch Logs. The filter log events The CLI command can be used as follows:

aws logs filter-log-events --log-group-name '' --start-time  --filter-pattern '

The CLI allows other fields, such as end time, log-stream-name, log-stream-name-prefix and max items to further refine the search parameters.

CloudWatch Logs Insights and Other Tools

CloudWatch Logs Insights is also a popular option for searching logs. It offers a query syntax language that can be used to filter, analyze, and aggregate log records over a given time period. This tool allows a more detailed search functionality compared to the standard text search.

Since log data can also be exported to S3, this allows for detailed analysis using Athena or other data analysis platforms available through AWS EMR. This is a feasible option for applications that produce a large volume of log data or require long-term retention of log data. To analyze log data in S3 using tools such as Athena, ensure that log records have a consistent pattern that can be parsed into table columns that will be used by log language statements. definition of data required by data analysis tools.

Subscription filters are another useful feature that can be used to transform and export log data across multiple services and perform other research and analysis processes. Target services include the following:

  • Amazon Kinesis data streams. This serverless real-time data stream service collects, processes and stores large amounts of data
  • Amazon Kinesis Data Firehose. This extract, transform, and load service captures, transforms, and loads streaming data to data lakes, warehouses, and analytics services.
  • Amazon OpenSearch. This open source research and analysis suite allows developers to visualize and analyze data.
  • AWS Lambda. With this serverless service, developers can use Lambda functions to transform and export log data to other services, such as S3 or Amazon Redshift, or external data analytics platforms.

Comments are closed.