What is CloudWatch Logs, and How Does it Differ from Metrics?

• CloudWatch Logs: A service that monitors, stores, and manages log files from various AWS resources (like EC2 instances, Lambda functions, VPC Flow Logs, etc.). It allows you to store logs and analyze them using CloudWatch Logs Insights.
• Difference from Metrics:
  • Logs contain raw, detailed information about system events, actions, or messages produced by applications or services.
  • Metrics are numerical data points that describe the performance or health of AWS resources (e.g., CPU utilization, network traffic). Metrics provide summarized data, while logs provide granular, detailed information.

Configuring Log Groups and Streams

• Log Groups: A container for logs that allows you to organize and manage logs. Each log group contains multiple log streams.
• Log Streams: A sequence of log events from a single source (e.g., an EC2 instance or Lambda function). Each log stream is part of a log group.

Example:

• You can create a log group called EC2-Logs, and within it, have separate log streams for each instance like i-12345678 and i-87654321.

How to Send Logs to CloudWatch

From EC2

Use amazon-cloudwatch-agent for EC2 Logging (the newer solution)

1. Install the CloudWatch Agent:

   • Amazon Linux:

     sudo yum install -y amazon-cloudwatch-agent
     

   • Ubuntu:

     sudo apt-get update
     sudo apt-get install -y amazon-cloudwatch-agent
     

2. Create the Configuration File:

   • Define your logging and metrics configuration in amazon-cloudwatch-agent.json (usually placed in /opt/aws/amazon-cloudwatch-agent/bin).

   • Example configuration:

     {
       "logs": {
         "logs_collected": {
           "files": {
             "collect_list": [
               {
                 "file_path": "/var/log/syslog",
                 "log_group_name": "MyAppLogGroup",
                 "log_stream_name": "{instance_id}/syslog"
               },
               {
                 "file_path": "/var/log/app.log",
                 "log_group_name": "MyAppLogGroup",
                 "log_stream_name": "{instance_id}/app"
               }
             ]
           }
         }
       }
     }
     

3. Start the CloudWatch Agent:

   • Apply the configuration and start the agent:

     sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \\
     -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent.json -s
     

4. Verify in CloudWatch Console:

   • Check CloudWatch > Logs for MyAppLogGroup and ensure logs are streaming from EC2.

From Lambda

Lambda functions automatically send logs to CloudWatch, which you can view in the Lambda console.

From VPC Flow Logs