CodeWithAbdessamad

Logging

Logging

In the world of backend systems, logging is the unsung hero of observability. It’s the lifeline that helps us understand what’s happening in our applications, diagnose issues, and ensure smooth operations. But raw, unstructured logs can quickly become a nightmare to parse and analyze—especially as systems scale. This section dives into two critical practices that transform logging from a simple record-keeping tool into a powerful asset for building resilient systems: structured logs and centralized logging.

Structured Logs

Structured logs are logs that follow a consistent, machine-readable format—typically JSON—enabling automated processing, analysis, and integration with monitoring tools. Unlike plain text logs (which are human-readable but machine-unfriendly), structured logs provide a clear schema for data, making it easy to extract specific fields, correlate events, and feed into analytics pipelines.

The key benefit of structured logs is reproducibility and machine-actionability. For example, a structured log for a user request might include fields like userid, requestmethod, status_code, and timestamp. This allows tools like Elasticsearch or Splunk to query these fields without parsing messy text.

Here’s how to implement structured logs in a Node.js application using the winston library:

<code class="language-javascript">const winston = require('winston');

<p>const logger = winston.createLogger({</p>
<p>  level: 'info',</p>
<p>  transports: [</p>
<p>    new winston.transports.Console(),</p>
<p>    new winston.transports.File({ filename: 'logs/error.log', level: 'error' })</p>
<p>  ]</p>
<p>});</p>

<p>// Log a structured message</p>
<p>logger.info({</p>
<p>  message: 'User request processed',</p>
<p>  user<em>id: 'user</em>123',</p>
<p>  request_method: 'GET',</p>
<p>  request_url: '/api/users',</p>
<p>  response_status: 200</p>
<p>});</code>

This example logs a JSON object. The winston library automatically serializes the object to JSON, ensuring the log is structured and machine-readable. Note that the message field is optional—you can omit it if you prefer to use the entire object as the log message.

When using structured logs, always include a consistent set of fields that help with tracing and debugging. For instance, in distributed systems, add traceid and spanid to correlate requests across services.

Structured logs are the foundation for effective logging. Without them, even the most advanced centralized logging systems become a black box.

Centralized Logging

As applications grow, logs from multiple services and environments become scattered and difficult to manage. This is where centralized logging comes in: the practice of collecting logs from all sources into a single, unified system for analysis and monitoring.

Centralized logging provides several critical advantages:

  • Correlation: Linking logs across services to understand end-to-end user journeys.
  • Scalability: Handling massive log volumes from thousands of services without performance degradation.
  • Real-time analysis: Detecting and responding to issues as they happen.
  • Alerting: Setting up rules to trigger alerts based on specific patterns (e.g., failed authentication attempts).

A popular open-source solution for centralized logging is Loki (by Grafana Labs). Loki is designed to be lightweight and efficient, making it ideal for cloud-native environments. Here’s a minimal setup for Loki:

  1. Install Loki (and its companion, PromQL) on your infrastructure.
  2. Configure applications to send logs to Loki using a client library.

Here’s a simple example of a Node.js application logging to Loki:

<code class="language-javascript">const { LokiClient } = require('loki-client');

<p>const lokiClient = new LokiClient({</p>
<p>  url: 'http://loki:3100/loki/api/v1/push',</p>
<p>  headers: { 'Content-Type': 'application/json' }</p>
<p>});</p>

<p>// Log to Loki</p>
<p>lokiClient.push({</p>
<p>  streams: [</p>
<p>    {</p>
<p>      stream: {</p>
<p>        job: 'user-service',</p>
<p>        logs: [</p>
<p>          {</p>
<p>            timestamp: new Date().toISOString(),</p>
<p>            message: 'User request processed',</p>
<p>            user<em>id: 'user</em>123',</p>
<p>            request_method: 'GET',</p>
<p>            request_url: '/api/users',</p>
<p>            response_status: 200</p>
<p>          }</p>
<p>        ]</p>
<p>      }</p>
<p>    }</p>
<p>  ]</p>
<p>});</code>

Note: In production, you’d use a logging library that handles HTTP requests and authentication. Loki excels at low resource usage and seamless integration with Grafana.

Centralized logging also enables log aggregation—the process of gathering logs from all sources into a single repository. Tools like ELK (Elasticsearch, Logstash, Kibana), Splunk, and Datadog provide end-to-end solutions. For example:

  1. Logstash ingests logs from multiple sources.
  2. Elasticsearch indexes logs for fast searching.
  3. Kibana visualizes the data.

While ELK is powerful, it requires more resources and configuration. Loki is a great choice for smaller to medium-sized applications that need simplicity and scalability.

Summary

Structured logs transform raw log entries into actionable data using consistent, machine-readable formats like JSON. This enables efficient processing, correlation, and integration with monitoring tools. Centralized logging then brings all structured logs from distributed systems into a single, unified system for real-time analysis, alerting, and troubleshooting. Together, these practices form the backbone of observability in modern backend engineering.

By implementing structured logs and centralized logging, you empower your team to build systems that are not only scalable but also resilient and easy to debug. 🌟