Logging Documentation

Overview

This document outlines the comprehensive logging strategy for the Dispatch Center Application, covering logging frameworks, standards, storage, analysis, and compliance requirements.

Table of Contents

• Logging Architecture
• Logging Framework
• Log Levels and Standards
• Log Storage and Retention
• Security and Privacy
• Log Analysis and Monitoring
• Structured Logging
• Correlation and Tracing
• Performance Considerations
• Compliance and Auditing

Logging Architecture

flowchart TB
    app_components["Application<br/>Components"]
    serilog["Serilog<br/>Pipeline"]
    azure_log["Azure Log<br/>Analytics"]
    log_aggregation["Log<br/>Aggregation"]
    enrichment["Enrichment<br/>& Filtering"]
    long_term["Long-term<br/>Storage"]

    app_components --> serilog
    serilog --> azure_log
    app_components --> log_aggregation
    serilog --> enrichment
    azure_log --> long_term

Logging Framework

Serilog Configuration

Basic Setup

public static class LoggingConfiguration
{
    public static ILogger CreateLogger(IConfiguration configuration)
    {
        return new LoggerConfiguration()
            .MinimumLevel.Information()
            .MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
            .MinimumLevel.Override("System", LogEventLevel.Warning)
            .Enrich.FromLogContext()
            .Enrich.WithMachineName()
            .Enrich.WithEnvironmentName()
            .Enrich.WithProperty("Application", "DispatchCenter")
            .WriteTo.Console(new JsonFormatter())
            .WriteTo.AzureAnalytics(
                workspaceId: configuration["Logging:WorkspaceId"],
                authenticationId: configuration["Logging:AuthenticationId"])
            .CreateLogger();
    }
}

Dependency Injection Setup

public void ConfigureServices(IServiceCollection services)
{
    services.AddLogging(builder =>
    {
        builder.ClearProviders();
        builder.AddSerilog();
    });

    services.AddScoped<ICorrelationService, CorrelationService>();
    services.AddScoped<ILogEnrichmentService, LogEnrichmentService>();
}

Log Formatting

JSON Structured Format

{
    "@timestamp": "2025-11-03T10:30:00.123Z",
    "@level": "Information",
    "@message": "Service request {ServiceRequestId} assigned to technician {TechnicianId}",
    "@exception": null,
    "ServiceRequestId": "12345-67890",
    "TechnicianId": "TECH001",
    "CustomerId": "CUST001",
    "CorrelationId": "abc123-def456",
    "UserId": "user@company.com",
    "MachineName": "APP-SERVER-01",
    "Environment": "Production",
    "Application": "DispatchCenter"
}

Log Levels and Standards

Log Level Definitions

TRACE

• Purpose: Detailed execution flow for debugging
• Usage: Method entry/exit, detailed state information
• Environment: Development and testing only
• Example:

  _logger.LogTrace("Entering method {MethodName} with parameters {@Parameters}", 
      nameof(CreateServiceRequest), parameters);
  

DEBUG

• Purpose: Development and troubleshooting information
• Usage: Variable values, decision points, intermediate results
• Environment: Development and staging
• Example:

  _logger.LogDebug("Processing service request {ServiceRequestId} with status {Status}", 
      serviceRequestId, status);
  

INFORMATION

• Purpose: General application flow and business events
• Usage: Business process milestones, successful operations
• Environment: All environments
• Example:

  _logger.LogInformation("Service request {ServiceRequestId} successfully created for customer {CustomerId}", 
      serviceRequest.Id, serviceRequest.CustomerId);
  

WARNING

• Purpose: Potential issues that don't affect functionality
• Usage: Deprecated features, fallback scenarios, recoverable errors
• Environment: All environments
• Example:

  _logger.LogWarning("External service {ServiceName} responded slowly ({ResponseTime}ms), using cached data", 
      serviceName, responseTime);
  

ERROR

• Purpose: Error conditions that impact functionality
• Usage: Handled exceptions, business rule violations, integration failures
• Environment: All environments
• Example:

  _logger.LogError(ex, "Failed to process service request {ServiceRequestId} for customer {CustomerId}", 
      serviceRequestId, customerId);
  

CRITICAL/FATAL

• Purpose: Critical errors that may cause application termination
• Usage: Unhandled exceptions, security violations, data corruption
• Environment: All environments
• Example:

  _logger.LogCritical(ex, "Database connection failed. Application cannot continue");
  

Log Storage and Retention

Storage Strategy

Primary Storage - Azure Log Analytics

• Retention: 90 days for application logs
• Retention: 1 year for audit logs
• Retention: 2 years for security logs
• Query Performance: Optimized for real-time analysis
• Cost Management: Hot/warm/cold storage tiers

Archive Storage - Azure Storage

• Purpose: Long-term compliance and historical analysis
• Format: Compressed JSON files
• Retention: 7 years for audit compliance
• Access Pattern: Infrequent access with restore capability

Retention Policies

retention_policies:
  application_logs:
    hot_tier: "30 days"
    warm_tier: "60 days"
    archive_tier: "7 years"

  audit_logs:
    hot_tier: "90 days"
    warm_tier: "1 year"
    archive_tier: "7 years"

  security_logs:
    hot_tier: "180 days"
    warm_tier: "2 years"
    archive_tier: "10 years"

Security and Privacy

Personally Identifiable Information (PII) Protection

Data Classification

public class DataClassificationAttribute : Attribute
{
    public DataClassification Classification { get; }

    public DataClassificationAttribute(DataClassification classification)
    {
        Classification = classification;
    }
}

public enum DataClassification
{
    Public,
    Internal,
    Confidential,
    Secret
}

Automatic PII Masking

public class PiiMaskingEnricher : ILogEventEnricher
{
    private readonly Regex _emailPattern = new Regex(@"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b");
    private readonly Regex _phonePattern = new Regex(@"\b\d{3}-\d{3}-\d{4}\b");
    private readonly Regex _ssnPattern = new Regex(@"\b\d{3}-\d{2}-\d{4}\b");

    public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
    {
        var messageTemplate = logEvent.MessageTemplate.Text;

        messageTemplate = _emailPattern.Replace(messageTemplate, "***@***.***");
        messageTemplate = _phonePattern.Replace(messageTemplate, "***-***-****");
        messageTemplate = _ssnPattern.Replace(messageTemplate, "***-**-****");

        // Update message template with masked values
    }
}

Access Control

Role-Based Log Access

log_access_roles:
  developers:
    - application_logs:read
    - debug_logs:read
    environments: ["development", "staging"]

  operations:
    - application_logs:read
    - infrastructure_logs:read
    - security_logs:read
    environments: ["all"]

  security_team:
    - audit_logs:read
    - security_logs:read
    - compliance_logs:read
    environments: ["all"]

  auditors:
    - audit_logs:read
    environments: ["production"]

Audit Trail

Security Event Logging

public class SecurityLogger
{
    private readonly ILogger<SecurityLogger> _logger;

    public void LogSecurityEvent(SecurityEventType eventType, string userId, string details)
    {
        _logger.LogWarning("Security Event: {EventType} for user {UserId}. Details: {Details}",
            eventType, userId, details);
    }

    public void LogDataAccess(string userId, string resource, string operation)
    {
        _logger.LogInformation("Data Access: User {UserId} performed {Operation} on {Resource}",
            userId, operation, resource);
    }
}

Log Analysis and Monitoring

KQL Queries for Common Scenarios

Error Analysis

AppLogs
| where TimeGenerated > ago(24h)
| where Level == "Error"
| summarize count() by bin(TimeGenerated, 1h), tostring(Properties.ServiceName)
| render timechart

Performance Analysis

AppTraces
| where TimeGenerated > ago(1h)
| where Message contains "Service request"
| extend Duration = todouble(Properties.Duration)
| summarize avg(Duration), max(Duration), min(Duration) by bin(TimeGenerated, 5m)
| render timechart

User Activity Tracking

AppLogs
| where TimeGenerated > ago(24h)
| where Properties.UserId != ""
| summarize operations = count() by UserId = tostring(Properties.UserId)
| top 20 by operations desc

Automated Log Analysis

Anomaly Detection

public class LogAnomalyDetection
{
    public async Task<List<Anomaly>> DetectAnomalies()
    {
        var query = @"
            AppLogs
            | where TimeGenerated > ago(24h)
            | summarize count() by bin(TimeGenerated, 1h)
            | make-series count on TimeGenerated step 1h
            | extend anomalies = series_decompose_anomalies(count)";

        // Execute query and process results
    }
}

Structured Logging

Context Enrichment

Request Context

public class RequestContextMiddleware
{
    public async Task InvokeAsync(HttpContext context, RequestDelegate next)
    {
        var correlationId = context.Request.Headers["X-Correlation-ID"].FirstOrDefault() 
            ?? Guid.NewGuid().ToString();

        using (LogContext.PushProperty("CorrelationId", correlationId))
        using (LogContext.PushProperty("UserId", context.User?.Identity?.Name))
        using (LogContext.PushProperty("UserAgent", context.Request.Headers["User-Agent"]))
        {
            await next(context);
        }
    }
}

Business Context

public class ServiceRequestService
{
    public async Task<ServiceRequest> CreateServiceRequestAsync(CreateServiceRequestCommand command)
    {
        using (LogContext.PushProperty("ServiceRequestId", command.Id))
        using (LogContext.PushProperty("CustomerId", command.CustomerId))
        using (LogContext.PushProperty("ServiceType", command.ServiceType))
        {
            _logger.LogInformation("Creating service request for customer {CustomerId}", command.CustomerId);

            // Business logic here

            _logger.LogInformation("Service request created successfully");
            return serviceRequest;
        }
    }
}

Correlation and Tracing

Distributed Tracing

Correlation ID Management

public interface ICorrelationService
{
    string GetCorrelationId();
    void SetCorrelationId(string correlationId);
}

public class CorrelationService : ICorrelationService
{
    private readonly IHttpContextAccessor _httpContextAccessor;

    public string GetCorrelationId()
    {
        return _httpContextAccessor.HttpContext?.Items["CorrelationId"]?.ToString()
            ?? Guid.NewGuid().ToString();
    }

    public void SetCorrelationId(string correlationId)
    {
        if (_httpContextAccessor.HttpContext != null)
        {
            _httpContextAccessor.HttpContext.Items["CorrelationId"] = correlationId;
        }
    }
}

Activity Tracking

public class ActivityTrackingService
{
    private readonly ILogger<ActivityTrackingService> _logger;

    public async Task<T> TrackActivityAsync<T>(string activityName, Func<Task<T>> activity)
    {
        using var activitySource = new ActivitySource("DispatchCenter");
        using var activityListener = new Activity(activityName);

        var stopwatch = Stopwatch.StartNew();

        try
        {
            _logger.LogInformation("Starting activity {ActivityName}", activityName);
            var result = await activity();
            _logger.LogInformation("Completed activity {ActivityName} in {Duration}ms", 
                activityName, stopwatch.ElapsedMilliseconds);
            return result;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed activity {ActivityName} after {Duration}ms", 
                activityName, stopwatch.ElapsedMilliseconds);
            throw;
        }
    }
}

Performance Considerations

Asynchronous Logging

public static class LoggingConfiguration
{
    public static ILogger CreateAsyncLogger(IConfiguration configuration)
    {
        return new LoggerConfiguration()
            .WriteTo.Async(a => a.AzureAnalytics(
                workspaceId: configuration["Logging:WorkspaceId"],
                authenticationId: configuration["Logging:AuthenticationId"],
                bufferSize: 1000,
                batchPostingLimit: 100))
            .CreateLogger();
    }
}

Sampling Strategy

public class SamplingFilter : ILogEventFilter
{
    private readonly Random _random = new Random();

    public bool IsEnabled(LogEvent logEvent)
    {
        // Sample debug logs at 10%
        if (logEvent.Level == LogEventLevel.Debug)
            return _random.NextDouble() < 0.1;

        // Sample trace logs at 1%
        if (logEvent.Level == LogEventLevel.Verbose)
            return _random.NextDouble() < 0.01;

        // Always log info and above
        return true;
    }
}

Resource Management

• Buffer Management: Configure appropriate buffer sizes
• Network Resilience: Implement retry policies for log shipping
• Memory Usage: Monitor logging overhead
• Disk Space: Manage local log file sizes

Compliance and Auditing

Regulatory Compliance

• SOX Compliance: Financial transaction audit trails
• GDPR Compliance: Data processing activity logs
• HIPAA Compliance: Healthcare data access logs (if applicable)
• SOC 2 Compliance: Security and availability monitoring

Audit Requirements

public class AuditLogger
{
    public void LogDataAccess(string userId, string dataType, string recordId, string operation)
    {
        _logger.LogInformation("AUDIT: User {UserId} performed {Operation} on {DataType} record {RecordId}",
            userId, operation, dataType, recordId);
    }

    public void LogConfigurationChange(string userId, string component, object oldValue, object newValue)
    {
        _logger.LogWarning("AUDIT: User {UserId} changed {Component} from {OldValue} to {NewValue}",
            userId, component, oldValue, newValue);
    }
}

Compliance Monitoring

• Data Access Patterns: Monitor unusual access patterns
• Configuration Changes: Track all system modifications
• User Activity: Comprehensive user action logging
• Data Export: Track data extraction and sharing

---

Document Version: 1.0
Last Updated: January 2026
Next Review: April 2026