Kibana for ft_transcendence: Visualization and Management

What is Kibana?

Kibana is Elasticsearch's visualization layer, providing a user-friendly interface to search, view, and interact with data stored in Elasticsearch indices. For the ft_transcendence project, Kibana serves as the central dashboard for our observability and security monitoring.

Current Kibana Version: 8.17.4 (matching our Elasticsearch version)

Why Kibana for ft_transcendence?

Kibana provides several critical capabilities for our project:

1. Real-time Log Visualization: Monitor application health and performance
2. Security Dashboards: Track user activities and potential security issues
3. Alerting: Get notified of critical system events
4. Index Management: Efficiently manage Elasticsearch indices

Installation Approach

We're using a Docker-based approach with Docker Compose to integrate Kibana with our Elasticsearch instance. This provides a clean, reproducible environment without installing anything directly on the host.

Configuration Details

kibana.yml

# Server settings
server.name: kibana
server.host: '0.0.0.0'
server.port: 5601

# Elasticsearch connection
elasticsearch.hosts: ['http://elasticsearch:9200']
elasticsearch.username: '${ELASTIC_USER}'
elasticsearch.password: '${ELASTIC_PASSWORD}'

# Security settings
xpack.security.enabled: true
xpack.encryptedSavedObjects.encryptionKey: '${ENCRYPTION_KEY}'

# Monitoring
monitoring.ui.container.elasticsearch.enabled: true

# CORS and other access settings
server.cors.enabled: true
server.cors.allow_origin: ['*']

Docker Compose Configuration

Here's how we've integrated Kibana with our Docker Compose setup:

kibana:
  image: docker.elastic.co/kibana/kibana:8.17.4
  container_name: ft_kibana
  ports:
    - '5601:5601'
  environment:
    - ELASTIC_USER=elastic
    - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    - ENCRYPTION_KEY=${KIBANA_ENCRYPTION_KEY}
  volumes:
    - ./config/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
  networks:
    - elastic
  depends_on:
    - elasticsearch

Key Kibana Features for ft_transcendence

Discover

The Discover tab allows us to search through logs and events:

• Filter logs by service (Django, Nginx, etc.)
• Full-text search across all log messages
• Timeline view for tracking event sequences

Dashboards

We've configured several custom dashboards for ft_transcendence:

1. Application Health Dashboard:

   • API response times
   • Error rates
   • Request volume

2. User Activity Dashboard:

   • Login/logout events
   • User actions
   • Session durations

3. Security Monitoring Dashboard:

   • Failed authentication attempts
   • Unusual access patterns
   • Security alerts

Dev Tools

The Console interface enables direct interaction with Elasticsearch's REST API:

• Run direct queries against indices
• Manage index templates
• Test complex aggregations

Securing Kibana

While our development setup has basic security, here are important considerations:

1. Authentication: Kibana inherits user management from Elasticsearch
2. TLS: Enable HTTPS for all Kibana traffic
3. Space-Based Authorization: Create separate spaces for different teams/purposes
4. Minimal Privileges: Create role-based access with least privilege

Getting Started with Kibana

Once the stack is running, access Kibana at:

• URL: http://localhost:5601 (or https://transcendence.abdellahelmrabet.me/kibana)
• Username: elastic
• Password: The value of ELASTIC_PASSWORD environment variable

Initial Setup Steps

1. Define index patterns for your log sources
2. Import pre-configured dashboards (if available)
3. Create visualizations for specific metrics
4. Configure alerts for critical events

Performance Considerations

For our limited development environment:

• Keep dashboards simple with few visualizations
• Limit auto-refresh rates to reduce load
• Use time filters to narrow data ranges
• Consider using rollup indices for historical data

In the next article, we'll explore how Filebeat collects logs from our ft_transcendence services.