OpenObserve: A High-Performance Modern Observability

Good morning, everyone! Dimitri Bellini here, and welcome back to Quadrata, my channel dedicated to the open-source world and the IT I love. As you know, I’m a big fan of my friend Zabbix, but it’s crucial to keep our eyes on the horizon, understand where the world is moving, and explore new solutions that meet the demands of our customers and the community.

That’s why today, I want to introduce you to a solution I’ve had the pleasure of getting to know: OpenObserve. It’s another powerful tool in the observability space, but it approaches the task in a refreshingly different way.

What is OpenObserve and Why Should You Care?

OpenObserve is a cloud-native, open-source observability platform designed to be a unified backend for your logs, metrics, and traces. Think of it as a lightweight yet powerful alternative to heavyweights like Elasticsearch, Splunk, or Datadog. It tackles a key challenge many of us face: consolidating different monitoring tools into a single, cohesive platform.

Instead of juggling separate tools like Prometheus for metrics, Loki for logs, and Jaeger for traces, OpenObserve brings everything under one roof. This unified approach simplifies your workflow and provides a single pane of glass to view the health of your entire infrastructure.

The Game-Changing Features

What really caught my attention are the core functionalities that make OpenObserve stand out:

• Massive Cost Reduction: This is a big one. By using a specific format called Parquet and a stateless architecture that leverages object storage (like S3, MinIO, or even a local disk), OpenObserve can drastically reduce storage costs. They claim it can be up to 140 times lower than Elasticsearch! For anyone managing hundreds of gigabytes of data per day, this is a revolutionary benefit.
• Blazing-Fast Performance: The entire engine is written in Rust. We’ve heard a lot about Rust, especially in the Linux kernel world, and for good reason. It’s an incredibly optimized and efficient language. This means OpenObserve can ingest a massive amount of data with a significantly lower memory and CPU footprint compared to Java-based solutions.
• Simplified Querying: If you’re comfortable with SQL, you’ll feel right at home. OpenObserve allows you to query your logs using standard SQL-based syntax, which dramatically lowers the learning curve. For metrics, it also supports PromQL, giving you the best of both worlds.
• Native OpenTelemetry Support: It seamlessly integrates with OpenTelemetry, the emerging standard for collecting traces and metrics. This makes it incredibly easy to instrument your applications, whether they’re written in Go, Python, or another language, and start sending data to OpenObserve.
• Real-time Alerting: Right from the UI, you can define alerts based on log patterns or metric thresholds, similar to what you might do in Prometheus.

Under the Hood: The Technology Stack

I always believe it’s fundamental to understand the components of a solution to appreciate its engineering. OpenObserve is built on a stack of impressive open-source technologies:

• Rust: The core language, providing memory safety and high performance.
• Apache Arrow DataFusion: A powerful query engine that enables the SQL support on top of Parquet files.
• Apache Parquet: A columnar storage format developed by the Apache Foundation that allows for incredible data compression and efficient querying.
• NATS: A lightweight and high-performance messaging system used for communication and coordination between nodes in a clustered setup.
• Vue.js: The framework used to build the modern and reactive web interface.
• SQLite / PostgreSQL: SQLite is used for metadata in simple, standalone deployments, while PostgreSQL is recommended for robust, high-availability production environments.

Getting Started with OpenObserve

One of the best parts is how easy it is to get started. For testing and simple use cases, you just need Docker. The architecture is straightforward: collectors like FluentBit, Vector, or OpenTelemetry send data to your OpenObserve container, which writes to a local disk. This simple setup can already handle an impressive ingestion rate of over 2TB of data per day on a single machine.

For high-availability (HA) production environments, the architecture scales out using Kubernetes, with distinct roles for routers, ingesters, queriers, and more, all coordinated by NATS and backed by object storage.

A Quick Tutorial: Installation with Docker

You can get a test environment running in minutes. It’s as simple as running a single Docker command. Here is the command I used, which you can customize with your own user and password:

    docker run -d --name openobserve \

      -p 5080:5080 \

      -e ZO_ROOT_USER_EMAIL="admin@example.com" \

      -e ZO_ROOT_USER_PASSWORD="Complexpass#123" \

      -v /opt/openobserve-data:/data \

      public.ecr.aws/zinclabs/openobserve:latest

    

I manage my containers with a tool that simplifies deployment, where I just fill in the image, ports, environment variables, and volume. It’s incredibly straightforward!

A Look at the Dashboard and Final Thoughts

Once you log in, you’re greeted with a clean dashboard showing key stats like ingested events and storage size. The “Data Sources” section is fantastic, providing you with ready-to-use instructions for ingesting data from Kubernetes, Linux, Windows, various databases, and more. This makes the initial setup a breeze.

The log exploration interface will feel familiar to anyone who has used Splunk, with powerful SQL-based querying and on-the-fly filtering. You can visualize metrics, build custom dashboards, analyze application traces with service maps, and even dive into real user monitoring.

What truly impressed me, however, is their licensing model. For self-hosted deployments, you can use the full enterprise version for free for up to 200GB of data ingestion per day. This includes features like single sign-on (SSO) and role-based access control (RBAC). This is a brilliant move that allows smaller teams and environments to leverage the full power of the platform without a cost barrier. A big round of applause to the OpenObserve team for that!

Conclusion: Keep a Close Eye on This One

So, is OpenObserve an interesting solution? Absolutely. It’s a project to watch closely. It has a smart approach—a lightweight, non-pachydermic solution built with exciting technologies like Rust and Parquet. It seems to have a finesse that sets it apart from the many other open-source observability tools out there.

I encourage you to take a look at it. The project is moving fast, and it offers a compelling combination of performance, cost-efficiency, and user-friendliness.

That’s all for today! Let me know your thoughts in the comments below. Do you find these all-in-one observability solutions useful? I’d love to hear from you.

A greeting from Dimitri, see you next week!

Don’t forget to like this video and subscribe to my channel for more open-source content:

My YouTube Channel: Quadrata

Join the conversation on Telegram: Zabbix Italia