Posts Taged open-source-monitoring

Home Posts Tagged "Open Source Monitoring"
The Ultimate Guide to Open Source Backups: Kopia vs. Minarca
By Dimitri Bellini20 June, 2025Blog

The Ultimate Guide to Open Source Backups: Kopia vs. Minarca

Good morning, everyone! It’s Dimitri Bellini, and welcome back to Quadrata, my channel dedicated to the world of open source. A few weeks ago, I had one of those heart-stopping moments we all dread: a chunk of my data suddenly became inaccessible. While I had some backup systems in place, they weren’t as reliable as I thought. That scare sent me on a mission to find the best, most robust open-source backup solution out there.

We often don’t think about backups until it’s too late. Whether it’s a broken hard drive, corrupted files, or the dreaded ransomware attack, having a solid backup strategy is the only thing that stands between you and disaster. But remember, as the saying goes, “The backup is guaranteed, but the restore is another story.” A good tool needs to handle both flawlessly.

My Hunt for the Perfect Backup Tool: The Checklist

Before diving into the options, I set some clear requirements for my ideal solution. It needed to be more than just a simple file-copying utility. Here’s what I was looking for:

  • Truly Open Source: The solution had to be freely available and transparent.

  • Cross-Platform: It must work seamlessly across Windows, Linux, and macOS.

  • Powerful Features: I was looking for advanced capabilities like:

    • Deduplication: A smart feature that saves significant space by not storing the same file or block of data more than once. This is different from compression, which just shrinks individual files.

    • Encryption: End-to-end encryption is a must, especially if I’m sending my data to a cloud provider. My data should be for my eyes only.

  • Flexible & Versatile: It needed a Command Line Interface (CLI) for automation and a Graphical User Interface (GUI) for ease of use. It also had to support a wide range of storage backends, from a local NAS (using protocols like SSH, NFS, SFTP) to cloud storage like Amazon S3 and Backblaze B2.

  • Simplicity: Finally, it couldn’t be a convoluted mess of scripts. The setup and daily management had to be straightforward.

The Contenders: Kopia vs. Minarca

My research led me to several interesting projects like Restic, Borg, and UrBackup, but two stood out for meeting almost all my criteria: Kopia and Minarca. They represent two fundamentally different philosophies in the backup world, and I put both to the test.

Deep Dive #1: Kopia – The Power User’s Choice

Kopia is a modern, feature-rich, agent-based backup tool. This means all the intelligence—deduplication, encryption, scheduling—resides in the agent you install on your computer. It directly manages the data in your chosen storage repository without needing a server component on the other end.

What I Love About Kopia (Pros):

  • Advanced Features: It offers fantastic cross-device deduplication, strong end-to-end encryption, compression, and error correction to ensure your data is always valid.

  • Snapshot-Based: Kopia creates point-in-time “snapshots” of your directories. This is great for versioning and ensures a consistent state for each backup.

  • Incredibly Versatile: It supports virtually every storage backend you can think of.

  • Resilient by Design: All your configuration policies are stored within the remote repository itself. If your computer dies, you can install Kopia on a new machine, connect to your repository, and all your settings and backup history are immediately available for a restore.

Potential Hurdles (Cons):

  • Learning Curve: While the GUI simplifies things, the underlying policy system for managing snapshot retention, scheduling, and exclusions is incredibly detailed and can be a bit overwhelming for beginners.

  • No Simultaneous Multi-Repository: It doesn’t natively support backing up to multiple repositories at the exact same time in a single job.

A Quick Kopia Walkthrough

Using Kopia involves three main steps:

  1. Connect to a Repository: You start by pointing the Kopia UI to your storage location, whether it’s a local folder, an SFTP server, or a cloud bucket like Backblaze B2. You’ll set a password that encrypts everything.

  2. Create a Snapshot: You then select a folder you want to back up and Kopia will create a snapshot. The first run will upload everything, but subsequent snapshots are incredibly fast, as it only processes and uploads the changes, thanks to deduplication.

  3. Restore: To restore, you browse your snapshots by date and time, find the files or folders you need, and either download them or “mount” the entire snapshot as a local drive to browse through it.

Deep Dive #2: Minarca – Simplicity and Centralized Control

Minarca takes a more traditional client-server approach. You install a lightweight agent on your machines and a server component on a central server (which can be a simple Linux box in your home or office). This architecture is fantastic for managing multiple devices.

Where Minarca Shines (Pros):

  • Incredibly User-Friendly: Both the agent and the server have beautiful, simple interfaces. It’s very intuitive to set up and manage.

  • Centralized Management: The web-based server dashboard is the star of the show. It gives you a complete overview of all your users and devices, with stats on backup duration, space used, and new or modified files. You can even restore files for any user directly from the web interface!

  • Multi-User/Multi-Device: It’s built from the ground up to support multiple users, each with multiple devices, all backing up to one central location.

  • Easy Installation: Setting up the Minarca server on an Ubuntu or Debian machine takes just a few simple commands.

What’s Missing (Cons):

  • No Built-in Advanced Features: Minarca currently leaves features like deduplication and encryption to the underlying storage system (for example, using a ZFS filesystem on your server).

  • File-Based, Not Snapshot-Based: It works by copying files (incrementally) rather than creating atomic snapshots like Kopia.

A Quick Minarca Walkthrough

Getting started with Minarca is a breeze:

  1. Set Up the Server: Install the Minarca server package on a Linux machine.

  2. Configure the Agent: On your client PC, install the Minarca agent and point it to your server’s address with the credentials you created.

  3. Create a Backup Job: In the agent, you can easily select which folders to include or exclude and set a simple backup schedule.

  4. Restore: The agent’s restore interface presents a calendar. You just click a date to see the available backup and browse the files you want to recover. Or, as an admin, you can do this from the central web dashboard.

Kopia vs. Minarca: Which One Is for You?

After testing both extensively, it’s clear they serve different needs.

Choose Kopia if: You are a power user who wants the most advanced features like end-to-end encryption and high-efficiency deduplication built right in. You are comfortable with a more technical setup and prefer a decentralized, agent-only approach.

Choose Minarca if: You value simplicity, ease of use, and centralized control. It’s the perfect solution if you need to manage backups for your family or a small office, and you want a clean dashboard to monitor everything at a glance.

My Final Thoughts and Your Turn

Both Kopia and Minarca are fantastic, robust open-source solutions that put you in control of your data. They are both miles ahead of just hoping for the best. The most important takeaway is this: don’t be like me and wait for a scare. Set up your backups today!

I’d love to hear from you. Which of these tools sounds more appealing to you? Are you using another open-source solution that you love? Let me know in the comments below. Your feedback is always welcome!

Thank you for reading, and don’t forget to subscribe to my YouTube channel for more deep dives into the world of open source.

Until next time, this is Dimitri. Ciao!

Read More

0

SigNoz: A Powerful Open Source APM and Observability Tool
By Dimitri Bellini12 June, 2025Blog

Diving Deep into SigNoz: A Powerful Open Source APM and Observability Tool

Good morning everyone, I’m Dimitri Bellini, and welcome back to Quadrata, the channel where we explore the fascinating world of open source and IT. As I always say, I hope you enjoy these videos, and if you haven’t already, please consider subscribing and hitting that like button if you find the content valuable!

While Zabbix always holds a special place in our hearts for monitoring, today I want to introduce something different. I’ve been getting requests from customers about how to monitor their applications, and for that, you typically need an Application Performance Monitor (APM), or as it’s sometimes fancily called, an “Observability Tool.”

Introducing SigNoz: Your Open Source Observability Hub

The tool I’m excited to share with you today is called SigNoz. It’s an open-source solution designed for comprehensive observability, which means it helps you monitor metrics, traces (the calls made within your application), and even logs. This last part is a key feature of SigNoz, as it aims to incorporate everything you might need to keep a close eye on your applications.

One of its core strengths is that it’s built natively on OpenTelemetry. OpenTelemetry is becoming an industry standard for collecting telemetry data (metrics, traces, logs) from your applications and transmitting it to a backend like SigNoz. We’ll touch on the advantages of this later.

Why Consider SigNoz?

SigNoz positions itself as an open-source alternative to paid, proprietary solutions like Datadog or New Relic, which can be quite expensive. Of course, choosing open source isn’t just about avoiding costs; it’s also about flexibility and community. For home labs, small projects, or even just for learning, SigNoz can be incredibly useful.

Key Features of SigNoz

  • Application Performance Monitoring (APM): Out-of-the-box, you get crucial metrics like P99 latency, error rates, requests per second, all neatly presented in dashboards.

  • Distributed Tracing: This allows you to follow the path of a request as it travels through your application, helping you pinpoint bottlenecks and errors.

  • Log Management: A relatively recent but powerful addition, SigNoz can ingest logs, allowing you to search and analyze them, similar to tools like Greylog (though perhaps with fewer advanced log-specific features for now).

  • Metrics and Dashboards: SigNoz provides a user-friendly interface with customizable dashboards and widgets.

  • Alerting: You can set up alerts, much like triggers in Zabbix, to get notified via various channels when something goes wrong.

Under the Hood: The Architecture of SigNoz

Understanding how SigNoz is built is fundamental to appreciating its capabilities:

  • OpenTelemetry: As mentioned, this is the core component for collecting and transmitting data from your applications.

  • ClickHouse: This is the database SigNoz uses. ClickHouse is an open-source, column-oriented database management system that’s incredibly efficient for handling and querying millions of data points very quickly. It also supports high availability and horizontal scaling even in its open-source version, which isn’t always the case with other databases.

  • SigNoz UI: The web interface that allows you to visualize and interact with the data collected by OpenTelemetry and stored in ClickHouse.

For those wanting to try it out at home, you can easily get this all running with Docker.

The Power of OpenTelemetry

OpenTelemetry is a game-changer. It’s becoming a de facto standard, with even tools like Dynatrace now able to use OpenTelemetry as a data source. The community around it is very active, making it a solid foundation for a product like SigNoz.

Key advantages of OpenTelemetry include:

  • Standardization: It provides a consistent way to instrument applications.

  • Libraries and Agents: It offers out-of-the-box libraries and agents for most major programming languages, simplifying instrumentation.

  • Auto-Instrumentation (Monkey Patching): Theoretically, OpenTelemetry can automatically inject the necessary code into your application to capture telemetry data without you needing to modify your application’s source code significantly. You just invoke your application with certain environment parameters. I say “theoretically” because while I tried it with one of my Python applications, I couldn’t get it to trace anything. Let me know in the comments if you’d like a dedicated video on this; I’m curious to dig deeper into why it didn’t work for me straight away!

Getting Started: Installing SigNoz with Docker and a Demo App

For my initial tests, I used a demo application suggested by the SigNoz team. Here’s a rundown of how you can get started with a standalone Docker setup:

1. Install SigNoz

It’s straightforward:

  1. Clone the SigNoz repository: git clone https://github.com/SigNoz/signoz.git (or the relevant path from their docs).

  2. Navigate into the directory and run Docker Compose. This will pull up four containers:

    • SigNoz Hotel Collector (OpenTelemetry Collector): Gathers data from OpenTelemetry agents.

    • SigNoz Query Service/Frontend: The graphical interface.

    • ClickHouse Server: The database.

    • Zookeeper: Manages ClickHouse instances (similar to etcd).

You can usually find the exact commands in the official SigNoz documentation under the Docker deployment section.

2. Set Up the Sample FastAPI Application

To see SigNoz in action, I used their “Sample FastAPI App”:

  1. Clone the demo app repository: (You’ll find this on the SigNoz GitHub or documentation).

  2. Create a Python 3 virtual environment: It’s always good practice to isolate dependencies. 
    python3 -m venv .venv
    
source .venv/bin/activate

  3. Install dependencies: 
    pip install -r requirements.txt

  4. Install OpenTelemetry components for auto-instrumentation: 
    pip install opentelemetry-distro opentelemetry-exporter-otlp

  5. Bootstrap OpenTelemetry (optional, for auto-instrumentation): 
    opentelemetry-bootstrap --action=install

    This attempts to find requirements for your specific application.

  6. Launch the application with OpenTelemetry instrumentation:

    You’ll need to set a few environment variables:

    • OTEL_RESOURCE_ATTRIBUTES: e.g., service.name=MyFastAPIApp (This name will appear in SigNoz).

    • OTEL_EXPORTER_OTLP_ENDPOINT: The address of your SigNoz collector (e.g., http://localhost:4317 if running locally).

    • OTEL_EXPORTER_OTLP_TRACES_EXPORTER: Set to otlp.

    • OTEL_EXPORTER_OTLP_PROTOCOL: Can be grpc or http/protobuf.

    Then, run your application using the opentelemetry-instrument command:

    OTEL_RESOURCE_ATTRIBUTES=service.name=FastApp OTEL_EXPORTER_OTLP_ENDPOINT="http://:4317" OTEL_EXPORTER_OTLP_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_PROTOCOL=grpc opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000

    (Replace with the actual IP where SigNoz is running).
    The opentelemetry-instrument part is what attempts the “monkey patching” or auto-instrumentation. The application itself (uvicorn main:app...) starts as it normally would.

A Quick Look at SigNoz in Action

Once the demo app was running and sending data, I could see traces appearing in my terminal (thanks to console exporter settings). To generate some load, I used Locust with a simple configuration to hit the app’s HTTP endpoint. This simulated about 10 users.

Navigating to the SigNoz UI (typically on port 3301, or as configured, if you’re using the Docker setup that forwards to 8080 or another port for the frontend, but the collector often listens on 4317/4318), the dashboard immediately showed my “FastApp” service. Clicking on it revealed:

  • Latency, request rate, and error rate graphs.

  • A list of endpoints called.

Drilling down into the traces, I could see individual requests. For this simple “Hello World” app, the trace was trivial, just showing the HTTP request. However, if the application were more complex, accessing a database, for example, OpenTelemetry could trace those interactions too, showing you the queries and time taken. This is where it gets really interesting for debugging and performance analysis.

The SigNoz interface felt responsive and well-designed. I was quite impressed with how smoothly it all worked.

Final Thoughts and What’s Next

I have to say, SigNoz seems like a very capable and well-put-together tool. It’s definitely worth trying out, especially if you’re looking for an open-source observability solution.

I plan to test it further with a more complex application, perhaps one involving a database, to see how it handles more intricate call graphs and to really gauge if it can be a strong contender against established players for more demanding scenarios.

It’s also interesting to note that Zabbix has APM features on its roadmap, potentially for version 8. So, the landscape is always evolving! But for now, SigNoz is a noteworthy project, especially for those interested in comprehensive observability that includes metrics, traces, AND logs in one package. This log management capability could make it a simpler alternative to setting up a separate, more complex logging stack for many use cases, particularly in home labs or smaller environments.

So, what do you think? Have you tried SigNoz or other APM tools? Let me know in the comments below! If there’s interest, I can certainly make more videos exploring its features or trying out more complex scenarios.

Thanks for watching, and I’ll see you next week. A greeting from me, Dimitri!

—

Stay Connected with Quadrata:

📺 Subscribe to Quadrata on YouTube

💬 Join the Zabbix Italia Telegram Channel (Also great for general monitoring discussions!)

Read More

0