Deep Dive into Zabbix Low-Level Discovery & the Game-Changing 7.4 Update

Good morning, everyone, and welcome back to Quadrata! This is Dimitri Bellini, and on this channel, we explore the fascinating world of open source and IT. I’m thrilled you’re here, and if you enjoy my content, please give this video a like and subscribe if you haven’t already!

I apologize for missing last week; work had me on the move. But I’m back, and with the recent release of Zabbix 7.4, I thought it was the perfect time to revisit a powerful feature: Low-Level Discovery (LLD). There’s an interesting new function in 7.4 that I want to discuss, but first, let’s get a solid understanding of what LLD is all about.

What Exactly is Zabbix Low-Level Discovery?

Low-Level Discovery is a fantastic Zabbix feature that automates the creation of items, triggers, and graphs. Think back to the “old days” – or perhaps your current reality if you’re not using LLD yet. Manually creating monitoring items for every CPU core, every file system, every network interface on every host… it’s a painstaking and error-prone process, especially in dynamic environments.

Imagine:

• A new mount point is added to a server. If you forget to add it to Zabbix, you won’t get alerts if it fills up. Disaster!
• A network switch with 100 ports. Manually configuring monitoring for each one? A recipe for headaches.

LLD, introduced way back in Zabbix 2.0, came to rescue us from this. It allows Zabbix to automatically discover resources on a host or device and create the necessary monitoring entities based on predefined prototypes.

Why Do We Need LLD?

• Eliminate Manual Toil: Say goodbye to the tedious task of manually creating items, triggers, and graphs.
• Dynamic Environments: Automatically adapt to changes like new virtual machines, extended filesystems, or added network ports.
• Consistency: Ensures that all similar resources are monitored in the same way.
• Accuracy: Reduces the risk of human error and forgotten resources.

How Does Low-Level Discovery Work?

The core principle is quite straightforward:

1. Discovery Rule: You define a discovery rule on a host or template. This rule specifies how Zabbix should find the resources.
2. Data Retrieval: Zabbix (or a Zabbix proxy) queries the target (e.g., a Zabbix agent, an SNMP device, an HTTP API) for a list of discoverable resources.
3. JSON Formatted Data: The target returns the data in a specific JSON format. This JSON typically contains an array of objects, where each object represents a discovered resource and includes key-value pairs. A common format uses macros like {#FSNAME} for a filesystem name or {#IFNAME} for an interface name.

   
   
   {
   
     "data": [
   
       { "{#FSNAME}": "/", "{#FSTYPE}": "ext4" },
   
       { "{#FSNAME}": "/boot", "{#FSTYPE}": "ext4" },
   
       { "{#FSNAME}": "/var/log", "{#FSTYPE}": "xfs" }
   
     ]
   
   }
   
           

4. Prototype Creation: Based on the received JSON data, Zabbix uses prototypes (item prototypes, trigger prototypes, graph prototypes, and even host prototypes) to automatically create actual items, triggers, etc., for each discovered resource. For example, if an item prototype uses {#FSNAME} in its key, Zabbix will create a unique item for each filesystem name returned by the discovery rule.

The beauty of this is its continuous nature. Zabbix periodically re-runs the discovery rule, automatically creating entities for new resources and, importantly, managing resources that are no longer found.

Out-of-the-Box vs. Custom Discoveries

Zabbix comes with several built-in LLD rules, often found in default templates:

• File systems: Automatically discovers mounted file systems (e.g., on Linux and Windows).
• Network interfaces: Discovers network interfaces.
• SNMP OIDs: Discovers resources via SNMP.
• Others like JMX, ODBC, Windows services, and host interfaces.

But what if you need to discover something specific to your custom application or a unique device? That’s where custom LLD shines. Zabbix is incredibly flexible, allowing almost any item type to become a source for discovery:

• Zabbix agent (system.run[]): Execute a script on the agent that outputs the required JSON.
• External checks: Similar to agent scripts but executed on the Zabbix server/proxy.
• HTTP agent: Perfect for querying REST APIs that return lists of resources.
• JavaScript items: Allows for complex logic, multiple API calls, and data manipulation before outputting the JSON.
• SNMP agent: For custom SNMP OID discovery.

The key is that your custom script or check must output data in the LLD JSON format Zabbix expects.

Configuring a Discovery Rule: Key Components

When you set up a discovery rule, you’ll encounter several important configuration tabs:

• Discovery rule (main tab): Define the item type (e.g., Zabbix agent, HTTP agent), key, update interval, etc. This is where you also configure how Zabbix handles “lost” resources.
• Preprocessing: Crucial for custom discoveries! You can take the raw output from your discovery item and transform it. For example, convert CSV to JSON, use regular expressions, or apply JSONPath to extract specific parts of a complex JSON.
• LLD macros: Here, you map the keys from your discovery JSON (e.g., {#FSNAME}) to JSONPath expressions that tell Zabbix where to find the corresponding values in the JSON output from the preprocessing step.
• Filters: Include or exclude discovered resources based on regular expressions matching LLD macro values.
• Overrides: A more advanced feature allowing you to change specific attributes (like item status, severity of triggers, tags) for discovered objects that match certain criteria.

Managing Lost Resources: A Welcome Improvement

A critical aspect of LLD is how it handles resources that were previously discovered but are no longer present. For a long time, we had the “Keep lost resources period” setting. If a resource disappeared, Zabbix would keep its associated items, triggers, etc., for a specified duration (e.g., 7 days) before deleting them. During this period, the items would often go into an unsupported state as Zabbix tried to query non-existent resources, creating noise.

Starting with Zabbix 7.0, a much smarter option was introduced: “Disable lost resources.” Now, you can configure Zabbix to immediately (or after a period) disable items for lost resources. This is fantastic because:

• It stops Zabbix from trying to poll non-existent resources, reducing load and unsupported item noise.
• The historical data for these items is preserved until they are eventually deleted (if configured to do so via “Keep lost resources period”).
• If the resource reappears, the items can be automatically re-enabled.

You can use these two settings in combination: for example, disable immediately but delete after 7 days. This offers great flexibility and a cleaner monitoring environment.

Prototypes: The Blueprints for Monitoring

Once LLD discovers resources, it needs templates to create the actual monitoring entities. These are called prototypes:

• Item prototypes: Define how items should be created for each discovered resource. You use LLD macros (e.g., {#FSNAME}) in the item name, key, etc.
• Trigger prototypes: Define how triggers should be created.
• Graph prototypes: Define how graphs should be created.
• Host prototypes: This is a particularly powerful one, allowing LLD to create *new hosts* in Zabbix based on discovered entities (e.g., discovering VMs from a hypervisor).

The Big News in Zabbix 7.4: Nested Host Prototypes!

Host prototypes have been around for a while, evolving significantly from Zabbix 6.0 to 7.0, gaining features like customizable interfaces, tags, and macro assignments for the discovered hosts. However, there was a significant limitation: a template assigned to a host created by a host prototype could not, itself, contain another host prototype to discover further hosts. In essence, nested host discovery wasn’t supported.

Imagine trying to monitor a virtualized environment:

1. You discover your vCenter.
2. You want the vCenter discovery to create host objects for each ESXi hypervisor. (Possible with host prototypes).
3. Then, you want each discovered ESXi hypervisor host (using its assigned template) to discover all the VMs running on it and create host objects for those VMs. (This was the roadblock!).

With Zabbix 7.4, this limitation is GONE! Zabbix now supports nested host prototypes. This means a template applied to a discovered host *can* indeed contain its own host prototype rules, enabling multi-level, chained discoveries. This is a game-changer for complex environments like Kubernetes, container platforms, or any scenario with layered applications.

A Quick Look at How It Works (Conceptual Demo)

In the video, I demonstrated this with a custom LLD setup:

1. Initial Discovery: I used a simple system.run item that read a CSV file. This CSV contained information about “parent” entities (simulating, say, hypervisors).
2. Preprocessing: A “CSV to JSON” preprocessing step converted this data into the LLD JSON format.
3. LLD Macros: I defined LLD macros like {#HOST} and {#HOSTGROUP}.
4. Host Prototype (Level 1): A host prototype used these macros to create new hosts in Zabbix and assign them a specific template (let’s call it “Template A”).
5. The Change in 7.4:

   • In Zabbix 7.0 (and earlier): If “Template A” itself contained a host prototype (e.g., to discover “child” entities like VMs), that nested host prototype would simply not appear or function on the hosts created by the Level 1 discovery. The Zabbix documentation even explicitly stated this limitation.
   • In Zabbix 7.4: “Template A” *can* now have its own discovery rules and host prototypes. So, when the Level 1 discovery creates a host and assigns “Template A”, “Template A” can then kick off its *own* LLD process to discover and create further hosts (Level 2).

This allows for a much more dynamic and hierarchical approach to discovering and monitoring complex infrastructures automatically.

Conclusion: Embrace the Automation!

Low-Level Discovery is an indispensable Zabbix feature for anyone serious about efficient and comprehensive monitoring. It saves incredible amounts of time, reduces errors, and keeps your monitoring setup in sync with your ever-changing IT landscape.

The introduction of “Disable lost resources” in Zabbix 7.0 was a great step forward, and now, with nested host prototypes in Zabbix 7.4, the power and flexibility of LLD have reached new heights. This opens up possibilities for automating the discovery of deeply layered applications and infrastructure in a way that wasn’t easily achievable before.

I encourage you to explore LLD in your Zabbix environment. Start with the out-of-the-box discoveries, and then don’t be afraid to dive into custom LLDs to tailor Zabbix perfectly to your needs.

What are your thoughts on Low-Level Discovery or this new Zabbix 7.4 feature? Are there any specific LLD scenarios you’d like me to cover in a future video? Let me know in the comments below! Your feedback is always appreciated.

Thanks for watching, and I’ll see you next week!

All the best,
Dimitri Bellini

Connect with me and the community:

• My YouTube Channel: Quadrata (https://www.youtube.com/@quadrata)
• Zabbix Italia Telegram Channel: Join the discussion! (https://t.me/zabbixitalia)