How We Stopped Our AI From Lying About Network Status

We almost shipped a tool that would have told you your network was healthy while a router sat powered off in a rack. Not because of a bug. Because our AI made it up.

This is the story of how we discovered the most dangerous failure mode in AI-powered infrastructure tooling, and how we redesigned our platform from the ground up to make sure it never happens.

The Discovery

Early in development, we were testing our platform against a small lab network—three Cisco routers running OSPF, a FortiGate firewall, a couple of switches. Standard stuff. We asked the AI a straightforward question: "What's the OSPF status across all routers?"

The response came back clean. Three routers, all reporting FULL adjacencies. Interface stats looked normal. CPU utilization in the low teens. Uptime values were plausible—one at 47 days, another at 12 days, the third at 31 days. It even included the router IDs and the area assignments. If you had been reading this output on any monitoring dashboard in production, you would have moved on. Everything looks fine.

Except R3 was powered off. We had shut it down thirty minutes earlier to test a different failure scenario and forgot to bring it back up. The AI did not know this. It had no SSH connection to R3. It had no SNMP data from R3. It had absolutely nothing from R3.

So it invented everything.

That output looks exactly like what show ip ospf neighbor would produce on a healthy router. The CPU percentage is reasonable for an idle Cisco box. The uptime is plausible. The OSPF neighbor states are correctly formatted with DR/BDR designations. Every single detail is wrong, and every single detail is convincing.

We sat there staring at it for a long time.

Why This Is Dangerous

This is not the kind of AI hallucination where a chatbot confidently names a law firm that does not exist. That is embarrassing. This is the kind of hallucination that creates false operational confidence. An engineer looking at this output has no reason to question it. The data is formatted correctly. The values are within normal ranges. The status is green.

In a real environment, this means you think your network has fully converged OSPF adjacencies when it actually has a hole in it. You think three routers are participating in the routing domain when only two are. You think traffic is being load-balanced across all available paths when an entire path is missing. If a link fails on one of the two remaining routers, you expect the third to absorb the traffic. It will not, because it is off. You find this out when users start calling.

And this was not a one-off. We tested it again, deliberately. We unplugged a switch. The AI told us the switch was fine. We took down a firewall interface. The AI reported the interface was up with zero errors. Every time we removed real data from the equation, the AI filled the gap with something plausible. It was consistent. It was confident. It was always wrong.

The Root Cause

This is not a bug in the model. It is a fundamental property of how large language models work, and understanding it is critical if you are building AI tools for operations.

LLMs are trained on massive amounts of text—including Cisco documentation, forum posts on NetworkLessons and Cisco Learning Network, configuration guides, and thousands of examples of show command output shared by engineers troubleshooting problems. The model has seen so many examples of healthy OSPF output that it knows, statistically, exactly what that output should look like. It knows that CPU utilization on an idle Cisco ISR is typically between 5% and 15%. It knows that OSPF neighbors in a healthy state show FULL with a DR/BDR designation. It knows uptime is formatted as days, hours, and minutes.

When you ask "what is the OSPF status of R3" and the model has no real data to work with, it does what it is designed to do: it generates the most statistically likely completion. And the most statistically likely OSPF status for a router is "healthy." Most routers, most of the time, are working. The model is not lying. It is doing exactly what next-token prediction does—producing plausible text. But plausible is not the same as true, and in operations, plausible-but-false is the most dangerous output possible.

This is why the term "hallucination" undersells the problem. Hallucination implies something visibly wrong, something you might catch. What we observed was fabrication—generated content that is indistinguishable from real data to a human reader. You cannot tell by looking at it. There is no tell. The formatting is correct, the values are reasonable, and the model even generates appropriate contextual commentary ("network is fully converged"). It is a perfect forgery.

Our Solution: The Anti-Fabrication Layer

After that discovery, we tore the architecture apart. We could not ship a tool that might fabricate device status. Not sometimes. Not ever. The bar for infrastructure tooling is zero fabrication, and anything less than zero is a deal-breaker.

We built what we call the anti-fabrication layer—a set of enforced rules that sit between the data collectors and the AI model. These rules are not suggestions. They are not prompt guidelines that the model can choose to follow or ignore. They are architectural constraints enforced in code, injected into every single interaction context before the model sees the query.

There are five rules:

1. Data provenance tags. Every piece of data that enters the AI's context is tagged with its source, the collection method, and a timestamp. If the AI says R1's CPU is at 12%, the provenance tag shows that this came from an SSH session to 10.10.1.1 at 14:32:07 UTC via show processes cpu. The model cannot reference data that does not have a provenance tag.

2. Collector verification. Before the AI generates any response about a device, the system verifies that the relevant collector successfully reached that device. If the SSH collector for R3 returned a connection timeout, the AI is told: "R3 collector status: FAILED. SSH connection timed out at 14:31:58 UTC. No data available for this device." The model receives this as hard context, not as a suggestion.

3. Explicit failure reporting. When a collector fails, the AI must report the failure and the reason. It does not silently skip the device. It does not hedge with "R3 may be experiencing issues." It says: "R3 is UNREACHABLE. The SSH collector failed with: Connection refused on 10.10.1.3:22. No data was collected. Diagnosis: the device may be powered off, the SSH service may be down, or there may be a network path issue."

4. No AI-generated device output. The AI is never allowed to generate text that looks like device command output unless that text was directly returned by a collector. If someone asks for show ip ospf neighbor output and the collector did not return it, the AI cannot produce it. Period. This is the hardest rule to enforce because it requires the model to know what it does not know—but with explicit provenance tagging, the boundary is clear.

5. Logged failures. Every collection failure is logged with the device, the timestamp, the error, and the query that triggered the failed collection attempt. This creates an audit trail. If someone asks "was R3 reachable at 2pm yesterday?" we can answer that from logs, not from the model's imagination.

These five rules are injected as a system context file—anti_fabrication_context.txt—that is prepended to every interaction. The model sees this before it sees the user's question, before it sees any device data, before it generates a single token. It is not optional. It cannot be overridden by user prompting. It is the first thing the model reads, every time.

How It Works in Practice

Same scenario. Three routers, R3 is powered off. Same question: "What's the OSPF status across all routers?"

Here is what a typical AI tool returns versus what our platform returns:

The left side is green across the board. An engineer glancing at it moves on. The right side forces you to deal with reality: R3 is down, the OSPF topology is incomplete, and you need to investigate. One of these outputs leads to action. The other leads to a false sense of security.

Notice the details. Our platform reports the actual SSH error (timeout on 10.10.1.3:22), not a vague "connection issue." It tells you no data was collected, so you know the absence of information is deliberate, not an oversight. And it adjusts the OSPF summary accordingly—R1 and R2 show two neighbors instead of three, because R3 is not participating. The AI subtracted R3 from the topology instead of inventing its presence.

The Result

Trust. That is the result.

When our platform says a router is healthy, it actually checked. When it says an OSPF adjacency is FULL, that status came from a real show ip ospf neighbor command executed over a real SSH session with a real timestamp. When it says a device is offline, you get the collector error, the connection attempt details, and diagnostic guidance—not a guess.

This changes how you use the tool. You stop second-guessing the output. You stop opening a separate terminal to verify what the AI told you. You start trusting the data because you know, architecturally, that it cannot present data it does not have. The anti-fabrication layer is not a feature you toggle on. It is the foundation. Every response passes through it.

We have been running this in our own infrastructure—managing Cisco routers, FortiGate firewalls, Proxmox hypervisors, and Wazuh SIEM instances—with zero fabricated data points. Not low fabrication. Zero. Because the architecture does not allow it. The model physically cannot present data that did not come from a collector, and every collector failure is surfaced as a failure, not papered over with plausible fiction.

Update: We Went Further — Cryptographic Signing

The anti-fabrication context file worked. It reduced fabrication to near zero. But “near zero” is not zero, and in infrastructure operations, the bar is zero.

The fundamental problem remained: the anti-fabrication rules are text. The AI’s output is text. Real device data is text. Everything lives in the same Python process, the same memory space. No matter how strict the prompts, we were asking text to police text. The boundary between “real data” and “fabricated data” was enforced by convention, not by architecture.

So we built tli-executor — a compiled C binary that handles all SSH communication with infrastructure devices. It runs as a separate process, communicates with the Python platform over a Unix domain socket, and signs every piece of output with HMAC-SHA256 using a 256-bit key that exists only in the binary’s volatile memory.

The AI never sees the key. It cannot access the executor’s memory. It cannot forge the signature. If output does not carry a valid HMAC, it gets flagged as [UNVERIFIED] automatically — by math, not by asking the AI to please not lie.

This is the evolution from “please don’t fabricate” to “fabrication is mathematically detectable.” The prompt-based rules are still in place as defense in depth, but the cryptographic layer is now the primary enforcement. If every other safeguard fails, the HMAC check catches it.

Read the full technical breakdown on our anti-fabrication architecture page.

The Takeaway

If you are building AI tools for infrastructure—or evaluating them—ask one question before anything else: what happens when the AI has no data?

If the answer is “it does its best” or “it uses its training data to provide a reasonable response,” walk away. That is fabrication waiting to happen. The model will generate convincing, correctly formatted, completely fictional device output, and you will not be able to tell by looking at it.

Anti-fabrication cannot be an afterthought. It cannot be a prompt instruction that says “please don’t make things up.” It has to be an architectural constraint. We started with prompt engineering and validation layers. They helped but were not enough. We ended with a compiled C binary and HMAC-SHA256 signing — cryptographic proof that every piece of data is real or flagged as suspect.

We learned this the hard way, staring at perfect OSPF output from a router that was not plugged in. We built our platform to make sure nobody else has to learn it that way.