One of the advantages of running NASDEV for many years is that eventually you encounter problems that you simply wouldn't find in a textbook.
Some are complicated.
Some are caused by obscure configuration issues.
And occasionally, the problem is something incredibly simple that produces a surprisingly unhelpful error.
This was one of those problems.
A Solaris Zone refused to boot.
The error message wasn't particularly useful:
zoneadm: zone stevenzone: call to zoneadmd(1M) failed: door call to zoneadmd(1M) failed: Interrupted system call
At first glance, there wasn't much to go on.
The Zone was configured.
The system itself was running.
There were no obvious hardware problems.
And yet the Zone simply wouldn't start.
Start With the Obvious
The first step was to establish whether the problem was with the Zone itself or with the underlying Solaris environment.
The Zone configuration was checked.
The system services were checked.
The Zone state was checked.
Nothing immediately stood out.
This is where troubleshooting a complex system can become frustrating.
There is a natural tendency to start changing things.
Maybe the configuration needs adjusting.
Maybe a service needs restarting.
Maybe the Zone needs to be reconfigured.
Maybe something needs to be deleted and recreated.
That is usually the point where things can get worse.
Instead, I started looking at the filesystem.
The Filesystem Was the Clue
Solaris Zones have a very important security requirement.
The Zone root and the paths underneath it must conform to the rules Solaris expects.
In particular, Solaris does not allow arbitrary symbolic links to become part of the path leading into the Zone environment.
That makes sense when you think about what a Zone is trying to achieve.
A Zone is intended to provide a controlled environment.
If a directory underneath the Zone root could simply be replaced with a symbolic link pointing somewhere outside the Zone, it could potentially provide a way to escape the intended filesystem boundary.
So Solaris is deliberately strict about this.
Very strict.
The Problem
Eventually, a directory in the Zone's filesystem hierarchy was found to have been replaced by a symbolic link.
To a normal filesystem user, this can look completely harmless.
The path resolves.
The files are accessible.
Applications may even appear to work normally.
But from the perspective of the Solaris Zone framework, it is not acceptable.
The Zone's filesystem hierarchy must contain real directories where Solaris expects directories.
A symbolic link changes the meaning of the path.
And Solaris refuses to allow that ambiguity when constructing the Zone environment.
The Fix
The solution was surprisingly simple once the cause was understood.
The symbolic link had to be removed and replaced with the appropriate real directory or correctly configured filesystem mount.
Once the filesystem hierarchy was restored to the expected structure, the Zone could boot normally.
No special zoneadm option was required.
No obscure SMF configuration needed changing.
No kernel parameter needed to be modified.
The problem was simply a filesystem path that wasn't what Solaris expected it to be.
Why Solaris Does This
This is actually a good example of a security feature that can initially look like an inconvenience.
It would be easy to think:
"If the path resolves, why shouldn't Solaris allow it?"
Because the Zone boundary is part of the security model.
Allowing arbitrary symbolic links in critical parts of the Zone root could create paths that escape the intended filesystem hierarchy.
Solaris therefore checks the filesystem structure before allowing the Zone to start.
It is effectively saying:
If I can't prove that this path stays inside the Zone, I'm not going to start the Zone.
That's a reasonable security decision.
The Lesson
The important lesson wasn't simply "don't use symbolic links."
Symbolic links are perfectly legitimate and useful in Solaris.
The lesson was more specific:
Understand the assumptions made by the layer underneath your application.
The filesystem looked fine from one perspective.
The Zone framework saw something fundamentally wrong.
This is common in systems administration.
A configuration can be perfectly valid for one component while being completely invalid for another component that depends on it.
Don't Trust the Error Message Alone
The original error didn't say:
"A directory underneath the Zone root is a symbolic link."
It said:
Interrupted system call
That's technically an error.
It just isn't a particularly useful description of the underlying problem.
This is why troubleshooting complex infrastructure requires looking beyond the first error message.
The error tells you where the failure became visible.
It doesn't necessarily tell you where the failure originated.
That distinction is incredibly important.
A Useful Troubleshooting Pattern
This incident reinforced a troubleshooting approach I use frequently with NASDEV.
Start at the highest level where the failure is visible.
Then work down through the layers.
For a Zone, that might mean:
Zone configuration
↓
Zone filesystem
↓
Mounts and datasets
↓
Network configuration
↓
Solaris services
↓
Underlying operating system
↓
Hardware
Don't immediately assume the problem is at the layer reporting the error.
The Zone manager was reporting the failure.
The actual problem was in the filesystem.
Why I Keep These Stories
This is exactly the sort of problem I want to document on NASDEV.
A generic Solaris manual can tell you how Zones work.
It can tell you the syntax of zonecfg.
It can tell you how to boot a Zone.
But the really useful knowledge often comes from the failures.
What does a broken Zone actually look like?
Which parts of the system should you check?
Which error messages are misleading?
What assumptions does Solaris make about the filesystem?
And, most importantly, what should you not start changing before you understand the problem?
Those are lessons learned from actually operating the system.
The NASDEV Philosophy
NASDEV isn't a laboratory where everything works perfectly.
It is a real system.
It has accumulated years of configuration, upgrades, hardware changes, new services and old services that have eventually been retired.
Things occasionally break.
When they do, the objective isn't simply to get everything running again.
It is to understand why it broke.
That distinction is important.
Fixing the immediate problem gets the system running.
Understanding the problem makes the next failure easier to solve.
And sometimes, as in this case, the eventual solution is remarkably simple.
A directory was supposed to be a directory.
Solaris knew that.
I just had to figure out why it wasn't one anymore.
