Inside NASDEV: Why I Still Run a Global Zone
With Solaris Zones providing the ability to separate services, it would be tempting to think that everything on NASDEV should live inside a non-global Zone.
That isn't quite how I use them.
The Global Zone remains an important part of the architecture.
It is the foundation from which the other Zones are managed, and it is also where some of the underlying system functions belong.
Understanding why is important because the Global Zone isn't simply "another Zone".
It has a fundamentally different role.
The Global Zone Is the System
The simplest way to think about the Global Zone is that it represents the underlying Solaris system itself.
The physical hardware belongs to it.
The Solaris kernel belongs to it.
The overall system configuration belongs to it.
And, importantly, the Global Zone has visibility and administrative control over the non-global Zones.
A non-global Zone is deliberately restricted.
The Global Zone is not.
That distinction is central to the Solaris security model.
Why Not Put Everything in Zones?
If Zones provide such useful isolation, why not put every possible service into one?
There are several reasons.
Some functions need access to the underlying system.
Some administration tasks are inherently global.
Hardware management is global.
ZFS pool management is global.
Zone configuration is global.
There are also services where introducing another layer of isolation simply doesn't provide enough benefit to justify the additional complexity.
The goal isn't to maximise the number of Zones.
The goal is to use Zones where they provide a useful boundary.
ZFS and the Global Zone
This is particularly relevant to storage.
The ZFS pool itself is a global system resource.
The pool contains datasets, and those datasets can then be made available to individual Zones as required.
This gives NASDEV a useful separation.
The Global Zone manages the underlying storage architecture.
Individual Zones consume the storage they have been allocated.
That means a web service doesn't need to know anything about the physical disks.
It simply sees the filesystem it has been given.
The same applies to databases, applications and other services.
Administration From the Outside
One of the most useful characteristics of the Global Zone is the ability to administer the entire environment from one place.
If a Zone has stopped responding, the Global Zone can still see its state.
If a Zone needs to be rebooted, that can be done from the Global Zone.
If a Zone configuration needs to be inspected, the Global Zone is where that work is performed.
This provides an important recovery path.
Even if a non-global Zone is badly broken, the underlying Solaris system remains available to investigate it.
The Global Zone Is Not a Dumping Ground
There is an important distinction here.
The fact that the Global Zone has access to everything doesn't mean everything should run there.
In fact, I deliberately try to avoid treating it as a general-purpose application environment.
The more unrelated applications that are installed directly into the Global Zone, the more difficult it becomes to maintain.
It also weakens one of the major advantages of having Zones in the first place.
If everything runs globally, there isn't much isolation.
The Global Zone should primarily contain things that genuinely belong to the underlying system.
A Useful Boundary
The architecture can therefore be thought of as having a fairly simple boundary.
The Global Zone provides:
- the operating system;
- hardware access;
- global networking configuration;
- ZFS storage management;
- Zone management;
- core system services.
The non-global Zones provide:
- applications;
- isolated services;
- development environments;
- externally accessible services;
- workloads that benefit from separation.
This isn't an absolute rule.
Real systems rarely fit perfectly into a diagram.
But it is a useful design principle.
Why This Matters During Troubleshooting
The distinction becomes particularly important when something breaks.
Suppose a web application stops working.
The first question shouldn't necessarily be:
"What's wrong with Solaris?"
It might simply be something inside the web Zone.
Likewise, if a Zone refuses to boot, it doesn't necessarily mean the Zone itself is broken.
The problem could be the storage underneath it.
Or a filesystem mount.
Or a network configuration.
Or a dependency in the Global Zone.
Understanding the layers makes troubleshooting much more systematic.
The Global Zone as a Recovery Environment
There is another advantage that isn't always obvious until something goes wrong.
The Global Zone provides a kind of recovery environment for the rest of the system.
A non-global Zone can be stopped.
Its filesystem can be inspected.
Its configuration can be examined.
Its datasets can be accessed from the appropriate administrative context.
Its services can be investigated.
All without depending on the Zone itself being fully operational.
That is extremely useful.
It means that a broken Zone doesn't necessarily become a completely inaccessible black box.
The system underneath it remains available.
The Security Boundary
The Global Zone also represents the highest level of trust in the Solaris Zone architecture.
A process running in a non-global Zone should not be able to simply reach into another Zone or manipulate the underlying system.
The Global Zone, however, has the authority to manage those environments.
This is what makes Zones useful as an isolation mechanism.
The boundary isn't just administrative.
It is part of the security architecture.
That is also why Solaris can be surprisingly strict about things such as filesystem paths, mounts and symbolic links when constructing a Zone.
It is protecting the boundary.
NASDEV Has Evolved Around This Model
As NASDEV has grown, the Global Zone has remained relatively stable while the collection of Zones around it has changed.
That is exactly what I wanted from the architecture.
The foundation doesn't need to change every time a new application is introduced.
A new service can often be given its own Zone.
An old service can be removed.
A development environment can be rebuilt.
The underlying system can continue running.
This gives NASDEV a degree of architectural flexibility without requiring a completely new server every time the requirements change.
There Is a Cost
Nothing comes for free.
Zones add complexity.
You now have another layer of configuration to understand.
There are Zone states.
Zone filesystems.
Zone networking.
Zone dependencies.
And a failure can involve interactions between the Global Zone and the non-global Zone.
But I think the trade-off is worthwhile.
The alternative is often a single large environment where everything depends on everything else.
For a system that has been evolving for years, that can become considerably harder to manage.
Keeping the Foundation Boring
One of my goals with NASDEV is to keep the underlying foundation as predictable as possible.
The interesting experimentation should happen above it.
The Global Zone is therefore not where I want to constantly experiment.
It should be stable.
The Zones are where I can try things.
That gives me a useful separation between:
the system that supports the experiments
and
the experiments themselves.
It isn't perfect isolation.
But it is a very practical engineering model.
The Bigger Picture
NASDEV isn't a collection of independent servers.
It isn't simply one big Solaris installation either.
It sits somewhere in between.
The Global Zone provides the foundation.
ZFS provides the storage architecture.
Zones provide service isolation.
Applications run inside those environments.
And the network ties everything together.
That architecture has evolved over many years, but the underlying principle has remained surprisingly consistent:
Keep the foundation stable and put change at the edges.
That has turned out to be one of the most useful design decisions in NASDEV.
And it is probably one of the main reasons the system has managed to evolve for so long without becoming completely unmanageable.
