One of the features of Solaris that has had a major influence on the design of NASDEV is something called Zones.
At first glance, a Zone can look a little like a virtual machine.
It has its own filesystem environment, networking configuration, processes and services. From inside the Zone, it can feel very much like you are working with a separate server.
But there is an important difference.
A Solaris Zone does not require a separate operating-system kernel.
Instead, multiple Zones share the same underlying Solaris kernel while maintaining strong separation between their individual environments.
For NASDEV, this turned out to be an extremely useful way of building infrastructure.
The Problem With One Big Server
When NASDEV started, it would have been perfectly possible to install everything directly into the global Solaris environment.
The web server could run there.
The database could run there.
Network services could run there.
Storage services could run there.
Development software could run there.
It would work.
But over time, that approach becomes increasingly difficult to manage.
One application can have dependencies that conflict with another.
A software upgrade can affect something completely unrelated.
A configuration change made for one service can potentially affect the entire system.
And eventually you end up with a server where nobody really wants to change anything because nobody is quite sure what else might break.
Zones provide a way around that problem.
Think of a Zone as a Separate Server
One of the easiest ways to understand a Zone is to think of it as a lightweight server running inside the physical machine.
The Zone has its own identity.
It can have its own hostname.
It can have its own IP configuration.
It has its own filesystem hierarchy.
It has its own processes.
And services running inside the Zone generally have no visibility of processes belonging to other Zones.
From an administration perspective, this creates a very useful boundary.
If I am working on a web service inside one Zone, I don't need to treat the entire NASDEV system as the thing I am modifying.
I am working inside that particular environment.
The Global Zone
At the bottom of this architecture is the Global Zone.
The Global Zone is different from the non-global Zones because it has visibility and administrative control over the entire Solaris system.
This makes it the natural place for managing the underlying system.
The Global Zone is effectively the foundation on which the other Zones exist.
That separation is important.
The applications don't all need to know about the physical hardware.
They don't need to know about every other service.
They simply operate within the environment that has been provided to them.
Why This Works So Well for NASDEV
As NASDEV evolved, different services could be separated into different Zones.
For example, a database doesn't necessarily need to share its entire operating environment with a web application.
A public-facing service can be isolated from internal infrastructure.
A development environment can be changed without necessarily affecting production services.
A test environment can be created without requiring another physical computer.
This makes the system considerably more flexible.
It also means that NASDEV can behave like a collection of servers while still being physically one system.
Zones Versus Virtual Machines
This naturally raises the question:
Why not just use virtual machines?
Virtual machines are extremely useful, and there are situations where they are absolutely the right choice.
The difference is that a virtual machine normally includes its own operating-system kernel.
That means every virtual machine carries additional overhead.
A Solaris Zone shares the Solaris kernel with the other Zones.
This makes Zones considerably lighter.
For NASDEV, that is a particularly good fit.
I don't need to run ten complete operating systems simply to isolate ten services.
I can run multiple Zones on one Solaris installation and give each one the resources and environment it actually needs.
Isolation Without the Overhead
The real attraction isn't simply performance.
It is separation.
A service can be given its own environment without creating another physical or virtual server.
This makes it easier to experiment.
It also makes troubleshooting easier because the scope of a problem can often be narrowed down to a particular Zone.
That doesn't mean Zones magically prevent problems.
They don't.
But they provide useful boundaries around those problems.
Networking
Zones can also have their own network identities.
This means that services inside different Zones can communicate over the network in much the same way that completely separate physical servers would.
For NASDEV, this is particularly useful because different components can be treated as independent network services.
A reverse proxy can sit in front of other services.
A web server can operate in its own environment.
A database can have its own network presence.
The physical machine doesn't have to expose all of those services directly.
This provides another layer of architectural separation.
Storage and Zones
The relationship between Zones and ZFS is another particularly powerful aspect of Solaris.
ZFS datasets can be used to provide storage to individual Zones.
This means the storage architecture and the service architecture can be separated.
The underlying storage can be managed using ZFS while the applications simply see the filesystems they have been given.
That combination is one of the reasons I think Solaris and ZFS work so well together.
ZFS provides the storage architecture.
Zones provide the service architecture.
Together they allow NASDEV to be divided into logical components without requiring a collection of independent machines.
The NASDEV Approach
The current NASDEV environment has evolved into a collection of Zones serving different purposes.
Some are important production services.
Some are infrastructure components.
Some exist specifically for development or experimentation.
And some are historical remnants of how the system evolved.
That's an important point.
NASDEV wasn't designed from a blank sheet of paper.
It grew.
A Zone was created because there was a reason for it at the time.
Sometimes that reason remained relevant.
Sometimes it didn't.
This is one of the interesting characteristics of a long-running system: the architecture becomes a record of its history.
Zones Make Experimentation Safer
One of the biggest advantages for me personally is the ability to experiment.
If I want to test a configuration, install software or investigate a problem, having a separate Zone provides a much safer environment in which to work.
It isn't completely isolated from the underlying system, of course.
A Zone still depends on the Solaris environment underneath it.
But it provides enough separation to make experimentation much more practical.
That has been particularly valuable with NASDEV because the system isn't simply a server.
It is also a laboratory.
They Don't Eliminate the Need to Understand Solaris
There is a temptation to think that Zones make everything simple.
They don't.
In fact, they introduce another layer that needs to be understood.
When something goes wrong, you need to determine whether the problem is:
- inside the Zone;
- in the Zone configuration;
- in the underlying Solaris environment;
- in the network;
- in the storage;
- or somewhere else entirely.
A Zone that refuses to boot, for example, can sometimes produce an error that doesn't immediately identify the real cause.
That is where understanding the underlying Solaris architecture becomes important.
And those real-world problems are exactly the sort of things I intend to document here.
A Useful Mental Model
The simplest way I think about NASDEV is something like this:
Physical hardware
↓
Solaris
↓
ZFS storage
↓
Solaris Zones
↓
Applications and services
Each layer has a purpose.
The hardware provides the physical resources.
Solaris provides the operating environment.
ZFS manages the storage.
Zones provide isolation.
Applications provide the services.
Keeping those responsibilities separate has allowed NASDEV to evolve for many years without needing to completely redesign the system every time something changes.
Why Zones Still Matter
There are newer technologies.
There are container platforms.
There are sophisticated virtualisation systems.
There are cloud orchestration frameworks.
All of them have their place.
But there is something appealing about the simplicity of Zones.
They solve a real problem without requiring a huge infrastructure stack.
For a system like NASDEV, that is exactly what I need.
I don't need hundreds of containers.
I don't need an orchestration cluster.
I need a reliable operating environment where different services can coexist without becoming unnecessarily tangled together.
Solaris Zones have provided that for many years.
And they remain one of the defining architectural features of NASDEV.
The Next Problem: When Things Go Wrong
Of course, having multiple Zones also means that occasionally one of them decides it doesn't want to cooperate.
That has happened more than once on NASDEV.
And some of those incidents have produced some particularly interesting troubleshooting lessons.
One of the most memorable involved a Zone that simply refused to boot because of something that looked completely innocent in the filesystem hierarchy.
It turned out that a directory had been replaced with a symbolic link.
Solaris wasn't having any of it.
That story deserves its own article.
Because sometimes the most useful lessons aren't found in the design documentation.
They're found when the system says:
No. I'm not booting. And here's a deliberately unhelpful error message to explain why.
And that's where the real NASDEV troubleshooting stories begin.
