http://www.bradgoodman.com
Text, Illustrations and Misspellings ©2007-2011 Bradley K. Goodman, All Rights Reserved
This document is intended to be a quick introduction to Serial Attached SCSI. It was written primarily for those who are farmiliar with "traditional" SCSI implementations, but reviews these concepts to put them in perspective. It is not meant to be complete, and in the interest of simplicity and getting the basics accross skips a lot - but is geared towards providing a good basis of general understanding.
In the beginning, there was "SCSI". SCSI was a parallel, multipoint bus - meaning that many devices could "tap" or connect into the same physical bus. Each device on the bus had an identification number, or "SCSI ID". This ID was set was sometimes hard-coded, but usually set by a jumper or dip-switch on a device before connecting it. See Figure 1.
The SCSI protocol provided a method for these devices to connect to, send commands, and send data to each other, i.e. a computer on ID7 could send a command (and some data) to a device on ID0. On another "level" - SCSI defines what (some of) these commands actualy are. For example, a specific command to write a block of data to a disk drive.
An important command, for example, is the INQUIRY command, (which all devices must support.) The INQUIRY command returns data about a device - Its vendors's name, the product name, and what kind of device it is (a disk drive, a CD-ROM, a computer, etc).
So, for example, a computer could start up, and blindly send an inquiry command out to every possible SCSI ID on its bus. (In early SCSI, there were only 8 valid IDs - even in later parallel implementations, there can be only 16). Devices on each ID, if they existed, would return their INQUIRY data, and the computer would see the bus (in Figure 1) as follows:
| SCSI ID | Vendor Name | Product Name | Device Type |
| 0 | |||
| 1 | Quantum | Fireball 250 | Disk |
| 2 | IBM | Deathstar 75 | Disk |
| 3 | |||
| 4 | |||
| 5 | Toshiba | LM-750 | CD-ROM |
| 6 | |||
| 7 | Apple | Macintosh SE | Processor |
And so this is how it worked for many many years. And as such, all operating system software, application software, etc. was built around this model.
Because, as all the software of the past 15 or 20 years was built around this, this model stays the same today. Even as we enter the world of Serial Attached SCSI.
Though SCSI was not initially designed as a "layerd" architecture, it was condusive to it - and has evolved in a layered fasion. SAS conceptualy operates by replacing the lower layer in the SCSI "stack", leaving the reset of the model (and legacy software) unchanged.
Let's see how this works...
The goal of SAS is to retain the semantics of the existing SCSI architecture, allowing legacy software to work as-is, while replacing the underlying physical layer with a point-to-point switched topology. (See diagram below).
As you can see, these topologies are very different. So how does it work? Simple. A SAS initiator's (or "host device's") HBA ("Host Bust Adapter") is responsable for traversing the device tree connected to it, assigning SCSI IDs to each of the devices, and presenting the host with a "flat" - bus-like list of devices it's attached to. The host will then "see" and "reference" all of the underlying devices as if it were on a "flat" (legacy) bus.
Let's see an example
Figure 2 shows a typical "modern" PC connected (via. SAS) to two four-disk enclosures.
Now if SAS connections are all "point-to-point" - how do we connect a host to a chassis with four drives in it? Through use of a switch, or expander. Figure 3 shows the actual topology of this system.
The host connects to an expander inside the chassis, which is a (in this case) six-port "star" or "hub" switch. Thus, the expander has six-ports, each of which can provide point-to-point connections between it and another SAS device (or other switch).
As shown, the host connects to one port on the expander in the first chassis. Four more ports on the expander connect it to the four internal drives, and the sixth port is used to daisy-chain the chassis to another device - which in this case is another chassis.
In the real-world, there are practical limits to the amount of devices that may be connected, number of expanders that may be chained, and rules to how the devices can be interconnected, but we won't go into them here.
What is importaint, is that the lower "physical/link" later of SAS will "hide" this topology from the host, presenting the host a "flat" list of devices as in legacy SCSI. Figure 4 shows what this system would look like to the host.
Some important things to note are:
In figure 4, each disk has a SCSI ID assigned to it. Where did they come from? How are they numbered? The short answer is they host "makes them up". They are not "hard-assigned" such as in legacy SCSI. They are completely at the discretion of the host, and exist only with and for the host's use - so they may address and reference these devices in a way that is applicable to legacy SCSI software.
SAS devices however do have unique, permanant numbers called WWN, or "World-Wide-Numbers". These numbers are assinged by the factory when these devices are created, can never be changed, and are unique to each and every SAS device. A devices WWN can be used by the host to later identify it, for example, to remember which drives were last present on a RAID controller.
SAS Host Bus Adapters often "remember" the SCSI IDs that it assigned to each WWN on the system - and use this data to attempt to assign the same SCSI IDs devices the next time the system is started, even if (for example) other devices are added or removed, and the topology looks different. This, again, is done at the discresion of some host adapter, and done only to try to make life easier for the host/host software. For example, (as described earlier) in legacy SCSI, IDs are assigned by jumper/dip-switch, and would therefore not change, (unless explitly changed by the user). So if software were configured with certian IDs being a certian way, it would be good for a SAS HBA to retain these IDs.
Enclosre Management is generally a fancy way of describing a methodology to monitor or control an enclosure itself, as opposed to the devices within it. For example, in the system above, how could we monitor the power supplies, fans, or other elements of the drive cabinets themselves? This is done through Enclosre Management.
The predominant Enclosure Management method these days is "SES" - or "SCSI Enclosure Services". SES is not unique to SAS - but it tends to come up and confuse. SES can be used with Legacy (parallel) SCSI busses as well as SAS. So how does it work?
SES works by physically adding another device inside the enclosure. The device is a SAS or SCSI device, just like a disk. The device, like any other, has a WWN, receives a SCSI ID, and will respond to basic SCSI commands, such as the INQUIRY command described above. Obviouisly, it's INQUIRY command would report it as an "Enclosure Management Device", and host software would therefore treat it differently than it would a disk drive.
Figure 6 shows how a host would see the bus with chassis which provided SES support.
The expander and Managment Processor may be on the same board, or even the same chip. This does not change the underlying premise of their operation. They are two very different devices, that act and behave very differently:
SES also provides an important (and confusing) service - and that is to help establish a relationship between devices - which, as described above, know-no toplogy or relationship in the SCSI world. For example, figure 6 shows a bunch of disks - with their relationship to an enclosure (is in figure 5) being completely unknown. One might easily say the relationship is simple if it were viewed on the basis of which disks were connected to which expanders. This is not true, as a single cabanate may have multiple expanders - or external expanders could be used to aggrigate multple cabanates, or even individual drives, etc.
Part of SES's service is to know and understand the topology of their own chassis/hardware/environment - to relate that to what's happening on the bus, and to help convey that to the user. This goes on to do things they could never be infered at all from the topology, like which which expander port connects to disk on which physical slots on the enclosure - or which fans in the enclosure are cooling which disks, etc.
Up until now, we have described a SAS bus as something essentially identical to a legacy (parallel) SCSI bus, in that it carries the same traffic, but uses a different underlying topology. This is almost accurate.
SAS uses a few protocols to deal with a few different type of traffic flowing through it. It is worth mentioning them here because they are used a lot in talking about SAS.
I will describe this first, as it is the simplest and easiest to understand. SSP is a protocol which encapsulates "legacy" SCSI commands and data for transmission between nodes. For example, if node "x" sends node "y" a command to "read data block 54", and node "y" sends back the data from that disk block, this transaction is done with SSP, which encapsulates the SCSI Command Block (CDB), the data, the "sense data" (error data, if needed), and other basic information which is used in any SCSI transaction, parallel, serial, legacy, whatever.
SMP standards for "SAS Management Protocol". It is used only by expanders and initiators (hosts). SMP provides a set of very simple commands to allow initiators and expanders to query information from each other. This is only done at start-up, or when a devices is added or removed from the bus.
SMP is used to allow the initator/host to discover what devices are on the SAS bus, so it may assign SCSI IDs to them and present them to the host. It is also used to allow the expanders to see what devices (WWNs) are connected off which ports of other expanders, so they will know how to open routes to different devices/WWNs.
This sharing of information between initiators and expanders whenever the bus is new or changed, is called "discovery". It is simply, everyone asking their neighbors about who their neighbors are and collecting everyones addresses - so (for example) expanders will know through which port messages to different addresses should be routed.
Again, this is SAS-specific - and this is all "hidden" from the host, and SCSI software model. A host knows nothing about this, and just knows about different devices on different SCSI IDs. It is the SAS HBA and the expanders that use the information obtained via. SMP messages to tell the host the devices exist, and to route messages to them. After DISCOVERY has completed, SMP is no longer even used.
STP stands for "SAS Tunneling Protcol". This is simply the mechanism that a SAS topology uses to talk to, and route commands from/to SATA (Serial-ATA) devices. SATA devices use a wire-level signaling that it somewhat similar to SAS, but outside of that, are quire different. SATA devices can be connected to a SAS topology however, and STP is used to tunnel this different data from a host, through the SAS network, to a SATA device.
I have put great detail into emphasizing the complete apathy the host and it's software has to all things "SAS" - emphasizing everything looks, appears and acts as it were on a legacy SCSI bus. You undoubtibly will soon fire up some piece of software somwhere which promptly details the entire SAS toplogy, WWNs, expanders, etc.
This is obviously very possible for a SAS HBA to provide very detailed information of the SAS domain to the host. So if its there, present and available, what is the point of all this?
Simple - the "generic" or "legacy" SCSI model has been around a long time. Virtualy all of the filsystems, volume managers, RAID software, etc. that have been written in every operating system for the last 20 years adheres to it. It was not all re-written when SAS came out - or iSCSI, or Fibre Channel, or SATA, etc. The core of all services outside of software that specifically view SAS busses knows only "SCSI". SAS therefore adheres to SCSI - so everything else doesn't have to adhere to SAS. It is "backward compatibility".
One little thing did change - sort of. As we mentioned before, SCSI originaly only allowed SCSI IDs to be from 0-7, so a lot of software only allowed those values. With the advent of Wide SCSI, IDs of up to 15 were possible, or even 31 with the theoretical implementation of 32-bit wide SCSI, which never existed. Thus, most software and APIs were modified to allow higher IDs. 16? 32? (Many, many years ago.)
The standard "recommendation" was to just give the whole byte, and allow for 256. But then with Fibre Channel and later iSCSI, this could be even higher. I guess a lot of APIs and such can use a "short" - 16 bits, or 65536. For OS's that can't do this, HBAs (since they're just making up the IDs anyway) could like and say there are more busses than physically present, adding more devices that way. I think, that it this day-and-age, it's not an issue anymore.
We shall conclude with a last look at SCSI's layered architecture - which emphasises the differences, and relationships between everything discussed above.
If you come away with anything, understand these:
SCSI is a architecture, or model, that defines how transactions work between connected devices, and what (many of) those transactions are. There are many different ways these devices can be physically connected, and the SCSI protocol can be carried.
SAS is a SCSI layer which is a carrier for the SCSI protocol. To system-software, this is unimportiant. SAS is treaed indistinguishably from "legacy" (parallel) SCSI, Fibre-Channel, iSCSI or future physical layers. The connection symantics or topology of SAS (or any other carrier) is completley abstracted from the host.
SES is an application that runs on top of SCSI - just as disk or tape service is. An SES device is a target device on a SCSI bus like any other.
As the competing standard to SCSI has been IDE (also called "ATA") - there is a serial version of the ATA spec as well, called SATA or Serial-ATA.
SATA and SAS have a very special relationship. SAS was developed after SATA, and was built off of many of concepts of SATA - particularly at the very low-level serial-interconnecte. They both use they same type of 1.5G, or 3.0G 10b/10b-encoded links. In fact, they are close enough that SATA devices can be used on a SAS fabric itself. Let's take a look at how this works.
A SATA drive, (much like a standard, "paralell"-ATA disk, sometimes referd to as a "PATA" disk), has a very simple view of the world. SATA was designed to connect a single host port to a single disk drive. Thus, a SATA host and a SATA drive see the interconnect as a simple point-to-point, two way-interconnect. The goals of SATA are similar to those of SAS - preserve the "legacy" software architecture and model. IDE busses could basically do two different things:
A simple register interface allowed commands, status, error, setup, etc to be done by reading and writing nine different 8-bit registers. This was an extension of the old IBM-PC's internal "AT" bus, which is where "ATA", or "AT-Attachement" gets it's name.
A subsequent DMA interface allowed for higher-speed transfer of larger blocks of data, for example, the actual data to read/write to/from disks.
So the SATA protocol was designed to perform these two functions. It uses exchanges called "FIS"es to exchange register states, and other frames for the data payload.
If a SAS HBA was "smart" enough to "speak" SATA, it could open a connection to a SATA disk, "speak" SATA to it, and everything would work just fine, right? Almost, but now quite. Remember that an ATA disk thinks that it's port is a simple point-to-point connection betweenn it and it's interconnected HBA. It has no knowledge of expanders in-between, switches or topologies. So how would this work? It is up to the expander which connects to a SATA drive to make it appear as though the host is always connected to it. This is done through "STP", or the "SATA Tunneling Protocol". Basically, a SAS host can open a connection to a SATA drive, send it a command, and then close the connection. The SATA drive however, doesn't see any "connections" or undersand the concept of one being "closed". It therefore procedes to (for example) read the requested data from its disk and buffer it. When it is ready to send the data back to the host, it starts DMAing it back on the SATA link. The expander (whcih is actually physically connected directly to the disk and receiving the physical data it is sending) responds by sending a signal to disk which tells it to essentially wait momentarily. The expander then opens up a return channel from the drive back to the host. Once the return channel is opened, the expander tells the drive to resume the transmission, and the drive sends the data - this time all the way back to the host. When it is complete, the expander again closes the connection.
So the concept here is that a SAS HBA is smart enough to send standard SCSI commands to devices via the SSP protocol, and to SATA devices via. the STP protocol. Could a SATA HBA talk to a SAS disk? The answer is no - as SATA HBAs are much simpler, they only know how to send FIS and DATA frames between it, and a directly-connected SATA disk. It does not understand the expander topologies, etc.
Brad Goodman
http://www.bradgoodman.com
Text, Illustrations and Misspellings ©2007-2011 Bradley K. Goodman, All Rights Reserved