SAN Physical design: Cabling and Connections

Cable Types

An Optical Fiber consists of a very thin glass core surrounded by a cladding. The cladding has a lower refractive index than the core, so the boundary between core and cladding acts like a mirror and reflects light down the core. The light source is usually infra-red and might not be visible to the human eye. Optical Fibers come in two modes; single mode and multi mode with the thicknes if the light pipe measured in microns. Popular sizes are 9, 50 and 62.5 microns. In general the thinner the cable, the further the light goes. In laymans terms, single mode fiber is thin enough to restrict the light, so that it does not refract, or bounce off the fiber walls. This means that only one light wavelength can go down the pipe, but it suffers less interference and so can travel further without losing effective signal strength. Single mode also uses light emitters with a longer wavelength than multi-mode. In multi-mode fibres, the light can refract off the walls, so several wavelengths and paths are available. Single mode is more expensive than multi-mode and is typically used where transmission speed is essential, for example for multiplexed remote syncronous data mirroring.

If you are cabling up new kit to existing cables, then be aware that while you can join 50 micron multi-mode to 62.5 micron multi-mode, you will get some light loss when going from 62.5 to 50. Some of the light will leak away around the edges. If at all possible, you should keep a consistent cable size throughout a path. 62.5 micron cables are generally not used for new installations and are becoming obsolete.

The transmission speed and acceptable distance of a fiber depends on a number of factors, including light source and cable characteristics. In general, a single-mode 9 micron cable can trasmit up to 50km and a multi-mode 50 micron cable can transmit up to 900m.
One fiber term you will come across is 'dark fiber', usually used as if it is some special kind of fibre. When utility companies lay fiber in the ground, they always install more than is required for current needs, as the installation is by far the most expensive cost. This spare fiber has no light going down it, so it is 'dark'. And one other thing, I'm English and I like to use correct English spelling. Industry standards are that the architecture is called Fibre Channel and the cables are called fibers, which should explain why I spell Fibre two different ways.

Most machine halls use structured cabling these days, where individual fiber pairs are aggregated together into a cable, and terminated on a patch panel. Jumper cables are then used to connect appliances to the patch panel. Every connection will result in some light loss, so there are limits on how many patch panels can be in a circuit. It's best to check with individual vendors for details.


Fiber cables typically come with two different types of ends, SC or LC which stand for Subscriber Siemens and Lucent connectors. They are also sometimes called Square, Little or Local. The type of connector you need is determined by the type of socket in your SAN switch or HBA. SC connectors fit into GBICs (Gigabyte interface connectors) while LC connectors plug into SFPs (Small Form Factor Pluggable). SC terminations are generally used for slower speed fibers, and LC for faster speeds where space is important.
Fiber cables come as a pair of fibers, and each end will have a send and a receive fiber, known as TX and RX. An SC plug will only fit one way into a GBIC socket, and the most common error I've come across when recabling SAN switches is that the fiber pair is crossed. This means that the LED on one side is shining at the LED on the other side, and nothing is working. You will also see that the transmission light is orange instead of green. In this case, it is reasonably easy to unclip the two fibers from the plug and swap them over. However take care not to break the glass fiber itself.
The picture below shows the two common fiber connectors

Fiber connectors

Other types of connector include HSSDC (High-Speed Serial DataConnection) and DBm/DBf. You may also see MIAs or Media Interface Adaptors. They have a standard DB9 serial plug on one end and an SC socket on the other end. They are used to convert an electrical signal to optical.

Port types

There are several types of device ports:

  • N_PORTs are end-points in a FC network and are used to connect servers to storage devices to fibers.
  • F_PORTs or Fabric Port, used to connect fibers to switches, so typically a fiber will be connected to a device using an N_PORT on one end, then to a switch using an F_PORT on the other end. An F-PORT can only connect to one device.
  • An L-PORT is a loop capable node or switch port.
  • An NL-PORT or Node Loop Port is used to attach devices to a FC-AL arbitrated Loop.
  • An FL_Port is a fabric port used for connection to a loop. This enables loop nodes to link to fabric nodes.
  • An E-PORT or Extension Port is used for ISL links, so the other end will be an E-PORT on another switch
  • Ports can also be in unassigned status, when they are G-PORTS or U-PORTS. A G-Port is assigned but the link initialisation is not complete. A U-Port is unassigned and not connected.
  • An EX-Port connects a router to an edge fabric but does not allow the fabrics to merge.
  • A VE-Port is a virtual port which emulates an E-Port over FCIP
  • A VEX-Port is a virtual routed EX-Port
  • A TE-Port is a trunking E-Port which allows routing of multiple VSANs.

Node and Port Addresses

A Node is an appliance that is connected to a fabric SAN and every Node has a unique 64-bit address called 'World Wide Node Name'. Every Port also has a unique 64-bit address called the 'World Wide Port Name'. These addresses are usually writen as a sequence of 8 hex bytes separated by colons like this 10:00:00:60:69:50:60:02. Bytes 3-5 are assigned to each vendor by the IEE naming standards body. Every Node as a unique WWNN node name, and this can mean individual HBAs, switches, storage units, storage arrays, tape libraries and tape drives. Every port on every node has a unique WWWPN port name.
This unique set of names means that it is possible to specify exactly which port in which switch or HBA that you need to address, pretty much in the same way you can use a telephone to reach a person.

Buffer Credits

When a data record is transmitted down a Fibre Channel record it is split up into a number of varying size blocks or frames. These blocks are transmitted through the fabric, then re-assembled at the far end to create the data record again. Switches are store-and-forward devices, they do not just pass the data straight through. To make sure the ISL fibers are used efficiently, every switch port has a number of buffers associated with it, called buffer credits or BB-Credits. The switch can then store several blocks of incoming data, while waiting to pass it on to the next node. When a receiver is ready to take information, it signals to its sender, then decrements the BB-Credit. When the data block is passed on to the next receiver the BB-Credit is incremented again. This means that BB-Credits are also used to throttle back the data transmission flow when devices or links get too busy.
Buffer credits are consumed by frames, not data. A large frame and a small frame will both use one buffer, so it's frame count, not the amount of data that matters.
EE-Credits or End to End credits are established between two communicating N-Ports and control the overall flow of the data stream through the fabric.
On a Brocade switch, you can check status of these buffers with the command 'portbuffershow'. This will tell you, among other things, how many buffers are allocated to each port, how many are in use, and how many are needed for efficient channel usage.

Record Keeping

Record keeping is the boring, but absolutely essential part of installing and managing a SAN. A good set of records makes it much easier to fix things when they go wrong.
At a minimum, you should have a diagram that shows every port on every switch, what type of port it is and what it is connected to. The good news is that most SAN management packages will produce this documentation for you, but it is important that you extract a fresh copy every time you make a change. Export it off onto a laptop, or even print it out, so you can take it into the machine hall with you.

back to top