Server System Memory Hardware Guide
Server System Memory Hardware Guide
Memory for servers differs from desktop memory by way of added hardware for better reliability and efficient transfers of many small packets. However, server memory can be expensive and confusing in terms of technology and performance. This guide was designed to explain how server memory works and why it is different from desktop memory.
Hardware and Construction
 |
Current memory is made up of either 8 to 16 chips called DRAM (Dynamic Random Access Memory) chips. These chips are the actual storage areas for data. Each bit is comprised of either voltage or no voltage on a capacitor and transistor pair. Because capacitors lose voltage, they must be continually refreshed or else risk losing data.
Data densities range from 64 MB to 1 GB per DRAM chip.
|
 |
Those DRAM chips are then put into a module called a DIMM (Dual Inline Memory Module). It is called dual because both sides of the module use a connection to connect the the motherboard. Current DIMMs for desktop and server computers use 120 pins, but have 240 connections. Depending on how many and the density of of the DRAM chips, the module may have densities anywhere from 512 MB to 8 GB.
|
Server memory has technologies which differentiate itself from desktop memory. Server memory has to be reliable and stable, but also provide quick access to that data. These technologies fix errors and allows more memory usage than desktop memory technologies. Current server memory technology is based upon the JEDEC DDR2 standard.
Fully Buffered (FB) Memory
Fully buffered memory has an Advanced Memory Buffer (AMB) between the memory controller and the ram modules. Also unlike normal memory which operates in parallel, this memory operates in serial so that all data must travel through the AMB. This approach can increase bandwidth without increasing pin counts and possibly offer higher speeds. Uses standard DDR2 technology with the extra logic directly on the DIMM. This setup like the last also introduces latency but with the benefits of error correcting and compensation for address errors. Unlike current desktop memory, memory access get slower by using more DIMMS of the same total memory size. That means that 8x1GB DIMMS will offer slower performance than 4x2GB DIMMS. This is because the AMB has to control more DIMMs, and thus more latency. This is good for servers, as being able to use more dense memory modules means more total available space for even more memory. Also unlike a desktop in dual channel mode, FB memory allows for the use of quad channels due to the lowered pin count. It thus makes sense to populate your server motherboard with FB memory in groups of four.
Registered Memory
Registered memory places logic between the memory controller and memory modules to take the load off the memory controller. This allows the system to use more memory then the current memory controller is capable of. These types of memory place a latency penalty of one clock cycle on read and write instructions, so this memory is always slower then non-registered memory. Most server grade
Error Correcting Code (ECC) Memory
Normal memory has to continually refresh its memory banks. It does this by taking the current bit, and re-writing it into the same spot. ECC memory is able to use extra logic to compensate for a 1-bit error by using algorithms like the Hamming Code. Most memory controllers however already are able to detect and correct errors. Also, these types of memories are designed for environments where errors can happen where strong electromagnetic fields can influence the data in the memory. Ranked Memory
A rank is memory terms is how many 64-bit (72 bit with ECC) data areas exist on a single DIMM. All memory is ranked, but it is of great important in server memory as servers are prone to exceeding the rank limits of server hardware. Ranks were created to overcome the small DRAM chip densities which make up a memory DIMM. This allowed to create higher density DIMMs without the need to use more expensive higher density DRAM chips.
Single ranked (1R) memory works by using the highest density DRAM chips. Dual ranked (2R) memory works by using twice as many DRAM chips at half the density. The memory module decides which side of the chip to access (rank) based on what data is needed. This means that the maximum amount of storage a single ranked DIMM can access is 100% of its data at once. Also means that the dual ranked DIMM can only access 50% of its data at any given time. Because of the use of more expensive components, single ranked memory is more expensive than its dual ranked counterpart.
Quad ranked memory is available, but with the advent of larger DRAM chips, that much ranking was not needed.
Server hardware can only accept so many ranks per system. Ranking limits can happen when you exceed the maximum amount of available ranks for a given memory configuration. The example below shows a system with 6 slots and 8 ranks available. Notice how in the first path, the system cannot have more memory added without going over the ranking limit.
|
Slot 1 |
Slot 2 |
Slot 3 |
Slot 4 |
Slot 5 |
Slot 6 |
Config 1 |
2R |
2R |
2R |
2R |
X |
X |
Config 2 |
2R |
2R |
1R |
1R |
1R |
1R |
Memory Speed
Frequencies Ram manufacturers will list the speed of their memories with two different numbering schemes, DDR-xxx and PC-yyyy.
The first number xxx refers to the clock rate of the memory. DDR400 works at the maximum rate of 400 MHz. However, because DDR and up memory transmits on both the up and the down of the clock, its actual speed is half the posted speed, so DDR400 really runs at the speed of 200 MHz.
The second number yyyy, refers to the maximum transfer rate of the ram module. PC4200 ram will transmit 4,200 MB/s and PC5400 will transmit at 5,400 MB/s. Until ram starts sending data more than twice a clock cycle, the speed is directly connected to the transfer rate. The transfer rate of ram is equal to 8 * the clock rate. DDR-400 is also PC-3200 ram and DDR2-800 is also PC-6400.
Type |
Rating |
Bus Clock (MHz) |
Typical Timings |
Maximum Bandwidth (GB/s) |
Single-Channel |
Dual-Channel |
DDR2-400 |
PC2-3200 |
200 |
3-3-3-9 |
3.20 |
6.40 |
DDR2-533 |
PC2-4200 |
266 |
4-4-4-12 |
4.26 |
8.53 |
DDR2-667 |
PC2-5300 |
333 |
5-5-5-15 |
5.34 |
10.67 |
DDR2-800 |
PC2-6400 |
400 |
5-5-5-15 |
6.40 |
12.80 |
DDR2-1066 |
PC2-8500 |
533 |
5-5-5-15 |
8.50 |
17.00 |
Latency
Memory uses a few clock cycles between getting a command for retrieving memory and actually sending it back to the CPU. The time it takes to retrieve the data is called latency, and it can have a big effect on performance.
RAM manufacturers typically list the recommended timing for their RAM as a series of four integers separated by dashes (e.g 2-2-2-6 or 3-3-3-8 or 4-4-4-12 and so on). While there are many other settings related to ram, these four integers refer to the following settings, which are typically listed in this order: CL - Trcd - Trp - Tras.
CL: CAS Latency. The time it takes between a command having been sent to the memory and when it begins to reply to it. It is the time it takes between the processor asking for some data from the memory and it returning it.
tRCD: RAS to CAS Delay. The time it takes between the activation of the line (RAS) and the column (CAS) where the data are stored in the matrix.
tRP: RAS Precharge. The time it takes between disabling the access to a line of data and the begin of the access the another line of data.
tRAS: Active to Precharge Delay. How long the memory has to wait until the next access to the memory can be initiated.