Please see my other blog for Oracle EBusiness Suite Posts - EBMentors

Search This Blog

Note: All the posts are based on practical approach avoiding lengthy theory. All have been tested on some development servers. Please don’t test any post on production servers until you are sure.

Monday, August 04, 2014

Getting Flash Storage info

Exadata flash storage is provided by Sun Flash Accelerator F20 PCI flash cards. Each PCI flash card has a device partitioned per FDom, yielding 16 flash devices. These flash devices are manifested as ExadataStorage Server flash disks and used for Smart Flash Cache and Smart Flash Logging.


1- From the storage server host’s point of view, you can see flash devices using lsscsi:
[root@pk3-iub-cel-es01 ~]# lsscsi -v |grep ATA
[8:0:0:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdn
[8:0:1:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdo
[8:0:2:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdp
[8:0:3:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdq
[9:0:0:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdr
[9:0:1:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sds
[9:0:2:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdt
[9:0:3:0]    disk    ATA      3E128-TS2-550B01 UI39  /dev/sdu
[10:0:0:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdv
[10:0:1:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdw
[10:0:2:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdx
[10:0:3:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdy
[11:0:0:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdz
[11:0:1:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdaa
[11:0:2:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdab
[11:0:3:0]   disk    ATA      3E128-TS2-550B01 UI39  /dev/sdac

The flash devices are split into groups of four, 8:,9:, 10:, and 11:; this is because each of the four flash cards have four FMods. Thus, every ExadataStorage Server will have (4 x 4) = 16 flash devices. You can also use flash_dom –l to display details for the PCI flash devices:

[root@pk3-iub-cel-es01 ~]# flash_dom -l

Aura Firmware Update Utility, Version 1.2.7

Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved..

U.S. Government Rights - Commercial Software. Government users are subject
to the Sun Microsystems, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.

Use is subject to license terms.

This distribution may include materials developed by third parties.

Sun, Sun Microsystems, the Sun logo, Sun StorageTek and ZFS are trademarks
or registered trademarks of Sun Microsystems, Inc. or its subsidiaries,
in the U.S. and other countries.

2- From CellCLI we can see how these flash devices are mapped to usable Exadata Flash entities:

[root@pk3-iub-cel-es01 ~]# cellcli -e list physicaldisk where disktype='FlashDisk' attributes name,disktype,physicalSize,slotNumber
         FLASH_1_0       FlashDisk       93.13225793838501G      "PCI Slot: 1; FDOM: 0"
         FLASH_1_1       FlashDisk       93.13225793838501G      "PCI Slot: 1; FDOM: 1"
         FLASH_1_2       FlashDisk       93.13225793838501G      "PCI Slot: 1; FDOM: 2"
         FLASH_1_3       FlashDisk       93.13225793838501G      "PCI Slot: 1; FDOM: 3"
         FLASH_2_0       FlashDisk       93.13225793838501G      "PCI Slot: 2; FDOM: 0"
         FLASH_2_1       FlashDisk       93.13225793838501G      "PCI Slot: 2; FDOM: 1"
         FLASH_2_2       FlashDisk       93.13225793838501G      "PCI Slot: 2; FDOM: 2"
         FLASH_2_3       FlashDisk       93.13225793838501G      "PCI Slot: 2; FDOM: 3"
         FLASH_4_0       FlashDisk       93.13225793838501G      "PCI Slot: 4; FDOM: 0"
         FLASH_4_1       FlashDisk       93.13225793838501G      "PCI Slot: 4; FDOM: 1"
         FLASH_4_2       FlashDisk       93.13225793838501G      "PCI Slot: 4; FDOM: 2"
         FLASH_4_3       FlashDisk       93.13225793838501G      "PCI Slot: 4; FDOM: 3"
         FLASH_5_0       FlashDisk       93.13225793838501G      "PCI Slot: 5; FDOM: 0"
         FLASH_5_1       FlashDisk       93.13225793838501G      "PCI Slot: 5; FDOM: 1"
         FLASH_5_2       FlashDisk       93.13225793838501G      "PCI Slot: 5; FDOM: 2"
         FLASH_5_3       FlashDisk       93.13225793838501G      "PCI Slot: 5; FDOM: 3"

Observe the flash devices grouped in sets of four on PCI slots 1, 2, 4, and 5, with each device per PCI slot residing in FDOM 0, 1, 2, or 3. 

3- A fdisk output for one of the devices shows a slice of storage. 
[root@pk3-iub-cel-es01 ~]# fdisk -l /dev/sdn

Disk /dev/sdn: 100.0 GB, 100000000512 bytes
255 heads, 63 sectors/track, 12157 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdn doesn't contain a valid partition table
[root@pk3-iub-cel-es01 ~]#

No comments: