Build log: Titus/Andronicus

Discussion in 'Cases / Modding Discussion' started by stdPichu, Jul 19, 2013.

Share This Page

  1. MikeBuzz

    MikeBuzz New Member

    Joined:
    Feb 20, 2014
    Messages:
    11
    Likes Received:
    0
    I did look at the low power models but they are hard to find and normally more money, there are a number of people out there using this same CPU and case with out issue. The avoton boards did look promising but the lack of availability put me off, maybe on the next build I could use one
     
  2. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    From the stuff I've seen on the ServeTheHome forums, idling power between the normal E3's and the lowe-power models is identical, it's only peak power that's different - something that you're rarely, if ever, going to hit on a NAS anyway. From various people's testing, there's only 5-10W difference at idle between the Xeons and the Avotons (less if yo go for a dual core Pentium) which doesn't add up to much of a hill of beans when you factor in the rest of the kit.

    The Avoton boards have passive cooling but still require a fair degree of airflow in order to stay moderately cool, something I don't think the NSC 800 might be too great at in its stock configuration, which is why I thought the E3 Xeon and the E3C22xD2I boards represented great speed per quid.

    I've got the same 1230v3 running at stock speeds, it's never gone above 50 degrees despite the hilariously cramped chassis and a fan that never goes above 1000rpm, the hard discs seem stable at more-or-less 40 degrees depending on model. Not that I give much of a care about temperatures. Andronicus is still running eight 2TB hard drives that I accidentally cooked to over 70 degrees when I stacked them on top of each other to test the M1015 and forgot to turn them off and they're still running fine ;)
     
  3. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    OK, so the samba AD has more or less been behaving itself so the time has come to build my main RAID array and copy stuff to it from my old QNAP. For now, this is going to be a like-for-like replacement; I'm going to be moving the six 3TB discs out of my QNAP and into Titus.

    "But stdPichu!" I hear you wail, "won't that necessitate either length downtime, restoring from backup, or buying a whole bunch of extra hard drives you're not going to use?". Of course not - this is one of the great things I love about RAID10, it can lose up to half its discs before the array falls over and dies... so I've taken exactly the right half of my discs from the RAID10 array in my QNAP and plugged them into alternate slots in Titus. Please note this is not for the faint of heart and you'd be mad to attempt this without a backup. They're all Western Digital, so here's a look in the dev tree:

    Code:
    root@titus:~# ls -l /dev/disk/by-id/|grep -i ata-wdc
    lrwxrwxrwx 1 root root  9 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452 -> ../../sde
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452-part1 -> ../../sde1
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452-part2 -> ../../sde2
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452-part3 -> ../../sde3
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452-part4 -> ../../sde4
    lrwxrwxrwx 1 root root  9 May  8 20:33 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596 -> ../../sdc
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596-part1 -> ../../sdc1
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596-part2 -> ../../sdc2
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596-part3 -> ../../sdc3
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596-part4 -> ../../sdc4
    lrwxrwxrwx 1 root root  9 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536 -> ../../sdd
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536-part1 -> ../../sdd1
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536-part2 -> ../../sdd2
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536-part3 -> ../../sdd3
    lrwxrwxrwx 1 root root 10 May  8 20:32 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536-part4 -> ../../sdd4
    There we see the four partitions that QNAP sets discs up with; we could mount the RAID array as-is (it should have been auto-detected by mdadm) but cobblers to that, that would waste nearly 1.5GB of space per disc! Since these are >2TB discs, they need a GPT partition to use their full capacity and we can see their partition tables with gdisk:

    Code:
    root@titus:~# gdisk -l /dev/sdc
    GPT fdisk (gdisk) version 0.8.8
    
    Partition table scan:
      MBR: protective
      BSD: not present
      APM: not present
      GPT: present
    
    Found valid GPT with protective MBR; using GPT.
    Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
    Logical sector size: 512 bytes
    Disk identifier (GUID): 0ED4E1EC-9614-4E28-B8E0-5908A5C3A6EE
    Partition table holds up to 128 entries
    First usable sector is 34, last usable sector is 5860533134
    Partitions will be aligned on 8-sector boundaries
    Total free space is 21157 sectors (10.3 MiB)
    
    Number  Start (sector)    End (sector)  Size       Code  Name
       1              40         1060289   517.7 MiB   0700  primary
       2         1060296         2120579   517.7 MiB   0700  primary
       3         2120584      5859515969   2.7 TiB     0700  primary
       4      5859515976      5860511999   486.3 MiB   0700  primary
    I say tish and fipsy to that. I'm going to erase those partitions and create one big fat GPT partition on each disc to use for RAID10. First let's stop any new RAID array mdadm may have detected:

    Code:
    root@titus:~# mdadm --stop /dev/md127
    mdadm: stopped /dev/md127
    Now let's erase the partition tables from /dev/sdc:

    Code:
    root@titus:~# sgdisk --zap /dev/sdc
    root@titus:~# gdisk -l /dev/sdc
    GPT fdisk (gdisk) version 0.8.8
    
    Partition table scan:
      MBR: not present
      BSD: not present
      APM: not present
      GPT: not present
    
    Creating new GPT entries.
    Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
    Logical sector size: 512 bytes
    Disk identifier (GUID): FBFFBD5D-ADD2-4266-B621-213572348FED
    Partition table holds up to 128 entries
    First usable sector is 34, last usable sector is 5860533134
    Partitions will be aligned on 2048-sector boundaries
    Total free space is 5860533101 sectors (2.7 TiB)
    
    Number  Start (sector)    End (sector)  Size       Code  Name
    Now we can again use gdisk to create a single GPT partition:

    Code:
    root@titus:~# gdisk /dev/sdc
    GPT fdisk (gdisk) version 0.8.8
    
    Partition table scan:
      MBR: protective
      BSD: not present
      APM: not present
      GPT: not present
    
    Creating new GPT entries.
    
    Command (? for help): n
    Partition number (1-128, default 1):
    First sector (34-5860533134, default = 2048) or {+-}size{KMGTP}:
    Last sector (2048-5860533134, default = 5860533134) or {+-}size{KMGTP}:
    Current type is 'Linux filesystem'
    Hex code or GUID (L to show codes, Enter = 8300): l
    0700 Microsoft basic data  0c01 Microsoft reserved    2700 Windows RE
    4100 PowerPC PReP boot     4200 Windows LDM data      4201 Windows LDM metadata
    7501 IBM GPFS              7f00 ChromeOS kernel       7f01 ChromeOS root
    7f02 ChromeOS reserved     8200 Linux swap            8300 Linux filesystem
    8301 Linux reserved        8302 Linux /home           8400 Intel Rapid Start
    8e00 Linux LVM             a500 FreeBSD disklabel     a501 FreeBSD boot
    a502 FreeBSD swap          a503 FreeBSD UFS           a504 FreeBSD ZFS
    a505 FreeBSD Vinum/RAID    a580 Midnight BSD data     a581 Midnight BSD boot
    a582 Midnight BSD swap     a583 Midnight BSD UFS      a584 Midnight BSD ZFS
    a585 Midnight BSD Vinum    a800 Apple UFS             a901 NetBSD swap
    a902 NetBSD FFS            a903 NetBSD LFS            a904 NetBSD concatenated
    a905 NetBSD encrypted      a906 NetBSD RAID           ab00 Apple boot
    af00 Apple HFS/HFS+        af01 Apple RAID            af02 Apple RAID offline
    af03 Apple label           af04 AppleTV recovery      af05 Apple Core Storage
    be00 Solaris boot          bf00 Solaris root          bf01 Solaris /usr & Mac Z
    bf02 Solaris swap          bf03 Solaris backup        bf04 Solaris /var
    bf05 Solaris /home         bf06 Solaris alternate se  bf07 Solaris Reserved 1
    bf08 Solaris Reserved 2    bf09 Solaris Reserved 3    bf0a Solaris Reserved 4
    bf0b Solaris Reserved 5    c001 HP-UX data            c002 HP-UX service
    ea00 Freedesktop $BOOT     eb00 Haiku BFS             ed00 Sony system partitio
    ef00 EFI System            ef01 MBR partition scheme  ef02 BIOS boot partition
    Press the  key to see more codes:
    fb00 VMWare VMFS           fb01 VMWare reserved       fc00 VMWare kcore crash p
    fd00 Linux RAID
    Hex code or GUID (L to show codes, Enter = 8300): fd00
    Changed type of partition to 'Linux RAID'
    
    Command (? for help): w
    
    Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
    PARTITIONS!!
    
    Do you want to proceed? (Y/N): y
    OK; writing new GUID partition table (GPT) to /dev/sdc.
    The operation has completed successfully.
    Now view your handiwork:

    Code:
    root@titus:~# gdisk -l /dev/sdc
    GPT fdisk (gdisk) version 0.8.8
    
    Partition table scan:
      MBR: protective
      BSD: not present
      APM: not present
      GPT: present
    
    Found valid GPT with protective MBR; using GPT.
    Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
    Logical sector size: 512 bytes
    Disk identifier (GUID): 2A1D847A-54E2-4260-9902-93515B4E5816
    Partition table holds up to 128 entries
    First usable sector is 34, last usable sector is 5860533134
    Partitions will be aligned on 2048-sector boundaries
    Total free space is 2014 sectors (1007.0 KiB)
    
    Number  Start (sector)    End (sector)  Size       Code  Name
       1            2048      5860533134   2.7 TiB     FD00  Linux RAID
    As you can see from the start sector of 2048, the partition is aligned, unlike the QNAP partitions. Tut tut. Now you can either repeat the above steps for the other discs or use sgdisk (the GPT-compatible tools to replace sfdisk) to clone that partition table to the other two drives:

    Code:
    root@titus:~# sgdisk -R=/dev/sdc /dev/sdd
    The operation has completed successfully.
    root@titus:~# sgdisk -R=/dev/sdc /dev/sde
    The operation has completed successfully.
    If you opt for this method rather than creating the partitions manually, there's an additional step you have to do with GPT which is to set a new GUID on these partitions:

    Code:
    root@titus:~# sgdisk -G /dev/sdd
    The operation has completed successfully.
    root@titus:~# sgdisk -G /dev/sde
    The operation has completed successfully.
    ...or you can create them manually by repeating the gdisk steps above for each new drive. At the end of the day, we should now see only three partitions:

    Code:
    root@titus:~# ls -l /dev/disk/by-id/|grep -i ata-wdc
    lrwxrwxrwx 1 root root  9 May  8 20:55 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452 -> ../../sdc
    lrwxrwxrwx 1 root root 10 May  8 20:55 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0116452-part1 -> ../../sdc1
    lrwxrwxrwx 1 root root  9 May  8 20:56 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596 -> ../../sdd
    lrwxrwxrwx 1 root root 10 May  8 20:56 ata-WDC_WD30EZRS-00J99B0_WD-WCAWZ0157596-part1 -> ../../sdd1
    lrwxrwxrwx 1 root root  9 May  8 20:56 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536 -> ../../sde
    lrwxrwxrwx 1 root root 10 May  8 20:56 ata-WDC_WD30EZRS-00J99B0_WD-WMAWZ0053536-part1 -> ../../sde1
    Now we're ready to create a new RAID10 array. Note I've defined the array will consist of six discs but three of them are currently missing:

    Code:
    root@titus:~# mdadm --create --verbose /dev/md32 --level raid10 --name storage --raid-disks 6 /dev/sdc1 missing /dev/sdd1 missing /dev/sde1 missing
    mdadm: layout defaults to n2
    mdadm: layout defaults to n2
    mdadm: chunk size defaults to 512K
    mdadm: size set to 2930134016K
    mdadm: automatically enabling write-intent bitmap on large array
    mdadm: Defaulting to version 1.2 metadata
    mdadm: array /dev/md32 started.
    We can now check md32:

    Code:
    root@titus:~# cat /proc/mdstat
    Personalities : [raid1] [raid10]
    md32 : active raid10 sdc1[4] sdd1[2] sde1[0]
          8790402048 blocks super 1.2 512K chunks 2 near-copies [6/3] [U_U_U_]
          bitmap: 66/66 pages [264KB], 65536KB chunk
    
    md1 : active raid1 sda5[0] sdb5[1]
          233365080 blocks super 1.2 [2/2] [UU]
          bitmap: 1/1 pages [4KB], 131072KB chunk
    
    md0 : active raid1 sda1[0] sdb1[1]
          498368 blocks super 1.2 [2/2] [UU]
    
    unused devices: 
    For a belt'n'braces approach, we'll add the RAID data to the config in /etc/mdadm/mdadm.conf. You can print this like so:

    Code:
    root@titus:~# mdadm --detail --scan
    ARRAY /dev/md/0 metadata=1.2 name=titus:0 UUID=68eb4d67:0e14bf67:3061e653:fd7d53b0
    ARRAY /dev/md/1 metadata=1.2 name=titus:1 UUID=d00fe243:df2e7c05:b6ca7bb7:7afa3dec
    ARRAY /dev/md32 metadata=1.2 name=titus:storage UUID=ade58d4e:ed83fda3:6a9dd8b4:22dd52db
    We could format this now, but I'm going to create a new LVM setup for this.

    Code:
    root@titus:~# pvdisplay
      --- Physical volume ---
      PV Name               /dev/md1
      VG Name               vg_root
      PV Size               222.55 GiB / not usable 2.59 MiB
      Allocatable           yes
      PE Size               4.00 MiB
      Total PE              56973
      Free PE               30913
      Allocated PE          26060
      PV UUID               s77eyX-dkzr-Q6xT-FYI3-ydnS-Mf1z-vRhh1x
    
      "/dev/md32" is a new physical volume of "8.19 TiB"
      --- NEW Physical volume ---
      PV Name               /dev/md32
      VG Name
      PV Size               8.19 TiB
      Allocatable           NO
      PE Size               0
      Total PE              0
      Free PE               0
      Allocated PE          0
      PV UUID               tkojNA-o9ZT-0eBA-CY9C-3jPQ-k2QK-qT3ZHe
    Now a new volume group inside /dev/md32:

    Code:
    root@titus:~# vgcreate vg_storage /dev/md32
      Volume group "vg_storage" successfully created
    root@titus:~# vgdisplay
      --- Volume group ---
      VG Name               vg_storage
      System ID
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  1
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                0
      Open LV               0
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               8.19 TiB
      PE Size               4.00 MiB
      Total PE              2146093
      Alloc PE / Size       0 / 0
      Free  PE / Size       2146093 / 8.19 TiB
      VG UUID               guodkl-hh3D-UKmj-LH5O-7mnA-jPZX-1Fpv6V
    
      --- Volume group ---
      VG Name               vg_root
      System ID
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  10
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                6
      Open LV               5
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               222.55 GiB
      PE Size               4.00 MiB
      Total PE              56973
      Alloc PE / Size       26060 / 101.80 GiB
      Free  PE / Size       30913 / 120.75 GiB
      VG UUID               HqhwWr-gyKM-WTkz-SGjN-kgt0-WDeO-uTZ5cC
    ...and now a single logical volume within that. This makes things very easy to shrink or expand at a later date.

    Code:
    root@titus:~# lvcreate -l 100%FREE -n lv_bulk vg_storage
      Logical volume "lv_bulk" created
    root@titus:~# lvdisplay
      --- Logical volume ---
      LV Path                /dev/vg_storage/lv_bulk
      LV Name                lv_bulk
      VG Name                vg_storage
      LV UUID                BOrrgh-hcLP-BNTe-HfIe-esTu-91TK-vnRw3e
      LV Write Access        read/write
      LV Creation host, time titus, 2014-05-08 21:12:06 +0100
      LV Status              available
      # open                 0
      LV Size                8.19 TiB
      Current LE             2146093
      Segments               1
      Allocation             inherit
      Read ahead sectors     auto
      - currently set to     6144
      Block device           253:6
    
    Now I'm going to format as ext4. Long story short - after a lot of testing over the last week or so I still find btrfs too experimental and unreliable for my needs. I'll upgrade once it becomes a bit more stable and fuller-featured.

    Pay special attention to the 64bit option - an ext4 filesystem can't grow beyond 16TB without it.

    Code:
    root@titus:~# mkfs.ext4 -m 1 -L titus_bulk -O 64bit /dev/mapper/vg_storage-lv_bulk
    mke2fs 1.42.9 (4-Feb-2014)
    Filesystem label=titus_bulk
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=128 blocks, Stripe width=384 blocks
    274702336 inodes, 2197599232 blocks
    21975992 blocks (1.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=4294967296
    67066 block groups
    32768 blocks per group, 32768 fragments per group
    4096 inodes per group
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
            4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
            102400000, 214990848, 512000000, 550731776, 644972544, 1934917632
    
    Allocating group tables: done
    Writing inode tables: done
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done
    root@titus:~# mount /dev/mapper/vg_storage-lv_bulk /storage/
    root@titus:~# df -h|grep storage
    /dev/mapper/vg_storage-lv_bulk  8.2T   90M  8.1T   1% /storage
    Let's give the system a quick test for speed.

    Code:
    root@titus:~# hdparm -tT --direct /dev/mapper/vg_storage-lv_bulk
    
    /dev/mapper/vg_storage-lv_bulk:
     Timing O_DIRECT cached reads:   1002 MB in  2.00 seconds = 500.81 MB/sec
     Timing O_DIRECT disk reads: 798 MB in  3.01 seconds = 264.86 MB/sec
    Not bad at all for a half-full array. Now my storage is mounted I can begin the long, long process of rsyncing the new data over... even at the ~110MB/s I can expect to get over 1Gb/s ethernet this will take a while... see you in a month or two :)
     
  4. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    Now the data has been written, I've added the remaining three hard discs from my now-retired QNAP, blown away the old partitions and cloned the new GPT partitions on. Now to add them to the RAID array:

    Code:
    root@titus:~# mdadm --manage /dev/md32 --add /dev/sdf1
    mdadm: added /dev/sdf1
    root@titus:~# mdadm --manage /dev/md32 --add /dev/sdg1
    mdadm: added /dev/sdg1
    root@titus:~# mdadm --manage /dev/md32 --add /dev/sdh1
    mdadm: added /dev/sdh1
    You should now be able to see the array rebuilding:

    Code:
    root@titus:~# cat /proc/mdstat
    Personalities : [raid1] [raid10]
    md32 : active raid10 sdh1[8] sdg1[7] sdf1[6] sdc1[4] sdb1[2] sda1[0]
          8790402048 blocks super 1.2 512K chunks 2 near-copies [6/3] [U_U_U_]
          [>....................]  recovery =  0.0% (27008/2930134016) finish=48791.7min speed=1000K/sec
          bitmap: 66/66 pages [264KB], 65536KB chunk
    
    md1 : active raid1 sde5[0] sdd5[1]
          233365080 blocks super 1.2 [2/2] [UU]
          bitmap: 1/1 pages [4KB], 131072KB chunk
    
    md0 : active raid1 sde1[0] sdd1[1]
          498368 blocks super 1.2 [2/2] [UU]
    
    unused devices: 
    Hmm. Looks like I've not rebooted since I applied my speedy RAID rebuild to sysctl.conf. Let's do that manually now:

    Code:
    root@titus:~# echo 50000 > /proc/sys/dev/raid/speed_limit_min
    root@titus:~# echo 5000000 > /proc/sys/dev/raid/speed_limit_max
    Sync speed is much improved.

    Code:
    root@titus:~# cat /proc/mdstat
    Personalities : [raid1] [raid10]
    md32 : active raid10 sdh1[8] sdg1[7] sdf1[6] sdc1[4] sdb1[2] sda1[0]
          8790402048 blocks super 1.2 512K chunks 2 near-copies [6/3] [U_U_U_]
          [>....................]  recovery =  0.3% (9860736/2930134016) finish=958.7min speed=50766K/sec
          bitmap: 66/66 pages [264KB], 65536KB chunk
    
    md1 : active raid1 sde5[0] sdd5[1]
          233365080 blocks super 1.2 [2/2] [UU]
          bitmap: 0/1 pages [0KB], 131072KB chunk
    
    md0 : active raid1 sde1[0] sdd1[1]
          498368 blocks super 1.2 [2/2] [UU]
    
    unused devices: 
    Storage speed will take a hit as the RAID rebuild gets underway; bear in mind that all discs in the stripe are now being cloned at ~50MB/s:

    Code:
    root@titus:~# hdparm -tT --direct /dev/mapper/vg_storage-lv_bulk
    
    /dev/mapper/vg_storage-lv_bulk:
     Timing O_DIRECT cached reads:   120 MB in  2.01 seconds =  59.59 MB/sec
     Timing O_DIRECT disk reads: 228 MB in  3.10 seconds =  73.64 MB/sec
     
  5. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    OK, a minor update - the RAID sync didn't speed up any from the 40-50MB/s above (RAID rebuilds often tend to start off slow and then pick up in the middle as they go on), and I'd expect the array to go at at least 100MB/s. To top it off, iostat showed that the discs were being only about 20% utilised. I read a post about RAID arrays not picking up controls to their rebuild speeds if those speed changes are made after the array is started; a quick check through dmesg showed mdadm was still using the settings it booted with:

    Code:
    md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
    md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
    I could have probably gotten away with just unmounting the disc and stopping/starting the RAID array but I rebooted the box anyway, ensuring that the rebuild values were applied before mdadm started.

    Low and beer hole, iostat now shows discs peaking at 100% frequently*...

    Code:
    Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
    sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
    sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
    sdh               0.00   782.00    0.00  195.00     0.00 62528.00   641.31     0.14    0.72    0.00    0.72   0.64  12.40
    sde               0.00   782.00    0.00  197.00     0.00 62592.00   635.45     6.45   32.89    0.00   32.89   5.08 100.00
    sdg             778.00     0.00  195.00    0.00 62528.00     0.00   641.31     0.26    1.31    1.31    0.00   1.21  23.60
    sdc             778.00     0.00  195.00    0.00 62528.00     0.00   641.31     0.26    1.31    1.31    0.00   1.21  23.60
    sdd               0.00   782.00    0.00  194.00     0.00 62208.00   641.32    33.74  172.54    0.00  172.54   5.15 100.00
    sdf             778.00     0.00  195.00    0.00 62528.00     0.00   641.31     0.25    1.29    1.29    0.00   1.19  23.20
    ...and sync speed jumped to 150MB/s.

    Code:
    root@titus:~# cat /proc/mdstat
    Personalities : [raid1] [raid10]
    md32 : active raid10 sdf1[4] sdg1[2] sde1[7] sdc1[0] sdd1[8] sdh1[6]
          8790402048 blocks super 1.2 512K chunks 2 near-copies [6/3] [U_U_U_]
          [==================>..]  recovery = 91.5% (2683128448/2930134016) finish=38.2min speed=146782K/sec
          bitmap: 61/66 pages [244KB], 65536KB chunk
    
    md1 : active raid1 sdb5[0] sda5[1]
          233365080 blocks super 1.2 [2/2] [UU]
          bitmap: 1/1 pages [4KB], 131072KB chunk
    
    md0 : active raid1 sdb1[0] sda1[1]
          498368 blocks super 1.2 [2/2] [UU]
    
    unused devices: 
    As you'd expect with RAID10, CPU utilisation during the rebuild has been low:

    Code:
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     1244 root      20   0       0      0      0 D   1.7  0.0  13:24.39 md32_resync
      769 root      20   0       0      0      0 S   1.0  0.0   7:51.02 md32_raid10
    And... tadaa! RAID all nicely built.

    Code:
    root@titus:~# cat /proc/mdstat
    Personalities : [raid1] [raid10]
    md32 : active raid10 sdf1[4] sdg1[2] sde1[7] sdc1[0] sdd1[8] sdh1[6]
          8790402048 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
          bitmap: 0/66 pages [0KB], 65536KB chunk
    
    md1 : active raid1 sdb5[0] sda5[1]
          233365080 blocks super 1.2 [2/2] [UU]
          bitmap: 1/1 pages [4KB], 131072KB chunk
    
    md0 : active raid1 sdb1[0] sda1[1]
          498368 blocks super 1.2 [2/2] [UU]
    
    unused devices: 
    Let's do another brief test to see how it stacks up.

    Code:
    root@titus:~# hdparm -tT /dev/mapper/vg_storage-lv_bulk
    
    /dev/mapper/vg_storage-lv_bulk:
     Timing cached reads:   31892 MB in  2.00 seconds = 15963.49 MB/sec
     Timing buffered disk reads: 1160 MB in  3.00 seconds = 386.59 MB/sec
    ~380MB/s is OK for six spindles but we'll see what else we can do with that in the near future.

    * As you can probably tell, iostat is a great utility for an underperforming RAID array - you can run it under a workload and see which of your discs in the continually at 100% to help you spot the weakest link in the chain.
     
  6. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    A minor interlude as something terrifying happened last night - Eurosong 2014! So I'm making a small diversion to set up get-iplayer to pull down this televisual feast. Firstly, ensure libav-tools (the Debian version of ffmpeg, long story) is installed and secondly, set you default iplayer preferences:

    Code:
    std@titus:~$ get-iplayer --prefs-add --mode=best --subtitles --whitespace --fatfilename --file-prefix=" - " --subdir --subdir-format="" --output "/storage/download/iplayer"
    INFO: Added option 'subdirformat' = ''
    INFO: Added option 'subdir' = '1'
    INFO: Added option 'subtitles' = '1'
    INFO: Added option 'fileprefix' = ' - '
    INFO: Added option 'whitespace' = '1'
    INFO: Added option 'fatfilename' = '1'
    INFO: Added option 'output' = '/storage/download/iplayer'
    INFO: Added option 'modes' = 'best'
    INFO: Options file /home/std/.get_iplayer/options updated
    A quick search shows four euro-tastic shows, however we only want the grand finale so let's just download that:

    Code:
    std@titus:~$ get-iplayer "eurovision"
    get_iplayer v2.86, Copyright (C) 2008-2010 Phil Lewis
      This program comes with ABSOLUTELY NO WARRANTY; for details use --warranty.
      This is free software, and you are welcome to redistribute it under certain
      conditions; use --conditions for details.
    
    Matches:
    340:    Eurovision Song Contest - 2014: 1. Semi-Final One, BBC One, Entertainment,Guidance,Music,Pop & Chart,TV,Talent Shows,Variety Shows, default
    341:    Eurovision Song Contest - 2014: 2. Semi-Final Two, BBC One, Entertainment,Guidance,Music,Pop & Chart,Popular,TV,Talent Shows,Variety Shows, default,
    342:    Eurovision Song Contest - 2014: 3. Grand Final, BBC One, Entertainment,Guidance,Highlights,Music,Pop & Chart,TV,Talent Shows,Variety Shows, default,
    482:    How to Win Eurovision - -, BBC Three, Entertainment,Music,Pop & Chart,TV, default
    
    INFO: 4 Matching Programmes
    
    std@titus:~$ get-iplayer -g 342
    get_iplayer v2.86, Copyright (C) 2008-2010 Phil Lewis
      This program comes with ABSOLUTELY NO WARRANTY; for details use --warranty.
      This is free software, and you are welcome to redistribute it under certain
      conditions; use --conditions for details.
    
    Matches:
    342:    Eurovision Song Contest - 2014: 3. Grand Final, BBC One, Entertainment,Guidance,Highlights,Music,Pop & Chart,TV,Talent Shows,Variety Shows, default,
    
    INFO: 1 Matching Programmes
    INFO: Checking existence of default version
    INFO: flashhd1,flashhd2,flashvhigh1,flashvhigh2,flashhigh1,flashhigh2,flashstd1,flashstd2,flashlow1,flashlow2 modes will be tried for version default
    INFO: Trying flashhd1 mode to record tv: Eurovision Song Contest - 2014: 3. Grand Final
    INFO: File name prefix = Eurovision Song Contest 1-3 2014: Grand Final
    
    INFO: Downloading Subtitles to '/storage/download/iplayer/Eurovision Song Contest/Eurovision Song Contest 1-3 2014: Grand Final.srt'
    RTMPDump v2.4
    (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
    Connecting ...
    INFO: Connected...
    Starting download at: 0.000 kB
    INFO: Metadata:
    INFO:   duration              12918.61
    INFO:   moovPosition          36.00
    INFO:   width                 1280.00
    INFO:   height                720.00
    INFO:   videocodecid          avc1
    INFO:   audiocodecid          mp4a
    INFO:   avcprofile            100.00
    INFO:   avclevel              41.00
    INFO:   aacaot                2.00
    INFO:   videoframerate        25.00
    INFO:   audiosamplerate       48000.00
    INFO:   audiochannels         2.00
    INFO: trackinfo:
    INFO:   length                322965000.00
    INFO:   timescale             25000.00
    INFO:   language              und
    INFO: sampledescription:
    INFO:   sampletype            avc1
    INFO:   length                620093440.00
    INFO:   timescale             48000.00
    INFO:   language              und
    INFO: sampledescription:
    INFO:   sampletype            mp4a
    794901.252 kB / 2438.72 sec (18.8%)
    ERROR: RTMP_ReadPacket, failed to read RTMP packet header
    794928.881 kB / 2438.72 sec (18.8%)
    INFO: Connection timed out, trying to resume.
    
    
    Resuming download at: 794928.881 kB
    3789589.291 kB / 12918.59 sec (99.9%)
    Download complete
    avconv version 9.13-6:9.13-1, Copyright (c) 2000-2014 the Libav developers
      built on May  4 2014 21:37:19 with gcc 4.8 (Debian 4.8.2-21)
    [flv @ 0x20b58e0] max_analyze_duration reached
    Input #0, flv, from '/storage/download/iplayer/Eurovision Song Contest/Eurovision Song Contest 1-3 2014: Grand Final.partial.mp4.flv':
      Metadata:
        moovPosition    : 36
        avcprofile      : 100
        avclevel        : 41
        aacaot          : 2
        videoframerate  : 25
        audiochannels   : 2
      Duration: 03:35:18.61, start: 0.000000, bitrate: N/A
        Stream #0.0: Video: h264 (High), yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 25 fps, 25 tbr, 1k tbn, 50 tbc
        Stream #0.1: Audio: aac, 48000 Hz, stereo, fltp
    Output #0, mp4, to '/storage/download/iplayer/Eurovision Song Contest/Eurovision Song Contest 1-3 2014: Grand Final.partial.mp4':
      Metadata:
        moovPosition    : 36
        avcprofile      : 100
        avclevel        : 41
        aacaot          : 2
        videoframerate  : 25
        audiochannels   : 2
        encoder         : Lavf54.20.4
        Stream #0.0: Video: libx264, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], q=2-31, 1k tbn, 1k tbc
        Stream #0.1: Audio: aac, 48000 Hz, stereo
    Stream mapping:
      Stream #0:0 -> #0:0 (copy)
      Stream #0:1 -> #0:1 (copy)
    Press ctrl-c to stop encoding
    frame=322957 fps=34264 q=-1.0 Lsize= 3785136kB time=12918.56 bitrate=2400.3kbits/s
    video:3625834kB audio:147394kB global headers:0kB muxing overhead 0.315591%
    INFO: Recorded /storage/download/iplayer/Eurovision Song Contest/Eurovision Song Contest 1-3 2014: Grand Final.mp4
    
    INFO: Downloaded Thumbnail to '/storage/download/iplayer/Eurovision Song Contest/Eurovision Song Contest 1-3 2014: Grand Final.jpg'
    INFO: MP4 tagging MP4 file
    
     Started writing to temp file.
     Progress: ===========================================================================>100%|
     Finished writing to temp file.
    That's dumped a nice 720p MP4 file along with the subtitles (mandatory as Graham Norton gets increasingly squiffy on Baileys as the evening progresses), all I need now is a few beers. Scratch that, a lot of beers.

    Code:
    std@titus:~$ ll /storage/download/iplayer/Eurovision\ Song\ Contest/
    total 3.7G
    drwxr-xr-x 2 std std 4.0K May 11 12:49 .
    drwxrwx--- 4 std std 4.0K May 11 11:03 ..
    -rw-r--r-- 1 std std 3.7G May 11 12:49 Eurovision Song Contest 1-3 2014 Grand Final.mp4
    -rw-r--r-- 1 std std 126K May 11 11:54 Eurovision Song Contest 1-3 2014 Grand Final.srt
    Now no spoilers please, I won't be watching this until the evening.
     
  7. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    An update for you Mike - during the RAID rebuild when the discs were being thrashed, I noticed that the sides and top of the case were starting to get warm and it seemed like significant pockets of warm air were building up. Sticking my hand round the back I could barely feel any air being expelled.

    I've since changed my fan settings in UEFI as follows:
    Duty cycles were left unchanged from the default
    CPU fan remained at "smart fan" which maintains a fairly regular speed of ~1000rpm
    FAN1 and FAN2 headers were both set to "Level 5" which runs the BeQuiet fans at about 800-1000rpm with no apparent increase in volume over ambient but seems to expel a lot more air.

    Caused a significant drop in temperatures throughout the case and now the sides are as cool as a cucumber. I wasn't worried about the hard drives so much as the HBA and power supply which I think will be considerably more temperature sensitive.

    This is for v1.80, I've not tried upgrading to v2.00 yet.
     
  8. MikeBuzz

    MikeBuzz New Member

    Joined:
    Feb 20, 2014
    Messages:
    11
    Likes Received:
    0
    Thanks for the update, i still have not got round to starting the build, but i will bare the above info in mind :)
     
  9. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    Well hope you found some time this weekend, I've finally had enough free time to rebuild my backup server Andronicus with the S1200KPR and the eight-drive RAID6.

    Much of what I've said above for the configuration of Titus goes for this box as well (and indeed I just plugged the M4's originally from Titus into Andronicus to do a new install on) but I was annoyed that the old array was always detected as /dev/md127 - I want it as /dev/md64, damnit! So once the drives were plugged in, stop the RAID array...

    Code:
    root@andronicus:~# mdadm --stop /dev/md127
    mdadm: stopped /dev/md127
    ...and re-assemble it with the new name. Apologies for the disc letters being all over the place!

    Code:
    root@andronicus:~# mdadm --assemble /dev/md64 --name=backup_storage --update=name /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1
    mdadm: /dev/md64 has been started with 8 drives.
    root@andronicus:~# mdadm --detail --scan
    ARRAY /dev/md/0 metadata=1.2 name=titus:0 UUID=68ee4d67:0e14bf37:3061e653:fd7d53b0
    ARRAY /dev/md/1 metadata=1.2 name=titus:1 UUID=d00fe243:df8e7c05:b6ca7bb7:79fa3dec
    ARRAY /dev/md64 metadata=1.2 name=andronicus:backup_storage UUID=a06627cd:cb349053:3db74b5a:eaf078bf
    As you can see, the arrays brought over from the Titus install have retained their old names, I'll update these at some point as well. For now we'll just use that info to update our /etc/mdadm/mdadm.conf

    Code:
    root@andronicus:~# grep -v '#' /etc/mdadm/mdadm.conf
    CREATE owner=root group=disk mode=0660 auto=yes
    HOMEHOST 
    MAILADDR root
    ARRAY /dev/md/0  metadata=1.2 UUID=68ee4d67:0e14bf37:3061e653:fd7d53b0 name=titus:0
    ARRAY /dev/md/1  metadata=1.2 UUID=d00fe243:df8e7c05:b6ca7bb7:79fa3dec name=titus:1
    ARRAY /dev/md/64 metadata=1.2 UUID=a06627cd:cb349053:3db74b5a:eaf078bf name=andronicus:backup_storage
    I did wonder if I could just rename the name in mdadm.conf but I suspect not. Now we can grab the UUID of the filesystem (I'm not using LVM on this one) and add it to fstab.

    Code:
    root@andronicus:~# blkid /dev/md64
    /dev/md64: LABEL="andronicus_stor" UUID="f0395d6d-c62c-8fc5-94fe-1b9a5795b048" TYPE="ext4"
    I've also managed to create my change to the RAID6 stripe cache permanent (set to a size of 16384) by creating the following udev rule:

    Code:
    root@andronicus:~# cat /etc/udev/rules.d/65-md-stripe-cache.rules
    SUBSYSTEM=="block", KERNEL=="md*", ACTION=="change", TEST=="md/stripe_cache_size", ATTR{md/stripe_cache_size}="16384"
     
  10. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    One more thing of note that eagle-eyed readers might have seen - most of the drives in this array are WD greens; these were purchased before the reds became available. There's lots of cautionary tales about not using them in linux/NAS/always-on/RAID/anything configurations due to their aggressive head parking - this is a real issue (look for Load Cycle Count or LCC and you'll see what I mean) that caused these drives to fail prematurely. There's a DOS utility called wdidle3 that can be used to disable this overly aggressive heard parking (and hence negate the issue), but it requires a bootable DOS key.

    Not so under linux - some enterprising soul wrote a utility that does the same thing in a linux environment. In debian you can install the package idle3-tools to get a hold of it - no more futzing around with fiddling the drive separately. Here I'm double-checking the values in my new system:

    Code:
    root@titus:~# idle3ctl /dev/sd[a-f]
    Idle3 timer set to 252 (0xfc)
    Idle3 timer set to 252 (0xfc)
    Idle3 timer set to 252 (0xfc)
    Idle3 timer set to 252 (0xfc)
    Idle3 timer set to 252 (0xfc)
    Idle3 timer is disabled
    /dev/sdf is, of course, my sole WD red which doesn't suffer from the issue. You can see from the smartctl output below that my LCC count for the greens is quite low, since I noticed this issue years ago on my QNAP:

    Code:
    root@titus:~# smartctl -A /dev/sdc
    smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13-1-amd64] (local build)
    Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
    
    === START OF READ SMART DATA SECTION ===
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   156   156   021    Pre-fail  Always       -       9200
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       192
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   069   069   000    Old_age   Always       -       23003
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       19
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       14
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       181
    194 Temperature_Celsius     0x0022   109   109   000    Old_age   Always       -       43
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    root@titus:~# smartctl -A /dev/sdf
    smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13-1-amd64] (local build)
    Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
    
    === START OF READ SMART DATA SECTION ===
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       5
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   082   082   000    Old_age   Always       -       13736
     10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       5
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       2
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
    194 Temperature_Celsius     0x0022   113   112   000    Old_age   Always       -       37
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0
    The WD red at /dev/sdf is of course much newer (and runs cooler than the greens as well). More on SMART monitoring in a sec, but that LCC from the green would still have been in single digits if it wasn't for WD being money-grabulating cheapskates and nuking their RAID-friendly firmware.

    You can, of course, run SMART-grabbing info continually via running smartcrl as a daemon (called smartd) which most NAS units come equipped with out of the box. I was very wary about doing this on this tin however as I was bitten hard by a bug in the linux kernel used in Debian Squeeze and Wheezy; sending SMART data through these LSI HBAs would frequently cause them to lock up and/or drop the discs requiring a hard reset to fix. Eeek! Thankfully this seems to have been fixed since ages ago - you can trigger this bug reliably by running the following command for a few minutes:

    Code:
    while true; do smartctl -a /dev/sdX > /dev/null; done
    You can read an explanation of the bug here: https://lkml.org/lkml/2010/4/26/335

    I've run that for the past few hours and haven't had any problems so now I'm going to enable smartd itself via /etc/default/smartmontools. You can then customise with /etc/smartd.conf. Using the device scanning options, I don't need to specify the SAT option (the SAS -> SATA translation layer) so the default configuration of

    Code:
    DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
    is fine. This scans for all non-removable hard drives and sends an email to root@titus if it finds any problems. After it's been running for a while you'll see smartd messages turn up in syslog; my interval is set for every 30mins:

    Code:
    May 12 19:33:55 titus smartd[13628]: smartd has fork()ed into background mode. New PID=13628.
    May 12 19:33:55 titus smartd[13628]: file /var/run/smartd.pid written containing PID 13628
    May 12 20:03:55 titus smartd[13628]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 124 to 123
    May 12 21:03:55 titus smartd[13628]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 69 to 68
    May 12 22:03:56 titus smartd[13628]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 123 to 124
    May 12 22:33:55 titus smartd[13628]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 68 to 69
    May 12 23:03:55 titus smartd[13628]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 124 to 123
    May 12 23:33:55 titus smartd[13628]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 122 to 121
    May 12 23:33:55 titus smartd[13628]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 123 to 124
    Now hopefully those aren't actually values in °C otherwise I'm in big trouble ;) Let's see what ipmitool says now that I've tweaked my fan settings:

    Code:
    root@titus:~# !ipmitool
    ipmitool sensor
    ATX+5VSB         | 5.040      | Volts      | ok    | 4.050     | 4.260     | 4.500     | 5.490     | 5.760     | 6.030
    +3VSB            | 3.440      | Volts      | ok    | 2.660     | 2.800     | 2.960     | 3.620     | 3.800     | 3.980
    Vcore            | 1.790      | Volts      | ok    | 1.440     | 1.530     | 1.620     | 1.980     | 2.070     | 2.160
    VCCM             | 1.350      | Volts      | ok    | 1.160     | 1.230     | 1.280     | 1.650     | 1.730     | 1.810
    +1.05V           | 1.060      | Volts      | ok    | 0.850     | 0.900     | 0.940     | 1.150     | 1.210     | 1.270
    VCCIO_OUT        | 1.000      | Volts      | ok    | 0.850     | 0.900     | 0.940     | 1.150     | 1.210     | 1.270
    BAT              | 3.120      | Volts      | ok    | 2.500     | 2.640     | 2.780     | 3.400     | 3.540     | 3.680
    +3V              | 3.360      | Volts      | ok    | 2.660     | 2.800     | 2.960     | 3.620     | 3.800     | 3.980
    +5V              | 5.040      | Volts      | ok    | 4.050     | 4.260     | 4.500     | 5.490     | 5.760     | 6.030
    CPU_FAN1         | 1000.000   | RPM        | ok    | na        | na        | 300.000   | na        | na        | na
    REAR_FAN1        | 1000.000   | RPM        | ok    | na        | na        | 300.000   | na        | na        | na
    FRNT_FAN1        | 1000.000   | RPM        | ok    | na        | na        | 300.000   | na        | na        | na
    MB Temperature   | 31.000     | degrees C  | ok    | na        | na        | na        | 80.000    | na        | na
    CPU Temperature  | 31.000     | degrees C  | ok    | na        | na        | na        | 91.000    | na        | na
    +12V             | 12.200     | Volts      | ok    | 9.600     | 10.200    | 10.700    | 13.100    | 13.800    | 14.500
    root@titus:~# hddtemp /dev/sde
    /dev/sde: WDC WD30EZRS-00J99B0: 29°C
    root@titus:~# sensors
    coretemp-isa-0000
    Adapter: ISA adapter
    Physical id 0:  +31.0°C  (high = +80.0°C, crit = +100.0°C)
    Core 0:         +31.0°C  (high = +80.0°C, crit = +100.0°C)
    Core 1:         +30.0°C  (high = +80.0°C, crit = +100.0°C)
    Core 2:         +30.0°C  (high = +80.0°C, crit = +100.0°C)
    Core 3:         +30.0°C  (high = +80.0°C, crit = +100.0°C)
    
    nct6776-isa-0290
    Adapter: ISA adapter
    +3.30V:       +3.38 V  (min =  +2.98 V, max =  +3.63 V)
    +5.00V:       +5.04 V  (min =  +4.75 V, max =  +5.26 V)
    +12.00V:     +12.30 V  (min = +11.40 V, max = +12.62 V)
    M/B Temp:     +31.0°C  (high = +60.0°C, hyst = +50.0°C)  sensor = thermistor
    CPU Temp:     +33.0°C  (high = +80.0°C, hyst = +75.0°C)  sensor = thermistor
    Much better :)
     
  11. MikeBuzz

    MikeBuzz New Member

    Joined:
    Feb 20, 2014
    Messages:
    11
    Likes Received:
    0
    Hi stdPichu

    I setup the server this weekend, ran a memtest for a few hours and all good no errors reported. i have notice that the temps seem to be a bit higher than yours, the ipmi web gui is reporting the temp is at 54 while the system is just booted to the bios screen, have you made any change to your CPU settings in the BIOS?
     
  12. MikeBuzz

    MikeBuzz New Member

    Joined:
    Feb 20, 2014
    Messages:
    11
    Likes Received:
    0
    forget that, this morning just powered on the server from work, god i love IPMI, MB reporting temp at 34 and CPU 32
     
  13. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    I've not changed any CPU options other than, IIRC, putting it in "balanced power/performance" settin gin the BIOS or something like that, although that shouldn't make much difference to temperatures.

    Which OS are you using on it? If linux, can you check the pstates module is in use for power saving? Ah, that said, you say at the BIOS screen - I think it's possible that many of the power management features are disabled when the UEFI doohickey is running as I noticed the fans work harder during BIOS fiddling as well. Do the temps drop once you've booted an OS?

    Before I started taking pics of the build, I re-oriented my Noctua so that the fins ran front-to-back rather than top-to-bottom as I had them originally as this made a significant difference to system temperatures.

    Of course, the binning of the CPU and the mounting/TIM of the Noctua (it took me a couple of tries before I got the Noctua mounted perfectly flush across the whole CPU) will have a significant effect on temperatures as well, and are you also running the v1.80 UEFI/v13 BMC or have you upgraded to the v2.00 UEFI/v14 BMC? Things have been running hoopy so I've not gone to the bother of updating yet.
     
  14. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    It's nice, isn't it? I now wonder how I ever lived without it for my home servers before :)

    Glad your temps are more reasonable now anyhoo.
     
  15. MikeBuzz

    MikeBuzz New Member

    Joined:
    Feb 20, 2014
    Messages:
    11
    Likes Received:
    0
    I am using the latest BIOS and BMC, will try the trick with changing the fins round, did you find the fan a very tight fit in the case?
     
  16. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    It's a tight fit, yes - to make mounting the mobo easier I unscrewed the top mounting bar so I could bend it out of the way slightly which gave me a bit more leeway. I think in the end I had about 5mm space between the top of the fan and the edge of the disc cage.

    I would say that if your temps are fine changing the fins around would be an awful lot of bother for no gain, just at the time I was worried the chipset heatsink wouldn't get enough air over it. If the BMC says your CPU and system temps are fine once the chassis is all closed up, I wouldn't bother myself. As stated above, I only ended up doing it because I had some trouble getting the HSF mounted perfectly flat on top of the CPU.

    Might give the v2.00 BIOS a whirl this week but I wish to hell ASRock provided proper changelogs for the damn things...!
     
  17. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    'ello again folks, bit of a departure here but it's tangentially related- some of you may have noticed that, after my two NAS builds were done, I've still got a DH77DF with a low-power CPU left over... so what better use to put it to than as a new HTPC?

    Luckily enough, the DH77DF is compatible with the rather awesome Streacom FC8 EVO case. Now this is a bit of a special case - it's made of heavy-duty aluminium and features some honking great cooling fins on the side, and comes with a set of copper heatpipes to plumb from the CPU to the side of the case, making it completely fanless. I've decided to post the images here because, as well as being tengentially related to Titus and Andronicus, it's also a really nice case and also probably one of the trickiest ones I've ever done an install with.

    As ever, Kustom had all the gubbins needed to get this train wreck a-rollin':
    Streacom FC8 Evo passively cooled mITX case
    90W PicoPSU bundle
    Slot loading DVD drive

    First, here's the case as it comes out of the box, along with about twelvety billion screws.
    [​IMG]

    And here's the DH77DF along with the CPU block and heatpipe components. Thermal goop from the previous HSF hasn't been cleaned off yet.
    [​IMG]

    Now the instructions for this case say to install the mobo in the case and then fit the components on top of it... after a quick dry run I saw there was no way in hell this was going to work due to the way the components fitted together - or more accurately, required a little gentle persuasion before they fitted together. So instead here I'm about to start a dry run with assembling the CPU block and heatpipes and trying to jimmy it into the case then.
    [​IMG]

    As you can see from the above shot, the heatpipes are a little bit bent out of shape and this causes problems as the clamps that will bolt them to side heatsink-side of the case are quite a tight fit.
    [​IMG]

    A large amount of cursing later and I've managed to cram those clamps on top of the heatpipes - this required some very patient bending and some help from My Lovely Assistant to hold things in place. It's a good job I did the dry run else there would have been thermal goop smeared all over the place. Once that was done, I left the dry run in place for an hour just to make sure their positions would hold. Note the return of Mr. Martini in the background for moral support.
    [​IMG]

    So I then removed the board from the case again and dismantled the CPU block and put it together again, this time putting thermal goop on the ends of the heatpipe. Once the block was reattached, I put goop on t'other ends of the heatpipe and bolted the clamps back on again and ended up with this:
    [​IMG]

    Here's a close-up of the heatpipes along with the unavoidable smearing of some goop. Note that the fit of the heatpipes makes it push down right into the PicoPSU - I had to do some more elaborate bending of that bottom-most pipe to even get it to fit during the dry run phase.
    [​IMG]

    Everything fitted in and all wired up. Eagle-eyed readers might have spotted an extra doofer I've plugged in which is taped up above the USB port cluster, and if you didn't then I'm calling attention to it anyway.
    [​IMG]

    Here's a top-down shot of the same view. As you can see despite the confines of the case there's little in the way of clutter. You can also get a nice glimpse of the fins and the hefty thickness of the aluminium.
    [​IMG]

    And here's one just before the top goes on, complete with the slot-loading DVD drive fitted.
    [​IMG]

    So far it's been doing sterling service on the test bed as a proto-HTPC, more on that in a bit but first I know a bunch of people will want to know about temperatures so here's a quick peek from playing back a 1080p x264 in XBMC:

    Code:
    root@zoidberg:~# sensors
    acpitz-virtual-0
    Adapter: Virtual device
    temp1:        +27.8°C  (crit = +92.0°C)
    temp2:        +29.8°C  (crit = +92.0°C)
    
    coretemp-isa-0000
    Adapter: ISA adapter
    Physical id 0:  +41.0°C  (high = +87.0°C, crit = +91.0°C)
    Core 0:         +41.0°C  (high = +87.0°C, crit = +91.0°C)
    Core 1:         +39.0°C  (high = +87.0°C, crit = +91.0°C)
    
    nct6776-isa-0a00
    Adapter: ISA adapter
    Vcore:                  +0.84 V  (min =  +0.00 V, max =  +1.74 V)
    in1:                    +1.06 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
    AVCC:                   +3.34 V  (min =  +2.98 V, max =  +3.63 V)
    +3.3V:                  +3.34 V  (min =  +2.98 V, max =  +3.63 V)
    in4:                    +1.02 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
    in5:                    +0.99 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
    in6:                    +1.04 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
    3VSB:                   +3.15 V  (min =  +2.98 V, max =  +3.63 V)
    Vbat:                   +3.26 V  (min =  +2.70 V, max =  +3.63 V)
    fan1:                     0 RPM  (min =    0 RPM)
    fan2:                     0 RPM  (min =    0 RPM)
    fan3:                     0 RPM  (min =    0 RPM)
    SYSTIN:                 +37.0°C  (high =  +0.0°C, hyst =  +0.0°C)  ALARM  sensor = CPU diode
    CPUTIN:                 +39.0°C  (high = +80.0°C, hyst = +75.0°C)  sensor = CPU diode
    AUXTIN:                +123.0°C  (high = +80.0°C, hyst = +75.0°C)  sensor = CPU diode
    PECI Agent 0:           +39.5°C  (high = +80.0°C, hyst = +75.0°C)
                                     (crit = +91.0°C)
    PCH_CHIP_CPU_MAX_TEMP:  +63.0°C  (high = +80.0°C, hyst = +75.0°C)
    PECI Agent 1:            +0.0°C  (high = +80.0°C, hyst = +75.0°C)
                                     (crit =  +0.0°C)
    PCH_CHIP_TEMP:           +0.0°C
    intrusion0:            ALARM
    intrusion1:            ALARM
    beep_enable:           disabled
    Given that the ambient temperature is currently 32°C that's none too shabby at all.
     
  18. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    Oh dear, been very bad at getting this doco up in a timely fashion. Why didn't I use something like OpenElec...? Well, I like to geek and it's nice to roll my own system that allows me to do the stuff that customised distros might not let you do easily. Kinda the point of the whole NAS build as well.

    The Debian install was very straightforward - just a barebones netinstall onto the 32GB SSD with a /boot, a root and 512MB of swap. I then set about making a decent approximation at making a lean, mean media machine for using XBMC. Firstly I installed the LightDM login manager followed by the XFCE desktop environment. This gives me a fast, lean system that's nonetheless well equipped for light desktop work for when XBMC might not be up to it.

    Once LightDM is installed, enable auto-login for a local user by changing the following lines in /etc/lightdm/lightdm.conf which do what they say on the tin:
    Code:
    autologin-user=xbmc
    autologin-user-timeout=0
    Just replace "xbmc" with whatever your user is called. LightDM will then launch the default X session which should be XFCE. If it isn't, you can log out and configure the default session from the login manager. Alternatively try using the command line:

    Code:
    root@zoidberg:~# update-alternatives --config x-window-manager
    There is only one alternative in link group x-window-manager (providing /usr/bin/x-window-manager): /usr/bin/xfwm4
    Nothing to configure.
    root@zoidberg:~# update-alternatives --config x-session-manager
    There is only one alternative in link group x-session-manager (providing /usr/bin/x-session-manager): /usr/bin/xfce4-session
    Nothing to configure.
    Now after all this, I found that suspend to RAM worked - I could invoke pm_suspend or s2ram either as root or as my XBMC user and the system would suspend nicely. Similarly, this also worked perfectly for XFCE and lightDM... but when I tried to invoke suspend from within XBMC, the system would appear to go down (monitor would turn off, network connections would stop) but the system would never actually enter suspend - the power light would stay on solid and, when I temporarily hooked up a fan, it would keep running. So there was obviously something weird going on with the way XBMC invokes suspend. What it was I don't know, since its logs weren't enough for me to dejumble the problem.

    Thankfully I found there was an easy fix (although complicated to debug!) for this - disabling async suspend, a newish feature that's meant to speed up the suspend process. This was easy to do by adding the following to /etc/rc.local:

    Code:
    echo 0 > /sys/power/pm_async
    This has the rather colossal downside that all suspend and resume operations now happen in a synchronous fashion rather than asynchronous. Suspend and resume are now... uh... twice as slow (i.e. they take about half a second rather than a quarter of a second).

    Every boot should now take you straight into XBMC. I counted 5 seconds from GRUB starting to boot to XBMC fully loaded, giving it a 7s power-button-to-XBMC. Suspend works instantly; it takes longer for my monitor to wake from sleep.
     
  19. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    Now when I installed XBMC, I did the following.

    Code:
    root@zoidberg:~# aptitude install xbmc cec-utils libcec2
    That should clue you in as to what that random doohickey I wired into the build was - it's the PulseEight CEC adapter. For those of you unfamiliar with CEC, it's a protocol that runs alongside HDMI that allows the computer to control peripheral equipment - in my case, the AV receiver and the TV it's plugged into. It requires the motherboard to have a free USB header and what Intel call an HTPC header; the DH77DF was one of the first motherboards to get one although they're beginning to appear elsewhere now.

    Getting it working wasn't exactly plug'n'play though. I noticed that XBMC couldn't see it but I could see an Atmel chip through lsusb:

    Code:
    root@zoidberg:~# lsusb
    Bus 004 Device 003: ID 03eb:2ffa Atmel Corp.
    Bus 004 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 003 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 002: ID 046a:0023 Cherry GmbH CyMotion Master Linux Keyboard G230
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Apparently this is a common problem with the CEC adapter, and PulseEight only provide update packages for ubuntu and windows - lé sigh. Honestly, I'd expect these admittedly niche devices to be shipped running the correct firmware already. So let's pull down the newest Ubuntu .deb, extract it and see what we can see.

    Code:
    root@zoidberg:~# wget http://packages.pulse-eight.net/ubuntu/pool/stable/dists/raring/cec-firmware-upgrade_0.4_amd64.deb
    --2014-07-15 21:06:30--  http://packages.pulse-eight.net/ubuntu/pool/stable/dists/raring/cec-firmware-upgrade_0.4_amd64.deb
    Resolving packages.pulse-eight.net (packages.pulse-eight.net)... 82.165.139.231
    Connecting to packages.pulse-eight.net (packages.pulse-eight.net)|82.165.139.231|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 129782 (127K) [application/octet-stream]
    Saving to: ‘cec-firmware-upgrade_0.4_amd64.deb’
    
    100%[============================================================================================================>] 129,782      660KB/s   in 0.2s
    
    2014-07-15 21:06:30 (660 KB/s) - ‘cec-firmware-upgrade_0.4_amd64.deb’ saved [129782/129782]
    
    root@zoidberg:~# dpkg -x cec-firmware-upgrade_0.4_amd64.deb cec_firmware
    root@zoidberg:~# cd cec_firmware/usr/bin/
    root@zoidberg:~/cec_firmware/usr/bin# file cec-firmware-upgrade
    cec-firmware-upgrade: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=18b6e3478e6d6864f42bd9d2b2eeff7935dfe636, stripped
    Wahey, looks like we've got ourselves a binary. Now I know from trying to install this deb via dpkg -i that this has a dependency on a program called zenity, so I've installed that and now we try to run the firmware:

    Code:
    root@zoidberg:~/cec_firmware/usr/bin# ./cec-firmware-upgrade
    Pulse-Eight USB-CEC Adapter - firmware flash tool
    =================================================
    
    Preparing devices to be upgraded:
    
    Upgrading the firmware of adapter 1:
            * erasing previous flash contents: [done]
            * upgrading flash memory:          [writing] [validating] [done]
            * starting the new firmware        [done]
    
    
    The firmware of your Pulse-Eight USB-CEC adapter has been upgraded successfully to firmware version 4
    
    Press enter to close this application.
    Now we note that the USB ID has changed. Curious.

    Code:
    root@zoidberg:~# lsusb
    Bus 002 Device 003: ID 2548:1002
    Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 003 Device 003: ID 046d:c051 Logitech, Inc. G3 (MX518) Optical Mouse
    Bus 003 Device 002: ID 046a:0023 Cherry GmbH CyMotion Master Linux Keyboard G230
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Let's test the adapter is working now with cec-client:

    Code:
    root@zoidberg:~# cec-client
    No device type given. Using 'recording device'
    CEC Parser created - libCEC version 2.1.4
    no serial port given. trying autodetect:
     path:     /sys/devices/pci0000:00/0000:00:1d.0/usb4/4-1/4-1.5
     com port: /dev/ttyACM0
    
    opening a connection to the CEC adapter...
    DEBUG:   [              13]     unregistering all CEC clients
    DEBUG:   [              13]     Broadcast (F): osd name set to 'Broadcast'
    DEBUG:   [              14]     connection opened, clearing any previous input and waiting for active transmissions to end before starting
    DEBUG:   [              14]     communication thread started
    DEBUG:   [              25]     turning controlled mode on
    NOTICE:  [              45]     connection opened
    DEBUG:   [              46]      TV (0): POLL
    DEBUG:   [              46]     processor thread started
    TRAFFIC: [              46]      POLL not sent
    DEBUG:   [             118]     TV (0): device status changed into 'not present'
    NOTICE:  [             118]     registering new CEC client - v2.1.4
    DEBUG:   [             118]     detecting logical address for type 'recording device'
    DEBUG:   [             118]     trying logical address 'Recorder 1'
    DEBUG:   [             118]      Recorder 1 (1): POLL
    TRAFFIC: [             118]      POLL not sent
    DEBUG:   [             185]     using logical address 'Recorder 1'
    DEBUG:   [             185]     Recorder 1 (1): device status changed into 'handled by libCEC'
    DEBUG:   [             185]     Recorder 1 (1): power status changed from 'unknown' to 'on'
    DEBUG:   [             185]     Recorder 1 (1): vendor = Pulse Eight (001582)
    DEBUG:   [             185]     Recorder 1 (1): CEC version 1.4
    DEBUG:   [             185]     AllocateLogicalAddresses - device '0', type 'recording device', LA '1'
    DEBUG:   [             185]     setting ackmask to  2
    DEBUG:   [             190]     Recorder 1 (1): osd name set to 'CECTester'
    DEBUG:   [             190]     Recorder 1 (1): menu language set to 'eng'
    DEBUG:   [             190]     GetPhysicalAddress - trying to get the physical address via ADL
    DEBUG:   [             190]     GetPhysicalAddress - ADL returned physical address 0000
    DEBUG:   [             190]     GetPhysicalAddress - trying to get the physical address via nvidia driver
    DEBUG:   [             190]     GetPhysicalAddress - nvidia driver returned physical address 0000
    DEBUG:   [             190]     GetPhysicalAddress - trying to get the physical address from the OS
    DEBUG:   [             190]     GetPhysicalAddress - OS returned physical address 0000
    DEBUG:   [             190]     SetDevicePhysicalAddress - not setting invalid physical address 0000
    NOTICE:  [             191]     setting HDMI port to 1 on device TV (0)
    DEBUG:   [             191]      TV (0): POLL
    TRAFFIC: [             191]      POLL not sent
    DEBUG:   [             258]     Recorder 1 (1): physical address changed from ffff to 1000
    DEBUG:   [             258]      broadcast (F): physical adddress 1000
    TRAFFIC: [             258]
     
  20. stdPichu

    stdPichu Kustomer

    Joined:
    Nov 13, 2012
    Messages:
    34
    Likes Received:
    0
    One more additional observation - I was expecting to have to do a whole arseload of configuration tweakery to get the Intel driver working properly for tear-free video... and I was disappointed. Everything worked straight out of the box with various PAL, NTSC, SECAM and film sources.

    But... the hardware-accelerated playback looks... utterly terrible. Like, Divx3-from-2001-terrible, at least on some of my own encodes from DVD or BD. Quite possibly I'm using options that the Intel silicon/driver doesn't gel with, but I've had to revert to software decode for now to get a decent picture. Fine for all of my current films, which all play back on this CPU without a sweat, but a worry for any future 4k or H.265 decodes at any rate.

    Sadly no-one seems to make a low-profile, single slot, passively cooled nVidia card I can use for video decoding...