Love

From John's wiki
Jump to navigation Jump to search

My server 'love' was one of my computers in my lab. You can read about its disk configuration.

This was my primary file server and VM host. It stored archives, backups, and shared data (such as images, packages, downloads, media, etc.) and ran a bunch of network critical VM guests in VirtualBox. This server was replaced by longing and was then salvaged and re-purposed as companion.

Parts

Supplier Manufacturer Component Warranty Price Qty Type Total Notes Ordered Arrived
Scorptec HP Enterprise HP Enterprise ProLiant X3421 Gen10 Microserver 1 year $899.00 1 Server $899.00 2019-03-03 2019-03-11
Scorptec AMD AMD Opteron X3421 2.10 - 3.40GHz 1 CPU included in X3421 Gen10 Microserver 2019-03-03 2019-03-11
Memory.NET Samsung M391A2K43BB1-CRC - Samsung 1x16GB DDR4-2400 ECC UDIMM Lifetime $208.39 2 RAM 16GB $416.78 2019-03-19 yes
Scorptec Western Digital Western Digital WD Black 500GB PCIe NVMe M.2 (PCIe) SSD 5 years $229.00 1 NVMe 500GB $229.00 2019-03-05 2019-03-11
eBay Western Digital SSD M.2 250GB WD Blue 2280 SATA Solid State Drive $75.95 2 SSD 250GB $151.90 2019-03-21 yes
eBay Western Digital SSD M.2 500GB WD Blue 2280 SATA Solid State Drive $120.00 2 SSD 500GB $240.00 2019-03-05 2019-03-11
Scorptec Western Digital Western Digital WD Ultrastar DC HC310 4TB HDD 5 years $329.00 4 HDD 4TB $1,316.00 2019-03-03 2019-03-11
AusPCMarket AOC AOC 27" 2ms Full HD Narrow Bezel Monitor - HDMI/DVI/VGA,Tilt,VESA100 $245.03 1 Monitor $245.03 2017-08-13 yes
eBay PowerShield PowerShield Defender LCD 650VA UPS $115.00 1 UPS 650VA $115.00 2019-03-13 2019-03-19
Scorptec Silverstone Silverstone M.2 NVMe SSD NGFF to PCIeX4 Adapter Card 1 year $35 1 Adapter $35 2019-03-05 2019-03-11
eBay Delock Delock 89588 interface cards/adapter Internal SATA PCI Express Card 4x $167.00 1 Adapter $167.00 2019-03-05 2019-03-20
eBay Kmise Kmise Stereo Headphones Foldable Headset Super Bass Earphones for PC Phone $21.99 1 Headphones $21.99 2019-07-06 yes
WASD Keyboards WASD Keyboards CODE V3 104-Key Mechanical Keyboard - Cherry MX Green $251.51 1 Keyboard $251.51 Black case, no sound dampeners, US$175.00 2018-03-25 yes
WASD Keyboards WASD Keyboards 17" Soft Foam Wrist Pad $20.12 1 Wrist pad $20.12 US$14.00 2018-03-25 yes
eBay Professional G55 USB Gaming Microphone Mic Audio for Computer Desktop TH1133 $26.65 1 USB Microphone $26.65 2019-07-06 yes
Total: $

Status

Hard disks

According to this: 

# zpool status
  pool: data
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0B in 1 days 10:20:21 with 0 errors on Mon Nov 14 10:44:22 2022
config:

       NAME                     STATE     READ WRITE CKSUM
       data                     DEGRADED     0     0     0
         mirror-0               DEGRADED     0     0     0
           sda                  ONLINE       0     0     0
           9460704850353196665  OFFLINE      0     0     0  was /dev/sdb1
         mirror-1               DEGRADED     0     0     0
           2467357469475118468  OFFLINE      0     0     0  was /dev/sdc1
           sdd                  ONLINE       0     0     0
       cache
         nvme0n1p4              ONLINE       0     0     0

The good disks are sda and sdd. The potentially bad disks are sdb and sdc. Here is their info: 

lrwxrwxrwx 1 root root    9 Nov 29 13:26 scsi-SATA_HGST_HUS726T4TAL_V6GGTPKS -> ../../sda
lrwxrwxrwx 1 root root    9 Nov 29 13:26 scsi-SATA_HGST_HUS726T4TAL_V1JBN0GH -> ../../sdb
lrwxrwxrwx 1 root root    9 Nov 29 13:26 scsi-SATA_HGST_HUS726T4TAL_V6H42DES -> ../../sdc
lrwxrwxrwx 1 root root    9 Nov 29 13:26 scsi-SATA_HGST_HUS726T4TAL_V6GGRKVS -> ../../sdd

Configuration

I'm going to build the system with dedup and compression on, then do some testing. Then rebuild the system with dedup and compression off, and compare. My guess is that I want dedup and compression both off. That will save CPU time and RAM at the cost of some magnetic storage (which we have more than enough of).

ZFS config

See the ZFS documentation or the ZFS FAQ.

See love-disk.hml for info on disk partitions.

THINK: should /fast be ZFS or ext4 on MD-raid?

These partition commands adapted from here. They might need revision... 

# gpart create -s gpt sde
# gpart create -s gpt sdf
# gpart add -t freebsd-zfs -b 2048 -a 4k -l log0 -s 20G sde
# gpart add -t freebsd-zfs -b 2048 -a 4k -l log1 -s 20G sdf
# gpart add -t freebsd-zfs -a 4k -l fast0 sde
# gpart add -t freebsd-zfs -a 4k -l fast1 sdf

Note: with the above, if we can't put the /boot partition on the NVMe drive, the logs will become 18GB and /boot will be 2GB.

Note: /data:logs are mirrored, not striped, for 20GB usable. See Setting Up Separate ZFS Log Devices which says the log should be 1/2 the size of RAM (i.e. 16GB). 

# apt install zfsutils-linux
# zpool create fast mirror /dev/sde2 /dev/sdf2
# zpool add fast log /dev/nvme0n1p2
# zpool create data mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd
# zpool add data log /dev/sde1 /dev/sdf1
# zpool add data cache /dev/nvme0n1p3

See How to improve ZFS performance and ZFS Compression: lz4 VS lzjb. 

# zfs set compression=lz4 fast
# zfs set compression=lz4 data

According to How to improve ZFS performance you need to upgrade ZFS from time-to-time, like this: 

# zfs upgrade -a
# zpool upgrade -a

Also according to How to improve ZFS performance the ARC and L2ARC settings should be tuned. THINK: can we use the / SSD for L2ARC?

See Expanding a zpool and adding ZIL (log) and L2ARC (cache) for info on how to configure L2ARC and logs.

Also according to How to improve ZFS performance we should disable deduping for better performance and after reading ZFS: Stripe & Parity, or Stripe & Mirror? Also... SLOG & L2ARC? and Inappropriate use of deduplication I'm gonna leave dedup off: 

# zfs set dedup=off fast
# zfs set dedup=off data

According to ZFS FAULTED / kernel: (ada0:ahcich4:0:0:0): lost device in the BIOS configure AHCI instead of IDE.

Disable atime: 

# zfs set atime=off fast
# zfs set atime=off data

And storing xattrs as system attributes significantly decreases the amount of disk I/O: 

# zfs set xattr=sa fast
# zfs set xattr=sa data

From StackOverflow some handy commands:

  • ls: Shows the uncompressed size on files, but the compressed size on folders
  • zfs get used <pool>: Shows the compressed space of all files in the pool
  • zfs get logicalused <pool>: Shows the uncompressed space that all the files in the pool would use
  • zfs get compressratio <pool>: Shows the average compressratio of the pool
  • du -h --apparent-size: Shows the uncompressed size of the given files/folders
  • du -h: Shows the compressed size of the given files/folders

Note: I considered redundant_metadata=most, but I didn't entirely understand it, and it sounded safer to use the default setting 'all'.

Note: I need to do more reading on ARC config. I'm not certain but I'm thinking 4GB to 8GB RAM reserved for ARC. Actually, no, now I'm thinking 12GB for ARC leaving 20GB for other purposes.

Note: I need to think more about if I want to use sync=always. Might be good for safety if it doesn't kill perf too much.

Note: as we don't have a UPS on this host we're gonna use sync=always until we do. 

# zfs set sync=always fast
# zfs set sync=always data

See ZFS Configuration Part 2: ZVols, LZ4, ARC, and ZILs Explained for info about configuring ARC via GRUB. I'm thinking I will configure ARC with 12GB and keep 4GB for dedup leaving 16GB left for the system (and its VM guests).

Upgrades

KVM

2020-07-10 jj5: I got one of these CS64U 4-Port USB VGA/Audio Cable KVM Switch (1.8m) from eBay so I can upgrade to three systems on the KVM rather than just two.

32GB RAM

Get 2x16GB ECC Buffered RAM. CPU supports up to 2400Mhz.

RAM upgrade options:

Fast disk

When it comes time to upgrade disk capacity use M.2 drives and adapter, e.g.:

Best option for $647:

The above SATA options are probably best, here are some NVMe considerations:

Or consider:

Optical drive

Do we care for an optical drive?

Shopping

WASD Keyboards

Component Notes Unit Price Quantity Price Arrived
CODE V3 104-Key Mechanical Keyboard - Cherry MX Green Black case, no sound dampeners US$175.00 1 US$175.00
17" Soft Foam Wrist Pad US$14.00 1 US$14.00
Freight US$69.50
Total US$258.50

Memory.NET

Component Warranty Unit Price Quantity Price Arrived
M391A2K43BB1-CRC - Samsung 1x 16GB DDR4-2400 ECC UDIMM PC4-19200T-E Dual Rank x8 Module Lifetime $208.39 2 $416.78
Freight $69.41
Total $486.19

eBay

Component Note Price Quantity Total Arrived
SSD M.2 250GB WD Blue 2280 SATA Solid State Drive $75.95 2 $151.90
M.2 Nvme SSD NGFF TO PCI-E X4 3.0 PCB Adapter M Key B Key Dual Interface Ri C1G1 low form factor $27.46 1 $27.46 2019-03-18
WD 4TB Red PRO 3.5” 7200RPM SATA3 NAS Hard Drive airgap $268.51 2 $537.02 2019-03-18
Samsung 16GB 1Rx4 DDR4-2400MHZ PC4-2400T ECC Registered 288Pin Server Memory Ram $199.00 2 $398.00 2019-03-18
Delock 89588 interface cards/adapter Internal SATA PCI Express Card 4x $167.00 1 $167.00 2019-03-20
SSD M.2 500GB WD Blue 2280 SATA Solid State Drive $120.00 2 $240.00 2019-03-11
2x Compressed Air Duster Can Cleaner 400ml for Notebook Laptop PC Keyboard AU $18.65 1 $18.65 2019-03-18
Netgear GS108 ProSafe 8 Port Gigabit Home Desktop Ethernet Unmanaged Switch $53.00 1 $53.00 2019-03-11
Network Cable CAT6 1000Mbps 5m Green, 5m Grey $9.95 2 $19.90 2019-03-18
Network Cable CAT6 1000Mbps 3m Black, 3m Blue $6.95 2 $13.90 2019-03-18
2 Port HDMI KVM Switch with Cables EL-21UHC E6I2 $33.37 1 $33.37
KVM Switch 2 Port ATEN CS22D USB DVI with cable remote $64.90 1 $64.90
Power Shield Defender LCD 650VA UPS $115.00 1 $115.00 2019-03-19
Total: $1,840.1

Scorptec Computers

Gonna put Ubuntu LTS on the SSD and put the four mag disks in RAID10 (software raid) for 8TB usable for ZFS /data.

Component Warranty Unit Price Quantity Price Arrived
HP Enterprise ProLiant X3421 Gen10 Microserver 1 year $899.00 1 $899.00 2019-03-11
Western Digital WD Ultrastar DC HC310 4TB HDD 5 years $329.00 4 $1,316.00 2019-03-11
Western Digital WD Black 500GB PCIe NVMe M.2 (PCIe) SSD 5 years $229.00 1 $229.00 2019-03-11
Aten CS22U 2 Port KVM 2 years $32.00 1 $32.00 2019-03-11
Kingston DTSE9G2 16GB USB 3.0 DataTraveler 5 years $12.00 3 $36.00 2019-03-11
Sub-total $2,512.00
Freight $68.00
Surcharge $25.80
Total $2,605.80
Component Warranty Unit Price Quantity Price Arrived
Silverstone M.2 NVMe SSD NGFF to PCIeX4 Adapter Card 1 year $35 1 $35 2019-03-11
Tax and shipping $12.47
Total $47.47

Stone Computers

Gonna put Ubuntu LTS on the EVO and put the four WD Golds in RAID10 (software raid) for 8TB usable for ZFS /data.

Product Code Product Name Unit Price Quantity Price
PC-HP-873830-375 HPE ProLiant MicroServer Gen10 (873830-375) Server $729 1 $729
HD-WD4000FYYZ-WD4002FYYZ 4TB WD4002FYYZ WD Gold SATA3-6gb/s HDD (128MB, 7200rpm) $311 4 $1,244
SSD-SAMSUNG-MZ-V7E250BW 250GB Samsung 970 EVO M.2 SSD (MZ-V7E250BW) $119 1 $119
NET-ATEN-CS-22U Aten Petite CS-22U 2 Port USB KVM Switch $47 1 $47
Total Price: $2,139

Need to replace SATA drive with NVME drive:

Old SSD-WD-WDS120G2G0B 120GB WD GREEN 3D NAND SSD, M.2 SATA $53 1 $53
New SSD-SAMSUNG-MZ-V7E250BW 250GB Samsung 970 EVO M.2 SSD (MZ-V7E250BW) $119 1 $119

Amazon.com.au

Gonna put Ubuntu LTS on the WD Black and put the four WD Golds in RAID10 (software raid) for 8TB usable.

Component Price Units Total
HP ProLiant MicroServer Gen10 AMD Opteron X3216, 8GB ECC UDIMM $837 x1 $837
WD 4TB Gold Enterprise-Class Hard Drive - WD4002FYYZ $316 x4 $1,267
WD Black SN750 500GB NVMe Internal Gaming SSD - Gen3 PCIe, M.2 2280, 3D NAND - WDS500G3X0C $188 x1 $188
QNINE M.2 NVME to PCIe Adapter $15 x1 $15
Aten 2-Port USB VGA Cable KVM Switch $45 x1 $45
$2,355

staticICE

Component Vendor (price) Vendor (price) Vendor (price) Vendor (price)
HPE ProLiant MicroServer Gen10 TechBuy ($714) Stone Computers ($729) i.store ($729)
WD 4TB Gold ARC ($291) Stone Computers ($311) i.store ($311) RAM City ($319)
WD Black 500GB NVMe PC Case Gear ($179) CPLOnline ($199) BudgetPC ($199) PLE Computers ($199)
M.2 NVME to PCIe Adapter Scorptec ($35)
2-Port USB VGA Cable KVM Switch BudgetPC ($32) JW Computers ($33) i-Tech ($34) Dick Smith ($34)

Research

Prices

Component Unit Price Qty Total Price
Server $566 1 $566
SSD disk $130 1 $130
Mag disk $232 4 $928
KVM $23 1 $23
UPS $94 1 $94
$1,741

Server

CPU

SSD disk

Mag disk

KVM

UPS

RAM

Mag disk

12 TB disk

Newegg 2-item combo: Total: $133.50/ea $267

4 TB disk

Note: we almost certainly don't want 5400RPM drives.

Retrieved from "https://www.jj5.net/wiki/mediawiki/index.php?title=Love&oldid=6173"