お寿司を食べた猫

昨日もおすし食べてなかったっけ

「やはり再起動・・・！再起動がすべてを解決する・・・！！！」

拝啓山田様

　お久しぶりです。1度お会いしたきりなので先生は覚えていないでしょうが、私は覚えております。 　山田先生は世の中のソフトウェアリカバリーの手段を調査し数十年（？）される中で、今でもReboot-Based Recoveyの概念を学術的に提唱しつづけておられます。 過去のご登壇されていた発表の内容は、それはもう素晴らしいものでした。

ですが、今回は山田先生に謝らければなりません。

　↓　再起動後　↓

ストレージプールが無くなったとかいうエラーが出ていて

再起動したら復活するのはいかんでしょ？！？！？！？！？！？！？（大声）

　ストレージプールが無くなるだけでなく、なぜか元に戻る怪現象。 今年は肝試しにチャレンジするつもりじゃなかったんだけど、どうしてこんなことになるかな～。

ごめんなさい、山田先生。私は再起動で治るような計算機を信用できないんです。ここは再起動でヨシ！としちゃイカンでしょ！！！

どうしてこうなってしまったのか、じっくり調査していきたいと思います。

ソフトを直すのか、買い替えなのか対策を打てれば御の字です。

現在の構成

今回エラーが出たNASについてですが、SynologyのOSが入ったNASにストレージ用にHDDが2台、それからHDDは交換前提と考慮してOS用にSSDを1台搭載しています。

bash-4.3# fdisk -l
Disk /dev/sda: 238.5 GiB, 256060514304 bytes, 500118192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x887d857e

Device     Boot   Start       End   Sectors   Size Id Type
/dev/sda1          2048   4982527   4980480   2.4G fd Linux raid autodetect
/dev/sda2       4982528   9176831   4194304     2G fd Linux raid autodetect
/dev/sda3       9437184 499913375 490476192 233.9G fd Linux raid autodetect


Disk /dev/sdb: 10.9 TiB, 12000138625024 bytes, 23437770752 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 3488A54F-BCF5-42DB-B53B-F131BF556C05

Device           Start         End    Sectors  Size Type
/dev/sdb1         2048     4982527    4980480  2.4G Linux RAID
/dev/sdb2      4982528     9176831    4194304    2G Linux RAID
/dev/sdb5      9453280  7813830239 7804376960  3.6T Linux RAID
/dev/sdb6   7813846336 11720838239 3906991904  1.8T Linux RAID
/dev/sdb7  11720854336 15627846239 3906991904  1.8T Linux RAID
/dev/sdb8  15627862336 23437556799 7809694464  3.7T Linux RAID


Disk /dev/sdc: 12.8 TiB, 14000519643136 bytes, 27344764928 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 3689E169-ED17-4AB0-82A7-E447EC9DD52B

Device           Start         End    Sectors  Size Type
/dev/sdc1         2048     4982527    4980480  2.4G Linux RAID
/dev/sdc2      4982528     9176831    4194304    2G Linux RAID
/dev/sdc5      9453280  7813830239 7804376960  3.6T Linux RAID
/dev/sdc6   7813846336 11720838239 3906991904  1.8T Linux RAID
/dev/sdc7  11720854336 15627846239 3906991904  1.8T Linux RAID
/dev/sdc8  15627862336 23437556799 7809694464  3.7T Linux RAID

...（ややこしいので中略、後で続きを張ります）

このように3台の物理デバイスが接続されています。

パーティションがいくつかに分かれているのはＯＳ領域と、ストレージとしてのパーティションが分かれているためです。

なんなら、ストレージとしてのパーティションが4つぐらいに分割されていますが、 これはストレージプールとして1つとしてＮＡＳからは扱われているので図にまとめると下記のな１つのLinuxボリュームがあると考えてください。

図にまとめるとこんな感じです。

NASの機能を使って、随時HDDを足していっていたため、細切れになっているパーティションがいくらかある状態ですが、そこはご容赦いただければと思います。

すし on X: "これは、僕が片方だけのHDDを 容量アップするように取り替えていっている図 https://t.co/3NtQCrJvSm" / X

実際使ってる空き容量とかそのあたりもこんな感じです。

bash-4.3# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.3G  999M  1.2G  46% /
none            1.9G     0  1.9G   0% /dev
/tmp            1.9G  836K  1.9G   1% /tmp
/run            1.9G  3.3M  1.9G   1% /run
/dev/shm        1.9G  4.0K  1.9G   1% /dev/shm
none            4.0K     0  4.0K   0% /sys/fs/cgroup
cgmfs           100K     0  100K   0% /run/cgmanager/fs
/dev/vg1000/lv   11T  7.5T  3.0T  72% /volume1
shm              64M     0   64M   0% /volume1/@docker/containers/bde3d2b79705851efb7bfbec8f5a14685b4b9d0ee97d522f50e9256ee4dd374c/mounts/shm
shm              64M     0   64M   0% /volume1/@docker/containers/4ed7c3bcedfa3a395b37f46037c66372233384b1956aaf8797fea582e426496a/mounts/shm
bash-4.3#
bash-4.3# lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg1000/lv
  LV Name                lv
  VG Name                vg1000
  LV UUID                mBcNpR-kA8A-45jy-huLm-IEI7-KK2x-1GONx3
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                10.91 TiB
  Current LE             2859867
  Segments               4
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:0

bash-4.3#

さっき中略したfdiskについてもこんな感じです。

bash-4.3# fdisk -l

...（さっきの続きから）...

Disk /dev/md0: 2.4 GiB, 2549940224 bytes, 4980352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md1: 2 GiB, 2147418112 bytes, 4194176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


GPT PMBR size mismatch (102399 != 30965759) will be corrected by w(rite).
Disk /dev/synoboot: 14.8 GiB, 15854469120 bytes, 30965760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C94E55EA-A4D2-4E78-9D73-46CBAE7A03EF

Device         Start    End Sectors Size Type
/dev/synoboot1  2048  32767   30720  15M EFI System
/dev/synoboot2 32768  94207   61440  30M Linux filesystem
/dev/synoboot3 94208 102366    8159   4M BIOS boot


Disk /dev/zram0: 1.1 GiB, 1210056704 bytes, 295424 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/zram1: 1.1 GiB, 1210056704 bytes, 295424 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md4: 1.8 TiB, 2000378789888 bytes, 3906989824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md5: 3.7 TiB, 3998562516992 bytes, 7809692416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md2: 3.6 TiB, 3995839954944 bytes, 7804374912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md3: 1.8 TiB, 2000378789888 bytes, 3906989824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/vg1000-lv: 10.9 TiB, 11995151597568 bytes, 23428030464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
bash-4.3#

何が起きたのか

気が付いたのは9/7の23時のことでした、たまたまnasにログインするとまぁそれは過激な文章が画面上に出ていて、再起動したら直っていたというのが状況です。

その場ではHDDのエラーログが無かったことぐらいしか確認できませんでしたが、あらためてカーネルログなど漁っていきます。

HDDのエラーログを真っ先に見たのは、HDDの故障以外にも

こいつの何かしらの影響を考えてでしたが、今回はどうも違うようです。

2023-09-02T20:29:34+09:00 FStation kernel: [  216.521506] md: md1: resync done.
2023-09-02T20:29:34+09:00 FStation kernel: [  216.529229] md: md1: current auto_remap = 0
2023-09-07T23:17:23+09:00 FStation kernel: [    3.674156] scsi 0:0:0:0: Direct-Access     SanDisk  SD8SNAT256G1122          Z233 PQ: 0 ANSI: 5
2023-09-07T23:17:23+09:00 FStation kernel: [    3.674655] sd 0:0:0:0: [sda] 500118192 512-byte logical blocks: (256 GB/238 GiB)
2023-09-07T23:17:23+09:00 FStation kernel: [    3.674678] sd 0:0:0:0: [sda] Write Protect is off
2023-09-07T23:17:23+09:00 FStation kernel: [    3.674687] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
2023-09-07T23:17:23+09:00 FStation kernel: [    3.680802] sd 0:0:0:0: [sda] Attached SCSI disk
2023-09-07T23:17:23+09:00 FStation kernel: [    3.681815] ata2.00: supports DRM functions and may not be fully accessible
2023-09-07T23:17:23+09:00 FStation kernel: [    3.685778] scsi 1:0:0:0: Direct-Access     WDC      WD120EDAZ-11F3RA0        81.0 PQ: 0 ANSI: 5
2023-09-07T23:17:23+09:00 FStation kernel: [    3.688470] sd 1:0:0:0: [sdb] 23437770752 512-byte logical blocks: (12.0 TB/10.9 TiB)
2023-09-07T23:17:23+09:00 FStation kernel: [    3.688472] sd 1:0:0:0: [sdb] 4096-byte physical blocks
2023-09-07T23:17:23+09:00 FStation kernel: [    3.688488] sd 1:0:0:0: [sdb] Write Protect is off
2023-09-07T23:17:23+09:00 FStation kernel: [    3.688496] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
2023-09-07T23:17:23+09:00 FStation kernel: [    3.693380] scsi 2:0:0:0: Direct-Access     WDC      WD140EDGZ-11B1PA0        85.0 PQ: 0 ANSI: 5
2023-09-07T23:17:23+09:00 FStation kernel: [    3.697450] sd 2:0:0:0: [sdc] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB)
2023-09-07T23:17:23+09:00 FStation kernel: [    3.697452] sd 2:0:0:0: [sdc] 4096-byte physical blocks
2023-09-07T23:17:23+09:00 FStation kernel: [    3.697468] sd 2:0:0:0: [sdc] Write Protect is off
2023-09-07T23:17:23+09:00 FStation kernel: [    3.697477] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
2023-09-07T23:17:23+09:00 FStation kernel: [    3.760948] sd 2:0:0:0: [sdc] Attached SCSI disk
2023-09-07T23:17:23+09:00 FStation kernel: [    3.770141] sd 1:0:0:0: [sdb] Attached SCSI disk
2023-09-07T23:17:23+09:00 FStation kernel: [    4.169970] md: invalid raid superblock magic on sda3
2023-09-07T23:17:23+09:00 FStation kernel: [    4.175298] md: sda3 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.217748] md: invalid raid superblock magic on sdb5
2023-09-07T23:17:23+09:00 FStation kernel: [    4.222977] md: sdb5 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.232424] md: invalid raid superblock magic on sdb6
2023-09-07T23:17:23+09:00 FStation kernel: [    4.237594] md: sdb6 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.246860] md: invalid raid superblock magic on sdb7
2023-09-07T23:17:23+09:00 FStation kernel: [    4.252093] md: sdb7 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.268573] md: invalid raid superblock magic on sdb8
2023-09-07T23:17:23+09:00 FStation kernel: [    4.273782] md: sdb8 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.317760] md: invalid raid superblock magic on sdc5
2023-09-07T23:17:23+09:00 FStation kernel: [    4.322958] md: sdc5 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.335250] md: invalid raid superblock magic on sdc6
2023-09-07T23:17:23+09:00 FStation kernel: [    4.340465] md: sdc6 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.348447] md: invalid raid superblock magic on sdc7
2023-09-07T23:17:23+09:00 FStation kernel: [    4.353731] md: sdc7 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.372931] md: invalid raid superblock magic on sdc8
2023-09-07T23:17:23+09:00 FStation kernel: [    4.378155] md: sdc8 does not have a valid v0.90 superblock, not importing!
2023-09-07T23:17:23+09:00 FStation kernel: [    4.378162] md: sda2 has different UUID to sda1
2023-09-07T23:17:23+09:00 FStation kernel: [    4.378165] md: sdb2 has different UUID to sda1
2023-09-07T23:17:23+09:00 FStation kernel: [    4.378168] md: sdc2 has different UUID to sda1
2023-09-07T23:17:23+09:00 FStation kernel: [    8.316157] scsi 8:0:0:0: Direct-Access              Patriot Memory           PMAP PQ: 0 ANSI: 6
2023-09-07T23:17:23+09:00 FStation kernel: [    8.843270] sd 8:0:0:0: [synoboot] 30965760 512-byte logical blocks: (15.8 GB/14.7 GiB)
2023-09-07T23:17:23+09:00 FStation kernel: [    8.844348] sd 8:0:0:0: [synoboot] Write Protect is off
2023-09-07T23:17:23+09:00 FStation kernel: [    8.845404] sd 8:0:0:0: [synoboot] No Caching mode page found
2023-09-07T23:17:23+09:00 FStation kernel: [    8.851313] sd 8:0:0:0: [synoboot] Assuming drive cache: write through
2023-09-07T23:17:23+09:00 FStation kernel: [    8.861100] sd 8:0:0:0: [synoboot] No Caching mode page found
2023-09-07T23:17:23+09:00 FStation kernel: [    8.866943] sd 8:0:0:0: [synoboot] Assuming drive cache: write through
2023-09-07T23:17:23+09:00 FStation kernel: [    8.884445] sd 8:0:0:0: [synoboot] No Caching mode page found
2023-09-07T23:17:23+09:00 FStation kernel: [    8.890341] sd 8:0:0:0: [synoboot] Assuming drive cache: write through
2023-09-07T23:17:23+09:00 FStation kernel: [    8.897012] sd 8:0:0:0: [synoboot] Attached SCSI removable disk
2023-09-07T23:17:23+09:00 FStation kernel: [   11.950709] sd 0:0:0:0: Attached scsi generic sg0 type 0
2023-09-07T23:17:23+09:00 FStation kernel: [   11.950741] sd 1:0:0:0: Attached scsi generic sg1 type 0
2023-09-07T23:17:23+09:00 FStation kernel: [   11.950772] sd 2:0:0:0: Attached scsi generic sg2 type 0
2023-09-07T23:17:23+09:00 FStation kernel: [   11.950801] sd 8:0:0:0: Attached scsi generic sg3 type 0
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1345 ##### ACTION:add
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVNAME:sda
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVGUID:162075421401        0
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVPATH:sda
2023-09-07T23:15:57+09:00 FStation hotplugd: SUBSYSTEM:block
2023-09-07T23:15:57+09:00 FStation hotplugd: PHYSDEVPATH:/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0
2023-09-07T23:15:57+09:00 FStation hotplugd: scemd_connector/scemd_connector.c:134 Fail to sendto() for scemd connector client.
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1444 ==== SATA disk [sda] hotswap [add] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: disk/disk_config_single.c:234 apply /usr/syno/bin/DiskApmSet.sh 255  /dev/sda 1>/dev/null 2>&1
2023-09-07T23:15:57+09:00 FStation hotplugd: disk/disk_config_single.c:234 apply /usr/syno/bin/syno_disk_ctl --ncq-on  /dev/sda 1>/dev/null 2>&1
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1465 ==== SATA disk [sda] Model: [SD8SNAT256G1122         ] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1466 ==== SATA disk [sda] Serial number: [162075421401] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1467 ==== SATA disk [sda] Firmware version: [Z2333000] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1345 ##### ACTION:add
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVNAME:sdb
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVGUID:8CJS01UF            0
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVPATH:sdb
2023-09-07T23:15:57+09:00 FStation hotplugd: SUBSYSTEM:block
2023-09-07T23:15:57+09:00 FStation hotplugd: PHYSDEVPATH:/devices/pci0000:00/0000:00:17.0/ata2/host1/target1:0:0/1:0:0:0
2023-09-07T23:15:57+09:00 FStation hotplugd: scemd_connector/scemd_connector.c:134 Fail to sendto() for scemd connector client.
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1444 ==== SATA disk [sdb] hotswap [add] ====
2023-09-07T23:15:57+09:00 FStation kernel: [   35.164842] ata2.00: supports DRM functions and may not be fully accessible
2023-09-07T23:15:57+09:00 FStation kernel: [   35.172228] ata2.00: supports DRM functions and may not be fully accessible
2023-09-07T23:15:57+09:00 FStation hotplugd: disk/disk_config_single.c:234 apply /usr/syno/bin/DiskApmSet.sh 255  /dev/sdb 1>/dev/null 2>&1
2023-09-07T23:15:57+09:00 FStation hotplugd: disk/disk_config_single.c:234 apply /usr/syno/bin/syno_disk_ctl --ncq-on  /dev/sdb 1>/dev/null 2>&1
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1465 ==== SATA disk [sdb] Model: [WD120EDAZ-11F3RA0       ] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1466 ==== SATA disk [sdb] Serial number: [8CJS01UF] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1467 ==== SATA disk [sdb] Firmware version: [81.00A81] ====
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1345 ##### ACTION:add
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVNAME:sdc
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVGUID:Y6GXALGC            0
2023-09-07T23:15:57+09:00 FStation hotplugd: DEVPATH:sdc
2023-09-07T23:15:57+09:00 FStation hotplugd: SUBSYSTEM:block
2023-09-07T23:15:57+09:00 FStation hotplugd: PHYSDEVPATH:/devices/pci0000:00/0000:00:17.0/ata3/host2/target2:0:0/2:0:0:0
2023-09-07T23:15:57+09:00 FStation hotplugd: scemd_connector/scemd_connector.c:134 Fail to sendto() for scemd connector client.
2023-09-07T23:15:57+09:00 FStation hotplugd: hotplugd.c:1444 ==== SATA disk [sdc] hotswap [add] ====
2023-09-07T23:15:57+09:00 FStation kernel: [   35.484400] ata3.00: supports DRM functions and may not be fully accessible
2023-09-07T23:15:57+09:00 FStation kernel: [   35.490237] ata3.00: supports DRM functions and may not be fully accessible
2023-09-07T23:15:58+09:00 FStation hotplugd: disk/disk_config_single.c:234 apply /usr/syno/bin/DiskApmSet.sh 255  /dev/sdc 1>/dev/null 2>&1
2023-09-07T23:15:58+09:00 FStation hotplugd: disk/disk_config_single.c:234 apply /usr/syno/bin/syno_disk_ctl --ncq-on  /dev/sdc 1>/dev/null 2>&1
2023-09-07T23:15:58+09:00 FStation hotplugd: hotplugd.c:1465 ==== SATA disk [sdc] Model: [WD140EDGZ-11B1PA0       ] ====
2023-09-07T23:15:58+09:00 FStation hotplugd: hotplugd.c:1466 ==== SATA disk [sdc] Serial number: [Y6GXALGC] ====
2023-09-07T23:15:58+09:00 FStation hotplugd: hotplugd.c:1467 ==== SATA disk [sdc] Firmware version: [85.00A85] ====
2023-09-07T23:16:47+09:00 FStation kernel: [   85.258396] ata2.00: supports DRM functions and may not be fully accessible
2023-09-07T23:16:47+09:00 FStation kernel: [   85.264651] ata2.00: supports DRM functions and may not be fully accessible
2023-09-07T23:16:47+09:00 FStation kernel: [   85.338789] ata3.00: supports DRM functions and may not be fully accessible
2023-09-07T23:16:47+09:00 FStation kernel: [   85.346006] ata3.00: supports DRM functions and may not be fully accessible

23時に再起動したときには「md: invalid raid superblock magic on sda3」とか出ていますが、これは昔からでていたログで特に問題はありません。（少なくとも今回の事象では）

ほかのログも見てみると、どうも9/5の 5:14ぐらいに書き込みができなくなっているログが出始めて9/7の23時に再起動するまでずっとエラーが出続けているログが見つかります。

これは普段では起きていないログなのでおそらく、この5:14に何かしらのトラブルが起きたものと思っています。

/var/log/messages

2023-09-04T20:28:38+09:00 FStation [173147.854629] init: tty main process ended, respawning
2023-09-04T20:28:47+09:00 FStation kernel: [173157.178434] FAT-fs (synoboot2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
2023-09-04T20:28:47+09:00 FStation synoddsmd: utils.cpp:35 Fail to get synology account
2023-09-04T20:28:47+09:00 FStation synoddsmd: user.cpp:129 get account info fail [100]
2023-09-04T20:28:47+09:00 FStation synoddsmd: synoddsm-hostd.cpp:227 Fail to get DDSM licenses, errCode: 0x100
2023-09-04T20:28:48+09:00 FStation [173157.868358] init: tty main process (25102) terminated with status 1
2023-09-04T20:28:48+09:00 FStation [173157.868380] init: tty main process ended, respawning

.....

2023-09-05T05:14:57+09:00 FStation [204761.681357] init: tty main process (5939) terminated with status 1
2023-09-05T05:14:57+09:00 FStation [204761.681380] init: tty main process ended, respawning
2023-09-05T05:15:05+09:00 FStation kernel: [204769.826143] uart write failed
2023-09-05T05:15:06+09:00 FStation kernel: [204548.072049] [@72]
2023-09-05T05:15:07+09:00 FStation kernel: [204771.038304] uart write failed
2023-09-05T05:15:07+09:00 FStation kernel: [204549.282880] [@72]
2023-09-05T05:15:07+09:00 FStation [204771.695214] init: tty main process (5950) terminated with status 1
2023-09-05T05:15:07+09:00 FStation [204771.695238] init: tty main process ended, respawning
2023-09-05T05:15:08+09:00 FStation kernel: [204772.433770] uart write failed
2023-09-05T05:15:08+09:00 FStation kernel: [204551.676803] [@72]
2023-09-05T05:15:09+09:00 FStation kernel: [204773.646662] uart write failed
2023-09-05T05:15:09+09:00 FStation kernel: [204552.888402] [@72]
2023-09-05T05:15:11+09:00 FStation kernel: [204775.041561] uart write failed
2023-09-05T05:15:11+09:00 FStation kernel: [204553.281778] [@72]
2023-09-05T05:15:12+09:00 FStation kernel: [204554.283394] [@1000]
2023-09-05T05:15:12+09:00 FStation kernel: [204776.254951] uart write failed
2023-09-05T05:15:12+09:00 FStation kernel: [204554.493864] [@72]
2023-09-05T05:15:13+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:13+09:00 FStation kernel: [204555.207815] [@103]
2023-09-05T05:15:13+09:00 FStation kernel: [204555.496622] [@1000]
2023-09-05T05:15:13+09:00 FStation kernel: [204555.515651] [@103]
2023-09-05T05:15:14+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:15+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:16+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:17+09:00 FStation synoaudiod: SYSTEM:  Last message 'audiolib/json_util.c' repeated 3 times, suppressed by syslog-ng on FStation
2023-09-05T05:15:17+09:00 FStation [204781.710584] init: tty main process (6213) terminated with status 1
2023-09-05T05:15:17+09:00 FStation [204781.710609] init: tty main process ended, respawning
2023-09-05T05:15:18+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:19+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:20+09:00 FStation synoaudiod: audiolib/json_util.cpp:61 Failed to fopen [/tmp/AudioStation/player.list.json], [No such file or directory]
2023-09-05T05:15:27+09:00 FStation synoaudiod: SYSTEM:  Last message 'audiolib/json_util.c' repeated 9 times, suppressed by syslog-ng on FStation
2023-09-05T05:15:27+09:00 FStation [204791.725219] init: tty main process (6226) terminated with status 1

一つ目の疑問

とここで1つの疑問が浮かびます。/dev/ストレージプールが認識されていない状況で、ログは欠損していませんでした。

ということは、OSや/var/logは大丈夫で、/volumeにマウントされている、/dev/vg1000/lvだけロストしていたということでしょうか。 読んでいる人はどういうこと？と思うかもしれませんが、構成を再度確認にすると、 OSやログが保存されているストレージプールは/dev/md0で分かれています。 ※さっきのまとめ画像の通りです。

/含め /var/logについては、SSDも含めて構成されているので、よっっっっっぽどのことが無い限り影響がありません。 /volumeは死んでいても、きちんとログは保存されておりました。

bash-4.3# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sat Jan  1 09:00:05 2000
     Raid Level : raid1
     Array Size : 2490176 (2.37 GiB 2.55 GB)
  Used Dev Size : 2490176 (2.37 GiB 2.55 GB)
   Raid Devices : 12
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Sep  9 15:00:35 2023
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

           UUID : 1ac6abb2:1b359f68:1842b64a:93cf1d09 (local to host FStation)
         Events : 0.4815281

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       33        1      active sync   /dev/sdc1
       2       8       17        2      active sync   /dev/sdb1
       -       0        0        3      removed
       -       0        0        4      removed
       -       0        0        5      removed
       -       0        0        6      removed
       -       0        0        7      removed
       -       0        0        8      removed
       -       0        0        9      removed
       -       0        0       10      removed
       -       0        0       11      removed
bash-4.3#

2つ目の疑問：どうして再起動で治ったのか

わかりません。

作業をし始めたあと、/var/logのデータをすべてチェックして、該当の時間のエラーログを見つけられず。原因は不明です。 完全に不完全燃焼です。Synologyさんの動作ログいろいろ情報が足りてなくて難しい…。

というわけで状況証拠もそろわず、この状況だけから推察すると、

diskのエラーや、カーネルでのエラーが一切見つからなかったところから、おそらくボリュームが何らかのタイミングで刺さったか（しかしなぜカーネルエラーが残残ってない）。

あるいは、Synologyのパッケージ関係が止まって、壊れたように検知してしまったんでしょうか？

Storage Pool and Volume missing !! とか海外では書かれているが、これはHWなのか、それともSWなのか…。

つまり、今回の事象では…

ということなのかもしれません。

山田先生、これからも尊敬しております。