- ceph-users - lists.ceph.io

by Ansgar Jazdzewski

hi folks I have a cluster running ceph 14.2.22 on ubuntu 18.04 and some hours ago one of the mons stopped working and the on-call team rebooted the node; not the mon is is not joining the ceph-cluster. TCP ports of mons are open and reachable! ceph health detail HEALTH_WARN 1/3 mons down, quorum osd02,osd03 MON_DOWN 1/3 mons down, quorum osd02,osd03 mon.osd01 (rank 0) addr [v2:10.152.28.171:3300/0,v1:10.152.28.171:6789/0] is down (out of quorum) I like to add a new 3rd mon to the cluster on osd04 but I'm a bit scared as it can result in 50% of the mons are not in reach!? Question: should I remove the mon on osd01 first and recreate the demon before starting a new mon on osd04? Thanks for your input! Ansgar

5 hours, 49 minutes

2
6
0 0

unable to map device with krbd on el7 with ceph nautilus

by cek+ceph＠deepunix.net

Hi. I've followed the installation guide and got nautilus 14.2.22 running on el7 via https://download.ceph.com/rpm-nautilus/el7/x86_64/ yum repo. I'm now trying to map a device on an el7 and getting extremely weird errors: # rbd info test1/blk1 --name client.testing-rw rbd image 'blk1': size 50 GiB in 12800 objects order 22 (4 MiB objects) snapshot_count: 0 id: 2e0929313a08e block_name_prefix: rbd_data.2e0929313a08e format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Fri Jul 23 15:59:12 2021 access_timestamp: Fri Jul 23 15:59:12 2021 modify_timestamp: Fri Jul 23 15:59:12 2021 # rbd device map test1/blk1 --name client.testing-rw rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (3) No such process # dmesg | tail [91885.624859] libceph: resolve 'name=testing-rw' (ret=-3): failed [91885.624863] libceph: parse_ips bad ip 'name=testing-rw,key=client.testing-rw' # modinfo rbd |grep vers filename: /lib/modules/3.10.0-1160.31.1.el7.x86_64/kernel/drivers/block/rbd.ko.xz rhelversion: 7.9 srcversion: 5386BBBD00C262C66CB81F5 vermagic: 3.10.0-1160.31.1.el7.x86_64 SMP mod_unload modversions Is that because the kernel module is too old? el7's base is supplying ceph-common-10.2.5. Where can I get an updated kernel module for el7 and it would also be wonderful to have some form of warning in docs that user might encounter this problem. Thanks.

1 day, 16 hours

2
1
0 0

OSDs flapping with "_open_alloc loaded 132 GiB in 2930776 extents available 113 GiB"

by Dave Piper

Hi all, We've got a containerized test cluster with 3 OSDs and ~ 220GiB of data. Shortly after upgrading from nautilus -> octopus, 2 of the 3 OSDs have started flapping. I've also got alarms about the MDS being damaged, which we've seen elsewhere and have a recovery process for, but I'm unable to run this (I suspect because I've only got 1 functioning OSD). My RGWs are also failing to start, again I suspect because of the bad state of OSDs. I've tried restarting all OSDs, rebooting all servers, checked auth (all looks fine) - but I'm still in the same state. My OSDs seem to be failing at the "_open_alloc opening allocation metadata" step; looking at logs for each OSD restart, the OSD writes this log, then no logs for a few minutes and then logs: bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in 2930776 extents available 113 GiB rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work After that we seem to try starting up in a slightly different state and get a different set of errors: bluefs _allocate failed to allocate 0x100716 on bdev 1, free 0xd0000; fallback to bdev 2 bluefs _allocate unable to allocate 0x100716 on bdev 2, free 0xffffffffffffffff; fallback to slow device expander and eventually crash and log a heap of stack dumps. I don't know what extents are but I seem to have a lot of them, and more than I've got capacity for? Maybe I'm running out of RAM or disk space somewhere, but I've got 21GB of free RAM on the server, and each OSD has a 350GiB device attached to it. I'm wondering if anyone has seen anything like this before or can suggest next debug steps to take? Cheers, Dave Full OSD logs surrounding the "_open_alloc opening allocation metadata" step: Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1626998833819439, "job": 1, "event": "recovery_started", "log_files": [392088, 392132]} Jul 23 00:07:13 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:13.818+0000 7f3de111bf40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #392088 mode 0 Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:17.240+0000 7f3de111bf40 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #392132 mode 0 Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:17.486+0000 7f3de111bf40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1626998837486404, "job": 1, "event": "recovery_finished"} Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:17.486+0000 7f3de111bf40 1 bluestore(/var/lib/ceph/osd/ceph-1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2 Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist init Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:17.524+0000 7f3de111bf40 1 freelist _init_from_label Jul 23 00:07:17 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:17.529+0000 7f3de111bf40 1 bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc opening allocation metadata Jul 23 00:07:18 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:18.238+0000 7f3de111bf40 1 HybridAllocator _spillover_range constructing fallback allocator Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:20.563+0000 7f3de111bf40 1 bluestore(/var/lib/ceph/osd/ceph-1) _open_alloc loaded 132 GiB in 2930776 extents available 113 GiB Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:20.563+0000 7f3de111bf40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work Jul 23 00:07:20 condor_sc0 container_name/ceph-osd-1[1709]: 2021-07-23T00:07:20.565+0000 7f3de111bf40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete

2 days, 2 hours

2
1
0 0

Luminous won't fully recover

by Shain Miley

We recently had a few Ceph nodes go offline which required a reboot. I have been able to get the cluster back to the state listed below however it does not seem like it will progress past the point of 23473/287823588 objects misplaced. Yesterday it was about 13% of the data that was misplaced…however this morning it has goteen to 0.008% but has not moved past this point in about an hour. Does anyone see anything in the output below that points to the problem and/or are there any suggestions that I can follow in order to figure out why the cluster health is not moving beyond this point? --------------------------------------------------- root@rbd1:~# ceph -s cluster: id: 504b5794-34bd-44e7-a8c3-0494cf800c23 health: HEALTH_ERR crush map has legacy tunables (require argonaut, min is firefly) 23473/287823588 objects misplaced (0.008%) 14 scrub errors Reduced data availability: 2 pgs inactive Possible data damage: 8 pgs inconsistent services: mon: 3 daemons, quorum hqceph1,hqceph2,hqceph3 mgr: hqceph2(active), standbys: hqceph3 osd: 288 osds: 270 up, 270 in; 2 remapped pgs rgw: 1 daemon active data: pools: 17 pools, 9411 pgs objects: 95.95M objects, 309TiB usage: 936TiB used, 627TiB / 1.53PiB avail pgs: 0.021% pgs not active 23473/287823588 objects misplaced (0.008%) 9369 active+clean 30 active+clean+scrubbing+deep 8 active+clean+inconsistent 2 activating+remapped 2 active+clean+scrubbing io: client: 1000B/s rd, 0B/s wr, 0op/s rd, 0op/s wr root@rbd1:~# ceph health detail HEALTH_ERR crush map has legacy tunables (require argonaut, min is firefly); 1 osds down; 23473/287823588 objects misplaced (0.008%); 14 scrub errors; Reduced data availability: 3 pgs inactive, 13 pgs peering; Possible data damage: 8 pgs inconsistent; Degraded data redundancy: 408658/287823588 objects degraded (0.142%), 38 pgs degraded OLD_CRUSH_TUNABLES crush map has legacy tunables (require argonaut, min is firefly) see http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables OSD_DOWN 1 osds down osd.95 (root=default,host=hqosd8) is down OBJECT_MISPLACED 23473/287823588 objects misplaced (0.008%) OSD_SCRUB_ERRORS 14 scrub errors PG_AVAILABILITY Reduced data availability: 3 pgs inactive, 13 pgs peering pg 3.b41 is stuck peering for 106.682058, current state peering, last acting [204,190] pg 3.c33 is stuck peering for 103.403643, current state peering, last acting [228,274] pg 3.d15 is stuck peering for 128.537454, current state peering, last acting [286,24] pg 3.fa9 is stuck peering for 106.526146, current state peering, last acting [286,47] pg 3.fb7 is stuck peering for 105.878878, current state peering, last acting [62,97] pg 3.13a2 is stuck peering for 106.491138, current state peering, last acting [270,219] pg 3.1521 is stuck inactive for 170180.165265, current state activating+remapped, last acting [94,186,188] pg 3.1565 is stuck peering for 106.782784, current state peering, last acting [121,60] pg 3.157c is stuck peering for 128.557448, current state peering, last acting [128,268] pg 3.1744 is stuck peering for 106.639603, current state peering, last acting [192,142] pg 3.1ac8 is stuck peering for 127.839550, current state peering, last acting [221,190] pg 3.1e24 is stuck peering for 128.201670, current state peering, last acting [118,158] pg 3.1e46 is stuck inactive for 169121.764376, current state activating+remapped, last acting [87,199,170] pg 18.36 is stuck peering for 128.554121, current state peering, last acting [204] pg 21.1ce is stuck peering for 106.582584, current state peering, last acting [266,192] PG_DAMAGED Possible data damage: 8 pgs inconsistent pg 3.1ca is active+clean+inconsistent, acting [201,8,180] pg 3.56a is active+clean+inconsistent, acting [148,240,8] pg 3.b0f is active+clean+inconsistent, acting [148,260,8] pg 3.b56 is active+clean+inconsistent, acting [218,8,240] pg 3.10ff is active+clean+inconsistent, acting [262,8,211] pg 3.1192 is active+clean+inconsistent, acting [192,8,187] pg 3.124a is active+clean+inconsistent, acting [123,8,222] pg 3.1c55 is active+clean+inconsistent, acting [180,8,287] PG_DEGRADED Degraded data redundancy: 408658/287823588 objects degraded (0.142%), 38 pgs degraded pg 3.8f is active+undersized+degraded, acting [163,149] pg 3.ba is active+undersized+degraded, acting [68,280] pg 3.1aa is active+undersized+degraded, acting [176,211] pg 3.29e is active+undersized+degraded, acting [241,194] pg 3.323 is active+undersized+degraded, acting [78,194] pg 3.343 is active+undersized+degraded, acting [242,144] pg 3.4ae is active+undersized+degraded, acting [153,237] pg 3.524 is active+undersized+degraded, acting [252,222] pg 3.5c9 is active+undersized+degraded, acting [272,252] pg 3.713 is active+undersized+degraded, acting [273,80] pg 3.730 is active+undersized+degraded, acting [235,212] pg 3.88f is active+undersized+degraded, acting [222,285] pg 3.8cb is active+undersized+degraded, acting [285,20] pg 3.9a0 is active+undersized+degraded, acting [240,200] pg 3.c19 is active+undersized+degraded, acting [165,276] pg 3.ec8 is active+undersized+degraded, acting [158,40] pg 3.1025 is active+undersized+degraded, acting [258,274] pg 3.1058 is active+undersized+degraded, acting [38,68] pg 3.14e4 is active+undersized+degraded, acting [185,39] pg 3.150c is active+undersized+degraded, acting [138,140] pg 3.1545 is active+undersized+degraded, acting [222,55] pg 3.15a6 is active+undersized+degraded, acting [242,272] pg 3.1620 is active+undersized+degraded, acting [200,164] pg 3.1710 is active+undersized+degraded, acting [176,285] pg 3.1792 is active+undersized+degraded, acting [190,11] pg 3.17bd is active+undersized+degraded, acting [207,15] pg 3.17da is active+undersized+degraded, acting [5,160] pg 3.183e is active+undersized+degraded, acting [273,136] pg 3.197d is active+undersized+degraded, acting [241,139] pg 3.1a3d is active+undersized+degraded, acting [184,121] pg 3.1ba6 is active+undersized+degraded, acting [47,249] pg 3.1c2b is active+undersized+degraded, acting [268,80] pg 3.1ca2 is active+undersized+degraded, acting [280,152] pg 3.1cd4 is active+undersized+degraded, acting [2,129] pg 3.1e13 is active+undersized+degraded, acting [247,114] pg 12.56 is active+undersized+degraded, acting [54] pg 18.8 is undersized+degraded+peered, acting [260] pg 21.9f is active+undersized+degraded, acting [215,201] -------------------------------------------------------------------------------------------------- Thanks, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org

2 days, 2 hours

2
1
0 0

ceph-Dokan on windows 10 not working after upgrade to pacific

by Robert W. Eckert

Hi- I recently upgraded to pacific, and I am now getting an error connecting on my windows 10 machine: The error is the handle_auth_bad_method, I tried a few combinations of cephx,none on the monitors, but I keep getting the same error. The same config(With paths updated) and key ring works on my WSL instance running an old luminous client (I can't seem to get it to install a newer client ) Do you have any suggestions on where to look? Thanks, Rob. ----------------------------------------------------- PS C:\Program Files\Ceph\bin> .\ceph-dokan.exe --id rob -l Q 2021-05-14T12:19:58.172Eastern Daylight Time 5 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] failed to fetch mon config (--no-mon-config to skip) PS C:\Program Files\Ceph\bin> cat c:/ProgramData/ceph/ceph.client.rob.keyring [client.rob] key = <REDACTED> caps mon = "allow rwx" caps osd = "allow rwx" PS C:\Program Files\Ceph\bin> cat C:\ProgramData\Ceph\ceph.conf # minimal ceph.conf [global] log to stderr = true ; Uncomment the following in order to use the Windows Event Log log to syslog = true run dir = C:/ProgramData/ceph/out crash dir = C:/ProgramData/ceph/out ; Use the following to change the cephfs client log level debug client = 2 [global] fsid = <redacted> mon_host = [<redacted>] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx [client] keyring = c:/ProgramData/ceph/ceph.client.rob.keyring log file = C:/ProgramData/ceph/out/$name.$pid.log admin socket = C:/ProgramData/ceph/out/$name.$pid.asok

2 days, 3 hours

4
7
0 0

OSD failed to load OSD map for epoch

by Johan Hattne

Dear all; We have 3-node cluster that has two OSDs on separate nodes, each with wal on NVMe. It's been running fine for quite some time, albeit under very light load. This week, we moved from package-based Octopus to container-based ditto (15.2.13, all on Debian stable). Within a few hours of that change, both OSDs crashed and dmesg filled up with stuff like: DMAR: DRHD: handling fault status reg 2 DMAR: [DMA Read] Request device [06:00.0] PASID ffffffff fault addr ffbc0000 [fault reason 06] PTE Read access is not set where 06:00.0 is the NVMe with the wal. This happened at the same time on *both* OSD nodes, but I'll worry about why this happened later. I would first like to see if I can get the cluster back up. From cephadm shell, I activate OSD 1 and try to start it (I did create a minimal /etc/ceph/ceph.conf with global "fsid" and "mon host" for that purpose): # ceph-volume lvm activate 1 cce125b2-2597-4be9-bd17-23eb059d2778 --no-systemd # ceph-osd -d --cluster ceph --id 1 This gives "osd.1 0 OSD::init() : unable to read osd superblock", and the subsequent output indicates that this due to checksum errors. So ignore checksum mismatches and try again: # CEPH_ARGS="--bluestore-ignore-data-csum" ceph-osd -d --cluster ceph --id 1 which results in "osd.1 0 failed to load OSD map for epoch 4372, got 0 bytes". The monitors are at 4378, as per: # ceph osd stat 2 osds: 0 up (since 47h), 1 in (since 47h); epoch: e4378 Is there any way to get past this? For instance, could I coax the OSDs into epoch 4378? This is the first time I deal a ceph disaster, so there may be all kinds of obvious things I'm missing. // Best wishes; Johan

2 days, 4 hours

1
0
0 0

[ceph] [pacific] cephadm cannot create OSD

by Gargano Andrea

Hi all, we are trying to install ceph on ubuntu 20.04 but we are not able to create OSD. Entering in cephadm shell we can see the following: root@tst2-ceph01:/# ceph -s cluster: id: 8b937a98-eb86-11eb-8509-c5c80111fd98 health: HEALTH_ERR Module 'cephadm' has failed: No filters applied OSD count 0 < osd_pool_default_size 3 services: mon: 3 daemons, quorum tst2-ceph01,tst2-ceph03,tst2-ceph02 (age 2h) mgr: tst2-ceph01.kwyejx(active, since 3h), standbys: tst2-ceph02.qrpuzp osd: 0 osds: 0 up (since 115m), 0 in (since 105m) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: root@tst2-ceph01:/# ceph orch device ls Hostname Path Type Serial Size Health Ident Fault Available tst2-ceph01 /dev/sdb hdd 600508b1001c1960d834c222fb64f2ea 1200G Unknown N/A N/A Yes tst2-ceph01 /dev/sdc hdd 600508b1001c36e812fb5d14997f5f47 1200G Unknown N/A N/A Yes tst2-ceph01 /dev/sdd hdd 600508b1001c01a0297ac2c5e8039063 1200G Unknown N/A N/A Yes tst2-ceph01 /dev/sde hdd 600508b1001cf4520d0f0155d0dd31ad 1200G Unknown N/A N/A Yes tst2-ceph01 /dev/sdf hdd 600508b1001cc911d4f570eba568a8d0 1200G Unknown N/A N/A Yes tst2-ceph01 /dev/sdg hdd 600508b1001c410bd38e6c55807bea25 1200G Unknown N/A N/A Yes tst2-ceph01 /dev/sdh ssd 600508b1001cdb21499020552589eadb 400G Unknown N/A N/A Yes tst2-ceph02 /dev/sdb hdd 600508b1001ce1f33b63f8859aeac9b4 1200G Unknown N/A N/A Yes tst2-ceph02 /dev/sdc hdd 600508b1001c0b4dbfa794d2b38f328e 1200G Unknown N/A N/A Yes tst2-ceph02 /dev/sdd hdd 600508b1001c145b8de4e4e7cc9129d5 1200G Unknown N/A N/A Yes tst2-ceph02 /dev/sde hdd 600508b1001c1d81d0aaacfdfd20f5f1 1200G Unknown N/A N/A Yes tst2-ceph02 /dev/sdf hdd 600508b1001c28d2a2c261449ca1a3cc 1200G Unknown N/A N/A Yes tst2-ceph02 /dev/sdg hdd 600508b1001c1f9a964b1513f70b51b3 1200G Unknown N/A N/A Yes tst2-ceph02 /dev/sdh ssd 600508b1001c8040dd5cf17903940177 400G Unknown N/A N/A Yes tst2-ceph03 /dev/sdb hdd 600508b1001c900ef43d7745db17d5cc 1200G Unknown N/A N/A Yes tst2-ceph03 /dev/sdc hdd 600508b1001cf1b79f7dc2f79ab2c90b 1200G Unknown N/A N/A Yes tst2-ceph03 /dev/sdd hdd 600508b1001c83c09fe03eb17e555f5f 1200G Unknown N/A N/A Yes tst2-ceph03 /dev/sde hdd 600508b1001c9c4c5db12fabf54a4ff3 1200G Unknown N/A N/A Yes tst2-ceph03 /dev/sdf hdd 600508b1001cdaa7dc09d751262e2cc9 1200G Unknown N/A N/A Yes tst2-ceph03 /dev/sdg hdd 600508b1001c8f435a08b7eae4a1323e 1200G Unknown N/A N/A Yes tst2-ceph03 /dev/sdh ssd 600508b1001c5e24f822d6790a5df65b 400G Unknown N/A N/A Yes we wrote the following spec file: service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: size: '1200GB' db_devices: size: '400GB' but running, the following appears: root@tst2-ceph01:/# ceph orch apply osd -i /spec.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. ################ OSDSPEC PREVIEWS ################ Preview data is being generated.. Please re-run this command in a bit. root@tst2-ceph01:/# ceph orch apply osd -i /spec.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. ################ OSDSPEC PREVIEWS ################ Preview data is being generated.. Please re-run this command in a bit. It's seems that yml file is not read. Any help please? Thank you, Andrea

2 days, 5 hours

3
5
0 0

Is there any way to obtain the maximum number of node failure in ceph without data loss?

by Jerry Lee

Hello, I would like to know the maximum number of node failures for a EC8+3 pool in a 12-node cluster with 3 OSDs in each node. The size and min_size of the EC8+3 pool is configured as 11 and 8, and OSDs of each PG are selected by host. When there is no node failure, the maximum number of node failures is 3, right? After unplugging a OSD (osd.14) in the cluster, I check the PG acting set changes and one of the results is shown as below: T0: [15,31,11,34,28,1,8,26,14,19,5] T1: after unplugging a OSD (osd.14) and recovery started [15,31,11,34,28,1,8,26,NONE,19,5] T2: [15,31,11,34,21,1,8,26,19,29,5] T3: [15,31,11,34,NONE,1,8,26,NONE,NONE,5] T4: recovery was done [15,31,11,34,21,1,8,26,19,29,5] For the PG, 3 OSD peers changed during the recovery progress ([_,_,_,_,28->21,_,_,_,14->19,19->29,_]). It seems that min_size (8) of chunks of the EC8+3 pool are kept during recovery. Does it mean that no more node failures are bearable during T3 to T4? Can we calculate the maximum number of node failures by examining all the acting sets of the PGs? Is there some simple way to obtain such information? Any ideas and feedback are appreciated, thanks! - Jerry

2 days, 7 hours

2
1
0 0

Files listed in radosgw BI but is not available in ceph

by Boris Behrens

Hi everybody, a customer mentioned that he got problems in accessing hist rgw data. I checked the bucket index and the file should be available. Then I pulled a list with radosgw-admin radoslist --bucket BUCKET and it seems that the file is gone. beside the "yaiks, is there a way the file might be somewhere else in ceph?" how can this happen? We do occational orphan objects cleanups but this does not pull the bucket index into account. It is a large bucket with 2.1m files in it and with 34 shards. Cheers and happy weekend Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 days, 12 hours

7
24
0 0

Can't clear UPGRADE_REDEPLOY_DAEMON after fix

by Arnaud MARTEL

Hi, I just upgraded my cluster from 16.2.4 to 16.2.5 and I had an error during the upgrade of the first osd daemon (cf below). I fixed the error (I just created the missing directory on all hosts), then resume the upgrade. Now, everything is OK but I still have a warning:" [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.0 on host cephp01 failed" and "ceph health detail" continues to show the old error: HEALTH_OK (muted: UPGRADE_REDEPLOY_DAEMON) (MUTED) [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.0 on host cephp01 failed. Upgrade daemon: osd.0: cephadm exited with an error code: 1, stderr:Redeploy daemon osd.0 ... Traceback (most recent call last): File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in <module> main() File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main r = ctx.func(ctx) File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image return func(ctx) File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4326, in command_deploy ports=daemon_ports) File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon c, osd_fsid=osd_fsid, ports=ports) File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2801, in deploy_daemon_units install_sysctl(ctx, fsid, daemon_type) File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2963, in install_sysctl _write(conf, lines) File "/var/lib/ceph/70126dbe-cdd7-11eb-a661-b49691b21313/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2948, in _write with open(conf, 'w') as f: FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/sysctl.d/90-ceph-70126dbe-cdd7-11eb-a661-b49691b21313-osd.conf' I checked that the osd.0 daemon has been correctly upgraded to v16.2.5 and is running (using dashboard and "cephadm ls" on the node), and "ceph orch upgrade status" returns that upgrade is completed. I use "ceph health mute UPGRADE_REDEPLOY_DAEMON" to hide the warning but I would prefer to clean it and I can't find something related to this warning in the documentation. Do you know if there is a command to clean this warning while the cluster stills up and serving cephfs clients ? Regards, Arnaud

2 days, 13 hours

1
1
0 0

2021

2020

2019

ceph-users