- ceph-users - lists.ceph.io

16.2.6 OSD down, out but container running....

by Marco Pizzolo

Hello Everyone, I'm seeing an issue where the podman container is running, but the osd is being reported as down and out. restarting service doesn't help, neither does rebooting the host. What am I missing? Thanks, Marco

2 minutes

3
2
0 0

Re: Upgrade to 16.2.6 and osd+mds crash after bluestore_fsck_quick_fix_on_mount true

by Beard Lionel

Hi, I had exactly the same behaviour: - upgrade from nautilus to pacific - same warning message - set config option - restart osd. I've first restarted one osd and it was fine, so I decided to restart all osds of same host, and about half of osds can't start anymore with same error as you. We didn't find any workaround, apart deleting and recreating failed osds ☹ For MDS, which was also crashing, I had to follow the recovery procedure to recover my data: https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#using-an-… Cordialement, Regards, Lionel BEARD CLS - IT & Operations -----Message d'origine----- De : Marek Grzybowski <marek.grzybowski(a)gmail.com> De la part de mgrzybowski Envoyé : mercredi 20 octobre 2021 23:56 À : ceph-users(a)ceph.io Objet : [ceph-users] Upgrade to 16.2.6 and osd+mds crash after bluestore_fsck_quick_fix_on_mount true CAUTION: This message comes from an external server, do not click on links or open attachments unless you know the sender and are sure the content is safe. Hi Recently I did perform upgrades on single node cephfs server i have. # ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ecpoolk3m1osd ecpoolk5m1osd ecpoolk4m2osd ~# ceph osd pool ls detail pool 20 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 10674 lfor 0/0/5088 flags hashpspool stripe_width 0 application cephfs pool 21 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 10674 lfor 0/0/5179 flags hashpspool stripe_width 0 application cephfs pool 22 'ecpoolk3m1osd' erasure profile myprofilek3m1osd size 4 min_size 3 crush_rule 3 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 10674 lfor 0/0/1442 flags hashpspool,ec_overwrites stripe_width 12288 compression_algorithm zstd compression_mode aggressive application cephfs pool 23 'ecpoolk5m1osd' erasure profile myprofilek5m1osd size 6 min_size 5 crush_rule 5 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 12517 lfor 0/0/7892 flags hashpspool,ec_overwrites stripe_width 20480 compression_algorithm zstd compression_mode aggressive application cephfs pool 24 'ecpoolk4m2osd' erasure profile myprofilek4m2osd size 6 min_size 5 crush_rule 6 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10674 flags hashpspool,ec_overwrites stripe_width 16384 compression_algorithm zstd compression_mode aggressive application cephfs pool 25 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 11033 lfor 0/0/10991 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth I started this upgrade from ubuntu 16.04 and luminous ( there were upgrades in the past and some osd's could be started in Kraken ) ): - first i upgraded ceph to Nautilus, all seems to went well and accoording to the docs, no warning in status - then i did "do-release-upgrade" to ubuntu to 18.04 ( ceph packaged were not touch by that upgrade ) - then i did "do-release-upgrade" to ubuntu to 20.04 ( this upgrade bumped ceph packages to 15.2.1-0ubuntu1, before each do-release-upgrade i removed /etc/ceph/ceph.conf, so at least mon deamon was down. osd should not start ( siple volumes are encrypted ) - next i upgraded ceph packages to 16.2.6-1focal m started deamons . All seems to work well, only what left was warning: 10 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats I found on the list that it is recommend to set: ceph config set osd bluestore_fsck_quick_fix_on_mount true and rolling restart OSDs. After first restart+fsck i got crash on OSD ( and on MDS to) : -1> 2021-10-14T22:02:45.877+0200 7f7f080a4f00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7f7f080a4f00 time 2021-10-14T22:02:45.878154+0200 /build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2) ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55e29cd0ce61] 2: /usr/bin/ceph-osd(+0xac6069) [0x55e29cd0d069] 3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x55e29ce97057] 4: (OSD::load_pgs()+0x6b4) [0x55e29ce07ec4] 5: (OSD::init()+0x2b4e) [0x55e29ce14a6e] 6: main() 7: __libc_start_main() 8: _start() The same went on next restart+fsck osd: -1> 2021-10-17T22:47:49.291+0200 7f98877bff00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7f98877bff00 time 2021-10-17T22:47:49.292912+0200 /build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2) ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x560e09af7e61] 2: /usr/bin/ceph-osd(+0xac6069) [0x560e09af8069] 3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x560e09c82057] 4: (OSD::load_pgs()+0x6b4) [0x560e09bf2ec4] 5: (OSD::init()+0x2b4e) [0x560e09bffa6e] 6: main() 7: __libc_start_main() 8: _start() Once crashed OSDs could not be bring back online, they will crash again if i try start them. Deep fsck did not found anything: ~# ceph-bluestore-tool --command fsck --deep yes --path /var/lib/ceph/osd/ceph-2 fsck success Any ideas what could cause this crashes and is it possible to bring online crashed osd this way ? -- mgrzybowski _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ________________________________ Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l'intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur ou s'il ne vous est pas destiné, merci de le détruire ainsi que toute copie de votre système et d'en avertir immédiatement l'expéditeur. Toute lecture non autorisée, toute utilisation de ce message qui n'est pas conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer l'intégrité de ce message électronique susceptible d'altération, l’expéditeur (et ses filiales) décline(nt) toute responsabilité au titre de ce message dans l'hypothèse où il aurait été modifié ou falsifié. This message and any attachments (the "message") is intended solely for the intended recipient(s) and is confidential. If you receive this message in error, or are not the intended recipient(s), please delete it and any copies from your systems and immediately notify the sender. Any unauthorized view, use that does not comply with its purpose, dissemination or disclosure, either whole or partial, is prohibited. Since the internet cannot guarantee the integrity of this message which may not be reliable, the sender (and its subsidiaries) shall not be liable for the message if modified or falsified.

7 hours, 3 minutes

2
1
0 0

Upgrade to 16.2.6 and osd+mds crash after bluestore_fsck_quick_fix_on_mount true

by mgrzybowski

Hi Recently I did perform upgrades on single node cephfs server i have. # ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ecpoolk3m1osd ecpoolk5m1osd ecpoolk4m2osd ~# ceph osd pool ls detail pool 20 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 10674 lfor 0/0/5088 flags hashpspool stripe_width 0 application cephfs pool 21 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 10674 lfor 0/0/5179 flags hashpspool stripe_width 0 application cephfs pool 22 'ecpoolk3m1osd' erasure profile myprofilek3m1osd size 4 min_size 3 crush_rule 3 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 10674 lfor 0/0/1442 flags hashpspool,ec_overwrites stripe_width 12288 compression_algorithm zstd compression_mode aggressive application cephfs pool 23 'ecpoolk5m1osd' erasure profile myprofilek5m1osd size 6 min_size 5 crush_rule 5 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 12517 lfor 0/0/7892 flags hashpspool,ec_overwrites stripe_width 20480 compression_algorithm zstd compression_mode aggressive application cephfs pool 24 'ecpoolk4m2osd' erasure profile myprofilek4m2osd size 6 min_size 5 crush_rule 6 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10674 flags hashpspool,ec_overwrites stripe_width 16384 compression_algorithm zstd compression_mode aggressive application cephfs pool 25 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 11033 lfor 0/0/10991 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth I started this upgrade from ubuntu 16.04 and luminous ( there were upgrades in the past and some osd's could be started in Kraken ) ): - first i upgraded ceph to Nautilus, all seems to went well and accoording to the docs, no warning in status - then i did "do-release-upgrade" to ubuntu to 18.04 ( ceph packaged were not touch by that upgrade ) - then i did "do-release-upgrade" to ubuntu to 20.04 ( this upgrade bumped ceph packages to 15.2.1-0ubuntu1, before each do-release-upgrade i removed /etc/ceph/ceph.conf, so at least mon deamon was down. osd should not start ( siple volumes are encrypted ) - next i upgraded ceph packages to 16.2.6-1focal m started deamons . All seems to work well, only what left was warning: 10 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats I found on the list that it is recommend to set: ceph config set osd bluestore_fsck_quick_fix_on_mount true and rolling restart OSDs. After first restart+fsck i got crash on OSD ( and on MDS to) : -1> 2021-10-14T22:02:45.877+0200 7f7f080a4f00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7f7f080a4f00 time 2021-10-14T22:02:45.878154+0200 /build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2) ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55e29cd0ce61] 2: /usr/bin/ceph-osd(+0xac6069) [0x55e29cd0d069] 3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x55e29ce97057] 4: (OSD::load_pgs()+0x6b4) [0x55e29ce07ec4] 5: (OSD::init()+0x2b4e) [0x55e29ce14a6e] 6: main() 7: __libc_start_main() 8: _start() The same went on next restart+fsck osd: -1> 2021-10-17T22:47:49.291+0200 7f98877bff00 -1 /build/ceph-16.2.6/src/osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*, spg_t, epoch_t*)' thread 7f98877bff00 time 2021-10-17T22:47:49.292912+0200 /build/ceph-16.2.6/src/osd/PG.cc: 1009: FAILED ceph_assert(values.size() == 2) ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x560e09af7e61] 2: /usr/bin/ceph-osd(+0xac6069) [0x560e09af8069] 3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0xa17) [0x560e09c82057] 4: (OSD::load_pgs()+0x6b4) [0x560e09bf2ec4] 5: (OSD::init()+0x2b4e) [0x560e09bffa6e] 6: main() 7: __libc_start_main() 8: _start() Once crashed OSDs could not be bring back online, they will crash again if i try start them. Deep fsck did not found anything: ~# ceph-bluestore-tool --command fsck --deep yes --path /var/lib/ceph/osd/ceph-2 fsck success Any ideas what could cause this crashes and is it possible to bring online crashed osd this way ? -- mgrzybowski

7 hours, 6 minutes

3
6
1 0

MDS not becoming active after migrating to cephadm

by Petr Belyaev

Hi, We’ve recently upgraded from Nautilus to Pacific, and tried moving our services to cephadm/ceph orch. For some reason, MDS nodes deployed through orch never become active (or at least standby-replay). Non-dockerized MDS nodes can still be deployed and work fine. Non-dockerized mds version is 16.2.6, docker image version is 16.2.5-387-g7282d81d (came as a default). In the MDS log, the only related message is monitors assigning MDS as standby. Increasing the log level does not help much, it only adds beacon messages. Monitor log also contains no differences compared to a non-dockerized MDS startup. Mds metadata command output is identical to that of a non-dockerized MDS. The only difference I can see in the log is the value in curly braces after the node name, e.g. mds.storage{0:1234ff}. For dockerized MDS, the first value is ffffffff, for non-dockerized it’s zero. Compat flags are identical. Could someone please advise me why the dockerized MDS is being stuck as a standby? Maybe some config values missing or smth? Best regards, Petr

8 hours, 36 minutes

3
5
0 0

cephadm does not find podman objects for osds

by Magnus Harlander

Hi, after converting my 2 node cluster to cephadm I'm in lots of trouble. - containerized mds are not available in the cluster. I must run mds from systemd to make my fs available. - osd podman objects are not found after a reboot of one node. I don't want to test it on the second node, because if it looses the podman config as well, my cluster is dead! Can anybody help me? Best regards Magnus Output from cephadm.log: cephadm ['--image', 'docker.io/ceph/ceph:v15', '--no-container-init', 'ls'] 2021-10-25 22:47:01,929 DEBUG container_init=False 2021-10-25 22:47:01,929 DEBUG Running command: systemctl is-enabled ceph-mds@s1 2021-10-25 22:47:01,935 DEBUG systemctl: stdout enabled 2021-10-25 22:47:01,935 DEBUG Running command: systemctl is-active ceph-mds@s1 2021-10-25 22:47:01,940 DEBUG systemctl: stdout active 2021-10-25 22:47:01,940 DEBUG Running command: ceph -v 2021-10-25 22:47:02,009 DEBUG ceph: stdout ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) 2021-10-25 22:47:02,016 DEBUG Running command: systemctl is-enabled ceph-osd@1 2021-10-25 22:47:02,024 DEBUG systemctl: stdout disabled 2021-10-25 22:47:02,024 DEBUG Running command: systemctl is-active ceph-osd@1 2021-10-25 22:47:02,031 DEBUG systemctl: stdout inactive 2021-10-25 22:47:02,031 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)mon.s1 2021-10-25 22:47:02,036 DEBUG systemctl: stdout enabled 2021-10-25 22:47:02,036 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)mon.s1 2021-10-25 22:47:02,041 DEBUG systemctl: stdout active 2021-10-25 22:47:02,041 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:02,068 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:02,069 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mon.s1 2021-10-25 22:47:02,111 DEBUG /bin/podman: stdout 1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25 22:45:39.315992691 +0200 CEST, 2021-10-25 22:47:02,203 DEBUG Running command: /bin/podman exec 1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48 ceph -v 2021-10-25 22:47:02,356 DEBUG /bin/podman: stdout ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) 2021-10-25 22:47:02,434 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)mgr.s1 2021-10-25 22:47:02,440 DEBUG systemctl: stdout enabled 2021-10-25 22:47:02,440 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)mgr.s1 2021-10-25 22:47:02,445 DEBUG systemctl: stdout active 2021-10-25 22:47:02,445 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:02,468 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:02,470 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mgr.s1 2021-10-25 22:47:02,513 DEBUG /bin/podman: stdout 2b08ddb6182e14985939e50a00e2306a77c3068a65ed122dab8bd5604c91af65,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25 22:45:39.535973449 +0200 CEST, 2021-10-25 22:47:02,601 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.0 2021-10-25 22:47:02,607 DEBUG systemctl: stdout enabled 2021-10-25 22:47:02,607 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.0 2021-10-25 22:47:02,612 DEBUG systemctl: stdout activating 2021-10-25 22:47:02,612 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:02,636 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:02,637 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0 2021-10-25 22:47:02,709 DEBUG /bin/podman: stderr Error: error inspecting object: no such object: "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0" 2021-10-25 22:47:02,712 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.2 2021-10-25 22:47:02,718 DEBUG systemctl: stdout enabled 2021-10-25 22:47:02,718 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.2 2021-10-25 22:47:02,723 DEBUG systemctl: stdout activating 2021-10-25 22:47:02,723 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:02,747 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:02,748 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.2 2021-10-25 22:47:02,821 DEBUG /bin/podman: stderr Error: error inspecting object: no such object: "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.2" 2021-10-25 22:47:02,823 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.6 2021-10-25 22:47:02,830 DEBUG systemctl: stdout enabled 2021-10-25 22:47:02,830 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.6 2021-10-25 22:47:02,835 DEBUG systemctl: stdout activating 2021-10-25 22:47:02,835 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:02,860 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:02,861 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.6 2021-10-25 22:47:02,948 DEBUG /bin/podman: stderr Error: error inspecting object: no such object: "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.6" 2021-10-25 22:47:02,950 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.7 2021-10-25 22:47:02,956 DEBUG systemctl: stdout enabled 2021-10-25 22:47:02,957 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.7 2021-10-25 22:47:02,961 DEBUG systemctl: stdout activating 2021-10-25 22:47:02,962 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:02,986 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:02,988 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.7 2021-10-25 22:47:03,063 DEBUG /bin/podman: stderr Error: error inspecting object: no such object: "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.7" 2021-10-25 22:47:03,066 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.8 2021-10-25 22:47:03,072 DEBUG systemctl: stdout enabled 2021-10-25 22:47:03,072 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)osd.8 2021-10-25 22:47:03,077 DEBUG systemctl: stdout activating 2021-10-25 22:47:03,077 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:03,099 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:03,101 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.8 2021-10-25 22:47:03,174 DEBUG /bin/podman: stderr Error: error inspecting object: no such object: "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.8" 2021-10-25 22:47:03,176 DEBUG Running command: systemctl is-enabled ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)mds.fs.s1.ajxyaf 2021-10-25 22:47:03,182 DEBUG systemctl: stdout enabled 2021-10-25 22:47:03,182 DEBUG Running command: systemctl is-active ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524(a)mds.fs.s1.ajxyaf 2021-10-25 22:47:03,187 DEBUG systemctl: stdout active 2021-10-25 22:47:03,187 DEBUG Running command: /bin/podman --version 2021-10-25 22:47:03,210 DEBUG /bin/podman: stdout podman version 3.2.3 2021-10-25 22:47:03,212 DEBUG Running command: /bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mds.fs.s1.ajxyaf 2021-10-25 22:47:03,256 DEBUG /bin/podman: stdout 818588b3702cbed9ff38abb07ae39e08f9700b8c8740f40f81641d0129bd8338,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25 22:45:39.314981699 +0200 CEST, -- Dr. Magnus Harlander Mail: harlan(a)harlan.de Web: www.harlan.de Stiftung: www.harlander-stiftung.de Ceterum censeo bitcoin esse delendam!

8 hours, 36 minutes

1
0
0 0

Re: jj's "improved" ceph balancer

by Jonas Jelten

Hi Josh, yes, there's many factors to optimize... which makes it kinda hard to achieve an optimal solution. I think we have to consider all these things, in ascending priority: * 1: Minimize distance to CRUSH (prefer fewest upmaps, and remove upmap items if balance is better) * 2: Relocation of PGs in remapped state (since they are not fully moved yet, hence 'easier' to relocate) * 3: Per-Pool PG distribution, respecting OSD device size -> ideal_pg_count = osd_size * (pg_num / sum(possible_osd_sizes) * 4: Primary/EC-N distribution (all osds have equal primary/EC-N counts, for workload balancing, not respecting device size (for hdd at least?), else this is just 3) * 5: Capacity balancing (all osds equally full) * 6: And of course CRUSH constraints Beautiful optimization problem, which could be fed into a solver :) My approach currently optimizes for 3, 5, 6, iteratively... > My only comment about what you did is that it should somehow work pool by pool and manage the +-1 globally. I think this is already implemented! Since each iteration I pick the "fullest" device first, it has to have more pools (or data) than other OSDs (e.g. through +1), and we try to migrate a PG off it. And we only migrate a particular PG of a pool from such a source OSD if it has >ideal_amount_for_pool (float, hence we allow moving +1s or worse). Same for a destination OSD, it's only selected if has other PGs of that pool <ideal_amount_for_pool (float, hence allowing it to become a +1 but not more). So we eliminate global imbalance, and respect equal PG distribution per pool. I can try to hack in (optional) constraints so it also supports optimization 4, but this works very much against the CRUSH placement (because we'd have to ignore OSD size). But since this is basically bypassing CRUSH-weights, it could also be done by placing all desired devices in a custom crush hierarchy with identical weighted buckets (even though "wasting" storage). Then we don't have to fight CRUSH and it's a 'simple' optimization 3 again. To achieve 2 and 1 it's just an re-ordering of candidate PGs. So in theory it should be doable™. -- Jonas On 25/10/2021 11.12, Josh Salomon wrote: > Hi Jonas, > > I want to clarify a bit my thoughts (it may be long) regarding balancing in general. > > 1 - Balancing the capacity correctly is of top priority, this is because we all know that the system is as full as the fullest device and as a storage system we can't allow large capacity which is wasted and can't be used. This is a top functional requirement. > 2 - Workload balancing is a performance requirement, and an important one, but we should not optimize workload on behalf of capacity so the challenge is how to do both simultaneously. (hint: it is not always possible, but when this is not possible the system performs less than the aggregated performance of the devices) > > Assumption 1: Per pool the workload on a PG is linear with the capacity, which means either all PGs have the same workload (#PGs is a power of 2) or some PGs has exactly twice the load as the others. From now on I will assume the number of PGs is a power of 2, since the adjustments to the other case are pretty simple. > > Conclusion 1: Balancing capacity based on all the PGs is the system may cause workload imbalance - balancing capacity should be done on a pool by pool basis. (assume 2 pools H(ot) and C(old) with exactly the same settings (#PGs, capacity and protection scheme). If you balance per PG capacity only you can have a device with all the PGs from C pool and a device with all the PGs from the H pool - > This will cause the second device to be fully loaded while the first device is idle). > > On the other hand, your point about the +-1 PGs when working on a pool by pool basis is correct and should be fixed (when working on pool by pool basis) > > When all the devices are identical, the other thing we need to do for balancing the workload is balancing the primaries (on a pool by pool basis) - this means that when the capacity is balanced (every OSD has the same number of PGs per pool) every OSD has also the same number of primaries (+-1) per pool. This is mainly important for replicated pools, for EC pools it is important (but less > critical) when working without "fast read" mode, and does not have any effect with EC pools with "fast read" mode enabled. (For EC pools we need to balance the N OSDs from N+K and not only the primaries - think about replica-3 as a special case of EC with 1+2) > > Now what happens when the devices are not identical - > In case of mixing technologies (SSD and HDD) - (this is not recommended, but you can see some use cases for this in my SDC presentation <https://www.youtube.com/watch?v=dz53aH2XggE&feature=emb_imp_woyt>) - without going into deep details the easiest solution is make all the faster (I mean much faster such as HDD/SSD or SSD/PM) devices always primaries and all the slow devices never primaries > (assuming you always keep at least one copy on a fast device). More on this in the presentation. > > The last case is when there are relatively minor performance differences between the devices (HDD with different RPM rate, or devices with the same technology and not the same size, but not a huge difference - I believe that when on device has X times the capacity as others when X > replica-count, we can't balance any more, but I need to complete my calculations). In these cases, assuming we know > something about the workload (R/W ratio) we can balance workload by giving more primaries to the faster or smaller devices relative to the slower or larger devices. This may not be optimal but can improve the performance, obviously it will not work for write only workloads, but it can improve performance as the ratio of reads is higher. > > So to summarize - we need first to balance capacity as perfectly as possible, but if we care about performance we should make sure that the capacity per each pool is balanced almost perfectly. Then we change the primaries based on the devices we have and on the workloads per pool in order to split the workload evenly among the devices. When we have large variance in the devices in the same pool, > perfect workload balancing may not be achievable, but we can try and find an optimal one for the configuration and workload we have. > > Having said all that - I really appreciate your work, and I went briefly over it. My only comment about what you did is that it should somehow work pool by pool and manage the +-1 globally. > > Regards, > > Josh

11 hours, 41 minutes

1
0
0 0

jj's "improved" ceph balancer

by Jonas Jelten

Hi! I've been working on this for quite some time now and I think it's ready for some broader testing and feedback. https://github.com/TheJJ/ceph-balancer It's an alternative standalone balancer implementation, optimizing for equal OSD storage utilization and PG placement across all pools. It doesn't change your cluster in any way, it just prints the commands you can run to apply the PG movements. Please play around with it :) Quickstart example: generate 10 PG movements on hdd to stdout ./placementoptimizer.py -v balance --max-pg-moves 10 --only-crushclass hdd | tee /tmp/balance-upmaps When there's remapped pgs (e.g. by applying the above upmaps), you can inspect progress with: ./placementoptimizer.py showremapped ./placementoptimizer.py showremapped --by-osd And you can get a nice Pool and OSD usage overview: ./placementoptimizer.py show --osds --per-pool-count --sort-utilization Of course there's many more features and optimizations to be added, but it served us very well in reclaiming terrabytes of until then unavailable storage already where the `mgr balancer` could no longer optimize. What do you think? Cheers -- Jonas

11 hours, 56 minutes

3
6
0 0

Rebooting one node immediately blocks IO via RGW

by Troels Hansen

I have a strange issue...... Its a 3 node cluster, deployed on Ubuntu, on containers, running version 15.2.4, docker.io/ceph/ceph:v15 Its only running RGW, and everything seems fine, and everyting works. No errors and the cluster is healthy. As soon as one node is restarted all IO is blocked, apparently because of slow ops, but I see no reason for it. Its running as simple as possible, with a replica count of 3. The second the OSD's on the halted node dissapears I see slow ops, but its blocking everything, and there is no IO to the cluster. The slow requests are spread accross all of the remaining OSD's. 2021-10-20T05:07:02.554282+0200 mon.prodceph-mon1 [WRN] Health check failed: 0 slow ops, oldest one blocked for 30 sec, osd.4 has slow ops (SLOW_OPS) 2021-10-20T05:07:04.652756+0200 osd.13 [WRN] slow request osd_op(client.394115.0:2994408 4.d 4:b4812045:::notify.4:head [watch ping cookie 94796974922496] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:34.010528+0000 currently delayed 2021-10-20T05:07:05.585995+0200 osd.25 [WRN] slow request osd_op(client.394158.0:62776921 7.1f3 7:cfb51b5f:::5a288701-a65a-45c0-97c7-edfb38f2f487.124110.147864_b19283e9-c7bd-448e-952d-2f172467fa5c:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e18084) initiated 2021-10-20T03:06:35.106815+0000 currently delayed 2021-10-20T05:07:05.629622+0200 osd.13 [WRN] slow request osd_op(client.394115.0:2994408 4.d 4:b4812045:::notify.4:head [watch ping cookie 94796974922496] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:34.010528+0000 currently delayed 2021-10-20T05:07:05.629660+0200 osd.13 [WRN] slow request osd_op(client.394158.0:62776924 4.d 4:b4812045:::notify.4:head [watch ping cookie 94141521019648] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:35.165999+0000 currently delayed 2021-10-20T05:07:05.629690+0200 osd.13 [WRN] slow request osd_op(client.305099.0:3244269 4.d 4:b4812045:::notify.4:head [watch ping cookie 94522369776384] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:35.402403+0000 currently delayed 2021-10-20T05:07:06.555735+0200 osd.25 [WRN] slow request osd_op(client.394158.0:62776921 7.1f3 7:cfb51b5f:::5a288701-a65a-45c0-97c7-edfb38f2f487.124110.147864_b19283e9-c7bd-448e-952d-2f172467fa5c:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e18084) initiated 2021-10-20T03:06:35.106815+0000 currently delayed 2021-10-20T05:07:06.677696+0200 osd.13 [WRN] slow request osd_op(client.394115.0:2994408 4.d 4:b4812045:::notify.4:head [watch ping cookie 94796974922496] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:34.010528+0000 currently delayed 2021-10-20T05:07:06.677732+0200 osd.13 [WRN] slow request osd_op(client.394158.0:62776924 4.d 4:b4812045:::notify.4:head [watch ping cookie 94141521019648] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:35.165999+0000 currently delayed 2021-10-20T05:07:06.677750+0200 osd.13 [WRN] slow request osd_op(client.305099.0:3244269 4.d 4:b4812045:::notify.4:head [watch ping cookie 94522369776384] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:35.402403+0000 currently delayed 2021-10-20T05:07:07.553717+0200 osd.25 [WRN] slow request osd_op(client.394158.0:62776921 7.1f3 7:cfb51b5f:::5a288701-a65a-45c0-97c7-edfb38f2f487.124110.147864_b19283e9-c7bd-448e-952d-2f172467fa5c:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e18084) initiated 2021-10-20T03:06:35.106815+0000 currently delayed 2021-10-20T05:07:07.643135+0200 osd.13 [WRN] slow request osd_op(client.394115.0:2994408 4.d 4:b4812045:::notify.4:head [watch ping cookie 94796974922496] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:34.010528+0000 currently delayed 2021-10-20T05:07:07.643159+0200 osd.13 [WRN] slow request osd_op(client.394158.0:62776924 4.d 4:b4812045:::notify.4:head [watch ping cookie 94141521019648] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:35.165999+0000 currently delayed 2021-10-20T05:07:07.643175+0200 osd.13 [WRN] slow request osd_op(client.305099.0:3244269 4.d 4:b4812045:::notify.4:head [watch ping cookie 94522369776384] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:35.402403+0000 currently delayed 2021-10-20T05:07:08.368877+0200 mon.prodceph-mon1 [WRN] Health check update: 0 slow ops, oldest one blocked for 35 sec, osd.4 has slow ops (SLOW_OPS) 2021-10-20T05:07:08.570167+0200 osd.25 [WRN] slow request osd_op(client.394158.0:62776921 7.1f3 7:cfb51b5f:::5a288701-a65a-45c0-97c7-edfb38f2f487.124110.147864_b19283e9-c7bd-448e-952d-2f172467fa5c:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e18084) initiated 2021-10-20T03:06:35.106815+0000 currently delayed 2021-10-20T05:07:08.570200+0200 osd.25 [WRN] slow request osd_op(client.394158.0:62776930 7.1f3 7:cfb51b5f:::5a288701-a65a-45c0-97c7-edfb38f2f487.124110.147864_b19283e9-c7bd-448e-952d-2f172467fa5c:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e18084) initiated 2021-10-20T03:06:38.518576+0000 currently delayed 2021-10-20T05:07:08.598671+0200 osd.13 [WRN] slow request osd_op(client.394115.0:2994408 4.d 4:b4812045:::notify.4:head [watch ping cookie 94796974922496] snapc 0=[] ondisk+write+known_if_redirected e18084) initiated 2021-10-20T03:06:34.010528+0000 currently delayed When then the node comes back up the slow ops dissapears, and IO resumes. I have tried to replicate it in a test environment using same Ceph version, as the other cluster is now running production, but not succeeded to reproduce. Any insights or ideas would be appreciated. -- Med venlig hilsen *Troels Hansen* Senior Linux Konsulent Tlf.: 22 43 71 57 tha(a)miracle.dk www.miracle.dk

13 hours

3
2
0 0

failing dkim

by mj

Hi, This is not about ceph, but about this ceph-users mailinglist. We have recently started using DKIM/DMARC/SPF etc, and since then we notice that the emails from this ceph-users mailinglist come with either a - failing DKIM signature or - no DKIM signature at all. Many of the other mailinglists I am subscribed to (like postfix, samba, sogo) generally pass the DKIM verification. Does this say something about how this particular ceph-users mailinglist is setup, or is there something we can do about it..? Sorry for being off-topic, please reply privately if this is not allowed and or appreciated here. MJ

13 hours, 1 minute

2
1
0 0

RGW/multisite sync traffic rps

by Stefan Schueffler

Hi, i have a question on RGW/multisite. The sync traffic is running a lot of requests per second (around 1500), which seems to be high, especially compared to the actual volume of user/client-requests. We have a rather simple multisite-setup with - two ceph clusters (16.2.6), 1 realm, 1 zonegroup, and one zone on each side, one of them ist the master zone. - latency between those cluster around 0.3ms - each cluster has 3 RGW/beast daemons running. - a handful of buckets (around 20), and a check script which creates one bucket per second (and deletes it after validating the successful bucket creation). - one of the buckets has a few million (smaller) objects, the others are (more or less) empty. - from the client side, there are just a few requests per second (mostly PUT objects into the one larger bucket), writing a few kilobytes per second. - roughly 5 GB in total disk size consumed currently, with the idea to increase the total consumption to a few TB over time. Both clusters are in sync (after the initial full sync, they now do incremental sync). Although they do sync the new objects from cluster A (master, to which the clients connect to) to B, we see a lot of „internal“ sync requests in our monitoring: each rgw daemon does about 500 requests per second to a rgw daemon on cluster A, especially to "/admin/log?…", which leads to a total of 1500 requests per second just for the sync, and this results in almost 60% cpu usage for the rgw/beast processes. When stopping and restarting the rgw-instances on cluster-B, it first catches up with the delta, and as soon as it finishes, it starts to request in this endless loop "/admin/log…" Is this amount of internal, sync-related requests normal and expected? Thanks for any ideas how to debug / introspect this. Best Stefan

13 hours, 14 minutes

2
2
0 0

2021

2020

2019

ceph-users