- ceph-users - lists.ceph.io

Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
by Dan van der Ster 14 Jul '20

14 Jul '20

Thanks for this info -- adding it to our list of reasons never to use FileStore again. In your case, are you able to migrate? On Tue, Jul 14, 2020 at 3:13 PM Eric Smith <Eric.Smith(a)vecima.com> wrote: > > FWIW Bluestore is not affected by this problem! > > -----Original Message----- > From: Eric Smith <Eric.Smith(a)vecima.com> > Sent: Saturday, July 11, 2020 6:40 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot > > It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues. > > Eric > > -----Original Message----- > From: Eric Smith <Eric.Smith(a)vecima.com> > Sent: Friday, July 10, 2020 1:46 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot > > For what it's worth - all of our objects are generating LONG named object files like so... > > \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long > > Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad? > > -----Original Message----- > From: Eric Smith <Eric.Smith(a)vecima.com> > Sent: Friday, July 10, 2020 6:59 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot > > I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: > > 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 > 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 > 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool > 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 > 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 > 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] > 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# > 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 > 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D > 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA > 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 > 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 > 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 > 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 > 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 > 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 > 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 > 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 > 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed > 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do > 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 > 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 > 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# > 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A > > Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Luminous 12.2.12 - filestore OSDs take an hour to boot
by Eric Smith 14 Jul '20

14 Jul '20

FWIW Bluestore is not affected by this problem! -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Saturday, July 11, 2020 6:40 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot It does appear that long file names and filestore seem to be a real problem. We have a cluster where 99% of the objects have names longer than N (220+?) characters such that it truncates the file name (as seen below with "_<sha-sum>_0_long") and stores the full object name in xattrs for the object. During boot the OSD goes out to lunch for increasing amounts of time based on the number of objects on disk you have that meet this criteria (With 2.4 million ish objects that meet this criteria, the OSD takes over an hour to boot). I plan on testing this same scenario with BlueStore to see if it's also susceptible to these boot / read issues. Eric -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 1:46 PM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Luminous 12.2.12 - filestore OSDs take an hour to boot For what it's worth - all of our objects are generating LONG named object files like so... \uABCD\ucontent.\srecording\swzdchd\u\utnda-trg-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\swzdchd\u\utpc2-tp1-1008007-wzdchd-216203706303281120-230932949-1593482400-159348660000000001\u\uwzdchd3._0bfd7c716b839cb7b3ad_0_long Does this matter? AFAICT it sees this as a long file name and has to lookup the object name in the xattrs ? Is that bad? -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Friday, July 10, 2020 6:59 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Luminous 12.2.12 - filestore OSDs take an hour to boot I have a cluster running Luminous 12.2.12 with Filestore and it takes my OSDs somewhere around an hour to start (They do start successfully - eventually). I have the following log entries that seem to show the OSD process attempting to descend into the PG directory on disk and create an object list of some sort: 2020-07-09 18:29:28.017207 7f3b680afd80 20 osd.1 137390 clearing temps in 8.14ads3_head pgid 8.14ads3 2020-07-09 18:29:28.017211 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is 8 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017213 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5020): first checking temp pool 2020-07-09 18:29:28.017215 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5012): pool is -10 shard is 3 pgid 8.14ads3 2020-07-09 18:29:28.017221 7f3b680afd80 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.017263 7f3b680afd80 20 filestore(/var/lib/ceph/osd/ceph-1) objects: [] 2020-07-09 18:29:28.017268 7f3b680afd80 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list(5028): fall through to non-temp collection, start 3#-1:00000000::::0# 2020-07-09 18:29:28.017272 7f3b680afd80 20 _collection_list_partial start:3#-1:00000000::::0# end:GHMAX-64 ls.size 0 2020-07-09 18:29:28.038124 7f3b680afd80 20 list_by_hash_bitwise prefix D 2020-07-09 18:29:28.058679 7f3b680afd80 20 list_by_hash_bitwise prefix DA 2020-07-09 18:29:28.069432 7f3b680afd80 20 list_by_hash_bitwise prefix DA4 2020-07-09 18:29:29.789598 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000074 2020-07-09 18:29:29.789634 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:29.789639 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:29.789641 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:29.789663 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:34.789815 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000109 2020-07-09 18:29:34.789898 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:34.789902 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:34.789906 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:34.789939 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:38.651689 7f3b680afd80 20 list_by_hash_bitwise prefix DA41 2020-07-09 18:29:39.790069 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000128 2020-07-09 18:29:39.790090 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:39.790092 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:39.790093 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:39.790102 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:44.790200 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000095 2020-07-09 18:29:44.790256 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:44.790265 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:44.790268 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:44.790286 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:49.790353 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(4010): woke after 5.000066 2020-07-09 18:29:49.790374 7f3b51a87700 10 journal commit_start max_applied_seq 53085082, open_ops 0 2020-07-09 18:29:49.790376 7f3b51a87700 10 journal commit_start blocked, all open_ops have completed 2020-07-09 18:29:49.790378 7f3b51a87700 10 journal commit_start nothing to do 2020-07-09 18:29:49.790387 7f3b51a87700 20 filestore(/var/lib/ceph/osd/ceph-1) sync_entry(3994): waiting for max_interval 5.000000 2020-07-09 18:29:50.564479 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 2020-07-09 18:29:50.564501 7f3b680afd80 20 list_by_hash_bitwise prefix DA410000 ob 3#8:b5280000::::head# 2020-07-09 18:29:50.564508 7f3b680afd80 20 list_by_hash_bitwise prefix DA41002A Any idea what's going on here? I can run a find of every file on the filesystem in under 12 minutes so I'm not sure what's taking so long. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

1 pg inconsistent
by Abhimnyu Dhobale 14 Jul '20

14 Jul '20

Good Day, Ceph is showing below error frequently. every time after pg repair it is resolved. [root@vpsapohmcs01 ~]# ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.574 is active+clean+inconsistent, acting [19,25,2] [root@vpsapohmcs02 ~]# cat /var/log/ceph/ceph-osd.19.log | grep error 2020-07-12 11:42:11.824 7f864e0b2700 -1 log_channel(cluster) log [ERR] : 1.574 shard 25 soid 1:2ea0a7a3:::rbd_data.515c96b8b4567.0000000000007a7c:head : candidate had a read error 2020-07-12 11:42:15.035 7f86520ba700 -1 log_channel(cluster) log [ERR] : 1.574 deep-scrub 1 errors [root@vpsapohmcs01 ~]# ceph --version ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable) Request you to please suggest. -- Thanks & Regards Abhimnyu Dhobale

4 3

v14.2.10 Nautilus crash
by Markus Binz 14 Jul '20

14 Jul '20

Hello, yesterday we upgraded a mimic cluster to v14.2.10, everything was running and ok. There was this new warning, 2 pool(s) have non-power-of-two pg_num and to get a HEALTH_OK state until we can expand this pools, i found this config option to suppress the warning: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false which resulted in a crash of 40 osd processes (about 60% of the cluster). no restart possible, always the same crash. 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -1547> 2020-06-30 21:13:51.171 7fd2b7708c00 -1 missing 'type' file, inferring filestore from current/ dir -738> 2020-06-30 21:13:56.179 7fd2b7708c00 -1 osd.30 385679 log_to_monitors {default=true} 0> 2020-06-30 21:13:56.199 7fd2a5813700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd2a5813700 thread_name:fn_odsk_fstore ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) 1: (()+0x11390) [0x7fd2b53a3390] 2: /usr/bin/ceph-osd() [0x87fd12] 3: (OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*)+0x5e1) [0x8f0f91] 4: (C_OnMapCommit::finish(int)+0x17) [0x946897] 5: (Context::complete(int)+0x9) [0x8fbfb9] 6: (Finisher::finisher_thread_entry()+0x15e) [0xeb2b8e] 7: (()+0x76ba) [0x7fd2b53996ba] 8: (clone()+0x6d) [0x7fd2b49a041d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. This is a mixed cluster of ubuntu xenial and bionic, it happens on both. It look's like, it happens when the new monmap arrived at the osd. The only fix i was able to come up with, downgrade ceph-osd to v14.2.9. Should i open a bug report? Regards Markus

3 7

missing ceph-mgr-dashboard and ceph-grafana-dashboards rpms for el7 and 14.2.10
by Joel Davidow 13 Jul '20

13 Jul '20

https://download.ceph.com/rpm-nautilus/el8/noarch/ contains ceph-mgr-dashboard-14.2.10-0.el8.noarch.rpm and ceph-grafana-dashboards-14.2.10-0.el8.noarch.rpm but there is no 14.2.10.0-el7.noarch.rpm for either ceph-mgr-dashboard or ceph-grafana-dashboards in https://download.ceph.com/rpm-nautilus/el7/noarch/. I tried using 14.2.9 rpms but there is a dependency of ceph-mgr 14.2.9 and I just installed 14.2.10 which I want to continue running. # rpm -Uvh https://download.ceph.com/rpm-nautilus/el7/noarch/ceph-mgr-dashboard-14.2.9… Retrieving https://download.ceph.com/rpm-nautilus/el7/noarch/ceph-mgr-dashboard-14.2.9… error: Failed dependencies: ceph-grafana-dashboards = 2:14.2.9-0.el7 is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch ceph-mgr = 2:14.2.9-0.el7 is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch python-jwt is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch python-routes is needed by ceph-mgr-dashboard-2:14.2.9-0.el7.noarch Is this the appropriate forum to raise this issue or should I use a different process to get the missing rpms addressed?

1 0

multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD) and CephFS
by Bobby 13 Jul '20

13 Jul '20

Hi, I have a question regarding support for multiple BLK-MQ queues for Ceph's RADOS Block Device (RBD). The below given link says that the driver has been using the BLK-MQ interface for a while but not actually multiple queues until now with having a queue per-CPU. A change to not hold onto caps that aren't actually needed. These improvements and more can be found as part of the Ceph changes for Linux 5.7, which should be released as stable in early June. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.7-Ceph-Performa… My question is: Is it possible that through Ceph FS (Filesystem in User Space) I can develop a multi-queue driver for Ceph? Asking because this way I can avoid kernel space. ( https://docs.ceph.com/docs/nautilus/start/quick-cephfs/) Looking forward for some help BR Bobby

1 0

Ceph stuck at: objects misplaced (0.064%)
by Ml Ml 13 Jul '20

13 Jul '20

Hello, ceph is stuck since 4 days with 0.064% misplaced and i dunno why. Can anyone help me to get it fixed? I did restart some OSDs and reweight them again to get some data moving but that did not help. root@node01:~ # ceph -s cluster: id: 251c937e-0b55-48c1-8f34-96e84e4023d4 health: HEALTH_WARN 1803/2799972 objects misplaced (0.064%) mon node02 is low on available space services: mon: 3 daemons, quorum node01,node02,node03 mgr: node03(active), standbys: node01, node02 osd: 16 osds: 16 up, 16 in; 1 remapped pgs data: pools: 1 pools, 512 pgs objects: 933.32k objects, 2.68TiB usage: 9.54TiB used, 5.34TiB / 14.9TiB avail pgs: 1803/2799972 objects misplaced (0.064%) 511 active+clean 1 active+clean+remapped io: client: 131KiB/s rd, 8.57MiB/s wr, 28op/s rd, 847op/s wr root@node01:~ # ceph health detail HEALTH_WARN 1803/2800179 objects misplaced (0.064%); mon node02 is low on available space OBJECT_MISPLACED 1803/2800179 objects misplaced (0.064%) MON_DISK_LOW mon node02 is low on available space mon.node02 has 28% avail root@node01:~ # ceph versions { "mon": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 3 }, "mgr": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 3 }, "osd": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 16 }, "mds": {}, "overall": { "ceph version 12.2.13 (98af9a6b9a46b2d562a0de4b09263d70aeb1c9dd) luminous (stable)": 22 } } root@node02:~ # df -h Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs 13G 1.3G 12G 11% /run /dev/sda3 46G 31G 14G 70% / tmpfs 63G 57M 63G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda1 922M 206M 653M 24% /boot /dev/fuse 30M 144K 30M 1% /etc/pve /dev/sde1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-11 /dev/sdf1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-14 /dev/sdc1 889G 676G 214G 77% /var/lib/ceph/osd/ceph-3 /dev/sdb1 889G 667G 222G 76% /var/lib/ceph/osd/ceph-2 /dev/sdd1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-7 tmpfs 13G 0 13G 0% /run/user/0 root@node02:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 14.34781 root default -2 4.25287 host node01 0 hdd 0.85999 osd.0 up 0.80005 1.00000 1 hdd 0.86749 osd.1 up 0.85004 1.00000 6 hdd 0.87270 osd.6 up 0.90002 1.00000 12 hdd 0.78000 osd.12 up 0.95001 1.00000 13 hdd 0.87270 osd.13 up 0.95001 1.00000 -3 3.91808 host node02 2 hdd 0.70000 osd.2 up 0.80005 1.00000 3 hdd 0.59999 osd.3 up 0.85004 1.00000 7 hdd 0.87270 osd.7 up 0.85004 1.00000 11 hdd 0.87270 osd.11 up 0.75006 1.00000 14 hdd 0.87270 osd.14 up 0.85004 1.00000 -4 6.17686 host node03 4 hdd 0.87000 osd.4 up 1.00000 1.00000 5 hdd 0.87000 osd.5 up 1.00000 1.00000 8 hdd 0.87270 osd.8 up 1.00000 1.00000 10 hdd 0.87270 osd.10 up 1.00000 1.00000 15 hdd 0.87270 osd.15 up 1.00000 1.00000 16 hdd 1.81879 osd.16 up 1.00000 1.00000 root@node01:~ # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE VAR PGS TYPE NAME -1 14.55780 - 14.9TiB 9.45TiB 7.46TiB 1.47GiB 23.2GiB 5.43TiB 63.52 1.00 - root default -2 4.27286 - 4.35TiB 3.15TiB 2.41TiB 486MiB 7.62GiB 1.21TiB 72.32 1.14 - host node01 0 hdd 0.85999 0.80005 888GiB 619GiB 269GiB 92.3MiB 0B 269GiB 69.72 1.10 89 osd.0 1 hdd 0.86749 0.85004 888GiB 641GiB 248GiB 109MiB 0B 248GiB 72.12 1.14 92 osd.1 6 hdd 0.87270 0.90002 894GiB 634GiB 632GiB 98.9MiB 2.65GiB 259GiB 70.99 1.12 107 osd.6 12 hdd 0.79999 0.95001 894GiB 664GiB 661GiB 94.4MiB 2.52GiB 230GiB 74.31 1.17 112 osd.12 13 hdd 0.87270 0.95001 894GiB 665GiB 663GiB 91.7MiB 2.46GiB 229GiB 74.43 1.17 112 osd.13 -3 4.10808 - 4.35TiB 3.17TiB 2.18TiB 479MiB 6.99GiB 1.18TiB 72.86 1.15 - host node02 2 hdd 0.78999 0.75006 888GiB 654GiB 235GiB 95.6MiB 0B 235GiB 73.57 1.16 94 osd.2 3 hdd 0.70000 0.80005 888GiB 737GiB 151GiB 114MiB 0B 151GiB 82.98 1.31 105 osd.3 7 hdd 0.87270 0.85004 894GiB 612GiB 610GiB 88.9MiB 2.43GiB 281GiB 68.50 1.08 103 osd.7 11 hdd 0.87270 0.75006 894GiB 576GiB 574GiB 81.8MiB 2.19GiB 317GiB 64.47 1.01 97 osd.11 14 hdd 0.87270 0.85004 894GiB 669GiB 666GiB 98.8MiB 2.37GiB 225GiB 74.85 1.18 112 osd.14 -4 6.17686 - 6.17TiB 3.13TiB 2.86TiB 541MiB 8.58GiB 3.04TiB 50.73 0.80 - host node03 4 hdd 0.87000 1.00000 888GiB 504GiB 384GiB 124MiB 0B 384GiB 56.72 0.89 72 osd.4 5 hdd 0.87000 1.00000 888GiB 520GiB 368GiB 96.2MiB 0B 368GiB 58.57 0.92 75 osd.5 8 hdd 0.87270 1.00000 894GiB 508GiB 505GiB 80.2MiB 2.07GiB 386GiB 56.80 0.89 85 osd.8 10 hdd 0.87270 1.00000 894GiB 374GiB 373GiB 51.9MiB 1.73GiB 519GiB 41.88 0.66 63 osd.10 15 hdd 0.87270 1.00000 894GiB 504GiB 502GiB 60.1MiB 1.99GiB 390GiB 56.37 0.89 84 osd.15 16 hdd 1.81879 1.00000 1.82TiB 797GiB 794GiB 129MiB 2.79GiB 1.04TiB 42.77 0.67 134 osd.16 TOTAL 14.9TiB 9.45TiB 7.46TiB 1.47GiB 23.2GiB 5.43TiB 63.52 MIN/MAX VAR: 0.66/1.31 STDDEV: 11.59 root@node02:~ # df -h Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs 13G 1.4G 12G 11% /run /dev/sda3 46G 31G 14G 70% / tmpfs 63G 63M 63G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda1 922M 206M 653M 24% /boot /dev/fuse 30M 140K 30M 1% /etc/pve /dev/sde1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-11 /dev/sdf1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-14 /dev/sdc1 889G 738G 152G 83% /var/lib/ceph/osd/ceph-3 /dev/sdb1 889G 654G 235G 74% /var/lib/ceph/osd/ceph-2 /dev/sdd1 93M 5.4M 88M 6% /var/lib/ceph/osd/ceph-7 tmpfs 13G 0 13G 0% /run/user/0 Any idea? Thanks, Michael

4 3

cephfs: creating two subvolumegroups with dedicated data pool...
by Christoph Ackermann 13 Jul '20

13 Jul '20

Hello list, i have a strange issue creating cephfs subvolumegroup with dedicated pools.... We have four fs (volumes); one is "backup" I would like to set up two subvolumegroup "veeampool" and "bareospool" each with dedicated data pools "cephfs.backup.veeam.data" and "cephfs.backup.bareos.data". Both EC42 with overwrites enabled and application cephfs. Pls see below. isceph@ceph-deploy:~$ sudo ceph fs subvolumegroup create backup veeampool --pool_layout cephfs.backup.veeam.data ...First one is ok. isceph@ceph-deploy:~$ sudo ceph fs subvolumegroup create backup bareospool --pool_layout cephfs.backup.bareos.data Error EINVAL: Invalid pool layout 'cephfs.backup.bareos.data'. It must be a valid data pool Second one throws this error... Is there only one data pool allowed per volume? Many thanks for clarification Christoph ...pool ls detail [..] pool 44 'cephfs.backup.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1132068 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 45 'cephfs.backup.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1132068 flags hashpspool stripe_width 0 application cephfs pool 46 'cephfs.backup.veeam.data' erasure profile ec-42-profile size 6 min_size 5 crush_rule 9 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 1174754 lfor 0/0/1174708 flags hashpspool,ec_overwrites stripe_width 16384 application cephfs pool 48 'cephfs.backup.bareos.data' erasure profile ec-42-profile size 6 min_size 5 crush_rule 10 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 1176205 flags hashpspool,ec_overwrites stripe_width 16384 application cephfs -- Christoph Ackermann | System Engineer INFOSERVE GmbH | Am Felsbrunnen 15 | D-66119 Saarbrücken Fon +49 (0)681 88008-59 | Fax +49 (0)681 88008-33 | mailto:C.Ackermann@infoserve.de | https://www.infoserve.de INFOSERVE Datenschutzhinweise: https://infoserve.de/datenschutz Handelsregister: Amtsgericht Saarbrücken, HRB 11001 | Erfüllungsort: Saarbrücken Geschäftsführer: Dr. Stefan Leinenbach | Ust-IdNr.: DE168970599

2 2

"task status" section in ceph -s output new?
by Rainer Krienke 13 Jul '20

13 Jul '20

Hello, today I upgraded from 14.2.9 to 14.2.10. Everything worked just fine. I think at the time I upgraded the last of my three MDS servers something new appeard in the output of ceph -s: # ceph -s cluster: id: xyz health: HEALTH_OK services: ... ... task status: scrub status: mds.myceph3: idle The "task status" section is new to me. Is it there to stay and what kind of task status messages might appear there? At the moment it displays information about ceph fs scrub status telling me nothig is going on here. Since I recently started using cephfs reading the message I started asking myself if I should scrub cephfs regularly or does ceph do this on its own or is scrubbing only needed in case cephs has been damaged? Does anyone know more about this? Thanks for your help Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312

2 1

ceph fs resize
by hw 13 Jul '20

13 Jul '20

Hi! My ceph version is ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) I have ceph fs and I add new osd to my cluster. ceph pg stat: 289 pgs: 1 active+clean+scrubbing+deep, 288 active+clean; 873 GiB data, 2.7 TiB used, 3.5 TiB / 6.2 TiB avail How I can extend my ceph fs from 3.5 TiB to 6.2 TiB avail Detail information: ceph fs status static - 2 clients ====== RANK STATE MDS ACTIVITY DNS INOS 0 active static.ceph02.sgpdiv Reqs: 0 /s 136k 128k POOL TYPE USED AVAIL static_metadata metadata 10.1G 973G static data 2754G 973G STANDBY MDS static.ceph05.aylgvy static.ceph04.wsljnw MDS version: ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE device_health_metrics 896.1k 3.0 6399G 0.0000 1.0 1 on static 874.0G 3.0 6399G 0.4097 1.0 256 on static_metadata 3472M 3.0 6399G 0.0016 4.0 32 on ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 6.2 TiB 3.5 TiB 2.7 TiB 2.7 TiB 43.55 TOTAL 6.2 TiB 3.5 TiB 2.7 TiB 2.7 TiB 43.55 --- POOLS --- POOL ID STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 896 KiB 20 2.6 MiB 0 973 GiB static 14 874 GiB 1.44M 2.7 TiB 48.54 973 GiB static_metadata 15 3.4 GiB 2.53M 10 GiB 0.35 973 GiB ceph fs volume ls [ { "name": "static" } ] ceph fs subvolume ls static []

2 1