April 2020 - ceph-users - lists.ceph.io

PG deep-scrub does not finish
by Andras Pataki 21 Apr '20

21 Apr '20

On a cluster running Nautilus (14.2.8), we are getting a complaint about a PG not being deep-scrubbed on time. Looking at the primary OSD's logs, it looks like it tries to deep-scrub the PG every hour or so, emits some complaints that I don't understand, but the deep scrub does not finish (either with or without a scrub error). Here is the PG from pg dump: 1.43f 31794 0 0 0 0 66930087214 0 0 3004 3004 active+clean+scrubbing+deep 2020-04-20 04:48:13.055481 46286'483734 46286:563439 [354,694,851] 354 [354,694,851] 354 39594'348643 2020-04-10 12:39:16.261088 38482'314644 2020-04-04 10:37:03.121638 0 Here is a section of the primary OSD logs (osd.354): 2020-04-18 08:21:08.322 7fffd2e2b700 0 log_channel(cluster) log [DBG] : 1.43f deep-scrub starts 2020-04-18 08:37:53.362 7fffd2e2b700 1 osd.354 pg_epoch: 45910 pg[1.43f( v 45909'449615 (45801'446525,45909'449615] local-lis/les=45908/45909 n=30862 ec=874/871 lis/c 45908/45908 les/c/f 45909/45909/0 45910/45910/42988) [354,851] r=0 lpr=45910 pi=[45908,45910)/1 luod=0'0 crt=45909'449615 lcod 45909'449614 mlcod 0'0 active+scrubbing+deep mbc={}] start_peering_interval up [354,694,851] -> [354,851], acting [354,694,851] -> [354,851], acting_primary 354 -> 354, up_primary 354 -> 354, role 0 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-04-18 08:37:53.362 7fffd2e2b700 1 osd.354 pg_epoch: 45910 pg[1.43f( v 45909'449615 (45801'446525,45909'449615] local-lis/les=45908/45909 n=30862 ec=874/871 lis/c 45908/45908 les/c/f 45909/45909/0 45910/45910/42988) [354,851] r=0 lpr=45910 pi=[45908,45910)/1 crt=45909'449615 lcod 45909'449614 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-04-18 08:38:01.002 7fffd2e2b700 1 osd.354 pg_epoch: 45912 pg[1.43f( v 45909'449615 (45801'446525,45909'449615] local-lis/les=45910/45911 n=30862 ec=874/871 lis/c 45910/45908 les/c/f 45911/45909/0 45912/45912/42988) [354,694,851] r=0 lpr=45912 pi=[45908,45912)/1 luod=0'0 crt=45909'449615 lcod 45909'449614 mlcod 0'0 active mbc={}] start_peering_interval up [354,851] -> [354,694,851], acting [354,851] -> [354,694,851], acting_primary 354 -> 354, up_primary 354 -> 354, role 0 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-04-18 08:38:01.002 7fffd2e2b700 1 osd.354 pg_epoch: 45912 pg[1.43f( v 45909'449615 (45801'446525,45909'449615] local-lis/les=45910/45911 n=30862 ec=874/871 lis/c 45910/45908 les/c/f 45911/45909/0 45912/45912/42988) [354,694,851] r=0 lpr=45912 pi=[45908,45912)/1 crt=45909'449615 lcod 45909'449614 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-04-18 08:40:04.219 7fffd2e2b700 0 log_channel(cluster) log [DBG] : 1.43f deep-scrub starts 2020-04-18 08:56:49.095 7fffd2e2b700 1 osd.354 pg_epoch: 45914 pg[1.43f( v 45913'449735 (45812'446725,45913'449735] local-lis/les=45912/45913 n=30868 ec=874/871 lis/c 45912/45912 les/c/f 45913/45913/0 45914/45914/42988) [354,851] r=0 lpr=45914 pi=[45912,45914)/1 luod=0'0 crt=45913'449735 lcod 45913'449734 mlcod 0'0 active+scrubbing+deep mbc={}] start_peering_interval up [354,694,851] -> [354,851], acting [354,694,851] -> [354,851], acting_primary 354 -> 354, up_primary 354 -> 354, role 0 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-04-18 08:56:49.095 7fffd2e2b700 1 osd.354 pg_epoch: 45914 pg[1.43f( v 45913'449735 (45812'446725,45913'449735] local-lis/les=45912/45913 n=30868 ec=874/871 lis/c 45912/45912 les/c/f 45913/45913/0 45914/45914/42988) [354,851] r=0 lpr=45914 pi=[45912,45914)/1 crt=45913'449735 lcod 45913'449734 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-04-18 08:56:55.627 7fffd2e2b700 1 osd.354 pg_epoch: 45916 pg[1.43f( v 45913'449735 (45812'446725,45913'449735] local-lis/les=45914/45915 n=30868 ec=874/871 lis/c 45914/45912 les/c/f 45915/45913/0 45916/45916/42988) [354,694,851] r=0 lpr=45916 pi=[45912,45916)/1 luod=0'0 crt=45913'449735 lcod 45913'449734 mlcod 0'0 active mbc={}] start_peering_interval up [354,851] -> [354,694,851], acting [354,851] -> [354,694,851], acting_primary 354 -> 354, up_primary 354 -> 354, role 0 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-04-18 08:56:55.627 7fffd2e2b700 1 osd.354 pg_epoch: 45916 pg[1.43f( v 45913'449735 (45812'446725,45913'449735] local-lis/les=45914/45915 n=30868 ec=874/871 lis/c 45914/45912 les/c/f 45915/45913/0 45916/45916/42988) [354,694,851] r=0 lpr=45916 pi=[45912,45916)/1 crt=45913'449735 lcod 45913'449734 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-04-18 08:56:56.867 7fffd2e2b700 0 log_channel(cluster) log [DBG] : 1.43f deep-scrub starts 2020-04-18 09:13:37.419 7fffd2e2b700 1 osd.354 pg_epoch: 45918 pg[1.43f( v 45917'449808 (45812'446725,45917'449808] local-lis/les=45916/45917 n=30876 ec=874/871 lis/c 45916/45916 les/c/f 45917/45917/0 45918/45918/42988) [354,851] r=0 lpr=45918 pi=[45916,45918)/1 luod=0'0 crt=45917'449808 lcod 45917'449807 mlcod 0'0 active+scrubbing+deep mbc={}] start_peering_interval up [354,694,851] -> [354,851], acting [354,694,851] -> [354,851], acting_primary 354 -> 354, up_primary 354 -> 354, role 0 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-04-18 09:13:37.419 7fffd2e2b700 1 osd.354 pg_epoch: 45918 pg[1.43f( v 45917'449808 (45812'446725,45917'449808] local-lis/les=45916/45917 n=30876 ec=874/871 lis/c 45916/45916 les/c/f 45917/45917/0 45918/45918/42988) [354,851] r=0 lpr=45918 pi=[45916,45918)/1 crt=45917'449808 lcod 45917'449807 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-04-18 09:13:44.959 7fffd2e2b700 1 osd.354 pg_epoch: 45920 pg[1.43f( v 45917'449808 (45812'446725,45917'449808] local-lis/les=45918/45919 n=30876 ec=874/871 lis/c 45918/45916 les/c/f 45919/45917/0 45920/45920/42988) [354,694,851] r=0 lpr=45920 pi=[45916,45920)/1 luod=0'0 crt=45917'449808 lcod 45917'449807 mlcod 0'0 active mbc={}] start_peering_interval up [354,851] -> [354,694,851], acting [354,851] -> [354,694,851], acting_primary 354 -> 354, up_primary 354 -> 354, role 0 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-04-18 09:13:44.959 7fffd2e2b700 1 osd.354 pg_epoch: 45920 pg[1.43f( v 45917'449808 (45812'446725,45917'449808] local-lis/les=45918/45919 n=30876 ec=874/871 lis/c 45918/45916 les/c/f 45919/45917/0 45920/45920/42988) [354,694,851] r=0 lpr=45920 pi=[45916,45920)/1 crt=45917'449808 lcod 45917'449807 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary I've tried restarting all 3 OSDs in question, it didn't help. I'm running the upmap balancer on this cluster (if that matters). Any ideas? Andras

2 3

Rebuilding the Ceph.io site with Jekyll
by Lars Marowsky-Bree 21 Apr '20

21 Apr '20

Hi all, as part of the Ceph Foundation, we're considering to re-launch the Ceph website and migrate it away from a dated WordPress to Jekyll, backed by Git et al. (Either hosted on our own infrastructure or even GitHub pages.) This would involve building/customizing a Jekyll theme, providing feedback on the site structure proposal and usability, migrating content (where appropriate) from the existing site, and working with the Ceph infra team on getting it hosted/deployed. Some help with improving the design would be welcome. Content creation isn't necessarily part of the requirements, but working with stakeholders on filling in blanks is; and if we could get someone savvy with Ceph who wants to fill in a few pages, that's a plus! After the launch, we should be mostly self-sufficient again for day-to-day tasks. If that's the kind of contract work you or a friend is interested in, please reach out to me. (The Foundation hasn't yet approved the budget, we're still trying to get a feeling for the funding required. But I'd be fairly optimistic.) Regards, Lars -- SUSE Software Solutions Germany GmbH, MD: Felix Imendörffer, HRB 36809 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)

1 0

Re: some ceph general questions about the design
by Anthony D'Atri 21 Apr '20

21 Apr '20

> > 1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd? > > If your OSD servers have HDDs, buy a good RAID Controller with a battery-backed write cache and configure it using multiple RAID-0 volumes (1 physical disk per volume). That way, reads and write will be accelerated by the cache on the HBA. I’ve lived this scenario and hated it. Multiple firmware and manufacturing issues, batteries/supercaps can fail and need to be monitored, bugs causing staged data to be lost before writing to disk, another bug that required replacing the card if there was preserved cache for a failed drive, because it would refuse to boot, difficulties in drive monitoring, HBA monitoring utility that would lock the HBA or peg the CPU, the list goes on. For the additional cost of RoC, cache RAM, supercap to (fingers crossed) protect the cache, all the additional monitoring and hands work … you might find that SATA SSDs on a JBOD HBA are no more expensive. > 3. if i have a 3 physically node osd cluster, did i need 5 physicall mons? > No. 3 MON are enough If you have good hands and spares. If your cluster is on a different continent and colo hands can’t find their own butts ….. it’s nice to survive a double failure. ymmv

2 1

Nautilus cluster damaged + crashing OSDs
by Robert Sander 21 Apr '20

21 Apr '20

Hi, one of our customers had his Ceph cluster crashed due to a power or network outage (they still try to figure out what happened). The cluster is very unhealthy but recovering: # ceph -s cluster: id: 1c95ca5d-948b-4113-9246-14761cb9a82a health: HEALTH_ERR 1 filesystem is degraded 1 mds daemon damaged 1 osds down 1 pools have many more objects per pg than average 1/115117480 objects unfound (0.000%) Reduced data availability: 71 pgs inactive, 53 pgs down, 18 pgs peering, 27 pgs stale Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 7303464/230234960 objects degraded (3.172%), 693 pgs degraded, 945 pgs undersized 14 daemons have recently crashed services: mon: 3 daemons, quorum maslxlabstore01,maslxlabstore02,maslxlabstore04 (age 64m) mgr: maslxlabstore01(active, since 69m), standbys: maslxlabstore03, maslxlabstore02, maslxlabstore04 mds: cephfs:2/3 {0=maslxlabstore03=up:resolve,1=maslxlabstore01=up:resolve} 2 up:standby, 1 damaged osd: 140 osds: 130 up (since 4m), 131 in (since 4m); 847 remapped pgs rgw: 4 daemons active (maslxlabstore01.rgw0, maslxlabstore02.rgw0, maslxlabstore03.rgw0, maslxlabstore04.rgw0) data: pools: 6 pools, 8328 pgs objects: 115.12M objects, 218 TiB usage: 425 TiB used, 290 TiB / 715 TiB avail pgs: 0.853% pgs not active 7303464/230234960 objects degraded (3.172%) 13486/230234960 objects misplaced (0.006%) 1/115117480 objects unfound (0.000%) 7311 active+clean 338 active+undersized+degraded+remapped+backfill_wait 255 active+undersized+degraded+remapped+backfilling 215 active+undersized+remapped+backfilling 99 active+undersized+degraded 44 down 37 active+undersized+remapped+backfill_wait 13 stale+peering 9 stale+down 5 stale+remapped+peering 1 active+recovery_unfound+undersized+degraded+remapped 1 active+clean+remapped io: client: 168 B/s rd, 0 B/s wr, 0 op/s rd, 0 op/s wr recovery: 1.9 GiB/s, 15 keys/s, 948 objects/s The MDS cluster is unable to start because one of them is damaged. 10 of the OSDs do not start. They crash very early in the boot process: 2020-04-20 16:26:14.935 7f818ec8cc00 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-04-20 16:26:14.935 7f818ec8cc00 0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable), process ceph-osd, pid 69463 2020-04-20 16:26:14.935 7f818ec8cc00 0 pidfile_write: ignore empty --pid-file 2020-04-20 16:26:15.503 7f818ec8cc00 0 starting osd.42 osd_data /var/lib/ceph/osd/ceph-42 /var/lib/ceph/osd/ceph-42/journal 2020-04-20 16:26:15.523 7f818ec8cc00 0 load: jerasure load: lrc load: isa 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compression = kNoCompression 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compression = kNoCompression 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compression = kNoCompression 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compression = kNoCompression 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 2020-04-20 16:26:18.131 7f818ec8cc00 0 _get_class not permitted to load lua 2020-04-20 16:26:18.131 7f818ec8cc00 0 _get_class not permitted to load kvs 2020-04-20 16:26:18.131 7f818ec8cc00 0 _get_class not permitted to load sdk 2020-04-20 16:26:18.131 7f818ec8cc00 0 <cls> /build/ceph-14.2.9/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2020-04-20 16:26:18.131 7f818ec8cc00 0 <cls> /build/ceph-14.2.9/src/cls/hello/cls_hello.cc:296: loading cls_hello 2020-04-20 16:26:18.131 7f818ec8cc00 0 osd.42 6008 crush map has features 288514051259236352, adjusting msgr requires for clients 2020-04-20 16:26:18.131 7f818ec8cc00 0 osd.42 6008 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons 2020-04-20 16:26:18.131 7f818ec8cc00 0 osd.42 6008 crush map has features 3314933000852226048, adjusting msgr requires for osds 2020-04-20 16:26:22.023 7f818ec8cc00 0 osd.42 6008 load_pgs 2020-04-20 16:26:22.499 7f818ec8cc00 0 osd.42 6008 load_pgs opened 109 pgs 2020-04-20 16:26:22.499 7f818ec8cc00 0 osd.42 6008 using weightedpriority op queue with priority op cut off at 64. 2020-04-20 16:26:22.499 7f818ec8cc00 -1 osd.42 6008 log_to_monitors {default=true} 2020-04-20 16:26:22.511 7f818ec8cc00 0 osd.42 6008 done with init, starting boot process 2020-04-20 16:26:23.883 7f815331c700 -1 /build/ceph-14.2.9/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f815331c700 time 2020-04-20 16:26:23.884183 /build/ceph-14.2.9/src/osd/PGLog.cc: 368: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail) ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564fbd9349d2] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad] 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930] 4: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4] 5: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47] 6: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6] 7: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21] 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3] 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227] 10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90] 16: (()+0x76db) [0x7f818c8666db] 17: (clone()+0x3f) [0x7f818b60688f] 2020-04-20 16:26:23.887 7f815331c700 -1 *** Caught signal (Aborted) ** in thread 7f815331c700 thread_name:tp_osd_tp ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (()+0x12890) [0x7f818c871890] 2: (gsignal()+0xc7) [0x7f818b523e97] 3: (abort()+0x141) [0x7f818b525801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x564fbd934a23] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad] 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930] 7: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4] 8: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47] 9: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6] 10: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21] 11: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3] 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227] 13: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12] 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54] 15: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc] 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90] 19: (()+0x76db) [0x7f818c8666db] 20: (clone()+0x3f) [0x7f818b60688f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -105> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command assert hook 0x564fc83ae510 -104> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command abort hook 0x564fc83ae510 -103> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perfcounters_dump hook 0x564fc83ae510 -102> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command 1 hook 0x564fc83ae510 -101> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perf dump hook 0x564fc83ae510 -100> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perfcounters_schema hook 0x564fc83ae510 -99> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perf histogram dump hook 0x564fc83ae510 -98> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command 2 hook 0x564fc83ae510 -97> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perf schema hook 0x564fc83ae510 -96> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perf histogram schema hook 0x564fc83ae510 -95> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command perf reset hook 0x564fc83ae510 -94> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config show hook 0x564fc83ae510 -93> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config help hook 0x564fc83ae510 -92> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config set hook 0x564fc83ae510 -91> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config unset hook 0x564fc83ae510 -90> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config get hook 0x564fc83ae510 -89> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config diff hook 0x564fc83ae510 -88> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command config diff get hook 0x564fc83ae510 -87> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command log flush hook 0x564fc83ae510 -86> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command log dump hook 0x564fc83ae510 -85> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command log reopen hook 0x564fc83ae510 -84> 2020-04-20 16:26:14.923 7f818ec8cc00 5 asok(0x564fc8472000) register_command dump_mempools hook 0x564fc9076068 -83> 2020-04-20 16:26:14.935 7f818ec8cc00 0 set uid:gid to 64045:64045 (ceph:ceph) -82> 2020-04-20 16:26:14.935 7f818ec8cc00 0 ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable), process ceph-osd, pid 69463 -81> 2020-04-20 16:26:14.935 7f818ec8cc00 0 pidfile_write: ignore empty --pid-file -80> 2020-04-20 16:26:15.503 7f818ec8cc00 0 starting osd.42 osd_data /var/lib/ceph/osd/ceph-42 /var/lib/ceph/osd/ceph-42/journal -79> 2020-04-20 16:26:15.523 7f818ec8cc00 0 load: jerasure load: lrc load: isa -78> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB -77> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel -76> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 -75> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compression = kNoCompression -74> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 -73> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 -72> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 -71> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 -70> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 -69> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 -68> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 -67> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 -66> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 -65> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 -64> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 -63> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 -62> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB -61> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel -60> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 -59> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option compression = kNoCompression -58> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 -57> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 -56> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 -55> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 -54> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 -53> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 -52> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 -51> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 -50> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 -49> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 -48> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 -47> 2020-04-20 16:26:16.339 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 -46> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB -45> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel -44> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 -43> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compression = kNoCompression -42> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 -41> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 -40> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 -39> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 -38> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 -37> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 -36> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 -35> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 -34> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 -33> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 -32> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 -31> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 -30> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_readahead_size = 2MB -29> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_style = kCompactionStyleLevel -28> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compaction_threads = 32 -27> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option compression = kNoCompression -26> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option flusher_threads = 8 -25> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_file_num_compaction_trigger = 8 -24> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_slowdown_writes_trigger = 32 -23> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option level0_stop_writes_trigger = 64 -22> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_background_compactions = 31 -21> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_base = 536870912 -20> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_bytes_for_level_multiplier = 8 -19> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option max_write_buffer_number = 32 -18> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option min_write_buffer_number_to_merge = 2 -17> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option recycle_log_file_num = 32 -16> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option target_file_size_base = 67108864 -15> 2020-04-20 16:26:17.731 7f818ec8cc00 0 set rocksdb option write_buffer_size = 67108864 -14> 2020-04-20 16:26:18.131 7f818ec8cc00 0 _get_class not permitted to load lua -13> 2020-04-20 16:26:18.131 7f818ec8cc00 0 _get_class not permitted to load kvs -12> 2020-04-20 16:26:18.131 7f818ec8cc00 0 _get_class not permitted to load sdk -11> 2020-04-20 16:26:18.131 7f818ec8cc00 0 <cls> /build/ceph-14.2.9/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs -10> 2020-04-20 16:26:18.131 7f818ec8cc00 0 <cls> /build/ceph-14.2.9/src/cls/hello/cls_hello.cc:296: loading cls_hello -9> 2020-04-20 16:26:18.131 7f818ec8cc00 0 osd.42 6008 crush map has features 288514051259236352, adjusting msgr requires for clients -8> 2020-04-20 16:26:18.131 7f818ec8cc00 0 osd.42 6008 crush map has features 288514051259236352 was 8705, adjusting msgr requires for mons -7> 2020-04-20 16:26:18.131 7f818ec8cc00 0 osd.42 6008 crush map has features 3314933000852226048, adjusting msgr requires for osds -6> 2020-04-20 16:26:22.023 7f818ec8cc00 0 osd.42 6008 load_pgs -5> 2020-04-20 16:26:22.499 7f818ec8cc00 0 osd.42 6008 load_pgs opened 109 pgs -4> 2020-04-20 16:26:22.499 7f818ec8cc00 0 osd.42 6008 using weightedpriority op queue with priority op cut off at 64. -3> 2020-04-20 16:26:22.499 7f818ec8cc00 -1 osd.42 6008 log_to_monitors {default=true} -2> 2020-04-20 16:26:22.511 7f818ec8cc00 0 osd.42 6008 done with init, starting boot process -1> 2020-04-20 16:26:23.883 7f815331c700 -1 /build/ceph-14.2.9/src/osd/PGLog.cc: In function 'void PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 7f815331c700 time 2020-04-20 16:26:23.884183 /build/ceph-14.2.9/src/osd/PGLog.cc: 368: FAILED ceph_assert(log.head >= olog.tail && olog.head >= log.tail) ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564fbd9349d2] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad] 3: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930] 4: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4] 5: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47] 6: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6] 7: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21] 8: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3] 9: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227] 10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90] 16: (()+0x76db) [0x7f818c8666db] 17: (clone()+0x3f) [0x7f818b60688f] 0> 2020-04-20 16:26:23.887 7f815331c700 -1 *** Caught signal (Aborted) ** in thread 7f815331c700 thread_name:tp_osd_tp ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) 1: (()+0x12890) [0x7f818c871890] 2: (gsignal()+0xc7) [0x7f818b523e97] 3: (abort()+0x141) [0x7f818b525801] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x564fbd934a23] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x564fbd934bad] 6: (PGLog::merge_log(pg_info_t&, pg_log_t&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x1cc0) [0x564fbdaff930] 7: (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_shard_t)+0x64) [0x564fbda4eca4] 8: (PG::proc_master_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, pg_missing_set<false>&, pg_shard_t)+0x97) [0x564fbda7fe47] 9: (PG::RecoveryState::GetLog::react(PG::RecoveryState::GotLog const&)+0xa6) [0x564fbda9d4f6] 10: (boost::statechart::simple_state<PG::RecoveryState::GetLog, PG::RecoveryState::Peering, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x191) [0x564fbdaf0e21] 11: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_queued_events()+0xb3) [0x564fbdabdfc3] 12: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x87) [0x564fbdabe227] 13: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x122) [0x564fbdaada12] 14: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x564fbd9e1f54] 15: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x564fbdc710c0] 16: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x564fbd9d5995] 17: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x564fbdfdb8cc] 18: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fbdfdea90] 19: (()+0x76db) [0x7f818c8666db] 20: (clone()+0x3f) [0x7f818b60688f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 0 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 0/ 0 mds 0/ 0 mds_balancer 0/ 0 mds_locker 0/ 0 mds_log 0/ 0 mds_log_expire 0/ 0 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 0 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 0 rbd_mirror 0/ 0 rbd_replay 0/ 0 journaler 0/ 0 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 0/ 0 crypto 0/ 0 finisher 0/ 0 reserver 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/ 5 rgw_sync 0/ 0 civetweb 0/ 0 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 0/ 0 xio 0/ 0 compressor 0/ 0 bluestore 0/ 0 bluefs 0/ 0 bdev 0/ 0 kstore 0/ 0 rocksdb 0/ 0 leveldb 0/ 0 memdb 0/ 0 kinetic 0/ 0 fuse 0/ 0 mgr 0/ 0 mgrc 0/ 0 dpdk 0/ 0 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.42.log --- end dump of recent events --- It would be nice if anybody could give me a hint on where to look further. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

5 9

ceph-ansible osd sizing and configuration
by Hemant Sonawane 21 Apr '20

21 Apr '20

Hi All I am facing some issue related to osd config in ceph-ansible I have 2 nvme disks where i have wal and db configure and i configure lvm like this in osd.yamL osd.yaml devices: - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf - /dev/sdg - /dev/sdh - /dev/sdi - /dev/sdj - /dev/sdk - /dev/sdl - /dev/sdm lvm_volumes: - data: /dev/sdb wal: wal-lv1 wal_vg: vg1 db: journal-lv1 db_vg: vg1 - data: /dev/sdb wal: wal-lv2 wal_vg: vg1 db: journal-lv2 db_vg: vg1 - data: /dev/sdb wal: wal-lv3 wal_vg: vg1 db: journal-lv3 db_vg: vg1 - data: /dev/sdb wal: wal-lv4 wal_vg: vg1 db: journal-lv4 db_vg: vg1 - data: /dev/sdb wal: wal-lv5 wal_vg: vg1 db: journal-lv5 db_vg: vg1 - data: /dev/sdc wal: wal-lv1 wal_vg: vg2 db: journal-lv1 db_vg: vg2 - data: /dev/sdc wal: wal-lv2 wal_vg: vg2 db: journal-lv2 db_vg: vg2 - data: /dev/sdc wal: wal-lv3 wal_vg: vg2 db: journal-lv3 db_vg: vg2 - data: /dev/sdc wal: wal-lv4 wal_vg: vg2 db: journal-lv4 db_vg: vg2 - data: /dev/sdc wal: wal-lv5 wal_vg: vg2 db: journal-lv5 db_vg: vg2 - data: /dev/sdd - data: /dev/sde - data: /dev/sdf - data: /dev/sdg - data: /dev/sdh - data: /dev/sdi - data: /dev/sdj - data: /dev/sdk - data: /dev/sdl - data: /dev/sdm I have also setup wal size and db size respectively in all.yaml all.yaml ntp_service_enabled: false mon_group_name: mons osd_group_name: osds mgr_group_name: mgrs ceph_origin: repository ceph_repository: community ceph_stable_release: octopus public_network: 10.18.0.0/22 cluster_network: 10.25.0.0/24 monitor_interface: ens3 configure_firewall: False ceph_test: true cephx: true ip_version: ipv4 osd_objectstore: bluestore dashboard_enabled: False grafana_server_group_name: grafana-server ceph_conf_overrides: global: bluestore block db size: 85899345920 (80GB) bluestore_block_wal_size: 1073741824 (1GB) i have also tried to to set values of db_size and journal_size in OSD options journal_size: 1024 # OSD journal size in MB block_db_size: 85899345920 But my osd partitions are not creating as I want and taking full disk space sdb 8:16 0 447.1G 0 disk └─ceph--3e81a883--86cd--4ddb--b0b0--be5892f6afe8-osd--block--b981ea8d--410d--4e2f--86f5--509112ff5ebf 253:0 0 447.1G 0 lvm sdc 8:32 0 447.1G 0 disk └─ceph--d3d82887--a7ad--40a4--9c3d--440cd9057fd9-osd--block--de6cae7d--daef--4144--a4bc--077f6044ef22 253:1 0 447.1G 0 lvm sdd 8:48 0 5.5T 0 disk └─ceph--e5aa3641--263c--45df--90d0--cd7e26772e69-osd--block--989fd7b2--d7b7--4170--8b33--cf4c8a29f5c1 253:2 0 5.5T 0 lvm sde 8:64 0 5.5T 0 disk └─ceph--286034b5--1c55--4fb6--8983--4ec6ef482dbd-osd--block--7cdc5ef4--8a88--4ff8--a08e--278985f4764c 253:3 0 5.5T 0 lvm sdf 8:80 0 5.5T 0 disk └─ceph--3bd8b866--b99e--41c4--a993--d69e85c1b658-osd--block--d32e42f4--1133--4c0d--aaf6--821b724fd220 253:4 0 5.5T 0 lvm sdg 8:96 0 5.5T 0 disk └─ceph--972ae2d1--2098--436f--a448--9015edfc4dc5-osd--block--79411ef4--2469--4243--9545--89cd266719f1 253:5 0 5.5T 0 lvm sdh 8:112 0 5.5T 0 disk └─ceph--92cd1c5f--3c67--420e--8a4d--c9ab0b75705b-osd--block--e643c7b3--cb10--49ab--86a8--31b60f4525dc 253:6 0 5.5T 0 lvm sdi 8:128 0 5.5T 0 disk └─ceph--0a27a900--0e81--4ef9--9b2c--692d0a609e7e-osd--block--30eddf48--5e12--4c2f--aed0--e06fb403ed33 253:7 0 5.5T 0 lvm sdj 8:144 0 5.5T 0 disk └─ceph--90156988--e0a3--4699--ae18--ad1f743521bb-osd--block--eb879f94--f289--4dd9--ba2e--48ff06a14708 253:8 0 5.5T 0 lvm sdk 8:160 0 5.5T 0 disk └─ceph--cdcabf84--e713--42c2--b2f0--f1bd6bd54924-osd--block--067418cf--1bd3--46a3--bdb9--b23216d6e8e3 253:9 0 5.5T 0 lvm sdl 8:176 0 5.5T 0 disk └─ceph--60829d44--ca66--44b3--b4be--05e8fe0c154e-osd--block--49a34fa3--c8ba--4fd9--ba3e--a6c4a889ecd5 253:10 0 5.5T 0 lvm sdm 8:192 0 5.5T 0 disk └─ceph--bd0a95e3--4d83--4c9e--a5fe--9f28414015fd-osd--block--abf5c032--eaf8--47b4--9966--67526009f68f 253:11 0 5.5T 0 lvm so after running playbook i get this result of osds with error failed: [10.11.2.202] (item={u'wal_vg': u'vg2', u'data': u'/dev/sdc', u'wal': u'wal-lv5', u'db': u'journal-lv5', u'db_vg': u'vg2'}) => changed=true ansible_loop_var: item cmd: - ceph-volume - --cluster - ceph - lvm - create - --bluestore - --data - /dev/sdc - --block.db - vg2/journal-lv5 - --block.wal - vg2/wal-lv5 delta: '0:00:03.260879' end: '2020-04-20 15:49:51.407784' item: data: /dev/sdc db: journal-lv5 db_vg: vg2 wal: wal-lv5 wal_vg: vg2 msg: non-zero return code rc: 1 start: '2020-04-20 15:49:48.146905' stderr: |- Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b26655b0-0859-4f8b-9471-0bafddfc36ef Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-b26655b0-0859-4f8b-9471-0bafddfc36ef ceph-b7e18568-6735-4a1a-9aa1-95094189e859 stderr: Calculated size of logical volume is 0 extents. Needs to be larger. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 77, in main self.create(args) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 246, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 327, in prepare block_lv = self.prepare_data_device('block', osd_fsid) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 223, in prepare_data_device **kwargs) File "/usr/lib/python3/dist-packages/ceph_volume/api/lvm.py", line 1185, in create_lv process.run(command) File "/usr/lib/python3/dist-packages/ceph_volume/process.py", line 153, in run raise RuntimeError(msg) RuntimeError: command returned non-zero exit status: 5 So some one could please help me to figure solution for this issue . May be i am doing somewhere something wrong. If yes then please tell me. Also how sizing can be done for osds i mean how we can allow ceph-ansible to not to use default sizing? -- Thanks and Regards, Hemant Sonawane

1 0

some ceph general questions about the design
by harald.freidhof＠gmail.com 21 Apr '20

21 Apr '20

Hello together, we want to create a productive ceph storage system in our datacenter in may this year with openstack and ucs and i tested a lot in my cep test enviroment, and i have some general questions. whats receommended? 1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd? 2. if i have a 2 pyhsicaly node osd cluster, did i need 3 physicall mons? 3. if i have a 3 physically node osd cluster, did i need 5 physicall mons? 3. where i should in install the mgr? on osd or mon 4. where i should in install the rgw? on osd or mon OR on 1 or 2 separate machines? in my testlab i created 3 VMs osds with mgr installed, and 5 VMs mons , and 1 VM as rgw -> is this correct? thx in advance hfreidhof

5 5

rbd device name reuse frequency
by Void Star Nill 21 Apr '20

21 Apr '20

Hello, How frequently do RBD device names get reused? For instance, when I map a volume on a client and it gets mapped to /dev/rbd0 and when it is unmapped, does a subsequent map reuse this name right away? I ask this question, because in our use case, we try to unmap a volume and we are thinking about adding some retries in case unmap fails for any reason. But I am concerned about race conditions such as the following: 1. thread 1 calls unmap, but the call times out and returns in the process, but in the background unmap request does go through and the device gets removed 2. thread 1 does a retry based on the device name. If between 1 and 2, another thread tries to map another volume and if it gets mapped to same device right after previous unmap was successful, then in step 2, we will be trying to unmap a device that doesn't belong to previous map. So I want to know how frequently do the device names get reused, and if there is a way to keep them using new names until they round back after a max limit. Thanks, Shridhar

4 5

How to migrate ceph-xattribs?
by Frank Schilder 20 Apr '20

20 Apr '20

De all, we are in the process of migrating a ceph file system from a 2-pool layout (rep meta+ec data) to the recently recommended 3-pool layout (rep meta, per primary data, ec data). As part of this, we need to migrate any ceph xattribs set on files and directories. As these are no longer discoverable, how would one go about this? Special cases: How to migrate quota settings? How to migrate dir- and file-layouts? Ideally, at least quota attributes should be transferable on the fly with tools like rsync. If automatic migration is not possible, is there at least an efficient way to *find* everything with special ceph attributes? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 5

ceph-ansible osd sizing and configuration
by Hemant Sonawane 20 Apr '20

20 Apr '20

Hi All I am facing some issue related to osd config in ceph-ansible I have 2 nvme disks where i have wal and db configure and i configure lvm like this in osd.yamL osd.yaml devices: - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf - /dev/sdg - /dev/sdh - /dev/sdi - /dev/sdj - /dev/sdk - /dev/sdl - /dev/sdm lvm_volumes: - data: /dev/sdb wal: wal-lv1 wal_vg: vg1 db: journal-lv1 db_vg: vg1 - data: /dev/sdb wal: wal-lv2 wal_vg: vg1 db: journal-lv2 db_vg: vg1 - data: /dev/sdb wal: wal-lv3 wal_vg: vg1 db: journal-lv3 db_vg: vg1 - data: /dev/sdb wal: wal-lv4 wal_vg: vg1 db: journal-lv4 db_vg: vg1 - data: /dev/sdb wal: wal-lv5 wal_vg: vg1 db: journal-lv5 db_vg: vg1 - data: /dev/sdc wal: wal-lv1 wal_vg: vg2 db: journal-lv1 db_vg: vg2 - data: /dev/sdc wal: wal-lv2 wal_vg: vg2 db: journal-lv2 db_vg: vg2 - data: /dev/sdc wal: wal-lv3 wal_vg: vg2 db: journal-lv3 db_vg: vg2 - data: /dev/sdc wal: wal-lv4 wal_vg: vg2 db: journal-lv4 db_vg: vg2 - data: /dev/sdc wal: wal-lv5 wal_vg: vg2 db: journal-lv5 db_vg: vg2 - data: /dev/sdd - data: /dev/sde - data: /dev/sdf - data: /dev/sdg - data: /dev/sdh - data: /dev/sdi - data: /dev/sdj - data: /dev/sdk - data: /dev/sdl - data: /dev/sdm I have also setup wal size and db size respectively in all.yaml all.yaml ntp_service_enabled: false mon_group_name: mons osd_group_name: osds mgr_group_name: mgrs ceph_origin: repository ceph_repository: community ceph_stable_release: octopus public_network: 10.18.0.0/22 cluster_network: 10.25.0.0/24 monitor_interface: ens3 configure_firewall: False ceph_test: true cephx: true ip_version: ipv4 osd_objectstore: bluestore dashboard_enabled: False grafana_server_group_name: grafana-server ceph_conf_overrides: global: bluestore block db size: 85899345920 (80GB) bluestore_block_wal_size: 1073741824 (1GB) i have also tried to to set values of db_size and journal_size in OSD options journal_size: 1024 # OSD journal size in MB block_db_size: 85899345920 But my osd partitions are not creating as I want and taking full disk space sdb 8:16 0 447.1G 0 disk └─ceph--3e81a883--86cd--4ddb--b0b0--be5892f6afe8-osd--block--b981ea8d--410d--4e2f--86f5--509112ff5ebf 253:0 0 447.1G 0 lvm sdc 8:32 0 447.1G 0 disk └─ceph--d3d82887--a7ad--40a4--9c3d--440cd9057fd9-osd--block--de6cae7d--daef--4144--a4bc--077f6044ef22 253:1 0 447.1G 0 lvm sdd 8:48 0 5.5T 0 disk └─ceph--e5aa3641--263c--45df--90d0--cd7e26772e69-osd--block--989fd7b2--d7b7--4170--8b33--cf4c8a29f5c1 253:2 0 5.5T 0 lvm sde 8:64 0 5.5T 0 disk └─ceph--286034b5--1c55--4fb6--8983--4ec6ef482dbd-osd--block--7cdc5ef4--8a88--4ff8--a08e--278985f4764c 253:3 0 5.5T 0 lvm sdf 8:80 0 5.5T 0 disk └─ceph--3bd8b866--b99e--41c4--a993--d69e85c1b658-osd--block--d32e42f4--1133--4c0d--aaf6--821b724fd220 253:4 0 5.5T 0 lvm sdg 8:96 0 5.5T 0 disk └─ceph--972ae2d1--2098--436f--a448--9015edfc4dc5-osd--block--79411ef4--2469--4243--9545--89cd266719f1 253:5 0 5.5T 0 lvm sdh 8:112 0 5.5T 0 disk └─ceph--92cd1c5f--3c67--420e--8a4d--c9ab0b75705b-osd--block--e643c7b3--cb10--49ab--86a8--31b60f4525dc 253:6 0 5.5T 0 lvm sdi 8:128 0 5.5T 0 disk └─ceph--0a27a900--0e81--4ef9--9b2c--692d0a609e7e-osd--block--30eddf48--5e12--4c2f--aed0--e06fb403ed33 253:7 0 5.5T 0 lvm sdj 8:144 0 5.5T 0 disk └─ceph--90156988--e0a3--4699--ae18--ad1f743521bb-osd--block--eb879f94--f289--4dd9--ba2e--48ff06a14708 253:8 0 5.5T 0 lvm sdk 8:160 0 5.5T 0 disk └─ceph--cdcabf84--e713--42c2--b2f0--f1bd6bd54924-osd--block--067418cf--1bd3--46a3--bdb9--b23216d6e8e3 253:9 0 5.5T 0 lvm sdl 8:176 0 5.5T 0 disk └─ceph--60829d44--ca66--44b3--b4be--05e8fe0c154e-osd--block--49a34fa3--c8ba--4fd9--ba3e--a6c4a889ecd5 253:10 0 5.5T 0 lvm sdm 8:192 0 5.5T 0 disk └─ceph--bd0a95e3--4d83--4c9e--a5fe--9f28414015fd-osd--block--abf5c032--eaf8--47b4--9966--67526009f68f 253:11 0 5.5T 0 lvm so after running playbook i get this result of osds with error failed: [10.11.2.202] (item={u'wal_vg': u'vg2', u'data': u'/dev/sdc', u'wal': u'wal-lv5', u'db': u'journal-lv5', u'db_vg': u'vg2'}) => changed=true ansible_loop_var: item cmd: - ceph-volume - --cluster - ceph - lvm - create - --bluestore - --data - /dev/sdc - --block.db - vg2/journal-lv5 - --block.wal - vg2/wal-lv5 delta: '0:00:03.260879' end: '2020-04-20 15:49:51.407784' item: data: /dev/sdc db: journal-lv5 db_vg: vg2 wal: wal-lv5 wal_vg: vg2 msg: non-zero return code rc: 1 start: '2020-04-20 15:49:48.146905' stderr: |- Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b26655b0-0859-4f8b-9471-0bafddfc36ef Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-b26655b0-0859-4f8b-9471-0bafddfc36ef ceph-b7e18568-6735-4a1a-9aa1-95094189e859 stderr: Calculated size of logical volume is 0 extents. Needs to be larger. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: purged osd.0 Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 39, in __init__ self.main(self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/main.py", line 150, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/main.py", line 42, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3/dist-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 77, in main self.create(args) File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 246, in safe_prepare self.prepare() File "/usr/lib/python3/dist-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 327, in prepare block_lv = self.prepare_data_device('block', osd_fsid) File "/usr/lib/python3/dist-packages/ceph_volume/devices/lvm/prepare.py", line 223, in prepare_data_device **kwargs) File "/usr/lib/python3/dist-packages/ceph_volume/api/lvm.py", line 1185, in create_lv process.run(command) File "/usr/lib/python3/dist-packages/ceph_volume/process.py", line 153, in run raise RuntimeError(msg) RuntimeError: command returned non-zero exit status: 5 So some one could please help me to figure solution for this issue . May be i am doing somewhere something wrong. If yes then please tell me. Also how sizing can be done for osds i mean how we can allow ceph-ansible to not to use default sizing? -- Thanks and Regards, Hemant Sonawane

1 0

Putting objects in Buckets
by jhamster＠chem.byu.edu 20 Apr '20

20 Apr '20

I have successfully installed a ceph storage cluster with a radosgw ceph object gateway daemon. I have been able to create an s3 bucket using a python boto3 script and a radosgw user, using the key and secret. I cannot find very much documentation concerning the usage of buckets to store objects, and since this is a new concept for me I was wondering if anyone with experience with this could help me out. I am hoping to use a frontend to manage ceph that requires s3 compatibility.

2 1

cephadm: how to override dashboard container image ?
by Dimitri Savineau 20 Apr '20

20 Apr '20

Hi, Is there a way to override the default dashboard container image hardcoded in cephadm [1] ? For the Ceph container image it possible to change the default value [2] either: - via cephadm cli and CEPHADM_IMAGE environment variable - via cephadm cli and the --image parameter - via container_image orchestrator parameter (ie: ceph config set global container_image xxx/yyy:zzz) But I'm not able to find something similar for the dashboard stack (alertmanager, grafana, node-exporter and prometheus) Any idea ? Regards, Dimitri [1] https://github.com/ceph/ceph/blob/master/src/cephadm/cephadm#L114-L160 [2] https://github.com/ceph/ceph/blob/master/src/cephadm/cephadm#L3

1 0

Goodbye and thanks for all the cuttlefish!
by Christopher Kunz 20 Apr '20

20 Apr '20

Hello ceph-{users,devel}, most of you might not know me, but I've been around a while. I've been on these lists since probably 2012, when my company, Filoo, started operating one of the early Ceph clusters for VM images. Much has changed since then, and today we shut down our biggest Ceph cluster. This was not for technical reasons, but purely for operational and commercial reasons. I still think Ceph is a superior concept and I'm sure it will continue to evolve and mature. The interactions on and off the Ceph lists, especially by Sage and many of the original Inktank team, have helped us tremendously, thanks a lot for that! I wish you all the best for the future of Ceph and will unsubscribe now. See you around! Best regads, Chris

1 0

Ceph 14.2.8 radosgw start failed due to handle_auth_bad_method
by Amit Ghadge 19 Apr '20

19 Apr '20

Hi All, I seen one issue while deploying radosgw on CentOS 7. monitor and data node deploy successfully but in radosgw seen below issue. I set ceph config set mon auth_cluster_required cephx ceph config set mon auth_service_required cephx ceph config set mon auth_client_required cephx Error: $ /usr/bin/radosgw -f --cluster ceph --name client.radosgw.masifgw01 --setuser ceph --setgroup ceph --debug_rgw 20 2020-04-19 09:05:05.421 7fe2eeea9700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [1] but i only support [2,1] failed to fetch mon config (--no-mon-config to skip mon dump: $ ceph mon dump dumped monmap epoch 4 epoch 4 fsid 10122b0b-8375-4a5a-9769-ff25a2d1d8b4 last_changed 2020-04-18 20:48:29.497461 created 2020-04-18 19:42:20.346011 min_mon_release 14 (nautilus) 0: [v2:10.44.127.xxx:3300/0,v1:10.44.127.xxx:6789/0] mon.node01 1: [v2:10.44.127.xxx:3300/0,v1:10.44.127.xxx:6789/0] mon.node02 2: [v2:10.44.127.xxx:3300/0,v1:10.44.127.xxx:6789/0] mon.node03 Anyone can face same issue, please help here. Thanks, Amit G

1 0

Re: Ghost usage on pool and unable to reclaim free space.
by Kári Bertilsson 19 Apr '20

19 Apr '20

Hi Khodayar That rados purge command did not actually seem to delete anything. It is still using 90GiB like before. I have 4 pools on these OSD's. But the "fast" pool still has 90GB but should be empty. Looking at the PG's in `ceph pg ls` LOG is using very little space on the pool. Are there any other logs ? # ceph osd pool ls detail pool 42 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186707 lfor 0/124366/185624 flags hashpspool stripe_width 0 application cephfs pool 51 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186709 lfor 0/186698/186696 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 application rbd removed_snaps [1~9] pool 53 'kube' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 186711 lfor 0/185831/185829 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~3] pool 54 'fast' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 186713 flags hashpspool,selfmanaged_snaps stripe_width 0 compression_algorithm lz4 compression_mode aggressive compression_required_ratio 0.9 application cephfs removed_snaps [52~2,55~2,59~2] # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 175 nvme 1.81898 1.00000 1.8 TiB 361 GiB 342 GiB 9.7 GiB 9.8 GiB 1.5 TiB 19.38 0.23 160 up osd.175 176 nvme 1.81898 1.00000 1.8 TiB 361 GiB 342 GiB 9.5 GiB 9.6 GiB 1.5 TiB 19.36 0.23 160 up osd.176 # ceph df POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs_metadata 42 320 MiB 2.82M 320 MiB 0.01 1.4 TiB N/A N/A 2.82M 0 B 0 B rbd 51 417 GiB 106.93k 417 GiB 12.32 1.4 TiB N/A N/A 106.93k 83 GiB 167 GiB kube 53 38 B 3 38 B 0 1.4 TiB N/A N/A 3 0 B 0 B fast 54 90 GiB 5.25M 90 GiB 2.93 1.4 TiB N/A N/A 5.25M 0 B 0 B # ceph pg ls PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP... ... 54.3a 81396 0 0 0 1512202599 0 0 3020 active+clean 9m 186704'339296 186722:615571 54.3b 82612 0 0 0 1556478139 0 0 3073 active+clean 9m 186706'340432 186722:609005 54.3c 82526 0 0 0 1553816829 0 0 3031 active+clean 9m 186704'806269 186722:1175811 54.3d 82144 0 0 0 1454278881 0 0 3016 active+clean 9m 186706'339167 186722:743880 54.3e 82740 0 0 0 1501081654 0 0 3056 active+clean 9m 186704'343212 186722:621323 54.3f 80846 0 0 0 1474357106 0 0 3048 active+clean 9m 186706'333741 186722:733149 On Sun, Apr 19, 2020 at 4:00 AM Khodayar Doustar <doustar(a)rayanexon.ir> wrote: > Hi Kári, > > You are purging only 2.6M objects with your purge command, so I guess that > 5.25M objects would be something else like logs. > Have you checked the osd df detail and osd metadata? > I have a case which osds bluestore logs are eating my whole space up, > maybe you are facing a similar one. > > Regards, > Khodayar > > On Sun, Apr 19, 2020 at 6:47 AM Kári Bertilsson <karibertils(a)gmail.com> > wrote: > >> Hello >> >> Running ceph v14.2.8 on everything. Pool is using replicated_rule with >> size/min_size 2 with 2 OSD's. I have scrubbed and deep scrubbed the OSDs. >> >> This pool was attached as data pool to cephfs containing alot of small >> files. I have since removed all files and detached the pool from the fs. >> Somehow the pool is still using 90GB. >> >> POOLS: >> POOL ID STORED OBJECTS USED %USED >> MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR >> UNDER COMPR >> fast 54 90 GiB 5.25M 90 GiB 2.93 >> 1.4 TiB N/A N/A 5.25M 0 B >> 0 B >> >> # rados -p fast ls|wc -l >> 1312030 >> >> # rados -p fast stat 100073acce3.00000000 >> error stat-ing fast/100073acce3.00000000: (2) No such file or directory >> >> # rados -p fast rm 100073acce3.00000000 >> error removing fast>100073acce3.00000000: (2) No such file or directory >> >> # rados -p fast get 100073acce3.00000000 test >> error getting fast/100073acce3.00000000: (2) No such file or directory >> >> I get the same error for every single object in the pool >> >> # rados purge fast --yes-i-really-really-mean-it >> Warning: using slow linear search >> Removed 2625749 objects >> successfully purged pool fast >> >> There was 5.25M objects in the pool before and after running this command. >> No change in ceph df. >> >> Any ideas how to reclaim the free space ? I can remove & recreate the >> pool, >> but i would like to know why and how to deal with this situation when i >> don't have that privilege >> >> Best regards >> Kári Bertilsson >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

Ghost usage on pool and unable to reclaim free space.
by Kári Bertilsson 19 Apr '20

19 Apr '20

Hello Running ceph v14.2.8 on everything. Pool is using replicated_rule with size/min_size 2 with 2 OSD's. I have scrubbed and deep scrubbed the OSDs. This pool was attached as data pool to cephfs containing alot of small files. I have since removed all files and detached the pool from the fs. Somehow the pool is still using 90GB. POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR fast 54 90 GiB 5.25M 90 GiB 2.93 1.4 TiB N/A N/A 5.25M 0 B 0 B # rados -p fast ls|wc -l 1312030 # rados -p fast stat 100073acce3.00000000 error stat-ing fast/100073acce3.00000000: (2) No such file or directory # rados -p fast rm 100073acce3.00000000 error removing fast>100073acce3.00000000: (2) No such file or directory # rados -p fast get 100073acce3.00000000 test error getting fast/100073acce3.00000000: (2) No such file or directory I get the same error for every single object in the pool # rados purge fast --yes-i-really-really-mean-it Warning: using slow linear search Removed 2625749 objects successfully purged pool fast There was 5.25M objects in the pool before and after running this command. No change in ceph df. Any ideas how to reclaim the free space ? I can remove & recreate the pool, but i would like to know why and how to deal with this situation when i don't have that privilege Best regards Kári Bertilsson

1 0

some ceph general questions about osd and pg
by harald.freidhof＠gmail.com 18 Apr '20

18 Apr '20

Hello togehter we want to implement a 3 nodes ceph cluster wirh nautilus. i already tested some ceph installations in our test enviroment and i have some generall questions. end of this month we will have three physical server with 256gb ram and two cpus and nerly 48 x 6tb disks. iam a little bit confused to calculate the pgs with the pgcalc on the ceph side. in the field osds what exactly meands that? the 3 physicaly osd nodes or the disks that will be used for the osds? what can you recomment us? we want later connect the ceph with rados and openstack and lvm thx in advance hfreidhof

3 3

Does ceph support CentOS8/RHEL8 now?
by FuLong Wang 18 Apr '20

18 Apr '20

Hi Everyone, We want to deploy ceph on CentOS8/RHEL8 environment, is it supported now? From below link, i see only CentOS7/RHEL7 supported, right? https://docs.ceph.com/docs/master/start/os-recommendations/ -- FuLong Wang _______________________________________________

1 0

Understanding PG peering count increase without osdmap changes
by Prasad Krishnan 18 Apr '20

18 Apr '20

Hi, I am trying to understand how the PG peering count could increase between two OSDMAP epochs when none of the OSDs went down or up, nor were there changes in their weights. This behaviour was seen on a Hammer cluster, but I guess it should be no different from the recent releases. I'm pasting the relevant piece of monitor logs below and attaching the relevant odmap dumps. As it can be seen, the changes to the osdmap between dump_e1053237 and dump_e10532398 seem to be restricted to a few pg_temp additions and changes to up_thru, but the OSD weights and OSD up/down state have remained the same. Kindly let me know what I'm missing here in my understanding. Thanks, K.Prasad Monitor logs -------------------- 2020-04-15 08:06:10.956681 mon.1 10.33.89.159:6789/0 52232955 : cluster [INF] pgmap v107549051: 18432 pgs: 18332 active+clean, 1 peering, 99 active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114 TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815 objects misplaced (0.105%) 2020-04-15 08:06:11.862613 mon.1 10.33.89.159:6789/0 52232956 : cluster [INF] osdmap e1053236: 388 osds: 315 up, 308 in 2020-04-15 08:06:11.868380 mon.1 10.33.89.159:6789/0 52232957 : cluster [INF] pgmap v107549052: 18432 pgs: 18332 active+clean, 1 peering, 99 active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114 TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815 objects misplaced (0.105%) 2020-04-15 08:06:10.460979 osd.57 10.33.249.161:6811/4066471 200448 : cluster [INF] 5.356s0 restarting backfill on osd.382(0) from (936954'62108,945456'62308] 0//0//-1 to 1053229'64769 2020-04-15 08:06:12.696839 mon.1 10.33.89.159:6789/0 52232959 : cluster [INF] osdmap e1053237: 388 osds: 315 up, 308 in 2020-04-15 08:06:12.703424 mon.1 10.33.89.159:6789/0 52232960 : cluster [INF] pgmap v107549053: 18432 pgs: 18332 active+clean, 1 peering, 99 active+remapped+backfill_toofull; 330 TB data, 518 TB used, 596 TB / 1114 TB avail; 34776/1684702815 objects degraded (0.002%); 1766213/1684702815 objects misplaced (0.105%) 2020-04-15 08:06:13.745554 mon.1 10.33.89.159:6789/0 52232961 : cluster [INF] osdmap e1053238: 388 osds: 315 up, 308 in 2020-04-15 08:06:13.899273 mon.1 10.33.89.159:6789/0 52232962 : cluster [INF] pgmap v107549054: 18432 pgs: 1 inactive, 1 active+remapped+wait_backfill+backfill_toofull, 2 activating, 17078 active+clean, 1251 peering, 2 activating+remapped, 5 remapped+peering, 1 activating+degraded, 91 active+remapped+backfill_toofull; 329 TB data, 518 TB used, 600 TB / 1118 TB avail; 32298/1678972195 objects degraded (0.002%); 1666716/1678972195 objects misplaced (0.099%) 2020-04-15 08:06:14.766039 mon.1 10.33.89.159:6789/0 52232963 : cluster [INF] osdmap e1053239: 388 osds: 315 up, 308 in 2020-04-15 08:06:14.833707 mon.1 10.33.89.159:6789/0 52232964 : cluster [INF] pgmap v107549055: 18432 pgs: 2 inactive, 1 active+remapped+wait_backfill+backfill_toofull, 3 activating, 16496 active+clean, 1794 peering, 4 activating+remapped, 42 remapped+peering, 3 activating+degraded, 87 active+remapped+backfill_toofull; 329 TB data, 518 TB used, 600 TB / 1118 TB avail; 30842/1678959760 objects degraded (0.002%); 1576530/1678959760 objects misplaced (0.094%) 2020-04-15 08:06:15.730301 mon.1 10.33.89.159:6789/0 52232965 : cluster [INF] osdmap e1053240: 388 osds: 315 up, 308 in 2020-04-15 08:06:15.774021 mon.1 10.33.89.159:6789/0 52232966 : cluster [INF] pgmap v107549056: 18432 pgs: 2 inactive, 1 active+remapped+wait_backfill+backfill_toofull, 3 activating, 16139 active+clean, 2103 peering, 6 activating+remapped, 92 remapped+peering, 3 activating+degraded, 83 active+remapped+backfill_toofull; 329 TB data, 518 TB used, 600 TB / 1118 TB avail; 29414/1678947337 objects degraded (0.002%); 1490753/1678947337 objects misplaced (0.089%) 2020-04-15 08:06:09.424940 osd.146 10.33.129.28:6800/2689 125912 : cluster [INF] 5.252fs0 restarting backfill on osd.382(3) from (935430'62507,945456'62707] 0//0//-1 to 1053229'65086 2020-04-15 08:06:16.713440 mon.1 10.33.89.159:6789/0 52232967 : cluster [INF] osdmap e1053241: 388 osds: 315 up, 308 in -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

HELP! Ceph( v 14.2.8) bucket notification dose not work!
by 曹海旺 17 Apr '20

17 Apr '20

Hi, I upgrade the ceph from 14.2.7 to the new version 14.2.8 . The bucket notification dose not work. I can’t create a TOPIC : I use post man to send a post flow by https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic REQUEST: POST http://rgw1:7480/?Action=CreateTopic&Name=webno&push-endpoint=https://192.1… RESPONSE: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>MethodNotAllowed</Code> <RequestId>tx000000000000000000008-005e6a0eab-cbcad-bj</RequestId> <HostId>cbcad-bj-bjz</HostId> </Error> The debug info on the node below: 2020-03-12 18:49:24.684 7fdde1e1d700 1 ====== starting new request req=0x55c91a51e8f0 ===== 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s initializing for trans_id = tx00000000000000000000e-005e6a13b4-cbc6a-bj 2020-03-12 18:49:24.684 7fdde1e1d700 10 rgw api priority: s3=8 s3website=7 2020-03-12 18:49:24.684 7fdde1e1d700 10 host=192.168.3.250 2020-03-12 18:49:24.684 7fdde1e1d700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2020-03-12 18:49:24.684 7fdde1e1d700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ 2020-03-12 18:49:24.684 7fdde1e1d700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2020-03-12 18:49:24.684 7fdde1e1d700 10 meta>> HTTP_X_AMZ_DATE 2020-03-12 18:49:24.684 7fdde1e1d700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2020-03-12 18:49:24.684 7fdde1e1d700 10 x>> x-amz-date:20200312T104924Z 2020-03-12 18:49:24.684 7fdde1e1d700 20 get_handler handler=26RGWHandler_REST_Service_S3 2020-03-12 18:49:24.684 7fdde1e1d700 10 handler=26RGWHandler_REST_Service_S3 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s getting op 4 2020-03-12 18:49:24.684 7fdde1e1d700 20 handler->ERRORHANDLER: err_no=-2003 new_err_no=-2003 2020-03-12 18:49:24.684 7fdde1e1d700 2 req 14 0.000s http status=405 2020-03-12 18:49:24.684 7fdde1e1d700 1 ====== req done req=0x55c91a51e8f0 op status=0 http_status=405 latency=0s ====== 2020-03-12 18:49:25.502 7fde0fe79700 2 RGWDataChangesLog::ChangesRenewThread: start The same post works in the version 14.2.7 What is the correct way to create a Topic in version 14.2.8 ？

3 3

Re: RGW and the orphans
by Eric Ivancich 17 Apr '20

17 Apr '20

> On Apr 16, 2020, at 1:58 PM, EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > > Hi Eric, > > Are there any ETA for get those script backported maybe in 14.2.10? > > Regards > Manuel There is a nautilus backport PR where the code works. It’s waiting on the added testing to be complete on master, so that can be backported as well. See: https://github.com/ceph/ceph/pull/34127 <https://github.com/ceph/ceph/pull/34127> -- J. Eric Ivancich he / him / his Red Hat Storage Ann Arbor, Michigan, US

1 0

radosgw-admin error: "could not fetch user info: no user info saved"
by Mathew Snyder 17 Apr '20

17 Apr '20

I'm running into a problem that I've found around the Internet, but for which I'm unable to find a solution: $ sudo radosgw-admin user info could not fetch user info: no user info saved I've seen others report this issue, but the troubleshooting has either failed to pinpoint the issue, or has gone from "I have a problem" to "I solved it" with no information as to how it was solved. Unfortunately, due to the nature of our network, I'm unable to provide debug information en masse. I can try to provide individual lines of output if someone knows what I should look for. I am able to run the user creation command: $ sudo radosgw-admin user create --uid="testuser2" --display-name="testuser2" { "user_id": "testuser2", "display_name": "testuser2", "email": "", "suspended": 0, "max_buckets": 1000, ...<snip>.... } After doing this, I'm still unable to see the user information instead getting the same "no user info saved" output. That said, I know the user is added: $ sudo radosgw-admin metadata list user [ "devtest1", "testuser", "testuser2", ...<snip>... ] I'm pretty new to this system and Ceph in general. I looked at /etc/ceph/ceph.conf but I don't see anything that would have anything to do with users. I also don't have an account on the Ansible server that controls the configurations so I can't see if anything is defined there. I know this isn't particularly detailed in its description of the issue, but I can't find any other useful help via search. Thank you, Mathew Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 3

Re: PGs unknown (osd down) after conversion to cephadm
by Dr. Marco Savoca 17 Apr '20

17 Apr '20

1 0

MDS: what's the purpose of using LogEvent with empty metablob?
by Xinying Song 17 Apr '20

17 Apr '20

Hi, cephers: What's the purpose of using LogEvent with empty metablob? For example in link/unlink operation cross two active mds, when slave receives OP_FINISH it will write an ESlaveUpdate::OP_COMMIT to the journal, then send OP_COMMITTED to master. When master receives OP_COMMITTED it will write an ECommitted to the journal then allow previously logged journal to be trimmed. Why are these two logevents necessary? I guess they are originally used for a scene that crashes happen, but in my opinion it seems not necessary. For example, if cash happens, after failed mds are brought up again, in resolve stage, master will resend OP_FINISH to slave, then things will continue as expected. Could anyone give some tips on this doubt? Sincerely thanks!

3 6

how to fix num_strays?
by Dan van der Ster 17 Apr '20

17 Apr '20

Hi all, Following some cephfs issues today we have a stable cluster but the num_strays is incorrect. After starting the mds, the values are reasonable, but they very soon underflow and start showing 18E (2^64 - a few) ---------------mds---------------- --mds_cache--- ------mds_log------ -mds_mem- ----mds_server----- mds_ ---objecter--- req rlat fwd inos caps exi imi |stry recy recd|subm evts segs repl|ino dn |hcr hcs hsr cre |sess|actv rd wr | 129 0 0 253k 1.9k 0 0 |246 0 0 | 5 4.0k 5 0 |253k 254k|129 0 0 0 |119 | 3 1 2 8.2k 0 0 253k 1.9k 0 0 |129 0 0 |395 4.4k 7 0 |253k 254k|8.2k 11 0 0 |119 | 0 33 517 9.7k 0 0 253k 1.8k 0 0 |181 0 0 |302 4.7k 7 0 |253k 254k|9.7k 5 0 0 |119 | 1 44 297 10k 0 0 253k 1.8k 0 0 |217 0 0 |382 5.1k 7 0 |253k 254k| 10k 11 0 0 |119 | 0 54 405 9.0k 0 0 253k 1.7k 0 0 |205 0 0 |386 5.5k 8 0 |253k 254k|9.0k 4 0 0 |119 | 1 46 431 8.2k 0 0 253k 1.7k 0 0 |161 0 0 |326 5.8k 8 0 |253k 254k|8.2k 6 0 0 |119 | 1 37 397 8.0k 0 0 253k 1.6k 0 0 |135 0 0 |279 6.1k 8 0 |253k 254k|8.0k 4 0 0 |119 | 1 31 317 9.2k 0 0 253k 1.6k 0 0 | 18E 0 0 |153 6.2k 8 0 |253k 254k|9.2k 6 0 0 |119 | 1 2 265 8.2k 0 0 253k 1.7k 0 0 | 18E 0 0 | 40 6.3k 8 0 |253k 254k|8.2k 5 0 0 |119 | 3 3 17 Is there a way to reset the num_strays to the correct number of strays ? # for i in `seq 600 609`; do rados --cluster=dwight listomapvals -p cephfs_metadata ${i}.00000000 2>&1 | wc -l; done 95 220 91 173 567 1578 1042 754 445 261 Cheers, Dan

2 5

cephadm adopting osd failed
by bbk＠nocloud.ch 17 Apr '20

17 Apr '20

Hi, From a healty nautilus cluster version 14.2.9 on a CentOS7 i try to follow the upgrade procedure to the containerized octopus setup with cephadm. * https://docs.ceph.com/docs/octopus/cephadm/adoption/ Everything step went fine until i wanted to adopt the osds, then i get a error. Does anybody have a idea, what my problem could be? ```````````````````` ~# cephadm adopt --style legacy --name osd.0 INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container... INFO:cephadm:objectstore_type is bluestore INFO:cephadm:Stopping old systemd unit ceph-osd@0... INFO:cephadm:Disabling old systemd unit ceph-osd@0... INFO:cephadm:Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 4282, in <module> r = args.func() File "/usr/sbin/cephadm", line 972, in _default_image return func() File "/usr/sbin/cephadm", line 2916, in command_adopt command_adopt_ceph(daemon_type, daemon_id, fsid); File "/usr/sbin/cephadm", line 2979, in command_adopt_ceph os.rmdir(data_dir_src) OSError: [Errno 39] Directory not empty: '//var/lib/ceph/osd/ceph-0' ```````````````````` Yours, bbk

3 5

Re: PGs unknown (osd down) after conversion to cephadm
by Dr. Marco Savoca 16 Apr '20

16 Apr '20

2 1

v13.2.9 Mimic released
by Abhishek Lekshmanan 16 Apr '20

16 Apr '20

We're glad to announce the availability of the ninth and very likely the last stable release in the Ceph Mimic stable release series. This release fixes bugs across all components and also contains a RGW security fix. We recommend all mimic users to upgrade to this version. We thank everyone for making this release a possibility. Notable Changes --------------- * CVE-2020-1760: Fixed XSS due to RGW GetObject header-splitting * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap balancing has been removed. Instead use the mgr balancer config `upmap_max_deviation` which now is an integer number of PGs of deviation from the target PGs per OSD. This can be set with a command like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default `upmap_max_deviation` is 1. There are situations where crush rules would not allow a pool to ever have completely balanced PGs. For example, if crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of the racks. In those cases, the configuration value can be increased. * The `cephfs-data-scan scan_links` command now automatically repair inotables and snaptable. For the full changelog please refer to the official release blog entry at https://ceph.io/releases/v13-2-9-mimic-released Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-13.2.9.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 58a2a9b31fd08d8bb3089fce0e312331502ff945 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer

1 0

Re: RGW and the orphans
by Katarzyna Myrek 16 Apr '20

16 Apr '20

Hi Thanks for the quick response. To be honest my cluster is getting full because of that trash and I am at the point where I have to do the removal manually ;/. Kind regards / Pozdrawiam, Katarzyna Myrek czw., 16 kwi 2020 o 13:09 EDH - Manuel Rios <mriosfer(a)easydatahost.com> napisał(a): > > Hi, > > From my experience orphans find didn't work since several releases ago, and command should be re-coded or deprecated because its not running. > > Im our cases it loops over generated shards until RGW daemon crash. > > Interested into this post, in our case orphans find takes more than 24 hours into start loop over shards, but never pass the shard 0 or 1. > > CEPH RGW devs, should provide any workaround script/ new tool or something to maintain our rgw clusters. Because with the last bugs all rgw cluster got a ton of trash, wasting resources and money. > > And manual cleaning is not trivial and easy. > > Waiting for more info, > > Manuel > > > -----Mensaje original----- > De: Katarzyna Myrek <katarzyna(a)myrek.pl> > Enviado el: jueves, 16 de abril de 2020 12:38 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] RGW and the orphans > > Hi > > Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? > > I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. > Does anyone have a good method for that? > > My cluster mostly has orphans from multipart uploads. > > > Kind regards / Pozdrawiam, > Katarzyna Myrek > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

MDS: cache pressure warnings with Ganesha exports
by Stolte, Felix 16 Apr '20

16 Apr '20

Hey folks, I keep getting ceph health warnings about clients failing to respond to cache pressure. They always refer to sessions from ganesha exports. I've read all threads regarding this issue, but none of my changes resolved it. What I’ve done so far: Ganesha.conf: MDCACHE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } Attr_Expiration_Time = 0 in every export mds_cache_memory_limit = 17179869184 on MDS Servers I even set “client_oc = false” on the ganesha server, but this doesn’t seem to be applied My setup is ceph version 14.2.8 (All Server and clients. One active MDS. Ganesha 2.8.3 runs on a dedicated Server. ceph daemon mds.<active_mds> dump_mempools (filtered out the empty pools): "mempool": { "by_pool": { "bloom_filter": { "items": 1414723, "bytes": 1414723 }, "buffer_anon": { "items": 180992, "bytes": 2404306271 }, "buffer_meta": { "items": 178660, "bytes": 15722080 }, "osdmap": { "items": 4121, "bytes": 75912 }, "mds_co": { "items": 221728924, "bytes": 16320868177 }, }, "total": { "items": 223507420, "bytes": 18742387163 Every hint how to resolve the issue are welcome. If more information is needed, I am glad to provide it. Regards Felix ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

3 7

RGW and the orphans
by Katarzyna Myrek 16 Apr '20

16 Apr '20

Hi Is there any new way to find and remove orphans from RGW pools on Nautilus? I have found info that "orphans find" is now deprecated? I can see that I have tons of orphans in one of our clusters. Was wondering how to safely remove them - make sure that they are really orphans. Does anyone have a good method for that? My cluster mostly has orphans from multipart uploads. Kind regards / Pozdrawiam, Katarzyna Myrek

2 1

radosgw multisite
by Ignazio Cassano 16 Apr '20

16 Apr '20

Hello All, Reading the documentation I created a multisite with luminous version. I would like to know if it sync in one way only. Using s3cmd if I put a file in a bucket on the primary zone I can see the file in the same bucket on the secondary zone. If I put a file in the bucket on secondary zone I cannot see the file on the primary. Is this the correct behaviour ? Thanks & Regards Ignazio

2 3

Re: New to ceph / Very unbalanced cluster
by Reed Dier 16 Apr '20

16 Apr '20

Copying the ML, because I forgot to reply-all. Reed > On Apr 15, 2020, at 3:58 PM, Reed Dier <reed.dier(a)focusvq.com> wrote: > > The problem is almost certainly stemming from unbalanced OSD distribution among your hosts, and assuming you are using a default 3x replication across hosts crush rule set. > > You are limited by your smallest bin size. > > In this case you have a 750GB HDD as the only OSD on node1, so when it wants 3 copies across 3 hosts, there are only ~750GB of space that can fulfill this requirement. > > Having lots of different size OSDs and differing OSDs in your topology is going to lead to issues of under/over utilization. > >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 21.54213 root default >> -3 0.75679 host node1 >> -5 5.39328 host node2 >> -10 15.39206 host node3 > > You either need to redistribute your OSDs across your hosts, or possibly rethink your disk strategy. > You could move osd.5 to node1, and osd.0 to node2, which would give you roughly 6TiB of usable hdd space across your three nodes. > > Reed > >> On Apr 15, 2020, at 10:50 AM, Simon Sutter <ssutter(a)hosttech.ch <mailto:ssutter@hosttech.ch>> wrote: >> >> Hello everybody, >> >> >> >> I'm very new to ceph and installed a testenvironment (nautilus). >> >> The current goal of this cluster is, to be a short period backup. >> >> For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?). >> >> I created for my cephfs two pools, one for metadata and one for storage data. >> >> >> >> I have three nodes and the ceph osd tree looks like this: >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> -1 21.54213 root default >> -3 0.75679 host node1 >> 0 hdd 0.75679 osd.0 up 0.00636 1.00000 >> -5 5.39328 host node2 >> 1 hdd 2.66429 osd.1 up 0.65007 1.00000 >> 3 hdd 2.72899 osd.3 up 0.65007 1.00000 >> -10 15.39206 host node3 >> 5 hdd 7.27739 osd.5 up 1.00000 1.00000 >> 6 hdd 7.27739 osd.6 up 1.00000 1.00000 >> 2 ssd 0.38249 osd.2 up 1.00000 1.00000 >> 4 ssd 0.45479 osd.4 up 1.00000 1.00000 >> >> >> The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview: >> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS >> 0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up >> 1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up >> 3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB 8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up >> 5 hdd 7.27739 1.00000 7.3 TiB 390 GiB 389 GiB 8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up >> 6 hdd 7.27739 1.00000 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up >> 2 ssd 0.38249 1.00000 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up >> 4 ssd 0.45479 1.00000 466 GiB 28 GiB 27 GiB 4 KiB 1024 MiB 438 GiB 6.03 0.52 4 up >> TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57 >> MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87 >> >> To counteract this, I tried to turn on the balancer module. >> >> The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects: >> >> 144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%) >> >> >> >> So my question is: is ceph supposed to do that? >> Why are all those objects misplaced? Because of those 112 PGs on osd0? >> Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512 >> >> >> >> Thank you very much >> Simon Sutter >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >

2 1

Issues with RGW PUT performance after upgrade to 14.2.8
by Katarzyna Myrek 16 Apr '20

16 Apr '20

Hi After upgrading to 14.2.8 i can see that PUT operations are significantly slower. GET and DELETE still have the same performance. I double checked OSD nodes and I cannot find anything suspicious there. No extreme iowaits etc. Anyone have the same problem? Kind regards / Pozdrawiam, Katarzyna Myrek

1 0

Cephadm and rados gateways
by Edward Garcia 16 Apr '20

16 Apr '20

Hello, I have a question. I’m trying to deploy a rados gateway and the container keeps crashing in less than a second on the host. Ceph/cephadm keeps trying to recreate it and its an endless loop. Looking at the log the gateway container is failing in trying to bind to port 80. How do I configure the settings for the gateway to change the port? I see on the documentation page that the settings are configured via the monitor configuration but what setting exactly? I tried adding rgw_frontends to ceph.conf and with cephadm config set. I initially created the realm, group, and some just like the cephadm page stated but every time I ceph orch apply rgw ... when I check with ceph config dump I see the rgw service defaults back to port 80.

1 0

Deletion of objects and garbage collector
by Priya Sehgal 15 Apr '20

15 Apr '20

Hi, I read at many places that when an object is deleted from ceph it is queued for deletion with the garbage collector (GC). However, when I delete objects of various sizes (both less than 4MB and large sized greater than 4MB i.e. MPU) I always find gc list as empty. I tried disabling GC also to make sure that it does not run and delete it, yet I did not find any object when I run the command. Here's the output: *radosgw-admin gc list* [ { "tag": "2~KsaJkJwSGeuVzeKpkHAe_5vJ3JqZmKc", "time": "2020-04-10 18:25:23.0.769037s", "objs": [] } ] NOTE: I tried to delete an object as on 14th April and 15th April. I issued "s3cmd del" command. Ceph Version I am using in Luminous. Please let me know how object deletes work. -- Regards, Priya

2 4

New to ceph / Very unbalanced cluster
by Simon Sutter 15 Apr '20

15 Apr '20

Hello everybody, I'm very new to ceph and installed a testenvironment (nautilus). The current goal of this cluster is, to be a short period backup. For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?). I created for my cephfs two pools, one for metadata and one for storage data. I have three nodes and the ceph osd tree looks like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.54213 root default -3 0.75679 host node1 0 hdd 0.75679 osd.0 up 0.00636 1.00000 -5 5.39328 host node2 1 hdd 2.66429 osd.1 up 0.65007 1.00000 3 hdd 2.72899 osd.3 up 0.65007 1.00000 -10 15.39206 host node3 5 hdd 7.27739 osd.5 up 1.00000 1.00000 6 hdd 7.27739 osd.6 up 1.00000 1.00000 2 ssd 0.38249 osd.2 up 1.00000 1.00000 4 ssd 0.45479 osd.4 up 1.00000 1.00000 The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up 1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up 3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB 8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up 5 hdd 7.27739 1.00000 7.3 TiB 390 GiB 389 GiB 8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up 6 hdd 7.27739 1.00000 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up 2 ssd 0.38249 1.00000 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up 4 ssd 0.45479 1.00000 466 GiB 28 GiB 27 GiB 4 KiB 1024 MiB 438 GiB 6.03 0.52 4 up TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57 MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87 To counteract this, I tried to turn on the balancer module. The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects: 144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%) So my question is: is ceph supposed to do that? Why are all those objects misplaced? Because of those 112 PGs on osd0? Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512 Thank you very much Simon Sutter

1 0

Re: MDS: obscene buffer_anon memory use when scanning lots of files
by John Madden 15 Apr '20

15 Apr '20

Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested: ~$ ceph tell mds.mds1 heap stats 2020-02-10 16:52:44.313 7fbda2cae700 0 client.59208005 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:44.337 7fbda3cb0700 0 client.59249562 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 tcmalloc heap stats:------------------------------------------------ MALLOC: 50000388656 (47684.1 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 174879528 ( 166.8 MiB) Bytes in central cache freelist MALLOC: + 14511680 ( 13.8 MiB) Bytes in transfer cache freelist MALLOC: + 14089320 ( 13.4 MiB) Bytes in thread cache freelists MALLOC: + 90534048 ( 86.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 50294403232 (47964.5 MiB) Actual memory used (physical + swap) MALLOC: + 50987008 ( 48.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 50345390240 (48013.1 MiB) Virtual address space used MALLOC: MALLOC: 260018 Spans in use MALLOC: 20 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ~$ ceph tell mds.mds1 heap release 2020-02-10 16:52:47.205 7f037eff5700 0 client.59249625 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:47.237 7f037fff7700 0 client.59249634 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 releasing free RAM back to system. The pools over 15 minutes or so: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2045, "bytes": 3069493686 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2445, "bytes": 3111162538 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 7850, "bytes": 7658678767 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 12274, "bytes": 11436728978 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 13747, "bytes": 11539478519 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 14615, "bytes": 13859676992 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 23267, "bytes": 22290063830 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 44944, "bytes": 40726959425 } And one about a minute after the heap release showing continued growth: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 50694, "bytes": 47343942094 } This is on a single active MDS with 2 standbys, scan for about a million files with about 20 parallel threads on two clients, open and read each if it exists. On Wed, Jan 22, 2020 at 8:25 AM John Madden <jmadden.com(a)gmail.com> wrote: > > > Couldn't John confirm that this is the issue by checking the heap stats and triggering the release via > > > > ceph tell mds.mds1 heap stats > > ceph tell mds.mds1 heap release > > > > (this would be much less disruptive than restarting the MDS) > > That was my first thought as well, but `release` doesn't appear to do > anything in this case. > > John

3 8

telemetry.ceph.com certificate expired
by Eneko Lacunza 15 Apr '20

15 Apr '20

Hi all, We're receiving a certificate error for telemetry module: Module 'telemetry' has failed: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /report (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)); Seems certificate expired yesterday (14th april). Cheers Eneko -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es

2 1

v14.2.9 Nautilus released
by Abhishek 15 Apr '20

15 Apr '20

This is the ninth bugfix release of Nautilus. This release fixes a couple of security issues in RGW & Messenger V2. We recommend all users to upgrade to this release. The official release blog entry is at https://ceph.io/releases/v14-2-9-nautilus-released/ Notable Changes: - CVE-2020-1759: Fixed nonce reuse in msgr V2 secure mode - CVE-2020-1760: Fixed XSS due to RGW GetObject header-splitting Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.9.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 581f22da52345dba46ee232b73b990f06029a2a0 Best, Abhishek

1 0

radosgw garbage collection seems stuck and mannual gc process didn't work
by 346415320＠qq.com 15 Apr '20

15 Apr '20

Ceph Version : Mimic 13.2.4 The cluster has been running steadily for more than a year, recently I found cluster usage grows faster than usual .And we figured out the problem is garbage collection. 'radosgw-admin gc list ' has millions of objects to gc. the earliest tag time is 2019-09 , but 99% of them are from 2020-03 to now `ceph df` GLOBAL: SIZE AVAIL RAW USED %RAW USED 1.7 PiB 1.1 PiB 602 TiB 35.22 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS .rgw.root 10 1.2 KiB 0 421 TiB 4 default.rgw.control 11 0 B 0 421 TiB 8 default.rgw.data.root 12 0 B 0 421 TiB 0 default.rgw.gc 13 0 B 0 421 TiB 0 default.rgw.log 14 4.8 GiB 0 421 TiB 6414 default.rgw.intent-log 15 0 B 0 421 TiB 0 default.rgw.meta 16 110 KiB 0 421 TiB 463 default.rgw.usage 17 0 B 0 421 TiB 0 default.rgw.users.keys 18 0 B 0 421 TiB 0 default.rgw.users.email 19 0 B 0 421 TiB 0 default.rgw.users.swift 20 0 B 0 421 TiB 0 default.rgw.users.uid 21 0 B 0 421 TiB 0 default.rgw.buckets.extra 22 0 B 0 421 TiB 0 default.rgw.buckets.index 23 0 B 0 421 TiB 118720 default.rgw.buckets.data 24 263 TiB 38.41 421 TiB 138902771 default.rgw.buckets.non-ec 25 0 B 0 421 TiB 16678 however we counted each bucket usage by ' radosgw-admin bucket stats ' ,it should cost only 160TiB , about 80TiB are in GC list former gc config setting before we find gc problem: rgw_gc_max_objs = 32 rgw_gc_obj_min_wait = 3600 rgw_gc_processor_period = 3600 rgw_gc_processor_max_time = 3600 yesterday we adjust our setting and restart rgw: rgw_gc_max_objs = 1024 rgw_gc_obj_min_wait = 300 rgw_gc_processor_period = 600 rgw_gc_processor_max_time = 600 rgw_gc_max_concurrent_io = 40 rgw_gc_max_trim_chunk = 1024 today we use ' rados -p default.rgw.log listomapkeys gc.$i --cluster ceph -N gc | wc -l ' (i from 0 to 1023) well , only gc.0 to gc.511 has data here are some result sorted -time 14:43 result: …… 36 gc_202004111443/gc.502.tag 38 gc_202004111443/gc.501.tag 40 gc_202004111443/gc.136.tag 46 gc_202004111443/gc.511.tag 212 gc_202004111443/gc.9.tag 218 gc_202004111443/gc.24.tag 21976 gc_202004111443/gc.13.tag 42956 gc_202004111443/gc.26.tag 71772 gc_202004111443/gc.25.tag 85766 gc_202004111443/gc.6.tag 104504 gc_202004111443/gc.7.tag 105444 gc_202004111443/gc.10.tag 106114 gc_202004111443/gc.3.tag 126860 gc_202004111443/gc.31.tag 127352 gc_202004111443/gc.23.tag 147942 gc_202004111443/gc.27.tag 148046 gc_202004111443/gc.15.tag 167116 gc_202004111443/gc.28.tag 167932 gc_202004111443/gc.21.tag 187986 gc_202004111443/gc.5.tag 188312 gc_202004111443/gc.22.tag 209084 gc_202004111443/gc.30.tag 209152 gc_202004111443/gc.18.tag 209702 gc_202004111443/gc.19.tag 231100 gc_202004111443/gc.8.tag 249622 gc_202004111443/gc.14.tag 251092 gc_202004111443/gc.2.tag 251366 gc_202004111443/gc.12.tag 251802 gc_202004111443/gc.0.tag 252158 gc_202004111443/gc.11.tag 272114 gc_202004111443/gc.1.tag 291518 gc_202004111443/gc.20.tag 293646 gc_202004111443/gc.16.tag 312998 gc_202004111443/gc.17.tag 352984 gc_202004111443/gc.29.tag 488232 gc_202004111443/gc.4.tag 5935806 total -time 16:53 result: …… 28 gc_202004111653/gc.324.tag 28 gc_202004111653/gc.414.tag 30 gc_202004111653/gc.350.tag 30 gc_202004111653/gc.456.tag 204 gc_202004111653/gc.9.tag 208 gc_202004111653/gc.24.tag 21986 gc_202004111653/gc.13.tag 42964 gc_202004111653/gc.26.tag 71780 gc_202004111653/gc.25.tag 85778 gc_202004111653/gc.6.tag 104512 gc_202004111653/gc.7.tag 105452 gc_202004111653/gc.10.tag 106122 gc_202004111653/gc.3.tag 126866 gc_202004111653/gc.31.tag 127372 gc_202004111653/gc.23.tag 147944 gc_202004111653/gc.27.tag 148058 gc_202004111653/gc.15.tag 167124 gc_202004111653/gc.28.tag 167936 gc_202004111653/gc.21.tag 187992 gc_202004111653/gc.5.tag 188320 gc_202004111653/gc.22.tag 209090 gc_202004111653/gc.30.tag 209170 gc_202004111653/gc.18.tag 209704 gc_202004111653/gc.19.tag 231108 gc_202004111653/gc.8.tag 249632 gc_202004111653/gc.14.tag 251096 gc_202004111653/gc.2.tag 251376 gc_202004111653/gc.12.tag 251806 gc_202004111653/gc.0.tag 252170 gc_202004111653/gc.11.tag 272118 gc_202004111653/gc.1.tag 291526 gc_202004111653/gc.20.tag 293656 gc_202004111653/gc.16.tag 313000 gc_202004111653/gc.17.tag 352988 gc_202004111653/gc.29.tag 488238 gc_202004111653/gc.4.tag 5932714 total -time 18:55 result: …… 22 gc_202004111855/gc.187.tag 22 gc_202004111855/gc.331.tag 24 gc_202004111855/gc.163.tag 24 gc_202004111855/gc.83.tag 194 gc_202004111855/gc.9.tag 208 gc_202004111855/gc.24.tag 21994 gc_202004111855/gc.13.tag 42966 gc_202004111855/gc.26.tag 71796 gc_202004111855/gc.25.tag 85788 gc_202004111855/gc.6.tag 104522 gc_202004111855/gc.7.tag 105458 gc_202004111855/gc.10.tag 106126 gc_202004111855/gc.3.tag 126872 gc_202004111855/gc.31.tag 127378 gc_202004111855/gc.23.tag 147954 gc_202004111855/gc.27.tag 148066 gc_202004111855/gc.15.tag 167136 gc_202004111855/gc.28.tag 167940 gc_202004111855/gc.21.tag 187996 gc_202004111855/gc.5.tag 188326 gc_202004111855/gc.22.tag 209092 gc_202004111855/gc.30.tag 209178 gc_202004111855/gc.18.tag 209720 gc_202004111855/gc.19.tag 231126 gc_202004111855/gc.8.tag 249646 gc_202004111855/gc.14.tag 251106 gc_202004111855/gc.2.tag 251378 gc_202004111855/gc.12.tag 251816 gc_202004111855/gc.0.tag 252176 gc_202004111855/gc.11.tag 272134 gc_202004111855/gc.1.tag 291528 gc_202004111855/gc.20.tag 293662 gc_202004111855/gc.16.tag 313010 gc_202004111855/gc.17.tag 352996 gc_202004111855/gc.29.tag 488250 gc_202004111855/gc.4.tag 5931710 total -time 22:54 result: …… 32 gc_202004112254/gc.191.tag 32 gc_202004112254/gc.196.tag 32 gc_202004112254/gc.486.tag 34 gc_202004112254/gc.382.tag 208 gc_202004112254/gc.9.tag 210 gc_202004112254/gc.24.tag 22012 gc_202004112254/gc.13.tag 43008 gc_202004112254/gc.26.tag 71828 gc_202004112254/gc.25.tag 85814 gc_202004112254/gc.6.tag 104544 gc_202004112254/gc.7.tag 105468 gc_202004112254/gc.10.tag 106146 gc_202004112254/gc.3.tag 126882 gc_202004112254/gc.31.tag 127396 gc_202004112254/gc.23.tag 147980 gc_202004112254/gc.27.tag 148088 gc_202004112254/gc.15.tag 167146 gc_202004112254/gc.28.tag 167960 gc_202004112254/gc.21.tag 188006 gc_202004112254/gc.5.tag 188352 gc_202004112254/gc.22.tag 209114 gc_202004112254/gc.30.tag 209204 gc_202004112254/gc.18.tag 209758 gc_202004112254/gc.19.tag 231148 gc_202004112254/gc.8.tag 249668 gc_202004112254/gc.14.tag 251126 gc_202004112254/gc.2.tag 251406 gc_202004112254/gc.12.tag 251850 gc_202004112254/gc.0.tag 252186 gc_202004112254/gc.11.tag 272154 gc_202004112254/gc.1.tag 291552 gc_202004112254/gc.20.tag 293684 gc_202004112254/gc.16.tag 313038 gc_202004112254/gc.17.tag 353014 gc_202004112254/gc.29.tag 488274 gc_202004112254/gc.4.tag 5935738 total it seems gc.0-gc.31 keeps on increasing. but gc.32 -gc.511 works fine. maybe some objects in gc.0-gc.31 stuck gc process, so gc thread on gc.0 - gc.31 cannot proceed after setting debug_rgw=20 , we find only a few process period (every 10 minutes as set) can log like '2020-04-11 23:02:18.724 7f7d1e61b700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009550000004491/6b97a8a61424129d996d7a13769e8c66.ts.2~KnoW62bPMfmWBWtH0THWWzV3urFJglW.1' most of process period only shows: '2020-04-11 23:32:14.814 7f7d1de1a700 2 object expiration: start 2020-04-11 23:32:14.814 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000000 2020-04-11 23:32:14.815 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000001 …… 2020-04-11 23:22:14.811 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000126 2020-04-11 23:22:14.813 7f7d1de1a700 2 object expiration: stop 2020-04-11 23:22:21.364 7f7d1e61b700 10 RGWGC::process() failed to acquire lock on gc.4 2020-04-11 23:22:31.074 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:22:53.074 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:23:15.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:23:37.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:23:59.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:24:21.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:24:43.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:25:05.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:25:27.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:25:49.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:26:11.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:26:33.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:26:55.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:27:17.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:27:39.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:28:01.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:28:23.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:28:45.075 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:29:07.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:29:29.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:29:51.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:30:13.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:30:35.077 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:30:57.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:31:19.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:31:41.076 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:32:03.077 7f7d1f61d700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-11 23:32:14.814 7f7d1de1a700 2 object expiration: start 2020-04-11 23:32:14.814 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000000 2020-04-11 23:32:14.815 7f7d1de1a700 20 proceeding shard = obj_delete_at_hint.0000000001 ……' and sometimes gc removes a few objects like: '2020-04-10 12:01:45.141 7f2acdd38700 2 object expiration: stop 2020-04-10 12:01:45.886 7f2ace539700 10 RGWGC::process() failed to acquire lock on gc.8 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009510000009744/8d5cb6722d478fd2395af1bc25b73880.ts.2~rDnSwcUmF-MKFwneDyMQRiI4tXuw6K7.1 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009510000009744/8d5cb6722d478fd2395af1bc25b73880.ts.2~astXMDje7aBnOLzHG7qIdofTrpHTsUB.1 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009550000003154/39abc9fe1bf0f289c6e02132cbc2d96f.ts.2~YW4Uv7qhbKqZoru7wIGvJXZAuytD7s6.1 2020-04-10 12:01:45.911 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.985987.4__multipart_playback/2009550000004519/296b0c23da59ff9eba8a522fc5febd11.ts.2~bL7CZCB-mKP7w2RHFomEsgBzAH1rZt9.1 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_1 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_2 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_3 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_4 2020-04-10 12:01:45.912 7f2ace539700 5 gc::process: removing default.rgw.buckets.data:d9d41307-13ce-4b43-aa24-2495540b67f5.2190746.7__shadow_.VYxHLF5VGGXMFsYy-CNiMOfDVI-NTfQ_5 2020-04-10 12:01:45.914 7f2ace539700 10 RGWGC::process() failed to acquire lock on gc.10 2020-04-10 12:01:45.915 7f2ace539700 10 RGWGC::process() failed to acquire lock on gc.11 2020-04-10 12:02:02.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-10 12:02:24.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-10 12:02:46.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-04-10 12:03:08.638 7f2acf53b700 2 RGWDataChangesLog::ChangesRenewThread: start ……' we scan those logs on 5 gc node( no other workload like object read/write or lifecycle, only for gc). all is the same. the "removing xxx" log appears only a few times within a few seconds in whole day's work so we suppose some objs in gc.0-gc.31 stuck gc process mannual ' radosgw-admin gc process ' not work , log is the same as above Has anyone encountered the same problem? what can i do to make gc thread proceed on gc.0-gc.31 or get everything back to normal

3 4

CephFS and Samba/CIFS permissions (xattr)
by Victor Rodriguez 14 Apr '20

14 Apr '20

Hello, I have a CephFS running on v14.2.8 correctly. I also have a VM which runs Samba as AD controller and fileserver (Zentyal). My plan was to mount a CephFS path on that VM and make Samba share those files to a Windows network. But I cant make the shares work as Samba is asking to mount the CephFS resource with "user_xattr" mount option, which the kernel driver doesnt support. Besides that, looks like I cant set CIFS permissions on directories/files in CephFS because extended attributes are not supported. Is that the expected behavior or am I overlooking something? Looking for information about this issue, I have read about ceph-vfs and Samba CTDB. Are those options really necessary to get Samba over CephFS with CIFS/AD domain permissions? Thanks a lot. Victor.

1 0

Announcing go-ceph v0.3.0
by John Mulligan 14 Apr '20

14 Apr '20

I'm happy to announce the another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.3.0 The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild and the ceph-csi project is starting to make use of this library. Specific questions, comments, bugs etc are best directed at our github issues tracker. --- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Fwd: Question on rbd maps
by Void Star Nill 14 Apr '20

14 Apr '20

Hello, Is there a way to find out all the clients where the volumes are mapped from a central point? We have a large fleet of machines that use ceph rbd volumes. For some maintenance purposes, we need to find out if a volume is mapped anywhere before acting on it. Right now we go and query each client machines with `rbd showmapped` commands. Is there a variant of this CLI that lists all mappings on a single node (ceph mon for instance)? Thanks, Shridhar

3 6

Fwd: question on rbd locks
by Void Star Nill 14 Apr '20

14 Apr '20

Hello All, Is there a way to specify that a lock (shared or exclusive) on an rbd volume be released if the client machine becomes unreachable or irresponsive? In one of our clusters, we use rbd locks on volumes to make sure provide a kind of shared or exclusive access - to make sure there are no writers when someone is reading and there are no readers when someone is writing. However, we often run into issues when one of the machines gets into kernel panic or something and the whole pipeline gets stalled. So is there a way to tell ceph to release the lock if the client becomes unavailable? Thanks, Shridhar

4 7

ceph-mgr with large connections in CLOSE_WAIT state
by Void Star Nill 14 Apr '20

14 Apr '20

Hi, I am seeing a large number of connections from ceph-mgr are stuck in CLOSE_WAIT state with data stuck in the receive queue. Looks like ceph-mgr process is not reading the data completely off the socket buffers and terminating the connections properly. I also notice that the access to dashboard plugin stops working after couple of days and I need to restart ceph-mgr for it to start working again. I haven't noticed any impact of volume I/O operations though. We are running nautilus version (14.2.4-1xenial) Is this a known issue? $ sudo netstat -anp|grep CLOSE_WAIT | wc -l 906 $ sudo netstat -anp|grep CLOSE_WAIT tcp6 77 0 x.x.x.x:8443 x.x.12.195:48624 CLOSE_WAIT 161716/ceph-mgr tcp6 210 0 x.x.x.x:8443 x.x.12.209:59220 CLOSE_WAIT 161716/ceph-mgr tcp6 2443 0 x.x.x.x:8443 x.x.0.103:65276 CLOSE_WAIT 161716/ceph-mgr tcp6 159 0 x.x.x.x:8443 x.x.12.202:60150 CLOSE_WAIT 161716/ceph-mgr tcp6 1 0 x.x.x.x:8443 x.x.12.161:51472 CLOSE_WAIT 161716/ceph-mgr tcp6 283 0 x.x.x.x:8443 x.x.12.202:44900 CLOSE_WAIT 161716/ceph-mgr tcp6 32 0 x.x.x.x:8443 x.x.12.207:58202 CLOSE_WAIT 161716/ceph-mgr tcp6 32 0 x.x.x.x:8443 x.x.12.200:52162 CLOSE_WAIT 161716/ceph-mgr tcp6 32 0 x.x.x.x:8443 x.x.12.167:43148 CLOSE_WAIT 161716/ceph-mgr tcp6 305 0 x.x.x.x:8443 x.x.12.167:60102 CLOSE_WAIT 161716/ceph-mgr tcp6 79 0 x.x.x.x:8443 x.x.12.195:48504 CLOSE_WAIT 161716/ceph-mgr tcp6 69 0 x.x.x.x:8443 x.x.12.167:50362 CLOSE_WAIT 161716/ceph-mgr tcp6 38 0 x.x.x.x:8443 x.x.12.167:36840 CLOSE_WAIT 161716/ceph-mgr tcp6 76 0 x.x.x.x:8443 x.x.12.209:54166 CLOSE_WAIT 161716/ceph-mgr tcp6 68 0 x.x.x.x:8443 x.x.12.207:34312 CLOSE_WAIT 161716/ceph-mgr tcp6 209 0 x.x.x.x:8443 x.x.12.209:52642 CLOSE_WAIT 161716/ceph-mgr tcp6 220 0 x.x.x.x:8443 x.x.12.202:41156 CLOSE_WAIT 161716/ceph-mgr tcp6 202 0 x.x.x.x:8443 x.x.12.209:55448 CLOSE_WAIT 161716/ceph-mgr tcp6 193 0 x.x.x.x:8443 x.x.12.207:56296 CLOSE_WAIT 161716/ceph-mgr tcp6 672 0 x.x.x.x:8443 x.x.12.207:35516 CLOSE_WAIT 161716/ceph-mgr tcp6 1186 0 x.x.x.x:8443 x.x.28.4:54074 CLOSE_WAIT 161716/ceph-mgr tcp6 163 0 x.x.x.x:8443 x.x.12.209:54458 CLOSE_WAIT 161716/ceph-mgr tcp6 214 0 x.x.x.x:8443 x.x.12.167:54912 CLOSE_WAIT 161716/ceph-mgr tcp6 129 0 x.x.x.x:8443 x.x.12.200:42494 CLOSE_WAIT 161716/ceph-mgr tcp6 86 0 x.x.x.x:8443 x.x.12.195:58618 CLOSE_WAIT 161716/ceph-mgr tcp6 103 0 x.x.x.x:8443 x.x.12.209:45408 CLOSE_WAIT 161716/ceph-mgr tcp6 77 0 x.x.x.x:8443 x.x.12.202:60792 CLOSE_WAIT 161716/ceph-mgr tcp6 189 0 x.x.x.x:8443 x.x.12.161:58640 CLOSE_WAIT 161716/ceph-mgr tcp6 68 0 x.x.x.x:8443 x.x.12.167:42004 CLOSE_WAIT 161716/ceph-mgr tcp6 1 0 x.x.x.x:8443 x.x.12.195:36348 CLOSE_WAIT 161716/ceph-mgr Thanks, Shridhar

1 1

PGs unknown (osd down) after conversion to cephadm
by Dr. Marco Savoca 14 Apr '20

14 Apr '20

2 1

Fw: Incompatibilities (implicit_tenants & barbican) with Openstack after migrating from Ceph Luminous to Nautilus.
by Scheurer François 14 Apr '20

14 Apr '20

(resending to the new maillist) Dear Casey, Dear All, We tested the migration from Luminous to Nautilus and noticed two regressions breaking the RGW integration in Openstack: 1) the following config parameter is not working on Nautilus but is valid on Luminous and on Master: rgw_keystone_implicit_tenants = swift In the log: parse error setting 'rgw_keystone_implicit_tenants' to 'swift' (Expected option value to be integer, got 'swift') This param is important to make RGW working for S3 and Swift. Setting it to false breaks swift/openstack and setting it to true makes S3 incompatible with dns-style bucketnames (with shared or public access). Please note that path-style bucketnames are deprecated by AWS and most clients are only supporting dns-style... Ref.: https://tracker.ceph.com/issues/24348 https://github.com/ceph/ceph/commit/3ba7be8d1ac7ee43e69eebb58263cd080cca1d38 2) the server-side encryption (SSE-KMS) is broken on Nautilus: to reproduce the issue: s3cmd --access_key $ACCESSKEY --secret_key $SECRETKEY --host-bucket "%(bucket)s.$ENDPOINT" --host "$ENDPOINT" --region="$REGION" --signature-v2 --no-preserve --no-ssl --server-side-encryption --server-side-encryption-kms-id ${SECRET##*/} put helloenc.txt s3://testenc/ output: upload: 'helloenc.txt' -> 's3://testenc/helloenc.txt' [1 of 1] 9 of 9 100% in 0s 37.50 B/s done ERROR: S3 error: 403 (AccessDenied): Failed to retrieve the actual key, kms-keyid: cd0903db-c613-49be-96d9-165c02544bc7 rgw log: see below TLDR: after investigating, I found that radosgw was actually getting the barbican secret correctly but the HTTP CODE (=200) validation was failing because of a bug in Nautilus. My understanding is following (please correct me): The bug in src/rgw/rgw_http_client.cc . Since Nautilus HTTP_CODE are converted into ERROR_CODE (200 becomes 0) in the request processing. This happens in RGWHTTPManager::reqs_thread_entry(), which centralizes the processing of (curl) HTTP Requests with multi-treading. This is fine but the member variable http_status of the class RGWHTTPClient is not updated with the resulting HTTP CODE, so the variable keeps its initial value of 0. Then in src/rgw/rgw_crypt.cc the logic is still verifying that http_status is in range [200,299] and this fails... I wrote the following oneliner bugfix for src/rgw/rgw_http_client.cc: diff --git a/src/rgw/rgw_http_client.cc b/src/rgw/rgw_http_client.cc index d0f0baead6..7c115293ad 100644 --- a/src/rgw/rgw_http_client.cc +++ b/src/rgw/rgw_http_client.cc @@ -1146,6 +1146,7 @@ void *RGWHTTPManager::reqs_thread_entry() status = -EAGAIN; } int id = req_data->id; + req_data->client->http_status = http_status; finish_request(req_data, status); switch (result) { case CURLE_OK: The s3cmd is then working fine with KMS server side encryption. Questions: * Could someone please write a fix for the regression of 1) and make a PR ? * Could somebody also make a PR for 2? Thank you for your help. :-) Cheers Francois Scheurer rgw log: export CLUSTER=ceph; /home/local/ceph/build/bin/radosgw -f --cluster ${CLUSTER} --name client.rgw.$(hostname) --setuser ceph --setgroup ceph & tail -fn0 /var/log/ceph/ceph-client.rgw.ewos1-osd1-stage.log | less -IS 2020-02-26 16:32:59.208 7fc1f1c54700 20 Getting KMS encryption key for key=cd0903db-c613-49be-96d9-165c02544bc7 2020-02-26 16:32:59.208 7fc1f1c54700 20 Requesting secret from barbican url=http://keystone.service.stage.i.ewcs.ch:5000/v3/auth/tokens 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPClient::process: http_status: 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::process 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::send 2020-02-26 16:32:59.208 7fc1f1c54700 20 sending request to http://keystone.service.stage.i.ewcs.ch:5000/v3/auth/tokens 2020-02-26 16:32:59.208 7fc1f1c54700 20 ssl verification is set to off 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: client->init_request(req_data): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 register_request mgr=0x56374b865540 req_data->id=4, curl_handle=0x56374c77c4a0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::signal_thread(): write(thread_pipe[1], (void *)&buf, sizeof(buf)): 4 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: signal_thread(): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::send: rgw_http_manager->add_request(req): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::process: send(req): 0 2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : int wait() : ret: 0 2020-02-26 16:32:59.208 7fc2184a1700 20 link_request req_data=0x56374c96a240 req_data->id=4, curl_handle=0x56374c77c4a0 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: http_status: 201 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: rgw_http_error_to_errno(http_status): 0 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: finish_request(req_data, status): status: 0 2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : void finish(int r) : ret: 0 2020-02-26 16:32:59.652 7fc1f1c54700 5 ewdebug: request_key_from_barbican: Accept application/octet-stream X-Auth-Token gAAAAABeVo-xxx 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPClient::process: http_status: 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::process 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::send 2020-02-26 16:32:59.652 7fc1f1c54700 20 sending request to http://barbican.service.stage.i.ewcs.ch:9311/v1/secrets/cd0903db-c613-49be-… 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: client->init_request(req_data): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 register_request mgr=0x56374b865540 req_data->id=5, curl_handle=0x56374c77c4a0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::signal_thread(): write(thread_pipe[1], (void *)&buf, sizeof(buf)): 4 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: signal_thread(): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::send: rgw_http_manager->add_request(req): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::process: send(req): 0 2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : int wait() : ret: 0 2020-02-26 16:32:59.652 7fc2184a1700 20 link_request req_data=0x56374c96a240 req_data->id=5, curl_handle=0x56374c77c4a0 => 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: http_status: 200 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: rgw_http_error_to_errno(http_status): 0 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: finish_request(req_data, status): status: 0 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : void finish(int r) : ret: 0 2020-02-26 16:32:59.752 7fc1f1c54700 5 ewdebug: request_key_from_barbican: secret_req.process: 0 => 2020-02-26 16:32:59.752 7fc1f1c54700 5 ewdebug: request_key_from_barbican: secret_req.get_http_status: 0 2020-02-26 16:32:59.752 7fc1f1c54700 5 ewdebug: request_key_from_barbican: secret_req.get_http_status not in [200,299] range! 2020-02-26 16:32:59.752 7fc1f1c54700 5 Failed to retrieve secret from barbican:cd0903db-c613-49be-96d9-165c02544bc7 2020-02-26 16:32:59.752 7fc1f1c54700 5 ERROR: failed to retrieve actual key from key_id: cd0903db-c613-49be-96d9-165c02544bc7 2020-02-26 16:32:59.752 7fc1f1c54700 2 req 1 1.092s s3:put_obj completing 2020-02-26 16:32:59.752 7fc1f1c54700 2 req 1 1.092s s3:put_obj op status=-13 2020-02-26 16:32:59.752 7fc1f1c54700 2 req 1 1.092s s3:put_obj http status=403 2020-02-26 16:32:59.752 7fc1f1c54700 1 ====== req done req=0x56374c9808d0 op status=-13 http_status=403 latency=1.092s ====== => we see that http_status is correct (200) but the variable secret_req.get_http_status (member of class RGWHTTPClient) is incorrect (0 instead of 200)

2 4

How to use ceph-volume raw ?
by hoannv46＠gmail.com 14 Apr '20

14 Apr '20

My cluster on 15.2.1 versions. I use ceph-volume raw to add new osd but it has error : ceph-volume raw prepare --bluestore --data /dev/vdb Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b3b79678-c40d-4a84-9c47-84c4d7234407 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/chown -R ceph:ceph /dev/vdb Running command: /usr/bin/ln -s /dev/vdb /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2020-04-14T09:36:40.234+0700 7ff302248700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bi n,: (2) No such file or directory 2020-04-14T09:36:40.234+0700 7ff302248700 -1 AuthRegistry(0x7ff2fc058388) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyr ing.bin,, disabling cephx stderr: got monmap epoch 1 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQC3IZVe+ZSrChAAhGUXJ5Nkg8K6UB9tiNaFxQ== stdout: creating /var/lib/ceph/osd/ceph-0/keyring added entity osd.0 auth(key=AQC3IZVe+ZSrChAAhGUXJ5Nkg8K6UB9tiNaFxQ==) Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --os d-uuid b3b79678-c40d-4a84-9c47-84c4d7234407 --setuser ceph --setgroup ceph stderr: 2020-04-14T09:36:40.742+0700 7f83483c4bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid stderr: 2020-04-14T09:36:40.742+0700 7f83483c4bc0 -1 bdev(0x556ebee86c00 /var/lib/ceph/osd/ceph-0//block) _lock flock failed on /var/lib/ceph/osd/ceph-0//block stderr: 2020-04-14T09:36:40.742+0700 7f83483c4bc0 -1 bdev(0x556ebee86c00 /var/lib/ceph/osd/ceph-0//block) open failed to lock /var/lib/ceph/osd/ceph-0//block: (11) Resource temporarily unavail able stderr: 2020-04-14T09:36:40.743+0700 7f83483c4bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (11) Resource temporarily unavailable stderr: 2020-04-14T09:36:40.743+0700 7f83483c4bc0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (11) Resource temporarily unavailable stderr: 2020-04-14T09:36:40.744+0700 7f83483c4bc0 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0/: (11) Resource temporarily unavailable --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it stderr: 2020-04-14T09:36:40.959+0700 7f0b569c3700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bi n,: (2) No such file or directory 2020-04-14T09:36:40.959+0700 7f0b569c3700 -1 AuthRegistry(0x7f0b50058388) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyr ing.bin,, disabling cephx stderr: purged osd.0 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --os d-data /var/lib/ceph/osd/ceph-0/ --osd-uuid b3b79678-c40d-4a84-9c47-84c4d7234407 --setuser ceph --setgroup ceph After that i create new parition of /dev/vdb it create osd successfully. then ceph-volume raw activate --device /dev/sdb1 --> systemd support not yet implemented

1 0

Re: Recommendation for decent write latency performance from HDDs
by Maged Mokhtar 12 Apr '20

12 Apr '20

On 12/04/2020 21:41, huxiaoyu(a)horebdata.cn wrote: > thanks again. > > I will try PetaSAN later. How big is the recommended cache size > (dm-writecache) for a OSD? > Actual number of partitions per SSD is more important, each partition serves 1 HDD/OSD, we allow 1-8. For size anything above 50GB is good, more will help specially with read caching. You need 2% of cache size in RAM, so a 100GB partitions requires 2GB in RAM, RAM is used internally to keep track of things, no data caching happens in RAM. /Maged > ------------------------------------------------------------------------ > huxiaoyu(a)horebdata.cn > > *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> > *Date:* 2020-04-12 21:34 > *To:* huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; Reed > Dier <mailto:reed.dier@focusvq.com>; jesper <mailto:jesper@krogh.cc> > *CC:* ceph-users <mailto:ceph-users@ceph.io> > *Subject:* Re: [ceph-users] Re: Recommendation for decent write > latency performance from HDDs > > > On 12/04/2020 20:35, huxiaoyu(a)horebdata.cn wrote: >> >> >> That said, with a recent kernel such as 4.19 stable release, and >> a decent enterprise SSD such as Intel D4510/4610, i do not need >> to worry about the data safety related to dm-writecache. >> >> thanks a lot. >> >> samuel >> > > The patch recently went in 5.4 and backported to 4.18+, so you > need to check your version has it. > > This will guarantee that a fua (force unit access)/sync write > generated by the OSD, will end up on media before successful > return. Enterprise drives + any drive that supports PLP will > guarantee stored data will not corrupt in a power failure. > > If you wish, you can install PetaSAN to test cache performance > yourself. > > /Maged > > >> ------------------------------------------------------------------------ >> huxiaoyu(a)horebdata.cn >> >> *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> >> *Date:* 2020-04-12 20:03 >> *To:* huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; >> Reed Dier <mailto:reed.dier@focusvq.com>; jesper >> <mailto:jesper@krogh.cc> >> *CC:* ceph-users <mailto:ceph-users@ceph.io> >> *Subject:* Re: [ceph-users] Re: Recommendation for decent >> write latency performance from HDDs >> >> >> On 12/04/2020 18:10, huxiaoyu(a)horebdata.cn wrote: >>> Dear Maged Mokhtar， >>> >>> It is very interesting to know that your experiment shows >>> dm-writecache would be better than other alternatives. I >>> have two questions: >> >> yes much better. >> >>> >>> 1 can one cache device serve multiple HDDs? I know bcache >>> can do this, which is convenient. dont know whether >>> dm-writecache has such a feature >> >> it works on a partition, so you can partition your disk to >> several partitions to support multiple OSDs,in our ui we >> allow from 1-8 partitions. >> >>> 2 Did you test whether write-back to disks from >>> dm-writecache is power-safe or not. As far as know, bcache >>> does not gurantee power-safe writebacks, thus i have to turn >>> off HDD write cache (otherwise a data loss may occur) >>> >> Get a recent kernel and insure it has the fua patch >> mentioned, this will correctly handle sync writes, else you >> may lose data. You also need a recent lvm tool set that >> support dm-writecache. You need also use an SSD with PLP >> support (enterprise models and some consumer models), some >> cheaper SSDs without PLP support can lose existing stored >> data on power loss, since their write cycle involves a >> read/erase/write block so a power loss can erase already >> stored data on such consumer devices. We also have another >> patch (see our source) that adds mirroring of metadata to >> dm-writecache to handle this, but that is not needed for >> decent drives. >> >>> best regards, >>> >>> samuel >>> >>> >>> >>> >>> ------------------------------------------------------------------------ >>> huxiaoyu(a)horebdata.cn >>> >>> *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> >>> *Date:* 2020-04-12 16:45 >>> *To:* Reed Dier <mailto:reed.dier@focusvq.com>; jesper >>> <mailto:jesper@krogh.cc> >>> *CC:* ceph-users <mailto:ceph-users@ceph.io> >>> *Subject:* [ceph-users] Re: Recommendation for decent >>> write latency performance from HDDs >>> On 10/04/2020 23:17, Reed Dier wrote: >>> > Going to resurrect this thread to provide another option: >>> > >>> > LVM-cache, ie putting a cache device in-front of the >>> bluestore-LVM LV. >>> > >>> > I only mention this because I noticed it in the SUSE >>> documentation for >>> > SES6 (based on Nautilus) here: >>> > >>> https://documentation.suse.com/ses/6/html/ses-all/lvmcache.html >>> in PetaSAN project, we support dm-writecache and it >>> works very well. We >>> had done tests with other cache devices such as bcache >>> and dm-cache, >>> and it is much better. it is mainly a write cache, but >>> reads are read >>> from cache device if present, but does not promote reads >>> from slow >>> device. Typically with hdd clusters, write latency is >>> the issue, reads >>> are helped by OSD cache and in case of reduplicated >>> pools, are much >>> faster anyways. >>> You need a recent kernel, we have an upstreamed patch: >>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/dri… >>> + depending on your distribution, you may need an >>> updated lvm tool set. >>> /Maged >>> > >>> >> * If you plan to use a fast drive as an LVM cache >>> for multiple >>> >> OSDs, be aware that all OSD operations (including >>> replication) >>> >> will go through the caching device. All >>> reads will be queried >>> >> from the caching device, and are only served from >>> the slow device >>> >> in case of a cache miss. Writes are always >>> applied to the caching >>> >> device first, and are flushed to the slow device >>> at a later time >>> >> ('writeback' is the default caching mode). >>> >> * When deciding whether to utilize an LVM cache, >>> verify whether the >>> >> fast drive can serve as a front for multiple OSDs >>> while still >>> >> providing an acceptable amount of IOPS. You >>> can test it by >>> >> measuring the maximum amount of IOPS that the >>> fast device can >>> >> serve, and then dividing the result by the number >>> of OSDs behind >>> >> the fast device. If the result is lower or close >>> to the maximum >>> >> amount of IOPS that the OSD can provide without >>> the cache, LVM >>> >> cache is probably not suited for this setup. >>> >> >>> >> * The interaction of the LVM cache device with OSDs is >>> >> important. Writes are periodically flushed from >>> the caching >>> >> device to the slow device. If the incoming >>> traffic is sustained >>> >> and significant, the caching device will struggle >>> to keep up with >>> >> incoming requests as well as the flushing >>> process, resulting in >>> >> performance drop. Unless the fast device can >>> provide much more >>> >> IOPS with better latency than the slow device, do >>> not use LVM >>> >> cache with a sustained high volume workload. >>> Traffic in a burst >>> >> pattern is more suited for LVM cache as it gives >>> the cache time >>> >> to flush its dirty data without interfering with >>> client traffic. >>> >> For a sustained low traffic workload, it is >>> difficult to guess in >>> >> advance whether using LVM cache will improve >>> performance. The >>> >> best test is to benchmark and compare the LVM >>> cache setup against >>> >> the WAL/DB setup. Moreover, as small writes >>> are heavy on the WAL >>> >> partition, it is suggested to use the fast device >>> for the DB >>> >> and/or WAL instead of an LVM cache. >>> >> >>> > >>> > So it sounds like you could partition your NVMe for >>> either LVM-cache, >>> > DB/WAL, or both? >>> > >>> > Just figured this sounded a bit more akin to what you >>> were looking for >>> > in your original post and figured I would share. >>> > >>> > I don't use this, but figured I would share it. >>> > >>> > Reed >>> > >>> >> On Apr 4, 2020, at 9:12 AM, jesper(a)krogh.cc >>> <mailto:jesper@krogh.cc> >>> >> wrote: >>> >> >>> >> Hi. >>> >> >>> >> We have a need for "bulk" storage - but with decent >>> write latencies. >>> >> Normally we would do this with a DAS with a Raid5 >>> with 2GB Battery >>> >> backed write cache in front - As cheap as possible >>> but still getting the >>> >> features of scalability of ceph. >>> >> >>> >> In our "first" ceph cluster we did the same - just >>> stuffed in BBWC >>> >> in the OSD nodes and we're fine - but now we're onto >>> the next one and >>> >> systems like: >>> >> >>> https://www.supermicro.com/en/products/system/1U/6119/SSG-6119P-ACR12N4L.cfm >>> >> Does not support a Raid controller like that - but is >>> branded as for >>> >> "Ceph >>> >> Storage Solutions". >>> >> >>> >> It do however support 4 NVMe slots in the front - So >>> - some level of >>> >> "tiering" using the NVMe drives should be what is >>> "suggested" - but what >>> >> do people do? What is recommeneded. I see multiple >>> options: >>> >> >>> >> Ceph tiering at the "pool - layer": >>> >> >>> https://docs.ceph.com/docs/master/rados/operations/cache-tiering/ >>> >> And rumors that it is "deprectated: >>> >> >>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2.0/html… >>> >> >>> >> Pro: Abstract layer >>> >> Con: Deprecated? - Lots of warnings? >>> >> >>> >> Offloading the block.db on NVMe / SSD: >>> >> >>> https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ >>> >> >>> >> Pro: Easy to deal with - seem heavily supported. >>> >> Con: As far as I can tell - this will only benefit >>> the metadata of the >>> >> osd- not actual data. Thus a data-commit to the osd >>> til still be >>> >> dominated >>> >> by the writelatency of the underlying - very slow HDD. >>> >> >>> >> Bcache: >>> >> >>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-June/027713.html >>> >> >>> >> Pro: Closest to the BBWC mentioned above - but with >>> way-way larger cache >>> >> sizes. >>> >> Con: It is hard to see if I end up being the only one >>> on the planet using >>> >> this >>> >> solution. >>> >> >>> >> Eat it - Writes will be as slow as hitting dead-rust >>> - anything that >>> >> cannot live >>> >> with that need to be entirely on SSD/NVMe. >>> >> >>> >> Other? >>> >> >>> >> Thanks for your input. >>> >> >>> >> Jesper >>> >> _______________________________________________ >>> >> ceph-users mailing list -- ceph-users(a)ceph.io >>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > >>> > >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users(a)ceph.io >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>

1 0

Re: Recommendation for decent write latency performance from HDDs
by Maged Mokhtar 12 Apr '20

12 Apr '20

On 12/04/2020 20:35, huxiaoyu(a)horebdata.cn wrote: > > > That said, with a recent kernel such as 4.19 stable release, and a > decent enterprise SSD such as Intel D4510/4610, i do not need to worry > about the data safety related to dm-writecache. > > thanks a lot. > > samuel > The patch recently went in 5.4 and backported to 4.18+, so you need to check your version has it. This will guarantee that a fua (force unit access)/sync write generated by the OSD, will end up on media before successful return. Enterprise drives + any drive that supports PLP will guarantee stored data will not corrupt in a power failure. If you wish, you can install PetaSAN to test cache performance yourself. /Maged > ------------------------------------------------------------------------ > huxiaoyu(a)horebdata.cn > > *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> > *Date:* 2020-04-12 20:03 > *To:* huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; Reed > Dier <mailto:reed.dier@focusvq.com>; jesper <mailto:jesper@krogh.cc> > *CC:* ceph-users <mailto:ceph-users@ceph.io> > *Subject:* Re: [ceph-users] Re: Recommendation for decent write > latency performance from HDDs > > > On 12/04/2020 18:10, huxiaoyu(a)horebdata.cn wrote: >> Dear Maged Mokhtar， >> >> It is very interesting to know that your experiment shows >> dm-writecache would be better than other alternatives. I have two >> questions: > > yes much better. > >> >> 1 can one cache device serve multiple HDDs? I know bcache can do >> this, which is convenient. dont know whether dm-writecache has >> such a feature > > it works on a partition, so you can partition your disk to several > partitions to support multiple OSDs,in our ui we allow from 1-8 > partitions. > >> 2 Did you test whether write-back to disks from dm-writecache is >> power-safe or not. As far as know, bcache does not gurantee >> power-safe writebacks, thus i have to turn off HDD write cache >> (otherwise a data loss may occur) >> > Get a recent kernel and insure it has the fua patch mentioned, > this will correctly handle sync writes, else you may lose data. > You also need a recent lvm tool set that support dm-writecache. > You need also use an SSD with PLP support (enterprise models and > some consumer models), some cheaper SSDs without PLP support can > lose existing stored data on power loss, since their write cycle > involves a read/erase/write block so a power loss can erase > already stored data on such consumer devices. We also have another > patch (see our source) that adds mirroring of metadata to > dm-writecache to handle this, but that is not needed for decent > drives. > >> best regards, >> >> samuel >> >> >> >> >> ------------------------------------------------------------------------ >> huxiaoyu(a)horebdata.cn >> >> *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> >> *Date:* 2020-04-12 16:45 >> *To:* Reed Dier <mailto:reed.dier@focusvq.com>; jesper >> <mailto:jesper@krogh.cc> >> *CC:* ceph-users <mailto:ceph-users@ceph.io> >> *Subject:* [ceph-users] Re: Recommendation for decent write >> latency performance from HDDs >> On 10/04/2020 23:17, Reed Dier wrote: >> > Going to resurrect this thread to provide another option: >> > >> > LVM-cache, ie putting a cache device in-front of the >> bluestore-LVM LV. >> > >> > I only mention this because I noticed it in the SUSE >> documentation for >> > SES6 (based on Nautilus) here: >> > https://documentation.suse.com/ses/6/html/ses-all/lvmcache.html >> in PetaSAN project, we support dm-writecache and it works >> very well. We >> had done tests with other cache devices such as bcache and >> dm-cache, >> and it is much better. it is mainly a write cache, but reads >> are read >> from cache device if present, but does not promote reads from >> slow >> device. Typically with hdd clusters, write latency is the >> issue, reads >> are helped by OSD cache and in case of reduplicated pools, >> are much >> faster anyways. >> You need a recent kernel, we have an upstreamed patch: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/dri… >> + depending on your distribution, you may need an updated lvm >> tool set. >> /Maged >> > >> >> * If you plan to use a fast drive as an LVM cache for >> multiple >> >> OSDs, be aware that all OSD operations (including >> replication) >> >> will go through the caching device. All reads will be >> queried >> >> from the caching device, and are only served from the >> slow device >> >> in case of a cache miss. Writes are always applied to >> the caching >> >> device first, and are flushed to the slow device at a >> later time >> >> ('writeback' is the default caching mode). >> >> * When deciding whether to utilize an LVM cache, verify >> whether the >> >> fast drive can serve as a front for multiple OSDs >> while still >> >> providing an acceptable amount of IOPS. You can test it by >> >> measuring the maximum amount of IOPS that the fast >> device can >> >> serve, and then dividing the result by the number of >> OSDs behind >> >> the fast device. If the result is lower or close to >> the maximum >> >> amount of IOPS that the OSD can provide without the >> cache, LVM >> >> cache is probably not suited for this setup. >> >> >> >> * The interaction of the LVM cache device with OSDs is >> >> important. Writes are periodically flushed from the >> caching >> >> device to the slow device. If the incoming traffic >> is sustained >> >> and significant, the caching device will struggle to >> keep up with >> >> incoming requests as well as the flushing process, >> resulting in >> >> performance drop. Unless the fast device can provide >> much more >> >> IOPS with better latency than the slow device, do not >> use LVM >> >> cache with a sustained high volume workload. Traffic >> in a burst >> >> pattern is more suited for LVM cache as it gives the >> cache time >> >> to flush its dirty data without interfering with >> client traffic. >> >> For a sustained low traffic workload, it is difficult >> to guess in >> >> advance whether using LVM cache will improve >> performance. The >> >> best test is to benchmark and compare the LVM cache >> setup against >> >> the WAL/DB setup. Moreover, as small writes are heavy >> on the WAL >> >> partition, it is suggested to use the fast device for >> the DB >> >> and/or WAL instead of an LVM cache. >> >> >> > >> > So it sounds like you could partition your NVMe for either >> LVM-cache, >> > DB/WAL, or both? >> > >> > Just figured this sounded a bit more akin to what you were >> looking for >> > in your original post and figured I would share. >> > >> > I don't use this, but figured I would share it. >> > >> > Reed >> > >> >> On Apr 4, 2020, at 9:12 AM, jesper(a)krogh.cc >> <mailto:jesper@krogh.cc> >> >> wrote: >> >> >> >> Hi. >> >> >> >> We have a need for "bulk" storage - but with decent write >> latencies. >> >> Normally we would do this with a DAS with a Raid5 with 2GB >> Battery >> >> backed write cache in front - As cheap as possible but >> still getting the >> >> features of scalability of ceph. >> >> >> >> In our "first" ceph cluster we did the same - just stuffed >> in BBWC >> >> in the OSD nodes and we're fine - but now we're onto the >> next one and >> >> systems like: >> >> >> https://www.supermicro.com/en/products/system/1U/6119/SSG-6119P-ACR12N4L.cfm >> >> Does not support a Raid controller like that - but is >> branded as for >> >> "Ceph >> >> Storage Solutions". >> >> >> >> It do however support 4 NVMe slots in the front - So - >> some level of >> >> "tiering" using the NVMe drives should be what is >> "suggested" - but what >> >> do people do? What is recommeneded. I see multiple options: >> >> >> >> Ceph tiering at the "pool - layer": >> >> >> https://docs.ceph.com/docs/master/rados/operations/cache-tiering/ >> >> And rumors that it is "deprectated: >> >> >> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2.0/html… >> >> >> >> Pro: Abstract layer >> >> Con: Deprecated? - Lots of warnings? >> >> >> >> Offloading the block.db on NVMe / SSD: >> >> >> https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ >> >> >> >> Pro: Easy to deal with - seem heavily supported. >> >> Con: As far as I can tell - this will only benefit the >> metadata of the >> >> osd- not actual data. Thus a data-commit to the osd til >> still be >> >> dominated >> >> by the writelatency of the underlying - very slow HDD. >> >> >> >> Bcache: >> >> >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-June/027713.html >> >> >> >> Pro: Closest to the BBWC mentioned above - but with >> way-way larger cache >> >> sizes. >> >> Con: It is hard to see if I end up being the only one on >> the planet using >> >> this >> >> solution. >> >> >> >> Eat it - Writes will be as slow as hitting dead-rust - >> anything that >> >> cannot live >> >> with that need to be entirely on SSD/NVMe. >> >> >> >> Other? >> >> >> >> Thanks for your input. >> >> >> >> Jesper >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > >> > >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

2 1

Re: Recommendation for decent write latency performance from HDDs
by Maged Mokhtar 12 Apr '20

12 Apr '20

On 12/04/2020 18:10, huxiaoyu(a)horebdata.cn wrote: > Dear Maged Mokhtar， > > It is very interesting to know that your experiment shows > dm-writecache would be better than other alternatives. I have two > questions: yes much better. > > 1 can one cache device serve multiple HDDs? I know bcache can do > this, which is convenient. dont know whether dm-writecache has such a > feature it works on a partition, so you can partition your disk to several partitions to support multiple OSDs,in our ui we allow from 1-8 partitions. > 2 Did you test whether write-back to disks from dm-writecache is > power-safe or not. As far as know, bcache does not gurantee power-safe > writebacks, thus i have to turn off HDD write cache (otherwise a data > loss may occur) > Get a recent kernel and insure it has the fua patch mentioned, this will correctly handle sync writes, else you may lose data. You also need a recent lvm tool set that support dm-writecache. You need also use an SSD with PLP support (enterprise models and some consumer models), some cheaper SSDs without PLP support can lose existing stored data on power loss, since their write cycle involves a read/erase/write block so a power loss can erase already stored data on such consumer devices. We also have another patch (see our source) that adds mirroring of metadata to dm-writecache to handle this, but that is not needed for decent drives. > best regards, > > samuel > > > > > ------------------------------------------------------------------------ > huxiaoyu(a)horebdata.cn > > *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> > *Date:* 2020-04-12 16:45 > *To:* Reed Dier <mailto:reed.dier@focusvq.com>; jesper > <mailto:jesper@krogh.cc> > *CC:* ceph-users <mailto:ceph-users@ceph.io> > *Subject:* [ceph-users] Re: Recommendation for decent write > latency performance from HDDs > On 10/04/2020 23:17, Reed Dier wrote: > > Going to resurrect this thread to provide another option: > > > > LVM-cache, ie putting a cache device in-front of the > bluestore-LVM LV. > > > > I only mention this because I noticed it in the SUSE > documentation for > > SES6 (based on Nautilus) here: > > https://documentation.suse.com/ses/6/html/ses-all/lvmcache.html > in PetaSAN project, we support dm-writecache and it works very > well. We > had done tests with other cache devices such as bcache and dm-cache, > and it is much better. it is mainly a write cache, but reads are read > from cache device if present, but does not promote reads from slow > device. Typically with hdd clusters, write latency is the issue, > reads > are helped by OSD cache and in case of reduplicated pools, are much > faster anyways. > You need a recent kernel, we have an upstreamed patch: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/dri… > + depending on your distribution, you may need an updated lvm tool > set. > /Maged > > > >> * If you plan to use a fast drive as an LVM cache for multiple > >> OSDs, be aware that all OSD operations (including replication) > >> will go through the caching device. All reads will be queried > >> from the caching device, and are only served from the slow > device > >> in case of a cache miss. Writes are always applied to the > caching > >> device first, and are flushed to the slow device at a later > time > >> ('writeback' is the default caching mode). > >> * When deciding whether to utilize an LVM cache, verify > whether the > >> fast drive can serve as a front for multiple OSDs while still > >> providing an acceptable amount of IOPS. You can test it by > >> measuring the maximum amount of IOPS that the fast device can > >> serve, and then dividing the result by the number of OSDs > behind > >> the fast device. If the result is lower or close to the maximum > >> amount of IOPS that the OSD can provide without the cache, LVM > >> cache is probably not suited for this setup. > >> > >> * The interaction of the LVM cache device with OSDs is > >> important. Writes are periodically flushed from the caching > >> device to the slow device. If the incoming traffic is sustained > >> and significant, the caching device will struggle to keep > up with > >> incoming requests as well as the flushing process, resulting in > >> performance drop. Unless the fast device can provide much more > >> IOPS with better latency than the slow device, do not use LVM > >> cache with a sustained high volume workload. Traffic in a burst > >> pattern is more suited for LVM cache as it gives the cache time > >> to flush its dirty data without interfering with client > traffic. > >> For a sustained low traffic workload, it is difficult to > guess in > >> advance whether using LVM cache will improve performance. The > >> best test is to benchmark and compare the LVM cache setup > against > >> the WAL/DB setup. Moreover, as small writes are heavy on > the WAL > >> partition, it is suggested to use the fast device for the DB > >> and/or WAL instead of an LVM cache. > >> > > > > So it sounds like you could partition your NVMe for either > LVM-cache, > > DB/WAL, or both? > > > > Just figured this sounded a bit more akin to what you were > looking for > > in your original post and figured I would share. > > > > I don't use this, but figured I would share it. > > > > Reed > > > >> On Apr 4, 2020, at 9:12 AM, jesper(a)krogh.cc > <mailto:jesper@krogh.cc> > >> wrote: > >> > >> Hi. > >> > >> We have a need for "bulk" storage - but with decent write > latencies. > >> Normally we would do this with a DAS with a Raid5 with 2GB Battery > >> backed write cache in front - As cheap as possible but still > getting the > >> features of scalability of ceph. > >> > >> In our "first" ceph cluster we did the same - just stuffed in BBWC > >> in the OSD nodes and we're fine - but now we're onto the next > one and > >> systems like: > >> > https://www.supermicro.com/en/products/system/1U/6119/SSG-6119P-ACR12N4L.cfm > >> Does not support a Raid controller like that - but is branded > as for > >> "Ceph > >> Storage Solutions". > >> > >> It do however support 4 NVMe slots in the front - So - some > level of > >> "tiering" using the NVMe drives should be what is "suggested" - > but what > >> do people do? What is recommeneded. I see multiple options: > >> > >> Ceph tiering at the "pool - layer": > >> https://docs.ceph.com/docs/master/rados/operations/cache-tiering/ > >> And rumors that it is "deprectated: > >> > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2.0/html… > >> > >> Pro: Abstract layer > >> Con: Deprecated? - Lots of warnings? > >> > >> Offloading the block.db on NVMe / SSD: > >> > https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ > >> > >> Pro: Easy to deal with - seem heavily supported. > >> Con: As far as I can tell - this will only benefit the metadata > of the > >> osd- not actual data. Thus a data-commit to the osd til still be > >> dominated > >> by the writelatency of the underlying - very slow HDD. > >> > >> Bcache: > >> > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-June/027713.html > >> > >> Pro: Closest to the BBWC mentioned above - but with way-way > larger cache > >> sizes. > >> Con: It is hard to see if I end up being the only one on the > planet using > >> this > >> solution. > >> > >> Eat it - Writes will be as slow as hitting dead-rust - anything > that > >> cannot live > >> with that need to be entirely on SSD/NVMe. > >> > >> Other? > >> > >> Thanks for your input. > >> > >> Jesper > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Recommendation for decent write latency performance from HDDs
by jesper＠krogh.cc 12 Apr '20

12 Apr '20

Hi. We have a need for "bulk" storage - but with decent write latencies. Normally we would do this with a DAS with a Raid5 with 2GB Battery backed write cache in front - As cheap as possible but still getting the features of scalability of ceph. In our "first" ceph cluster we did the same - just stuffed in BBWC in the OSD nodes and we're fine - but now we're onto the next one and systems like: https://www.supermicro.com/en/products/system/1U/6119/SSG-6119P-ACR12N4L.cfm Does not support a Raid controller like that - but is branded as for "Ceph Storage Solutions". It do however support 4 NVMe slots in the front - So - some level of "tiering" using the NVMe drives should be what is "suggested" - but what do people do? What is recommeneded. I see multiple options: Ceph tiering at the "pool - layer": https://docs.ceph.com/docs/master/rados/operations/cache-tiering/ And rumors that it is "deprectated: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2.0/html… Pro: Abstract layer Con: Deprecated? - Lots of warnings? Offloading the block.db on NVMe / SSD: https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/ Pro: Easy to deal with - seem heavily supported. Con: As far as I can tell - this will only benefit the metadata of the osd- not actual data. Thus a data-commit to the osd til still be dominated by the writelatency of the underlying - very slow HDD. Bcache: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-June/027713.html Pro: Closest to the BBWC mentioned above - but with way-way larger cache sizes. Con: It is hard to see if I end up being the only one on the planet using this solution. Eat it - Writes will be as slow as hitting dead-rust - anything that cannot live with that need to be entirely on SSD/NVMe. Other? Thanks for your input. Jesper

5 7

How to fix 1 pg stale+active+clean
by Marc Roos 12 Apr '20

12 Apr '20

How to fix 1 pg marked as stale+active+clean pg 30.4 is stuck stale for 175342.419261, current state stale+active+clean, last acting [31]

3 4

Re: Understanding monitor requirements
by Brian Topping 12 Apr '20

12 Apr '20

Hi again, after all, this appears to be an MTU issue: Baseline: 1) Two of the nodes have a straight ethernet with 1500MTU, the third (problem) node is on a WAN tunnel with a restricted MTU. It appears that the MTUs were not set up correctly, so no surprise some software has problems. 2) I decided I knew Ceph well enough that I could handle recovery from disaster cases in Rook and it has advantages I can use. So please keep that in mind as I discuss this issue. (For those who aren’t familiar, Rook just orchestrates containers that are built by the Ceph team.) 3) In Rook, monitors run as pods under a CNI. The CNI adds additional overhead for transit, in my case a VxLAN overlay network. This overhead is apparently not enough to cause problems when running between nodes on a full 1500MTU local net. So the first two monitors come up cleanly. After spending a lot of time looking at the logs, I could see the mon map of all three nodes properly distributed, but when it came to an election, all nodes knew the election epoch but the third was not joining. Comparing the logs of the second node as peon with the troubled third node on the other side of the restricted MTU, the difference appeared to be that the third node was not providing a feature proposal when in fact it probably was and it was being dropped. So the election would end without the third node being a part of the quorum. The third node stopped asking for a new election and that’s how things ended. What I did this morning was figure out the MTU of the WAN tunnel and then change the entire CNI to that number. My expectation was that everything would start working and the necessary fragmentation would be generated by the client end of any connection. Instead, the second node that was previously able to join as peon was no longer able to do so. It seems to follow that the smaller MTU (1340 to be exact) set on the overall CNI causes the elections to fail. There are a number of things that I can do to improve the behavior of the cluster (such as PMTUD), but if Ceph is not going to work with a small MTU, all bets are off. I tried looking for issues in tracker.ceph.com, but apparently I haven’t logged in there for a while and my account was deleted. I applied for a new one. Any ideas what I can do here? Thanks! Brian

1 1

MDS: what's the purpose for LogEvent with empty metablob?
by Xinying Song 11 Apr '20

11 Apr '20

Hi, everyone: what's the purpose for LogEvent with empty metablob? For example in link/unlink operation cross two active mds, the procedure may look like below: 1. master send PREPARE to slave 2. slave receives PREPARE, and journal an ESlaveUpdate::OP_PREPARE, then responds ACK to master 3. master receives ACK and journal an EUpdate, then send OP_FINISH to slave 4. slave receives OP_FINISH and journal ESlaveUpdate::OP_COMMIT, then send OP_COMMIT to master 5. master receives OP_COMMIT and journal an ECommitted Why logevents in step 4 and 5 are necessary? Both of them have empty metablob. I guess these two logevents are originally used for a scene that crashes happen, but actually it seems not necessary. For example, if cash happens, after failed mds are brought up again, in resolve stage, master will resend OP_FINISH to slave, then things will continue as expected. Any tips about this question are appreciated. Thanks!

1 0

Octopus 15.2.1
by gert.wieberdink＠ziggo.nl 11 Apr '20

11 Apr '20

I am trying to install a fresh Ceph cluster on CentOS 8. Using the latest Ceph repo for el8, it still is not possible because of certain dependencies: libleveldb.so.1 needed by ceph-osd. Even after manually downloading and installing the leveldb-1.20-1.el8.x86_64.rpm package, there are still dependencies: Problem: package ceph-mgr-2:15.2.1-0.el8.x86_64 requires ceph-mgr-modules-core = 2:15.2.1-0.el8, but none of the providers can be installed - conflicting requests - nothing provides python3-cherrypy needed by ceph-mgr-modules-core-2:15.2.1-0.el8.noarch - nothing provides python3-pecan needed by ceph-mgr-modules-core-2:15.2.1-0.el8.noarch Is there a way to perform a fresh Ceph install on CentOS 8? Thanking in advance for your answer.

3 2

ceph-df free discrepancy
by Reed Dier 10 Apr '20

10 Apr '20

Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most OSDs are .7) > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 530 TiB 163 TiB 366 TiB 367 TiB 69.19 > ssd 107 TiB 37 TiB 70 TiB 70 TiB 65.33 > TOTAL 637 TiB 201 TiB 436 TiB 437 TiB 68.54 > > POOLS: > POOL ID STORED OBJECTS USED %USED MAX AVAIL > fs-metadata 16 44 GiB 4.16M 44 GiB 0.25 5.6 TiB > cephfs-hdd-3x 17 46 TiB 109.54M 144 TiB 61.81 30 TiB > objects-hybrid 20 46 TiB 537.08M 187 TiB 91.71 5.6 TiB > objects-hdd 24 224 GiB 50.81k 676 GiB 0.74 30 TiB > rbd-hybrid 29 3.8 TiB 1.19M 11 TiB 40.38 5.6 TiB > device_health_metrics 33 270 MiB 327 270 MiB 0 30 TiB > rbd-ssd 34 4.2 TiB 1.19M 12 TiB 41.55 5.6 TiB > cephfs-hdd-ec73 37 42 TiB 30.35M 72 TiB 44.78 62 TiB I have a few pools that don't look like they are calculating the available storage for that pool correctly. Specifically, any of my hybrid pools (20,29) or all-SSD pools (16,34). For my hybrid pools, I have a crush rule of take 1 of host in the ssd root, take -1 chassis in the hdd root. For my ssd pools, I have a crush rule of take 0 of host in the ssd root. Now I have 60 ssd osds 1.92T each, and sadly distribution is imperfect (leaving those issues out of this), and I have plenty of underfull and overfull osds, which I am trying to manually reweighs to get my most full's down to free up space: > $ ceph osd df class ssd | sort -k17 > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS > MIN/MAX VAR: 0.80/1.21 STDDEV: 5.56 > TOTAL 107 TiB 70 TiB 68 TiB 163 GiB 431 GiB 37 TiB 65.33 > 28 ssd 1.77879 1.00000 1.8 TiB 951 GiB 916 GiB 7.1 GiB 5.8 GiB 871 GiB 52.20 0.80 68 up > 33 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1010 GiB 6.0 MiB 5.9 GiB 777 GiB 57.33 0.88 74 up > 47 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1011 GiB 6.7 GiB 6.4 GiB 776 GiB 57.38 0.88 75 up > [SNIP] > 57 ssd 1.77879 0.98000 1.8 TiB 1.4 TiB 1.3 TiB 6.2 GiB 8.6 GiB 417 GiB 77.08 1.18 102 up > 107 ssd 1.80429 1.00000 1.8 TiB 1.4 TiB 1.3 TiB 7.0 GiB 8.7 GiB 422 GiB 77.15 1.18 102 up > 50 ssd 1.77879 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 5.5 MiB 8.6 GiB 381 GiB 79.10 1.21 105 up > 60 ssd 1.77879 0.92000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 9.0 GiB 379 GiB 79.17 1.21 105 up That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't, which is still far short of 37TiB raw free, as expected. However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB. But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes. Meaning if my napkin math is right, should show ~22.8TiB free. Am I grossly mis-understanding how this is calculated? Maybe this is fixed in Octopus? Just trying to get a grasp on what I'm seeing not matching expectations. Thanks, Reed

2 2

remove S3 bucket with rados CLI
by Robert Sander 10 Apr '20

10 Apr '20

Hi, is it possible to remove an S3 bucket and its S3 objects with the rados CLI tool (removing the low level Ceph objects)? The situation is a nearfull cluster with OSDs filled more than 80% and the default.rgw.buckets.data being reported as 100% full. Read operations are still possible but no write. s3cmd rb … says operation halted and radosgw-admin bucket rm also waits for a healthy cluster. The other option would be to tune the nearfull_ratio and full_ratio temporarily to allow the cluster to use more space, correct? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 4

Understanding monitor requirements
by Brian Topping 10 Apr '20

10 Apr '20

Hi experts, question about monitors and latency. I am setting up a new cluster and I’d like to have more than one monitor. Unfortunately, the primary site only has two chassis, so to get the third mon, I’ve been trying to bring it up remotely. So far, it’s not working and I wonder if someone knows how I can fit the proverbial square peg in this round hole. This gist[1] has the logs that I have from the third (remote) monitor and a diagram of the network. The logs keep repeating the same errors about slow ops from the first log entry. For comparison, I’ve also included a performance test of etcd that is running on the same three nodes. The secondary site node is slower by an order of magnitude. As for the use case, I mostly expect the primary hardware to be stable and the third mon provides uptime when I need to upgrade either primary node. There will never be any OSD traffic over the site-to-site link. The storage profiles are primarily RBD with static sizes and limited changes. My understanding is that the monitors are only keeping track of PG locations for pools. If that’s the case and the storage is primarily fixed size RBDs, the amount of inter-monitor communication should also be limited since changes are few. Am I mistaken here? Thanks for any input that can be provided! Brian [1] https://gist.github.com/briantopping/3638d665bc60c19e6f3f2005eb555a9d

1 1

Profiling of CRUSH Ceph
by Bobby 09 Apr '20

09 Apr '20

Hi all, I am a novice Ceph user and I am going through Ceph CRUSH algorithm. I would like to do some profiling of CRUSH computations on a CPU. Are there any CRUSH benchmarks? Any profiling tool for CRUSH? I would like to examine what are the compute intensive parts of CRUSH on the CPU. May be there is some research paper on it already? The goal is hardware and software partioning. I am really looking forward for some helping tips. Bobby !

1 0

v15.2.1 Octopus released
by Abhishek 09 Apr '20

09 Apr '20

This is the first bugfix release of Ceph Octopus, we recommend all Octopus users upgrade. This release fixes an upgrade issue and also has 2 security fixes Notable Changes ~~~~~~~~~~~~~~~ * issue#44759: Fixed luminous->nautilus->octopus upgrade asserts * CVE-2020-1759: Fixed nonce reuse in msgr V2 secure mode * CVE-2020-1760: Fixed XSS due to RGW GetObject header-splitting For the full changelog please refer to the official release blog entry at https://ceph.io/releases/v15-2-1-octopus-released/, for upgrades from previous versions, please follow the relevant section of the upgrade docs at https://docs.ceph.com/docs/master/releases/octopus/#upgrading-from-mimic-or… Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.1.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 9fd2f65f91d9246fae2c841a6222d34d121680ee

1 0

Balancer not balancing (14.2.7, crush-compat)
by Vladimir Brik 09 Apr '20

09 Apr '20

Hello I am running ceph 14.2.7 with balancer in crush-compat mode (needed because of old clients), but it's doesn't seem to be doing anything. It used to work in the past. I am not sure what changed. I created a big pool, ~285TB stored, and it doesn't look like it ever got balanced: pool 43 'fs-data-k5m2-hdd' erasure size 7 min_size 6 crush_rule 7 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode warn last_change 48647 lfor 0/42080/42102 flags hashpspool,ec_overwrites,nearfull stripe_width 20480 application cephfs OSD utilization varies between ~50% and about ~80%, with about 60% raw used. I am using a mixture of 9TB and 14TB drives. Number of PGs/drive varies 103 and 207. # ceph osd df | grep hdd | sort -k 17 | (head -n 2; tail -n 2) 160 hdd 12.53519 1.00000 13 TiB 6.0 TiB 5.9 TiB 74 KiB 12 GiB 6.6 TiB 47.74 0.79 120 up 146 hdd 12.53519 1.00000 13 TiB 6.0 TiB 6.0 TiB 51 MiB 13 GiB 6.5 TiB 48.17 0.80 119 up 79 hdd 8.99799 1.00000 9.0 TiB 7.3 TiB 7.2 TiB 42 KiB 16 GiB 1.7 TiB 80.91 1.34 186 up 62 hdd 8.99799 1.00000 9.0 TiB 7.3 TiB 7.2 TiB 112 KiB 16 GiB 1.7 TiB 81.44 1.35 189 up # ceph balancer status { "last_optimize_duration": "0:00:00.339635", "plans": [], "mode": "crush-compat", "active": true, "optimize_result": "Some osds belong to multiple subtrees: {0: ['default', 'default~hdd'], ... "last_optimize_started": "Thu Apr 9 11:17:40 2020" } Does anybody know how to debug this? Thanks, Vlad

1 1

Bucket index entries containing unicode NULL - how to remove them?
by Maks Kowalik 09 Apr '20

09 Apr '20

Hello, since some time I've been investigating problems causing buckets' index corruption. In my case it's been because of numerous bugs related to index resharding and bucket lifecycle policies. One of those bugs present in versions prior to 14.2.8 made the index omapkeys' names contain unicode NULL which manifests like this: root@mach0122:~/mkw # radosgw-admin bi list --bucket=sysa-user-logs|grep idx |tail -5 "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031521.gz.u6_qPM0X2mXQRXD0hEfRK2dCc4dD4El\u0000.4", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031600.gz.bvZdIu5KnsAQyXoa_ZWpix-BYD-yvcz\u0000.15", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031600.gz.wX9jHyKnJ21w5DYL_rdfTsb7zH3tUEa\u0000.10", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031601.gz.K6GhZfNzcqDCcjkrn5GskRWS8ufXSbO\u0000.8", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031601.gz.vEr4VCu0QU07te_RHNJ9Wi1cb8tsiYq\u0000.7", I am unable to remove them by rmomapkey. Also I tried to use the method https://github.com/ceph/ceph/blob/8c63b26fe88bb02d894705cb1beec289668fb43d/… but it fails with: File "./idx_removal.py", line 96, in remove_key iocontext.remove_omap_keys(write_op, key) File "rados.pyx", line 516, in rados.requires.wrapper.validate_func (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:4992) File "rados.pyx", line 3607, in rados.Ioctx.remove_omap_keys (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:45059) File "rados.pyx", line 543, in rados.cstr_list (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:5665) File "rados.pyx", line 539, in rados.cstr (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:5463) TypeError: keys must be a string Does anyone know what else can be done to remove such omapkey? For the obvious reasons I can't use the clear_omap method... Kind regards, Maks Kowalik

1 0

Using M2 SSDs as osds
by Stolte, Felix 09 Apr '20

09 Apr '20

Hey guys, I am evaluating using M2 SSDs as osds for an all flash pool. Is anyone using that in production and can elaborate on his experience? I am a little bit concerned about the lifetime of the M2 disks. Best regards Felix IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

2 1

Nautilus 14.2.7 radosgw lifecycle not removing expired objects
by oneill.gs＠gmail.com 08 Apr '20

08 Apr '20

Hello, I'm running a cluster with Ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable). I've encountered an issue with my cluster where objects are marked as expired but are not removed during lifecycle processing. These buckets have a mix of objects with and without special characters in their names. I set debug_rgw to 20/5 overnight so I could get more detailed logs when the lifecycle processing happened. The lifecycle function seems to correctly identify expired objects that should be removed, however they are not deleted. This is happening across multiple buckets and users. This cluster does have dynamic bucket index resharding enabled, however this issue affects buckets regardless of their reshard status. I also have cache tiering set to readproxy for the buckets data pool. root@rgw01 # ceph daemon /var/run/ceph/ceph-rgw-rgw01.asok config show | grep -E "lifecycle|lc" "rgw_lifecycle_work_time": "00:00-06:00", "rgw_enable_lc_threads": "true", "rgw_lc_debug_interval": "-1", "rgw_lc_lock_max_time": "60", "rgw_lc_max_objs": "32", "rgw_lc_max_rules": "1000", "rgw_lc_thread_delay": "0", root@mon01 # radosgw-admin lc list [ { "bucket": ":testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50", "status": "COMPLETE" } ] root@mon01 # radosgw-admin lc get --bucket testgso { "prefix_map": { "": { "status": true, "dm_expiration": false, "expiration": 1, "noncur_expiration": 0, "mp_expiration": 1 } }, "rule_map": [ { "id": "111222333", "rule": { "id": "111222333", "prefix": "", "status": "Enabled", "expiration": { "days": "1", "date": "" }, "noncur_expiration": { "days": "", "date": "" }, "mp_expiration": { "days": "1", "date": "" }, "filter": { "prefix": "", "obj_tags": {} }, "dm_expiration": false } } ] } goneill@mbp $ s3cmd -c Documents/s3cfg/testgso.cfg ls s3://testgso DIR s3://testgso/lifecycle/ 2020-03-31 17:26 12 s3://testgso/file!test.txt 2020-03-31 17:29 12 s3://testgso/file$test.txt 2020-03-31 17:29 12 s3://testgso/file&test.txt 2020-03-31 17:26 12 s3://testgso/file'test.txt 2020-03-31 17:26 12 s3://testgso/file(test.txt 2020-03-31 17:26 12 s3://testgso/file)test.txt 2020-03-31 17:29 12 s3://testgso/file*test.txt 2020-03-31 17:30 12 s3://testgso/file+test.txt 2020-03-31 17:30 12 s3://testgso/file,test.txt 2020-03-31 17:26 12 s3://testgso/file-test.txt 2020-03-31 17:30 12 s3://testgso/file:test.txt 2020-03-31 17:30 12 s3://testgso/file;test.txt 2020-03-31 17:30 12 s3://testgso/file=test.txt 2020-03-31 17:30 12 s3://testgso/file?test.txt 2020-03-31 17:06 12 s3://testgso/file@test.txt 2019-08-05 21:02 12 s3://testgso/test.txt goneill@mbp $ s3cmd -c Documents/s3cfg/testgso.cfg getlifecycle s3://testgso <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>111222333</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> ### Logs from rgw01 2020-04-08 00:00:00.645 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc9bf0 obj=default.rgw.meta:root:testgso state=0x561a8bc0ac40 s->prefetch_data=0 2020-04-08 00:00:00.645 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+testgso : expiry miss 2020-04-08 00:00:00.645 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+testgso info.flags=0x16 2020-04-08 00:00:00.645 7fc55bbcc700 10 adding default.rgw.meta+root+testgso to cache LRU end 2020-04-08 00:00:00.645 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.645 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.645 7fc55bbcc700 20 Read xattr: user.rgw.idtag 2020-04-08 00:00:00.645 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+testgso : type miss (requested=0x13, cached=0x16) 2020-04-08 00:00:00.645 7fc55bbcc700 20 rados->read ofs=0 len=0 2020-04-08 00:00:00.646 7fc55bbcc700 20 rados_obj.operate() r=0 bl.length=181 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+testgso info.flags=0x13 2020-04-08 00:00:00.646 7fc55bbcc700 10 moving default.rgw.meta+root+testgso to cache LRU end 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.646 7fc55bbcc700 20 rgw_get_bucket_info: bucket instance: testgso[0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50] 2020-04-08 00:00:00.646 7fc55bbcc700 20 reading from default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.646 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc9bf0 obj=default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 state=0x561a9d27d540 s->prefetch_data=0 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : expiry miss 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 info.flags=0x16 2020-04-08 00:00:00.646 7fc55bbcc700 10 adding default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 to cache LRU end 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.acl bl.length()=153 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.lc bl.length()=130 2020-04-08 00:00:00.646 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.646 7fc55bbcc700 20 Read xattr: user.rgw.acl 2020-04-08 00:00:00.646 7fc55bbcc700 20 Read xattr: user.rgw.idtag 2020-04-08 00:00:00.646 7fc55bbcc700 20 Read xattr: user.rgw.lc 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : type miss (requested=0x13, cached=0x16) 2020-04-08 00:00:00.646 7fc55bbcc700 20 rados->read ofs=0 len=0 2020-04-08 00:00:00.646 7fc55bbcc700 20 rados_obj.operate() r=0 bl.length=281 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 info.flags=0x13 2020-04-08 00:00:00.646 7fc55bbcc700 10 moving default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 to cache LRU end 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.acl bl.length()=153 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.lc bl.length()=130 2020-04-08 00:00:00.646 7fc55bbcc700 10 chain_cache_entry: cache_locator=default.rgw.meta+root+testgso 2020-04-08 00:00:00.646 7fc55bbcc700 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.647 7fc55bbcc700 10 lifecycle: bucket_lc_process() prefix_map size=1 2020-04-08 00:00:00.647 7fc55bbcc700 20 lifecycle: bucket_lc_process(): prefix= 2020-04-08 00:00:00.647 7fc55bbcc700 10 cls_bucket_list_unordered testgso[0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50] start [] num_entries 1100 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file!test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file$test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file&test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file'test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file(test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file)test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file*test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file+test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file,test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file-test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file:test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file;test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file=test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file?test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file(a)test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got lifecycle/file.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 20 lifecycle: bucket_lc_process(): key=file!test.txt 2020-04-08 00:00:00.648 7fc55bbcc700 20 obj_has_expired(): mtime=2020-03-31 13:26:07.0.481384s days=1 base_time=2020-04-08 00:00:00.000000 timediff=642833 cmp=86400 2020-04-08 00:00:00.648 7fc55bbcc700 20 check(): key=file!test.txt: is_expired=1 2020-04-08 00:00:00.648 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file!test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.657 7fc44f1b3700 10 manifest: total_size = 49667 2020-04-08 00:00:00.657 7fc44f1b3700 20 get_obj_state: setting s->obj_tag to 0a6e5079-ad65-4ea7-a868-b2133099e0d0.22351662.19462946 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_obj_state: setting s->obj_tag to 0a6e5079-ad65-4ea7-a868-b2133099e0d0.25454121.7727201 2020-04-08 00:00:00.672 7fc55bbcc700 10 If-UnModified-Since: 2020-03-31 13:26:07.0.481384s Last-Modified: 2020-03-31 13:26:07.000000 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file!test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.672 7fc55bbcc700 20 prepare_atomic_modification: state is not atomic. state=0x561aa0c64aa0 2020-04-08 00:00:00.672 7fc55bbcc700 20 reading from default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc8330 obj=default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 state=0x561a9d27d0c0 s->prefetch_data=0 2020-04-08 00:00:00.672 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : type miss (requested=0x16, cached=0x13) 2020-04-08 00:00:00.672 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 info.flags=0x16 2020-04-08 00:00:00.672 7fc55bbcc700 10 moving default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 to cache LRU end 2020-04-08 00:00:00.672 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.672 7fc55bbcc700 10 updating xattr: name=user.rgw.acl bl.length()=153 2020-04-08 00:00:00.672 7fc55bbcc700 10 updating xattr: name=user.rgw.lc bl.length()=130 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.672 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : hit (requested=0x11, cached=0x17) 2020-04-08 00:00:00.672 7fc55bbcc700 20 bucket index object: .dir.0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.683 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.683 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.683 7fc55bbcc700 20 lifecycle: ERROR: orule.process() returned ret=-125 2020-04-08 00:00:00.683 7fc55bbcc700 20 lifecycle: bucket_lc_process(): key=file$test.txt 2020-04-08 00:00:00.683 7fc55bbcc700 20 obj_has_expired(): mtime=2020-03-31 13:29:57.0.916797s days=1 base_time=2020-04-08 00:00:00.000000 timediff=642603 cmp=86400 2020-04-08 00:00:00.683 7fc55bbcc700 20 check(): key=file$test.txt: is_expired=1 2020-04-08 00:00:00.683 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file$test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.684 7fc55bbcc700 10 manifest: total_size = 12 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_obj_state: setting s->obj_tag to 0a6e5079-ad65-4ea7-a868-b2133099e0d0.25454121.7731015 2020-04-08 00:00:00.684 7fc55bbcc700 10 If-UnModified-Since: 2020-03-31 13:29:57.0.916797s Last-Modified: 2020-03-31 13:29:57.000000 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file$test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.684 7fc55bbcc700 20 prepare_atomic_modification: state is not atomic. state=0x561aa0c64aa0 2020-04-08 00:00:00.684 7fc55bbcc700 20 reading from default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc8330 obj=default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 state=0x561a9d27d0c0 s->prefetch_data=0 2020-04-08 00:00:00.684 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : hit (requested=0x16, cached=0x17) 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.684 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : hit (requested=0x11, cached=0x17) 2020-04-08 00:00:00.684 7fc55bbcc700 20 bucket index object: .dir.0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.711 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.711 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.711 7fc55bbcc700 20 lifecycle: ERROR: orule.process() returned ret=-125

1 0

[Octopus] OSD overloading
by Jack 08 Apr '20

08 Apr '20

Hello, I've a issue, since my Nautilus -> Octopus upgrade My cluster has many rbd images (~3k or something) Each of them has ~30 snapshots Each day, I create and remove a least a snapshot per image Since Octopus, when I remove the "nosnaptrim" flags, each OSDs uses 100% of its CPU time The whole cluster collapses: OSDs no longer see each others, most of them are seens as down .. I do not see any progress being made : it does not appear the problem will solve by itself What can I do ? Best regards,

8 12

Multiple OSDs down, and won't come up (possibly related to other Nautilus issues)
by aoanla＠gmail.com 08 Apr '20

08 Apr '20

So, this is following on from a discussion in the #ceph IRC channel, where we seem to have reached the limit of what we can do. I have a ~15 node, 311 OSD cluster. (20 OSDs per node). The cluster is Nautilus - the 3 MONs + the first 8 OSD hosts were installed as Mimic and upgraded to Nautilus with ceph-ansible ; the remaining OSD hosts were added directly with Nautilus as they were only added in a few weeks ago. Yesterday, suddenly, about half of the OSDs (~140) were marked Down, and a number of slow operations were detected. Initially, examining the logs (and with a bit of help from IRC), I noticed that the ansible roles used to build the newer OSDs had configured chrony incorrectly, and their clocks were drifting. (There were BADAUTHORIZER errors in OSD logs, too.) I fixed the chrony configuration... and we (including people in IRC) expected everything to just... stabilise. Things have not stabilised, which leads me to suspect that there are other issues at play. After noticing a number of issues with mgrs deadlocking in Nautilus - eg https://tracker.ceph.com/issues/17170 https://tracker.ceph.com/issues/43048 - I tried stopping all mgrs and mons, and then slowly bringing them up. This has not helped. Interestingly, the OSDs with slow ops (some of which are marked down) report ops_in_flight which are "wait for new map", whilst the lead mon believes those same ops are timed out. (I can of course, telnet to every OSD, even the down ones, from other OSDs, including ones which report issues talking to them on the same port; and from the lead mon.) I am wondering if this is an example of: https://tracker.ceph.com/issues/44184 as we did create a new pool shortly after adding the new OSD host nodes... but it isn't clear from that ticket [or the discussion on this list] how to fix this, other than removing the pool - which I can't do, as we need this pool to exist, and the pool is replaces needs to be decomissioned. Can anyone advise what I should do next? At present, obviously, the cluster is unusable.

3 5

Maximum CephFS Filesystem Size
by DHilsbos＠performair.com 07 Apr '20

07 Apr '20

All; We set up a CephFS on a Nautilus (14.2.8) cluster in February, to hold backups. We finally have all the backups running, and are just waiting for the system reach steady-state. I'm concerned about usage numbers, in the Dashboard Capacity it shows the cluster as 37% used, while under Filesystems --> <FSName> --> Pools -_> <data> --> Usage, it shows 71% used. Does CephFS place a limit on the size of a CephFS? Is there a limit to how large a pool can be in Ceph? Where is the sizing discrepancy coming from, and do I need to address it? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

Re: ceph mds can't recall client caps anymore
by Patrick Donnelly 07 Apr '20

07 Apr '20

On Tue, Apr 7, 2020 at 3:36 AM alean Huang <woalean(a)gmail.com> wrote: > > hi, > i am appreciate you work very much, there is a problem about ceph mds i meet. > ceph version: luminous > the mds cache grow to 20G and limit is 4G after 'stat *' from client which mount by kernel(centos7 , 4.14). > there are 1000w files in this dir. mds log show recall_client_state does not work after recall some caps. > > log is like this: > 2020-04-07 17:08:36.841304 7f23ddd15700 10 mds.2.server recall_client_state: session client.2366013 172.16.200.20:0/4162573294 caps 6801282, leases 0 > 2020-04-07 17:08:36.841323 7f23ddd15700 15 mds.2.server session recall threshold (16384) hit at 0; skipping! > 2020-04-07 17:08:36.841326 7f23ddd15700 7 mds.2.server recalled (throttled) 0 client caps. > > mds_recall_max_decay_rate = 2.5 It looks like your setting for mds_recall_max_caps is larger than mds_recall_max_decay_threshold. Are you changing these configurations? If so, why? -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

Load on drives of different sizes in ceph
by Andras Pataki 07 Apr '20

07 Apr '20

Hi cephers, I'm looking for some advice on what to do about drives of different sizes in the same cluster. We have so far kept the drive sizes consistent on our main ceph cluster (using 8TB drives). We're getting some new hardware with larger, 12TB drives next, and I'm pondering on how best to configure them. If I just simply add them, they will have 1.5x the data (which is less of a problem), but will also get 1.5x the iops - so I presume it will slow the whole cluster down as a result (these drives will be busy, and the rest will not be as much). I'm wondering how people generally handle this. I'm more concerned about these larger drives being busier than the rest - so I'd like to be able to put for example 1/3 drive of less accessed data on them in addition to the usual data - to use the extra capacity but not increase the load on them. Is there an easy way to accomplish this? One possibility is to run two OSDs on the drive (in two crush hierarchies), which isn't ideal. Can I just run one OSD somehow and put it into two crush roots, or something similar? Andras

4 4

virtual machines crashes after upgrade to octopus
by Lomayani S. Laizer 06 Apr '20

06 Apr '20

Hello, After upgrade our ceph cluster to octopus few days ago we are seeing vms crashes with below error. We are using ceph with openstack(rocky). Everything running ubuntu 18.04 with kernel 5.3. We seeing this crashes in busy vms. this is cluster was upgraded from nautilus. kernel: [430751.176904] fn-radosclient[3905]: segfault at da0801 ip 00007fe78e076686 sp 00007fe7697f9470 erro r 4 in librbd.so.1.12.0[7fe78de73000+5cb000] Apr 6 03:26:00 compute6 kernel: [430751.176922] Code: 00 64 48 8b 04 25 28 00 00 00 48 89 44 24 18 31 c0 48 85 db 0f 84 fa 00 00 00 8 0 bf 38 01 00 00 00 48 89 fd 0f 84 ea 00 00 00 <83> bb 20 3f 00 00 ff 0f 84 dd 00 00 00 48 8b 83 18 3f 00 00 48 8d Apr 6 03:26:11 compute6 libvirtd[1671]: 2020-04-06 03:26:11.955+0000: 1671: error : qemuMonitorIO:719 : internal error: End of file f rom qemu monitor

3 4

rgw multisite with https endpoints
by Richard Kearsley 06 Apr '20

06 Apr '20

Hi there I have a fairly simple ceph multisite configuration with 2 ceph clusters in 2 different datacenters in the same city The rgws have this config for ssl: rgw_frontends = civetweb port=7480+443s ssl_certificate=/opt/ssl/ceph-bundle.pem The certificate is a real issued certificate, not self signed I configured the multisite with the guide from https://docs.ceph.com/docs/nautilus/radosgw/multisite/ More or less ok so far, some learning curve but that's ok I can access and upload to buckets at both endpoints with s3 client using https - https://ceph01cs1.domain.com and https://ceph01cs2.domain.com - all good Now the problem seems to be when my zones in the zonegroup use https endpoints, e.g. { "id": "4c6774fb-01eb-41fe-a74a-c2693f8e69fc", "name": "eu", "api_name": "eu", "is_master": "true", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "zones": [ { "id": "0c203df2-6f31-4ad1-a899-91f85bf34c4e", "name": "ceph01cs1", "endpoints": [ "https://ceph01cs1.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "fec1fec8-a3c1-454d-8ed2-2c1da45f9c33", "name": "ceph01cs2", "endpoints": [ "https://ceph01cs2.domain.com:443" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "08921dd5-1523-41b6-908f-2f58aa38c969" } Meta syncs ok - buckets and users get created, but data doesn't, and period can be commited and appears on both clusters I can also curl between the two clusters over 443 However, data sync gets stuck on 'init': realm 08921dd5-1523-41b6-908f-2f58aa38c969 (world) zonegroup 4c6774fb-01eb-41fe-a74a-c2693f8e69fc (eu) zone 0c203df2-6f31-4ad1-a899-91f85bf34c4e (ceph01cs2) metadata sync no sync (zone is master) data sync source: fec1fec8-a3c1-454d-8ed2-2c1da45f9c33 (ceph01cs1) init full sync: 128/128 shards full sync: 0 buckets to sync incremental sync: 0/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] I find errors like: 2020-03-31 20:27:11.372 7f60c84e1700 0 RGW-SYNC:data:sync: ERROR: failed to init sync, retcode=-16 2020-03-31 20:27:29.548 7f60c84e1700 0 RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote data log shards 2020-03-31 20:29:48.499 7f60c94e3700 0 RGW-SYNC:meta: ERROR: failed to fetch all metadata keys If I change the endpoints in the zonegroup to plain http, e.g. http://ceph01cs1.domain.com:7480 and http://ceph01cs2.domain.com:7480 then sync starts! So my question, and I couldn't find any examples of people using https to sync.. are https endpoints supported with multisite? and why would meta work over https but not data? Many thanks Richard

3 3

can not visit dashboard of ceph mgr
by sz_cuitao＠163.com 06 Apr '20

06 Apr '20

hi: I enabled the dashboard module of ceph mgr and configed it,but it does not works. why it is? [cephuser@cadmin ~]$ sudo ceph mgr module enable dashboard module 'dashboard' is already enabled [cephuser@cadmin ~]$ sudo ceph config set mgr mgr/dashboard/server_addr 192.168.137.90 [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ sudo ceph config set mgr mgr/dashboard/server_port 6000 [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ sudo ceph dashboard ac-user-create admin ceph123 administrator [root@cadmin ~]# systemctl restart ceph-mgr@cadmin [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# netstat -an | grep 6000 [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# ceph mgr services {} [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# systemctl status ceph-mgr@cadmin ● ceph-mgr(a)cadmin.service - Ceph cluster manager daemon Loaded: loaded (/usr/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2020-04-05 18:57:44 CST; 5min ago Main PID: 86161 (ceph-mgr) CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr(a)cadmin.service └─86161 /usr/bin/ceph-mgr -f --cluster ceph --id cadmin --setuser ceph --setgroup ceph Apr 05 18:57:44 cadmin systemd[1]: Stopping Ceph cluster manager daemon... Apr 05 18:57:44 cadmin systemd[1]: Stopped Ceph cluster manager daemon. Apr 05 18:57:44 cadmin systemd[1]: Started Ceph cluster manager daemon. [root@cadmin ~]# sz_cuitao(a)163.com

2 1

osd with specifiying directories
by Micha 06 Apr '20

06 Apr '20

Hi, I want to try using object storage with java. Is it possible to set up osds with "only" directories as data destination (using cephadmin) , instead of whole disks? I have read through much of the docu but didn't found how to do it (if it's possible). Thanks Michael

3 5

Commands on cephfs mounts getting stuck in uninterruptible sleep
by david.piper＠metaswitch.com 06 Apr '20

06 Apr '20

Hello, I am seeing some commands running on CephFS mounts getting stuck in an uninterruptible sleep, at which point I can only terminate them by rebooting the client. Has anyone experienced anything similar and found a way to safe-guard against this? My mount is using the ceph kernel driver, with the following config in fstab: 10.225.44.236,10.225.44.237,10.225.44.238:6789:/albacore/system/deploy on /opt/dcl/deploy type ceph (rw,noatime,name=albacore,secret=<hidden>,acl,wsize=32768,rsize=32768,_netdev) The vast majority of commands complete successfully on the mounted filesystem but on one occasion a "chmod -R +r *" command hung indefinitely (despite having run successfully numerous times before). Attempts to terminate the process using `kill` fail. Repeated attempts to run the same command also get blocked in the same state. A `ps` command shows the processes are stuck in uninterruptable sleep: [root@svr01 albacore] ~> ps -Al | grep chmod 4 D 0 18657 18656 0 80 0 - 26998 rwsem_ pts/2 00:00:00 chmod 4 D 0 21835 1 0 80 0 - 26998 rwsem_ ? 00:00:00 chmod Ceph seems to be unaware of the hung process. There are no slow ops / ops in flight in either of the dump_ops_in_flight output on the server, or under sys/kernel/debug/ceph/ on the client. Similarly there are no logs in dmesg for the command / process. Ceph health reports no MDS issues, and there's nothing in the logs from my MDS from when the processes hung. The only method I've found of clearing the processes is to reboot my client. Has anyone got experience with this? Are there ceph mount options that would guard against this? Some details of the current setup: • ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) • We're using the ceph kernel driver, kernel: 5.5.7-1.el7.elrepo.x86_64 • The client server has 38 separate directories mounted, all from the same CephFS filesystem. • All 38 directories are mounted with the same config by three separate clients. • Mount config (in fstab): 10.225.44.236,10.225.44.237,10.225.44.238:6789:/albacore/system/deploy on /opt/dcl/deploy type ceph (rw,noatime,name=albacore,secret=<hidden>,acl,wsize=32768,rsize=32768,_netdev) Kind regards, Dave

1 0

Can I fix the devive name,when using image map?
by sz_cuitao＠163.com 06 Apr '20

06 Apr '20

Hi: I use image map to mount a ceph device to localhost,but I found that,the device name I can not control. This is a problem,when it's name changed,the FS on it or the database using this device may get wrongs. Can I control the device name? For exapme: [root@gate2 ~]# rbd showmapped id pool namespace image snap device 0 testpool test_img - /dev/rbd0 Can I map the device like this,just directly give the deveice name?like this: rbd map testpool/test_img /dev/rbd0 sz_cuitao(a)163.com

2 2

Removing OSDs in Mimic
by ADRIAN NICOLAE 06 Apr '20

06 Apr '20

Hi all, I have a Ceph cluster with ~ 70 OSDs of different sizes running on Mimic . I'm using ceph-deploy for managing the cluster size. I have to remove some smaller drives and replace them with bigger drives. From your experience, are the removing an OSD guidelines from Mimic docs accurate ? I know that there were some changes from the older versions and I want to avoid any confusions. I'm talking about the following procedure : - take the OSD out of the cluster with "ceph osd out osd_no" - stopping the osd daemon if it's still running - purging the OSD from the cluster map running from the ceph-deploy host (or one of the mons ?) : ceph osd purge {id} --yes-i-really-mean-it I don't have any specific entries for these OSDs in ceph.conf so I guess I shouldn't change any conf file. Thank you.

1 0

Radosgw WAF
by m.kefayati＠afranet.com 06 Apr '20

06 Apr '20

Hi we deploy ceph object storage and want secure RGW. Is there any solution or any user experience about it? Is it common to use WAF ? tnx

2 1

v14.2.8 Nautilus released
by Abhishek Lekshmanan 06 Apr '20

06 Apr '20

This is the eighth update to the Ceph Nautilus release series. This release fixes issues across a range of subsystems. We recommend that all users upgrade to this release. Please note the following important changes in this release; as always the full changelog is posted at: https://ceph.io/releases/v14-2-8-nautilus-released Notable Changes --------------- * The default value of `bluestore_min_alloc_size_ssd` has been changed to 4K to improve performance across all workloads. * The following OSD memory config options related to bluestore cache autotuning can now be configured during runtime: - osd_memory_base (default: 768 MB) - osd_memory_cache_min (default: 128 MB) - osd_memory_expected_fragmentation (default: 0.15) - osd_memory_target (default: 4 GB) The above options can be set with:: ceph config set osd <option> <value> * The MGR now accepts `profile rbd` and `profile rbd-read-only` user caps. These caps can be used to provide users access to MGR-based RBD functionality such as `rbd perf image iostat` an `rbd perf image iotop`. * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap balancing has been removed. Instead use the mgr balancer config `upmap_max_deviation` which now is an integer number of PGs of deviation from the target PGs per OSD. This can be set with a command like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default `upmap_max_deviation` is 1. There are situations where crush rules would not allow a pool to ever have completely balanced PGs. For example, if crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of the racks. In those cases, the configuration value can be increased. * RGW: a mismatch between the bucket notification documentation and the actual message format was fixed. This means that any endpoints receiving bucket notification, will now receive the same notifications inside a JSON array named 'Records'. Note that this does not affect pulling bucket notification from a subscription in a 'pubsub' zone, as these are already wrapped inside that array. * CephFS: multiple active MDS forward scrub is now rejected. Scrub currently only is permitted on a file system with a single rank. Reduce the ranks to one via `ceph fs set <fs_name> max_mds 1`. * Ceph now refuses to create a file system with a default EC data pool. For further explanation, see: https://docs.ceph.com/docs/nautilus/cephfs/createfs/#creating-pools * Ceph will now issue a health warning if a RADOS pool has a `pg_num` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.8.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 2d095e947a02261ce61424021bb43bd3022d35cb -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer HRB 21284 (AG Nürnberg)

6 7

Cann't create ceph cluster
by sz_cuitao＠163.com 06 Apr '20

06 Apr '20

hi! why can not create custer using ceph-deploy? Please give me some advice,thanks! [root@cadmin ~]# cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [root@cadmin ~]# [root@cadmin ~]# python --version Python 2.7.5 [root@cadmin ~]# [root@cadmin ~]# [root@cadmin ~]# su - cephuser Last login: Sat Apr 4 23:44:23 CST 2020 on pts/0 [cephuser@cadmin ~]$ [cephuser@cadmin ~]$ cd cluster/ [cephuser@cadmin cluster]$ [cephuser@cadmin cluster]$ ceph-deploy new cadmin [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephuser/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.25): /bin/ceph-deploy new cadmin [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph_deploy][ERROR ] Traceback (most recent call last): [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/util/decorators.py", line 69, in newfunc [ceph_deploy][ERROR ] return f(*a, **kw) [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/cli.py", line 162, in _main [ceph_deploy][ERROR ] return args.func(args) [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/new.py", line 141, in new [ceph_deploy][ERROR ] ssh_copy_keys(host, args.username) [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/new.py", line 35, in ssh_copy_keys [ceph_deploy][ERROR ] if ssh.can_connect_passwordless(hostname): [ceph_deploy][ERROR ] File "/usr/lib/python2.7/site-packages/ceph_deploy/util/ssh.py", line 15, in can_connect_passwordless [ceph_deploy][ERROR ] if not remoto.connection.needs_ssh(hostname): [ceph_deploy][ERROR ] AttributeError: 'module' object has no attribute 'needs_ssh' [ceph_deploy][ERROR ] sz_cuitao(a)163.com

3 3

Re: Questions on Ceph cluster without OS disks
by Martin Verges 05 Apr '20

05 Apr '20

Hello Samuel, we from croit.io don't use NFS to boot up Servers. We copy the OS directly into the RAM (approximately 0.5-1GB). Think of it like a container, you start it and throw it away when you no longer need it. This way we can save the slots of OS harddisks to add more storage per node and reduce overall costs as 1GB ram is cheaper then an OS disk and consumes less power. If our management node is down, nothing will happen to the cluster. No impact, no downtime. However, you do need the mgmt node to boot up the cluster. So after a very rare total power outage, your first system would be the mgmt node and then the cluster itself. But again, if you configure your systems correct, no manual work is required to recover from that. For everything else, it is possible (but definitely not needed) to deploy our mgmt node in active/passive HA. We have multiple hundred installations worldwide in production environments. Our strong PXE knowledge comes from more than 20 years of datacenter hosting experience and it never ever failed us in the last >10 years. The main benefits out of that: - Immutable OS freshly booted: Every host has exactly the same version, same library, kernel, Ceph versions,... - OS is heavily tested by us: Every croit deployment has exactly the same image. We can find errors much faster and hit much fewer errors. - Easy Update: Updating OS, Ceph or anything else is just a node reboot. No cluster downtime, No service Impact, full automatic handling by our mgmt Software. - No need to install OS: No maintenance costs, no labor required, no other OS management required. - Centralized Logs/Stats: As it is booted in memory, all logs and statistics are collected on a central place for easy access. - Easy to scale: It doesn't matter if you boot 3 oder 300 nodes, all boot the exact same image in a few seconds. .. lots more Please do not hesitate to contact us directly. We always try to offer an excellent service and are strongly customer oriented. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Sa., 21. März 2020 um 13:53 Uhr schrieb huxiaoyu(a)horebdata.cn < huxiaoyu(a)horebdata.cn>: > Hello， Martin， > > I notice that Croit advocate the use of ceph cluster without OS disks, but > with PXE boot. > > Do you use a NFS server to serve the root file system for each node? such > as hosting configuration files, user and password, log files, etc. My > question is, will the NFS server be a single point of failure? If the NFS > server goes down, the network experience any outage, ceph nodes may not be > able to write to the local file systems, possibly leading to service outage. > > How do you deal with the above potential issues in production? I am a bit > worried... > > best regards, > > samuel > > > > > ------------------------------ > huxiaoyu(a)horebdata.cn > > >

6 14

Is there a better way to make a samba/nfs gateway?
by Seth Galitzer 04 Apr '20

04 Apr '20

I have a hybrid environment and need to share with both Linux and Windows clients. For my previous iterations of file storage, I exported nfs and samba shares directly from my monolithic file server. All Linux clients used nfs and all Windows clients used samba. Now that I've switched to ceph, things are a bit more complicated. I built a gateway to export nfs and samba as needed, and connect that as a client to my ceph cluster. After having file locking problems with kernel nfs, I made the switch to nfs-ganesha, which has helped immensely. For Linux clients that have high I/O needs, like desktops and some web servers, I connect to ceph directly for those shares. For all other Linux needs, I use nfs from the gateway. For all Windows clients (desktops and a small number of servers), I use samba exported from the gateway. Since my ceph cluster went live in August, I have had some kind of strange (to me) error at least once a week, almost always related to the gateway client. Last night, it was MDS_CLIENT_OLDEST_TID. Since we're on Spring Break at my university and not very busy, I decided to unmount/remount the ceph share, requiring stopping nfs and samba services. Stopping nfs-ganesha took a while, but it finally completed with no complaints from the ceph cluster. Stopping samba took longer and gave me MDS_SLOW_REQUEST and MDS_CLIENT_LATE_RELEASE on the mds. It finally finished, and I was able to unmount/remount the ceph share and that finally cleared all the errors. This is leading me to believe that samba on the gateway and all the clients attaching to that is putting a strain on the connection back to ceph. Which finally brings me to my question: is there a better way to export samba to my clients using the ceph back end? Or is this as good as it gets and I just have to put up with the seemingly frequent errors? I can live with the errors and have been able to handle them so far, but I know people who have much bigger clusters and many more clients than me (by an order of magnitude) and don't see nearly as many errors as I do. Which is why I'm trying to figure out what is special about my setup. All my ceph nodes are running latest nautilus on Centos 7 (I just updated last week to 14.2.8), as is the gateway host. I'm mounting ceph directly on the gateway (by way of the kernel using cephfs, not rados/rbd) to a single mount point and exporting from there. My searches so far have not turned up anything extraordinarily useful, so I'm asking for some guidance here. Any advice is welcome. Thanks. Seth -- Seth Galitzer Systems Coordinator Computer Science Department Kansas State University http://www.cs.ksu.edu/~sgsax sgsax(a)ksu.edu 785-532-7790

8 11

Resize Bluestore i.e. shrink?
by Udo Waechter 04 Apr '20

04 Apr '20

Hi all, I'm currently building a little ceph-cluster, with embedded devices. My OSD Nodes are constrained in RAM (1GB, but 5 SATA Ports, please don't kill me ;) ). Anyways. Each of those nodes have 2x256 GB SSD and 2x 1TB HDDs. I'm using nautilus and deployd the whole thing with ceph-deploy. Now, I'd like to add more mem (a.k.a SWAP space, as raid1 on either ssds/hdds) to those OSDs. My question is: Can I shrink a bluestore lvm? I've seen 'ceph-bluestore-tool bluefs-bdev-expand' which suggests to expand the underlying lvm. It seem to me that I would have to remove the specific OSD and then add it manually after resizing the underlying lvm volume. And yes, I know that ceph need RAM ;) Cheers, udo.

2 1

Upgrading from Mimic to Nautilus
by Paul Choi 03 Apr '20

03 Apr '20

Hi, How is the experience of upgrading from Mimic to Nautilus? Anything to watch out for? And is it a smooth transition for the ceph-fuse clients on Mimic when interacting with a Nautilus MDS? I'm reading the docs on https://docs.ceph.com/docs/nautilus/install/upgrading-ceph/ and it looks pretty straightforward. I'm currently on Mimic 13.2.8, and have been putting off upgrading to Nautilus, but I think it's time. Thanks in advance, -Paul Choi

3 2

Purpose of crush_ln() function
by Bobby 03 Apr '20

03 Apr '20

Hi, I am trying to understand Straw2 Bucket of Ceph CRUSH algorithm. Can someone please tell what is the purpose of* crush_ln?* And what does "*compute 2^44*log2(input+1)*" means in the comment section above above *crush_ln* function? Thanks in advance

1 0

different RGW Versions on same ceph cluster
by Scheurer François 03 Apr '20

03 Apr '20

Dear All One ceph cluster is running with all daemons (mon, mgr, osd, rgw) having the version 12.2.12. Let's say we configure an additional radosgw instance with version 14.2.8, configured with the same ceph cluster name, realm, zonegroup and zone as the existing instances. Is it dangerous to start a different version of rgw concurrently with the older rgw? Can both versions server PUT & GET requests concurrently on the same buckets? It is probably ok, because during an upgrade it is as well assumed to have temporarily different versions running concurrently... So the question is kind of the same as asking if one may cancel the radosgw upgrade from Luminous to Nautilus in the middle, rolling back the radosgw packages. Many thanks in advance. Cheers Francois Scheurer

2 2

[Octopus] Beware the on-disk conversion
by Jack 03 Apr '20

03 Apr '20

Hi, As the upgrade documentation tells: > Note that the first time each OSD starts, it will do a format > conversion to improve the accounting for “omap” data. This may > take a few minutes to as much as a few hours (for an HDD with lots > of omap data). You can disable this automatic conversion with: What the documentation does not say is that this process takes a lot of memory I am upgrading a rusty cluster from Nautilus, you can check out the ram consumption as attachment First, we have a 3TB osd conversion: it tooks ~15min, and 19GB of memory Then, we have a larger 6TB osd conversion: it tooks more than 2 hours, and 35GB of memory Finally, you have the largest 10TB osd: only 1H15, but 52GB of memory

5 15

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

Odd CephFS Performance
by Gabryel Mason-Williams 03 Apr '20

03 Apr '20

We have been benchmarking CephFS and comparing it Rados to see the performance difference and how much overhead CephFS has. However, we are getting odd results when using more than 1 OSD server (each OSDS has only one disk) using CephFS but using Rados everything appears normal. These tests are run on the same Ceph Cluster. CephFS Rados OSDS Thread 16 Thread 16 1 289 316 2 139 546 3 143 728 4 142 844 CephFS is being benchmarked using: fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4M --numjobs=16 --size=1G --group_reporting Rados is being benchmarked using: rados bench -p cephfs_data 10 write -t 16 If you could provide some help or insight into why this is happening or how to stop it, that would be much appreciated. Kind regards, Gabryel

2 2

v15.2.0 Octopus released
by Abhishek Lekshmanan 03 Apr '20

03 Apr '20

We're happy to announce the first stable release of Octopus v15.2.0. There are a lot of changes and new features added, we advise everyone to read the release notes carefully, and in particular the upgrade notes, before upgrading. Please refer to the official blog entry https://ceph.io/releases/v15-2-0-octopus-released/ for a detailed version with links & changelog. This release wouldn't have been possible without the support of the community, this release saw contributions from over 330 developers & 80 organizations, and we thank everyone for making this release happen. Major Changes from Nautilus --------------------------- General ~~~~~~~ * A new deployment tool called **cephadm** has been introduced that integrates Ceph daemon deployment and management via containers into the orchestration layer. * Health alerts can now be muted, either temporarily or permanently. * Health alerts are now raised for recent Ceph daemons crashes. * A simple 'alerts' module has been introduced to send email health alerts for clusters deployed without the benefit of an existing external monitoring infrastructure. * Packages are built for the following distributions: - CentOS 8 - CentOS 7 (partial--see below) - Ubuntu 18.04 (Bionic) - Debian Buster - Container images (based on CentOS 8) Note that the dashboard, prometheus, and restful manager modules will not work on the CentOS 7 build due to Python 3 module dependencies that are missing in CentOS 7. Besides this packages built by the community will also available for the following distros: - Fedora (33/rawhide) - openSUSE (15.2, Tumbleweed) Dashboard ~~~~~~~~~ The mgr-dashboard has gained a lot of new features and functionality: * UI Enhancements - New vertical navigation bar - New unified sidebar: better background task and events notification - Shows all progress mgr module notifications - Multi-select on tables to perform bulk operations * Dashboard user account security enhancements - Disabling/enabling existing user accounts - Clone an existing user role - Users can change their own password - Configurable password policies: Minimum password complexity/length requirements - Configurable password expiration - Change password after first login New and enhanced management of Ceph features/services: * OSD/device management - List all disks associated with an OSD - Add support for blinking enclosure LEDs via the orchestrator - List all hosts known by the orchestrator - List all disks and their properties attached to a node - Display disk health information (health prediction and SMART data) - Deploy new OSDs on new disks/hosts - Display and allow sorting by an OSD's default device class in the OSD table - Explicitly set/change the device class of an OSD, display and sort OSDs by device class * Pool management - Viewing and setting pool quotas - Define and change per-pool PG autoscaling mode * RGW management enhancements - Enable bucket versioning - Enable MFA support - Select placement target on bucket creation * CephFS management enhancements - CephFS client eviction - CephFS snapshot management - CephFS quota management - Browse CephFS directory * iSCSI management enhancements - Show iSCSI GW status on landing page - Prevent deletion of IQNs with open sessions - Display iSCSI "logged in" info * Prometheus alert management - List configured Prometheus alerts RADOS ~~~~~ * Objects can now be brought in sync during recovery by copying only the modified portion of the object, reducing tail latencies during recovery. * Ceph will allow recovery below *min_size* for Erasure coded pools, wherever possible. * The PG autoscaler feature introduced in Nautilus is enabled for new pools by default, allowing new clusters to autotune *pg num* without any user intervention. The default values for new pools and RGW/CephFS metadata pools have also been adjusted to perform well for most users. * BlueStore has received several improvements and performance updates, including improved accounting for "omap" (key/value) object data by pool, improved cache memory management, and a reduced allocation unit size for SSD devices. (Note that by default, the first time each OSD starts after upgrading to octopus it will trigger a conversion that may take from a few minutes to a few hours, depending on the amount of stored "omap" data.) * Snapshot trimming metadata is now managed in a more efficient and scalable fashion. RBD block storage ~~~~~~~~~~~~~~~~~ * Mirroring now supports a new snapshot-based mode that no longer requires the journaling feature and its related impacts in exchange for the loss of point-in-time consistency (it remains crash consistent). * Clone operations now preserve the sparseness of the underlying RBD image. * The trash feature has been improved to (optionally) automatically move old parent images to the trash when their children are all deleted or flattened. * The trash can be configured to automatically purge on a defined schedule. * Images can be online re-sparsified to reduce the usage of zeroed extents. * The `rbd-nbd` tool has been improved to use more modern kernel interfaces. * Caching has been improved to be more efficient and performant. * `rbd-mirror` automatically adjusts its per-image memory usage based upon its memory target. * A new persistent read-only caching daemon is available to offload reads from shared parent images. RGW object storage ~~~~~~~~~~~~~~~~~~ * New `Multisite Sync Policy` primitives for per-bucket replication. (EXPERIMENTAL) * S3 feature support: - Bucket Replication (EXPERIMENTAL) - `Bucket Notifications`_ via HTTP/S, AMQP and Kafka - Bucket Tagging - Object Lock - Public Access Block for buckets * Bucket sharding: - Significantly improved listing performance on buckets with many shards. - Dynamic resharding prefers prime shard counts for improved distribution. - Raised the default number of bucket shards to 11. * Added `HashiCorp Vault Integration`_ for SSE-KMS. * Added Keystone token cache for S3 requests. CephFS distributed file system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Inline data support in CephFS has been deprecated and will likely be removed in a future release. * MDS daemons can now be assigned to manage a particular file system via the new `mds_join_fs` option. * MDS now aggressively asks idle clients to trim caps which improves stability when file system load changes. * The mgr volumes plugin has received numerous improvements to support CephFS via CSI, including snapshots and cloning. * cephfs-shell has had numerous incremental improvements and bug fixes. Upgrading from Mimic or Nautilus -------------------------------- You can monitor the progress of your upgrade at each stage with the `ceph versions` command, which will tell you what ceph version(s) are running for each type of daemon. Instructions ~~~~~~~~~~~~ #. Make sure your cluster is stable and healthy (no down or recovering OSDs). (Optional, but recommended.) #. Set the `noout` flag for the duration of the upgrade. (Optional, but recommended.):: # ceph osd set noout #. Upgrade monitors by installing the new packages and restarting the monitor daemons. For example, on each monitor host,:: # systemctl restart ceph-mon.target Once all monitors are up, verify that the monitor upgrade is complete by looking for the `octopus` string in the mon map. The command:: # ceph mon dump | grep min_mon_release should report:: min_mon_release 15 (nautilus) If it doesn't, that implies that one or more monitors hasn't been upgraded and restarted and/or the quorum does not include all monitors. #. Upgrade `ceph-mgr` daemons by installing the new packages and restarting all manager daemons. For example, on each manager host,:: # systemctl restart ceph-mgr.target Verify the `ceph-mgr` daemons are running by checking `ceph -s`:: # ceph -s ... services: mon: 3 daemons, quorum foo,bar,baz mgr: foo(active), standbys: bar, baz ... #. Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts:: # systemctl restart ceph-osd.target Note that the first time each OSD starts, it will do a format conversion to improve the accounting for "omap" data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data). You can disable this automatic conversion with:: # ceph config set osd bluestore_fsck_quick_fix_on_mount false You can monitor the progress of the OSD upgrades with the `ceph versions` or `ceph osd versions` commands:: # ceph osd versions { "ceph version 13.2.5 (...) mimic (stable)": 12, "ceph version 15.2.0 (...) octopus (stable)": 22, } #. Upgrade all CephFS MDS daemons. For each CephFS file system, #. Reduce the number of ranks to 1. (Make note of the original number of MDS daemons first if you plan to restore it later.):: # ceph status # ceph fs set <fs_name> max_mds 1 #. Wait for the cluster to deactivate any non-zero ranks by periodically checking the status:: # ceph status #. Take all standby MDS daemons offline on the appropriate hosts with:: # systemctl stop ceph-mds@<daemon_name> #. Confirm that only one MDS is online and is rank 0 for your FS:: # ceph status #. Upgrade the last remaining MDS daemon by installing the new packages and restarting the daemon:: # systemctl restart ceph-mds.target #. Restart all standby MDS daemons that were taken offline:: # systemctl start ceph-mds.target #. Restore the original value of `max_mds` for the volume:: # ceph fs set <fs_name> max_mds <original_max_mds> #. Upgrade all radosgw daemons by upgrading packages and restarting daemons on all hosts:: # systemctl restart ceph-radosgw.target #. Complete the upgrade by disallowing pre-Octopus OSDs and enabling all new Octopus-only functionality:: # ceph osd require-osd-release octopus #. If you set `noout` at the beginning, be sure to clear it with:: # ceph osd unset noout #. Verify the cluster is healthy with `ceph health`. If your CRUSH tunables are older than Hammer, Ceph will now issue a health warning. If you see a health alert to that effect, you can revert this change with:: ceph config set mon mon_crush_min_required_version firefly If Ceph does not complain, however, then we recommend you also switch any existing CRUSH buckets to straw2, which was added back in the Hammer release. If you have any 'straw' buckets, this will result in a modest amount of data movement, but generally nothing too severe.:: ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 If there are problems, you can easily revert with:: ceph osd setcrushmap -i backup-crushmap Moving to 'straw2' buckets will unlock a few recent features, like the `crush-compat` :ref:`balancer <balancer>` mode added back in Luminous. #. If you are upgrading from Mimic, or did not already do so when you upgraded to Nautlius, we recommened you enable the new :ref:`v2 network protocol <msgr2>`, issue the following command:: ceph mon enable-msgr2 This will instruct all monitors that bind to the old default port 6789 for the legacy v1 protocol to also bind to the new 3300 v2 protocol port. To see if all monitors have been updated,:: ceph mon dump and verify that each monitor has both a `v2:` and `v1:` address listed. #. Consider enabling the :ref:`telemetry module <telemetry>` to send anonymized usage statistics and crash information to the Ceph upstream developers. To see what would be reported (without actually sending any information to anyone),:: ceph mgr module enable telemetry ceph telemetry show If you are comfortable with the data that is reported, you can opt-in to automatically report the high-level cluster metadata with:: ceph telemetry on For more information about the telemetry module, see :ref:`the documentation <telemetry>`. Upgrading from pre-Mimic releases (like Luminous) ------------------------------------------------- You *must* first upgrade to Mimic (13.2.z) or Nautilus (14.2.z) before upgrading to Octopus. Upgrade compatibility notes --------------------------- * Starting with Octopus, there is now a separate repository directory for each version on `download.ceph.com` (e.g., `rpm-15.2.0` and `debian-15.2.0`). The traditional package directory that is named after the release (e.g., `rpm-octopus` and `debian-octopus`) is now a symlink to the most recently bug fix version for that release. We no longer generate a single repository that combines all bug fix versions for a single named release. * The RGW "num_rados_handles" has been removed. If you were using a value of "num_rados_handles" greater than 1 multiply your current "objecter_inflight_ops" and "objecter_inflight_op_bytes" paramaeters by the old "num_rados_handles" to get the same throttle behavior. * Ceph now packages python bindings for python3.6 instead of python3.4, because python3 in EL7/EL8 is now using python3.6 as the native python3. see the `announcement`_ for more details on the background of this change. * librbd now uses a write-around cache policy be default, replacing the previous write-back cache policy default. This cache policy allows librbd to immediately complete write IOs while they are still in-flight to the OSDs. Subsequent flush requests will ensure all in-flight write IOs are completed prior to completing. The librbd cache policy can be controlled via a new "rbd_cache_policy" configuration option. * librbd now includes a simple IO scheduler which attempts to batch together multiple IOs against the same backing RBD data block object. The librbd IO scheduler policy can be controlled via a new "rbd_io_scheduler" configuration option. * RGW: radosgw-admin introduces two subcommands that allow the managing of expire-stale objects that might be left behind after a bucket reshard in earlier versions of RGW. One subcommand lists such objects and the other deletes them. Read the troubleshooting section of the dynamic resharding docs for details. * RGW: Bucket naming restrictions have changed and likely to cause InvalidBucketName errors. We recommend to set `rgw_relaxed_s3_bucket_names` option to true as a workaround. * In the Zabbix Mgr Module there was a typo in the key being send to Zabbix for PGs in backfill_wait state. The key that was sent was 'wait_backfill' and the correct name is 'backfill_wait'. Update your Zabbix template accordingly so that it accepts the new key being send to Zabbix. * zabbix plugin for ceph manager now includes osd and pool discovery. Update of zabbix_template.xml is needed to receive per-pool (read/write throughput, diskspace usage) and per-osd (latency, status, pgs) statistics * The format of all date + time stamps has been modified to fully conform to ISO 8601. The old format (`YYYY-MM-DD HH:MM:SS.ssssss`) excluded the `T` separator between the date and time and was rendered using the local time zone without any explicit indication. The new format includes the separator as well as a `+nnnn` or `-nnnn` suffix to indicate the time zone, or a `Z` suffix if the time is UTC. For example, `2019-04-26T18:40:06.225953+0100`. Any code or scripts that was previously parsing date and/or time values from the JSON or XML structure CLI output should be checked to ensure it can handle ISO 8601 conformant values. Any code parsing date or time values from the unstructured human-readable output should be modified to parse the structured output instead, as the human-readable output may change without notice. * The `bluestore_no_per_pool_stats_tolerance` config option has been replaced with `bluestore_fsck_error_on_no_per_pool_stats` (default: false). The overall default behavior has not changed: fsck will warn but not fail on legacy stores, and repair will convert to per-pool stats. * The disaster-recovery related 'ceph mon sync force' command has been replaced with 'ceph daemon <...> sync_force'. * The `osd_recovery_max_active` option now has `osd_recovery_max_active_hdd` and `osd_recovery_max_active_ssd` variants, each with different default values for HDD and SSD-backed OSDs, respectively. By default `osd_recovery_max_active` now defaults to zero, which means that the OSD will conditionally use the HDD or SSD option values. Administrators who have customized this value may want to consider whether they have set this to a value similar to the new defaults (3 for HDDs and 10 for SSDs) and, if so, remove the option from their configuration entirely. * monitors now have a `ceph osd info` command that will provide information on all osds, or provided osds, thus simplifying the process of having to parse `osd dump` for the same information. * The structured output of `ceph status` or `ceph -s` is now more concise, particularly the `mgrmap` and `monmap` sections, and the structure of the `osdmap` section has been cleaned up. * A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. New admin command `ceph daemon mgr.# dump_osd_network [threshold]` command will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. New admin command `ceph daemon osd.# dump_osd_network [threshold]` will do the same but only including heartbeats initiated by the specified OSD. * Inline data support for CephFS has been deprecated. When setting the flag, users will see a warning to that effect, and enabling it now requires the `--yes-i-really-really-mean-it` flag. If the MDS is started on a filesystem that has it enabled, a health warning is generated. Support for this feature will be removed in a future release. * `ceph {set,unset} full` is not supported anymore. We have been using `full` and `nearfull` flags in OSD map for tracking the fullness status of a cluster back since the Hammer release, if the OSD map is marked `full` all write operations will be blocked until this flag is removed. In the Infernalis release and Linux kernel 4.7 client, we introduced the per-pool full/nearfull flags to track the status for a finer-grained control, so the clients will hold the write operations if either the cluster-wide `full` flag or the per-pool `full` flag is set. This was a compromise, as we needed to support the cluster with and without per-pool `full` flags support. But this practically defeated the purpose of introducing the per-pool flags. So, in the Mimic release, the new flags finally took the place of their cluster-wide counterparts, as the monitor started removing these two flags from OSD map. So the clients of Infernalis and up can benefit from this change, as they won't be blocked by the full pools which they are not writing to. In this release, `ceph {set,unset} full` is now considered as an invalid command. And the clients will continue honoring both the cluster-wide and per-pool flags to be backward comaptible with pre-infernalis clusters. * The telemetry module now reports more information. First, there is a new 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:: ceph config set mgr mgr/telemetry/channel_device false Second, we now report more information about CephFS file systems, including: - how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached We have also added: - which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use If you had telemetry enabled, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: $ ceph telemetry show # see everything $ ceph telemetry show basic # basic cluster info (including all of the new info) * Following invalid settings now are not tolerated anymore for the command `ceph osd erasure-code-profile set xxx`. * invalid `m` for "reed_sol_r6_op" erasure technique * invalid `m` and invalid `w` for "liber8tion" erasure technique * New OSD daemon command dump_recovery_reservations which reveals the recovery locks held (in_progress) and waiting in priority queues. * New OSD daemon command dump_scrub_reservations which reveals the scrub reservations that are held for local (primary) and remote (replica) PGs. * Previously, `ceph tell mgr ...` could be used to call commands implemented by mgr modules. This is no longer supported. Since luminous, using `tell` has not been necessary: those same commands are also accessible without the `tell mgr` portion (e.g., `ceph tell mgr influx foo` is the same as `ceph influx foo`. `ceph tell mgr ...` will now call admin commands--the same set of commands accessible via `ceph daemon ...` when you are logged into the appropriate host. * The `ceph tell` and `ceph daemon` commands have been unified, such that all such commands are accessible via either interface. Note that ceph-mgr tell commands are accessible via either `ceph tell mgr ...` or `ceph tell mgr.<id> ...`, and it is only possible to send tell commands to the active daemon (the standbys do not accept incoming connections over the network). * Ceph will now issue a health warning if a RADOS pool as a `pg_num` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false * The format of MDSs in `ceph fs dump` has changed. * The `mds_cache_size` config option is completely removed. Since luminous, the `mds_cache_memory_limit` config option has been preferred to configure the MDS's cache limits. * The `pg_autoscale_mode` is now set to `on` by default for newly created pools, which means that Ceph will automatically manage the number of PGs. To change this behavior, or to learn more about PG autoscaling, see :ref:`pg-autoscaler`. Note that existing pools in upgraded clusters will still be set to `warn` by default. * The `upmap_max_iterations` config option of mgr/balancer has been renamed to `upmap_max_optimizations` to better match its behaviour. * `mClockClientQueue` and `mClockClassQueue` OpQueue implementations have been removed in favor of of a single `mClockScheduler` implementation of a simpler OSD interface. Accordingly, the `osd_op_queue_mclock*` family of config options has been removed in favor of the `osd_mclock_scheduler*` family of options. * The config subsystem now searches dot ('.') delimited prefixes for options. That means for an entity like `client.foo.bar`, its overall configuration will be a combination of the global options, `client`, `client.foo`, and `client.foo.bar`. Previously, only global, `client`, and `client.foo.bar` options would apply. This change may affect the configuration for clients that include a `.` in their name. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.0.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer

14 21

librados : handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
by Yoann Moulin 02 Apr '20

02 Apr '20

Hello, I have a Nautilus (14.2.8) cluster and I'd like to give access to a pool with librados to a user. Here what I have > # ceph osd pool ls detail | grep user1 > pool 5 'user1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 108 flags hashpspool max_bytes 1099511627776 stripe_width 0 application user1 > # ceph auth get client.user1 > exported keyring for client.user1> client.user1 > key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX== > caps: [mon] allow r > caps: [osd] allow rw pool=user1 namespace=user1 On the client > $ cat ~/ceph.conf> [global] > mon host = [v2:10.90.36.16:3300,v1:10.90.36.16:6789],[v2:10.90.36.17:3300,v1:10.90.36.17:6789],[v2:10.90.36.18:3300,v1:10.90.36.18:6789] > keyring = ~/user1.keyring > $ cat ~/user1.keyring > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX== > $ rados -c ~/ceph.conf -p pool ls > 2020-04-02 12:44:59.900 7fd78aea3700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] > 2020-04-02 12:44:59.900 7fd789ea1700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] > 2020-04-02 12:44:59.900 7fd78a6a2700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] > failed to fetch mon config (--no-mon-config to skip) Is there something I missed? Thanks for your help. Best regards, -- Yoann Moulin EPFL IC-IT

1 1

Logging remove duplicate time
by Marc Roos 02 Apr '20

02 Apr '20

I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean; I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean; I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean; I already have the time logged, I do not need it a second time. Mar 31 13:39:59 c01 ceph-mgr: 2020-03-31 13:39:59.518 7f554edc8700 0 log_channel(cluster) log [DBG] : pgmap v672065: 384 pgs: 384 active+clean;

2 2

LARGE_OMAP_OBJECTS 1 large omap objects
by Dietmar Rieder 02 Apr '20

02 Apr '20

Hi, I'm trying to understand the "LARGE_OMAP_OBJECTS 1 large omap objects" warning for out cephfs metadata pool. It seems that pg 5.26 has a large omap object with > 200k keys [WRN] : Large omap object found. Object: 5:654134d2:::mds0_openfiles.0:head PG: 5.4b2c82a6 (5.26) Key count: 286083 Size (bytes): 14043228 I guess this object is related to the open files on the (cephfs mds0_openfiles.0). But what exactly does it tell me? Is the number of keys the number of currently open files? If yes, this is not matching the sum of open files over all clients obtained with lsof (which is less than 1000). So how can I get rid of this? (Reboot the clients?) Thanks for your help Dietmar -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Innrain 80, 6020 Innsbruck Email: dietmar.rieder(a)i-med.ac.at Web: http://www.icbi.at

2 2

Re: Netplan bonding configuration
by James McEwan 02 Apr '20

02 Apr '20

Hi, I am currently building a 10 node Ceph cluster, each OSD node has 2x 25 Gbit/s nics, and I have 2 TOR switches (mlag not supported). enp179s0f0 -> sw1 enp179s0f1 -> sw2 vlan 323 is used for ‘public network’ vlan 324 is used for ‘cluster network’ My desired configuration is to create two bond interfaces in active-backup mode: bond0 - enp179s0f0.323 (active) - enp179s0f1.323 (backup) bond1 - enp179s0f0.324 (backup) - enp179s0f1.324 (active) This way, the public network will use switch1, and the cluster network will use switch2, under normal operation. I am, however, having an issue implementing this configuration in Ubuntu 18.04 with netplan (see configuration at the end of this post). When I reboot a node with the below netplan configuration, the bond interface is created, but the vlan interfaces are not added to the bond. I see the following errors in the log: systemd-networkd[1641]: enp179s0f0.323: Enslaving by 'bond0’ systemd-networkd[1641]: bond0: Enslaving link 'enp179s0f0.323’ systemd-networkd[1641]: enp179s0f1.323: Enslaving by 'bond0’ systemd-networkd[1641]: bond0: Enslaving link 'enp179s0f1.323’ systemd-networkd[1643]: enp179s0f1.323: Could not join netdev: Operation not permitted systemd-networkd[1643]: enp179s0f1.323: Failed systemd-networkd[1643]: enp179s0f0.323: Could not join netdev: Operation not permitted systemd-networkd[1643]: enp179s0f0.323: Failed If I manually run ’systemctl restart systemd-networkd’ after boot has completed, then the bond is successfully created with the vlan interfaces. Does anybody have a similar configuration working specifically with netplan/networkd? Could you please share your configuration? Netplan config that doesn’t work at boot time: network: version: 2 renderer: networkd ethernets: enp179s0f0: {} enp179s0f1: {} bonds: bond0: dhcp4: false dhcp6: false interfaces: - enp179s0f0.323 - enp179s0f1.323 parameters: mode: active-backup primary: enp179s0f0.323 mii-monitor-interval: 1 addresses: [insert address here] bond1: dhcp4: false dhcp6: false interfaces: - enp179s0f0.324 - enp179s0f1.324 parameters: mode: active-backup primary: enp179s0f1.324 mii-monitor-interval: 1 addresses: [insert address here] vlans: enp179s0f0.323: id: 323 link: enp179s0f0 enp179s0f1.323: id: 323 link: enp179s0f1 enp179s0f0.324: id: 324 link: enp179s0f0 enp179s0f1.324: id: 324 link: enp179s0f1

5 7

luminous： osd continue down because of the hearbeattimeout
by linghucongsong 02 Apr '20

02 Apr '20

HI! all! Thanks for reading this msg. I hava one ceph cluster installed with ceph V12.2.12. It runs well for about half a year. Last week we add anoher two meachine to this ceph cluster.Then all the osds became unstable. The osd ansync message complain can not hearbeat to eachother.But the network ping with no drop packages and no error packages. I use bond0 for the ceph cluster front and back netwrok.Now I set nodown noout the cluster became stable, but from the log I see a lot for error aysnc message.I have try simple message, It also the smae error. All the osd error like below: NG_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.989469 7f42794da700 0 -- 10.255.255.54:6814/1000006 >> 10.255.255.56:0/7 conn(0x55721e0e5800 :6814 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.989557 7f42784d8700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.52:0/7 conn(0x55721e0e8800 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.989728 7f4278cd9700 0 -- 10.255.255.54:6814/1000006 >> 10.255.255.55:0/7 conn(0x55722973b000 :6814 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.989872 7f42794da700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.55:0/7 conn(0x557225b15000 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.990111 7f42794da700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.55:0/7 conn(0x557228506000 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.990161 7f42784d8700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.56:0/7 conn(0x557226320000 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.990196 7f42794da700 0 -- 10.255.255.54:6814/1000006 >> 10.255.255.56:0/7 conn(0x55722650b000 :6814 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.991450 7f4278cd9700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.55:0/7 conn(0x5572298d7800 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.991458 7f42784d8700 0 -- 10.255.255.54:6814/1000006 >> 10.255.255.52:0/7 conn(0x557226f19000 :6814 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.991639 7f4278cd9700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.52:0/7 conn(0x557226867800 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.991798 7f42794da700 0 -- 10.255.255.54:6814/1000006 >> 10.255.255.56:0/7 conn(0x55722a20b000 :6814 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) 2020-04-02 09:59:17.991842 7f42784d8700 0 -- 10.255.255.54:6819/1000006 >> 10.255.255.56:0/7 conn(0x557226869000 :6819 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) The network config: bond0 Link encap:Ethernet HWaddr 6c:92:bf:c2:8e:e5 inet6 addr: fe80::6e92:bfff:fec2:8ee5/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:126155520073 errors:0 dropped:3217298 overruns:0 frame:0 TX packets:133297822313 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:57485747361080 (57.4 TB) TX bytes:71267041966300 (71.2 TB) bond0.38 Link encap:Ethernet HWaddr 6c:92:bf:c2:8e:e5 inet addr:192.168.38.54 Bcast:192.168.38.255 Mask:255.255.255.0 inet6 addr: fe80::6e92:bfff:fec2:8ee5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:60802363 errors:0 dropped:0 overruns:0 frame:0 TX packets:53614452 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:34857574617 (34.8 GB) TX bytes:23829455266 (23.8 GB) bond0.4000 Link encap:Ethernet HWaddr 6c:92:bf:c2:8e:e5 inet addr:10.255.255.54 Bcast:10.255.255.63 Mask:255.255.255.192 inet6 addr: fe80::6e92:bfff:fec2:8ee5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:107628762285 errors:0 dropped:0 overruns:0 frame:0 TX packets:96091921746 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:54705054842566 (54.7 TB) TX bytes:68763270985565 (68.7 TB) brq86d8e0ef-fa Link encap:Ethernet HWaddr 26:30:9e:96:7a:71 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:2512246 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:338012546 (338.0 MB) TX bytes:0 (0.0 B) docker0 Link encap:Ethernet HWaddr 02:42:67:87:8d:a7 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eno1 Link encap:Ethernet HWaddr 6c:92:bf:c2:8e:e5 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:62792632732 errors:0 dropped:614221 overruns:0 frame:0 TX packets:66647497482 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:28698305745018 (28.6 TB) TX bytes:35631476375125 (35.6 TB) eno2 Link encap:Ethernet HWaddr 6c:92:bf:c2:8e:e5 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:63362887347 errors:0 dropped:633400 overruns:0 frame:0 TX packets:66650324833 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:28787441616499 (28.7 TB) TX bytes:35635565591656 (35.6 TB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:3272039707 errors:0 dropped:0 overruns:0 frame:0 TX packets:3272039707 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:753581347467 (753.5 GB) TX bytes:753581347467 (753.5 GB) tap579ce88c-9e Link encap:Ethernet HWaddr fe:16:3e:32:3b:0d inet6 addr: fe80::fc16:3eff:fe32:3b0d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3322983 errors:0 dropped:0 overruns:0 frame:0 TX packets:3480283 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2221592585 (2.2 GB) TX bytes:1380408263 (1.3 GB) tapa32c35b1-87 Link encap:Ethernet HWaddr fe:16:3e:79:65:9f inet6 addr: fe80::fc16:3eff:fe79:659f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:22360161 errors:0 dropped:0 overruns:0 frame:0 TX packets:25585406 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3953751232 (3.9 GB) TX bytes:6985195478 (6.9 GB) vxlan-100 Link encap:Ethernet HWaddr 26:30:9e:96:7a:71 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:7671113091 errors:0 dropped:0 overruns:0 frame:0 TX packets:6732694121 errors:0 dropped:17713 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2959527236402 (2.9 TB) TX bytes:973862583509 (973.8 GB)

2 1

Replace OSD node without remapping PGs
by Nghia Viet Tran 02 Apr '20

02 Apr '20

Hi everyone, I'm working on replacing OSDs node with the newer one. The new host has the new hostname and new disk (faster one but the same size with old disk). My plan is - Reweight the OSD to zero to spread all existed data to the rest nodes to keep data availability - set flag noout, norebalance, norecover, nobackfill, destroy the OSD and join the new OSD as the same ID of the old one. By above approach, the cluster will remap PGs of all nodes. Each data will be moved twice times until it reach the new OSD (reweight and join new node as same ID) I also did the other way that only set flags and destroy OSD. But the result is still the same (degraded objects from destroyed osd and misplaced object after joining new osd) Are there any ways to replace the OSD node directly without remapping PGs of the whole cluster? Many thanks! Nghia.

3 3

Using sendfile on Ceph FS results in data stuck in client cache
by Mikael Öhman 01 Apr '20

01 Apr '20

Hi all, Using sendfile function to write data to cephfs, the data doesn't end up being written. From the client that writes the file, it looks correct at first, but from all other ceph clients, the size is 0 bytes. Re-mounting the filesystem, the data is lost. I didn't see any errors, the data just doesn't get written, as if it's just cached in cephfs client. Writing just an extra byte at the end of the file (without sendfile), it seems to trigger the actual write of all the data. Could someone else confirm if they are also seeing such issue? I'm on ceph 13.2.8, using kernel module for mounting on CentOS7. I've used this sendfile-example for the example below: https://github.com/pijewski/sendfile-example/blob/master/sendfile.c Using a small 27 byte source file. # ls -lh examples/ -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src # ./sendfile examples/src examples/dst 27 # ls -lh examples/ ------x--- 1 root c3-staff 27 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src But, directory is still on 27 bytes: # ls -lhd examples drwxr-sr-x 1 root c3-staff 27 Mar 24 18:15 examples and on all other cephfs clients, the file is empty: # ls -lh examples/ ------x--- 1 root c3-staff 0 Mar 24 18:12 dst -rw-r--r-- 1 root c3-staff 27 Mar 24 18:04 src Is this a bug in cephfs, or should I not expect sendfile to work (as it is not posix compliant). There are no error reported from what i can see, and it is 100% reproducible Best regards, Mikael

2 4

Re: osd can not start at boot after upgrade to octopus
by Eugen Block 01 Apr '20

01 Apr '20

Resending the response back to the list. Zitat von "Lomayani S. Laizer" <lomlaizer(a)gmail.com>: > Hello, > I have been running Nautilus from May last year so this is separate issue > from recent bug > > I think the problem is between systemd and ceph-volume. No any logs hitting > osd logs because osd dont start at all. > starting osd manually works fine (/usr/bin/ceph-osd -f --cluster ceph --id > 29 --setuser ceph --setgroup ceph) > > You can see starting osd just exit with no usable log(RuntimeError: command > returned non-zero exit status: 1) > > Below is the logs of ceph-volume-systemd.log > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev /dev/ceph-3811ddf5-02be-40f1-a53e-053131aa5712/osd-bl > ock-3e52d340-5416-46e6-b697-c15ca85f6883 --path /var/lib/ceph/osd/ceph-29 > --no-mon-config > Running command: /bin/ln -snf > /dev/ceph-3811ddf5-02be-40f1-a53e-053131aa5712/osd-block-3e52d340-5416-46e6-b697-c15ca85f6883 > /var/lib/c > eph/osd/ceph-29/block > Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-29/block > Running command: /bin/chown -R ceph:ceph /dev/dm-4 > Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-29 > Running command: /bin/systemctl enable > ceph-volume@lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 > Running command: /bin/systemctl enable --runtime ceph-osd@29 > stderr: Created symlink /run/systemd/system/ce > [2020-04-01 12:17:31,111][ceph_volume.process][INFO ] stderr > ph-osd.target.wants/ceph-osd(a)33.service → /lib/systemd/system/ceph-osd@. > service. > [2020-04-01 12:17:31,111][ceph_volume.process][INFO ] stderr > ph-osd.target.wants/ceph-osd(a)29.service → /lib/systemd/system/ceph-osd@. > service. > Running command: /bin/systemctl start ceph-osd@29 > stderr: Job for ceph-osd(a)29.service canceled. > --> RuntimeError: command returned non-zero exit status: 1 > > On Wed, Apr 1, 2020 at 1:22 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Hi, >> >> are you hitting [1]? Did you run Nautilus only for a short period of >> time before upgrading to Octopus? >> >> If this doesn't apply to you, can you see anything in the OSD logs >> (/var/log/ceph/ceph-osd.<ID>.log)? >> >> Regards, >> Eugen >> >> [1] https://tracker.ceph.com/issues/44770 >> >> Zitat von "Lomayani S. Laizer" <lomlaizer(a)gmail.com>: >> >> > Hello, >> > I have upgraded nautilus cluster to octopus few days ago. the cluster was >> > running ok and even after to octopus everything was running ok >> > >> > the issue came when i rebooted the servers for updating the kernel. Two >> > servers out of 6 osd's servers osd cant start. No error reported in >> > ceph-volume.log and ceph-volume-systemd.log >> > >> > Starting osd with /usr/bin/ceph-osd -f --cluster ceph --id 30 --setuser >> > ceph --setgroup ceph works just fine. the issue is starting osd in >> systemd >> > >> > ceph-volume-systemd.log >> > >> > 16:36:28,193][systemd][WARNING] failed activating OSD, retries left: 30 >> > [2020-03-31 16:36:28,196][systemd][WARNING] command returned non-zero >> exit >> > status: 1 >> > [2020-03-31 16:36:28,196][systemd][WARNING] failed activating OSD, >> retries >> > left: 30 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-31-a8efb7db-686b-4789-a9c4-01442c28577f >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 28-7f4113c8-c5cf-4f70-9f7a-7a32de9d6587 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 30-8a70ad95-1c79-4502-a9a3-d5d7b9df84b6 >> > [2020-03-31 16:41:25,054][systemd][INFO ] raw systemd input received: >> > lvm-33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 31-a8efb7db-686b-4789-a9c4-01442c28577f >> > [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: >> > lvm-29-3e52d340-5416-46e6-b697-c15ca85f6883 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 33-7d688fc1-ed7b-45ae-ac0e-7b1787e0b64f >> > [2020-03-31 16:41:25,068][systemd][INFO ] raw systemd input received: >> > lvm-32-3841a62d-d6bc-404a-8762-163530b2d5d4 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 29-3e52d340-5416-46e6-b697-c15ca85f6883 >> > [2020-03-31 16:41:25,096][systemd][INFO ] parsed sub-command: lvm, extra >> > data: 32-3841a62d-d6bc-404a-8762-163530b2d5d4 >> > [2020-03-31 16:41:25,108][ceph_volume.process][INFO ] Running command: >> > /usr/sbin/ceph-volume lvm trigger 29-3e52d340-5416-46e6-b697-c15ca85f6883 >> > >> > ceph-volume.log >> > 2-163530b2d5d4 >> > [2020-03-31 17:17:23,679][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@31 >> > [2020-03-31 17:17:23,863][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@30 >> > [2020-03-31 17:17:24,045][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@33 >> > [2020-03-31 17:17:24,241][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@32 >> > [2020-03-31 17:17:24,449][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@28 >> > [2020-03-31 17:17:24,629][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl enable --runtime ceph-osd@29 >> > [2020-03-31 17:17:24,652][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)31.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:24,664][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)30.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:24,872][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@31 >> > [2020-03-31 17:17:24,875][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)33.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,072][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)32.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,075][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@30 >> > [2020-03-31 17:17:25,282][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@33 >> > [2020-03-31 17:17:25,497][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)28.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,499][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@32 >> > [2020-03-31 17:17:25,520][ceph_volume.process][INFO ] stderr Created >> > symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)29.service → >> > /lib/systemd/system/ceph-osd@.service. >> > [2020-03-31 17:17:25,705][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@28 >> > [2020-03-31 17:17:25,887][ceph_volume.process][INFO ] Running command: >> > /bin/systemctl start ceph-osd@29 >> > >> > -- >> > Lomayani >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: [Octopus] Beware the on-disk conversion
by Dan van der Ster 01 Apr '20

01 Apr '20

Doh, I hope so! On Wed, Apr 1, 2020 at 5:35 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > April fools day!!!!!! :) > > > -----Original Message----- > Sent: 01 April 2020 17:28 > To: ceph-users(a)ceph.io > Subject: [ceph-users] [Octopus] Beware the on-disk conversion > > Hi, > > As the upgrade documentation tells: > > Note that the first time each OSD starts, it will do a format > > conversion to improve the accounting for omap data. This may take > a > > few minutes to as much as a few hours (for an HDD with lots of omap > > data). You can disable this automatic conversion with: > > What the documentation does not say is that this process takes a lot of > memory > > I am upgrading a rusty cluster from Nautilus, you can check out the ram > consumption as attachment > > First, we have a 3TB osd conversion: it tooks ~15min, and 19GB of memory > > Then, we have a larger 6TB osd conversion: it tooks more than 2 hours, > and 35GB of memory > > Finally, you have the largest 10TB osd: only 1H15, but 52GB of memory > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1