October 2019 - ceph-users

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

MDS rejects clients causing hanging mountpoint on linux kernel client
by Florian Pritz 08 Feb '21

08 Feb '21

Hi, We are running a ceph cluster on Ubuntu 18.04 machines with ceph 14.2.4. Our cephfs clients are using the kernel module and we have noticed that some of them are sometimes (at least once) hanging after an MDS restart. The only way to resolve this is to unmount and remount the mountpoint, or reboot the machine if unmounting is not possible. After some investigation, the problem seems to be that the MDS denies reconnect attempts from some clients during restart even though the reconnect interval is not yet reached. In particular, I see the following log entries. Note that there are supposedly 9 sessions. 9 clients reconnect (one client has two mountpoints) and then two more clients reconnect after the MDS already logged "reconnect_done". These two clients were hanging after the event. The kernel log of one of them is shown below too. Running `ceph tell mds.0 client ls` after the clients have been rebooted/remounted also shows 11 clients instead of 9. Do you have any ideas what is wrong here and how it could be fixed? I'm guessing that the issue is that the MDS apparently has an incorrect session count and stops the reconnect process to soon. Is this indeed a bug and if so, do you know what is broken? Regardless, I also think that the kernel should be able to deal with a denied reconnect and that it should try again later. Yet, even after 10 minutes, the kernel does not attempt to reconnect. Is this a known issue or maybe fixed in newer kernels? If not, is there a chance to get this fixed? Thanks, Florian MDS log: > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.server reconnect_clients -- 9 sessions > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24197043 v1:10.1.4.203:0/990008521 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.30487144 v1:10.1.4.146:0/483747473 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21019865 v1:10.1.7.22:0/3752632657 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.21020717 v1:10.1.7.115:0/2841046616 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24171153 v1:10.1.7.243:0/1127767158 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.23978093 v1:10.1.4.71:0/824226283 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.24209569 v1:10.1.4.157:0/1271865906 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190930 v1:10.1.4.240:0/3195698606 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 0 log_channel(cluster) log [DBG] : reconnect by client.20190912 v1:10.1.4.146:0/852604154 after 0 > 2019-09-26 16:08:27.479 7f9fdde99700 1 mds.0.59 reconnect_done > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.24167394 v1:10.1.67.49:0/1483641729 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1087700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.49:0/1483641729 conn(0x55af50053f80 0x55af50140800 :6801 s=OPENED pgs=21 cs=1 l=0).fault server, going to standby > 2019-09-26 16:08:27.483 7f9fdde99700 1 mds.0.server no longer in reconnect state, ignoring reconnect, sending close > 2019-09-26 16:08:27.483 7f9fdde99700 0 log_channel(cluster) log [INF] : denied reconnect attempt (mds is up:reconnect) from client.30586072 v1:10.1.67.140:0/3664284158 after 0.00400002 (allowed interval 45) > 2019-09-26 16:08:27.483 7f9fe1888700 0 --1- [v2:10.1.4.203:6800/806949107,v1:10.1.4.203:6801/806949107] >> v1:10.1.67.140:0/3664284158 conn(0x55af50055600 0x55af50143000 :6801 s=OPENED pgs=8 cs=1 l=0).fault server, going to standby Hanging client (10.1.67.49) kernel log: > 2019-09-26T16:08:27.481676+02:00 hostnamefoo kernel: [708596.227148] ceph: mds0 reconnect start > 2019-09-26T16:08:27.488943+02:00 hostnamefoo kernel: [708596.233145] ceph: mds0 reconnect denied > 2019-09-26T16:16:17.541041+02:00 hostnamefoo kernel: [709066.287601] libceph: mds0 10.1.4.203:6801 socket closed (con state NEGOTIATING) > 2019-09-26T16:16:18.068934+02:00 hostnamefoo kernel: [709066.813064] ceph: mds0 rejected session > 2019-09-26T16:16:18.068955+02:00 hostnamefoo kernel: [709066.814843] ceph: get_quota_realm: ino (10000000008.fffffffffffffffe) null i_snap_realm

3 6

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

ceph osd set-require-min-compat-client jewel failure
by 潘东元 14 May '20

14 May '20

hi,every one, my ceph version 12.2.12，I want to set require min compat client luminous,I use command #ceph osd set-require-min-compat-client luminous but ceph report:Error EPERM: cannot set require_min_compat_client to luminous: 4 connected client(s) look like jewel (missing 0xa00000000200000); add --yes-i-really-mean-it to do it anyway [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 4 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 168 } } } so,I run command: [root@node-1 gyt]# ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it set require_min_compat_client to luminous but now,I want to set require min compat client jewel,I use command： [root@node-1 gyt]# ceph osd set-require-min-compat-client jewel Error EPERM: osdmap current utilizes features that require luminous; cannot set require_min_compat_client below that to jewel what‘s the way we are set luminous chang to jewel？

3 2

ceph-mgr Module "zabbix" cannot send Data
by i.schmidt＠langeoog.de 07 Apr '20

07 Apr '20

Hi Folks We are using Ceph as our storage backend on our 6 Node Proxmox VM Cluster. To Monitor our systems we use Zabbix and i would like to get some Ceph Data into our Zabbix to get some alarms when something goes wrong. Ceph mgr has a module, "zabbix" that uses "zabbix-sender" to actively send data, but i cannot get the module working. It always responds with "failed to send data" The network side seems to be fine: root@vm-2:~# traceroute 192.168.15.253 traceroute to 192.168.15.253 (192.168.15.253), 30 hops max, 60 byte packets 1 192.168.15.253 (192.168.15.253) 0.411 ms 0.402 ms 0.393 ms root@vm-2:~# nmap -p 10051 192.168.15.253 Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-18 08:40 CEST Nmap scan report for 192.168.15.253 Host is up (0.00026s latency). PORT STATE SERVICE 10051/tcp open zabbix-trapper MAC Address: BA:F5:30:EF:40:EF (Unknown) Nmap done: 1 IP address (1 host up) scanned in 0.61 seconds root@vm-2:~# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "192.168.15.253", "identifier": "VM-2", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} root@vm-2:~# But if i try "ceph zabbix send" i get "failed to send data to zabbix" and this show up in the systems journal: Sep 18 08:41:13 vm-2 ceph-mgr[54445]: 2019-09-18 08:41:13.272 7fe360fe4700 -1 mgr.server reply reply (1) Operation not permitted The log of ceph-mgr on that machine states: 2019-09-18 08:42:18.188 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253392]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000179"}] 2019-09-18 08:43:18.217 7fe359fd6700 0 mgr[zabbix] Exception when sending: /usr/bin/zabbix_sender exited non-zero: zabbix_sender [3253629]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 44; total: 44; seconds spent: 0.000321"}] I'm guessing, this could have something to do with user rights. But i have no idea where to start to track this down. Maybe someone here has a hint? If more information is needed, i will gladly provide it. greetings Ingo

5 6

osd_pg_create causing slow requests in Nautilus
by Bryan Stillwell 03 Apr '20

03 Apr '20

We've run into a problem on our test cluster this afternoon which is running Nautilus (14.2.2). It seems that any time PGs move on the cluster (from marking an OSD down, setting the primary-affinity to 0, or by using the balancer), a large number of the OSDs in the cluster peg the CPU cores they're running on for a while which causes slow requests. From what I can tell it appears to be related to slow peering caused by osd_pg_create() taking a long time. This was seen on quite a few OSDs while waiting for peering to complete: # ceph daemon osd.3 ops { "ops": [ { "description": "osd_pg_create(e179061 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:34:46.556413", "age": 318.25234538000001, "duration": 318.25241895300002, "type_data": { "flag_point": "started", "events": [ { "time": "2019-08-27 14:34:46.556413", "event": "initiated" }, { "time": "2019-08-27 14:34:46.556413", "event": "header_read" }, { "time": "2019-08-27 14:34:46.556299", "event": "throttled" }, { "time": "2019-08-27 14:34:46.556456", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456901", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456903", "event": "wait for new map" }, { "time": "2019-08-27 14:40:01.292346", "event": "started" } ] } }, ...snip... { "description": "osd_pg_create(e179066 287.7a:177739 287.9a:177739 287.e2:177739 287.e7:177739 287.f6:177739 287.187:177739 287.1aa:177739 287.216:177739 287.306:177739 287.3e6:177739)", "initiated_at": "2019-08-27 14:35:09.908567", "age": 294.900191001, "duration": 294.90068416899999, "type_data": { "flag_point": "delayed", "events": [ { "time": "2019-08-27 14:35:09.908567", "event": "initiated" }, { "time": "2019-08-27 14:35:09.908567", "event": "header_read" }, { "time": "2019-08-27 14:35:09.908520", "event": "throttled" }, { "time": "2019-08-27 14:35:09.908617", "event": "all_read" }, { "time": "2019-08-27 14:35:12.456921", "event": "dispatched" }, { "time": "2019-08-27 14:35:12.456923", "event": "wait for new map" } ] } } ], "num_ops": 6 } That "wait for new map" message made us think something was getting hung up on the monitors, so we restarted them all without any luck. I'll keep investigating, but so far my google searches aren't pulling anything up so I wanted to see if anyone else is running into this? Thanks, Bryan

6 16

RGWReshardLock::lock failed to acquire lock ret=-16
by Josh Haft 12 Mar '20

12 Mar '20

Hi, Currently running Mimic 13.2.5. We had reports this morning of timeouts and failures with PUT and GET requests to our Ceph RGW cluster. I found these messages in the RGW log: RGWReshardLock::lock failed to acquire lock on bucket_name:bucket_instance ret=-16 NOTICE: resharding operation on bucket index detected, blocking block_while_resharding ERROR: bucket is still resharding, please retry Which were preceded by many of these, which I think are normal/expected. check_bucket_shards: resharding needed: stats.num_objects=6415879 shard max_objects=6400000 Our RGW cluster sits behind haproxy which notified me approx 90 seconds after the first 'resharding needed' message that no backends were available. It appears this dynamic reshard process caused the RGWs to lock up for a period of time. Roughly 2 minutes later the reshard error messages stop and operation returns to normal. Looking back through previous RGW logs, I see a similar event from about a week ago, on the same bucket. We have several buckets with shard counts exceeding 1k (this one only has 128), and much larger object counts, so clearly this isn't the first time dynamic sharding has been invoked on this cluster. Has anyone seen this? I expect it will come up again, and can turn up debugging if that'll help. Thanks for any assistance! Josh

1 1

official ceph.com buster builds?
by Chad W Seys 27 Feb '20

27 Feb '20

Hi all, Am I missing the ceph buster build built by ceph.com ? http://download.ceph.com/debian-nautilus/dists/ Should I be using the Croit supplied builds? Thanks! Chad.

2 2

ceph status reports: slow ops - this is related to long running process /usr/bin/ceph-osd
by Thomas 18 Feb '20

18 Feb '20

Hi, ceph status reports: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available 4 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 59 pgs inactive, 16 pgs peering Degraded data redundancy: 597/153910758 objects degraded (0.000%), 2 pgs degraded, 1 pg undersized Degraded data redundancy (low space): 23 pgs backfill_toofull 1 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 3 pools have too many placement groups 164 slow requests are blocked > 32 sec 1082 stuck requests are blocked > 4096 sec 1490 slow ops, oldest one blocked for 19711 sec, daemons [osd,0,osd,175,osd,186,osd,5,osd,6,osd,63,osd,68,osd,9,mon,ld5505,mon,ld5506]... have slow ops. services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 5h) mgr: ld5507(active, since 5h), standbys: ld5506, ld5505 mds: pve_cephfs:0/1, 1 failed osd: 419 osds: 416 up, 416 in; 6024 remapped pgs data: pools: 6 pools, 8864 pgs objects: 51.30M objects, 196 TiB usage: 594 TiB used, 907 TiB / 1.5 PiB avail pgs: 0.666% pgs not active 597/153910758 objects degraded (0.000%) 52964415/153910758 objects misplaced (34.412%) 5954 active+remapped+backfill_wait 2786 active+clean 40 active+remapped+backfilling 35 activating 23 active+remapped+backfill_wait+backfill_toofull 16 peering 7 activating+remapped 1 activating+undersized+degraded 1 active+clean+scrubbing 1 active+recovering+degraded io: client: 3.5 KiB/s wr, 0 op/s rd, 0 op/s wr recovery: 551 MiB/s, 137 objects/s I'm concerned about the slow ops on osd.0 and osd.9. On the relevant OSD node I can see 2 relevant services running for hours: ceph 14795 1 99 09:58 ? 08:49:22 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph ceph 15394 1 99 09:58 ? 07:10:00 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph In the relevant osd log I can find similar messages: root@ld5505:~# tail -f /var/log/ceph/ceph-osd.0.log 2019-10-08 15:35:32.830 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:33.806 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:34.842 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) 2019-10-08 15:35:35.862 7ff60c7cc700 -1 osd.0 233323 get_health_metrics reporting 236 slow ops, oldest is osd_pg_create(e233257 38.0:199987) root@ld5505:~# tail -f /var/log/ceph/ceph-osd.9.log 2019-10-08 15:35:38.822 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:39.854 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:40.850 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) 2019-10-08 15:35:41.862 7f8957599700 -1 osd.9 233407 get_health_metrics reporting 818 slow ops, oldest is osd_op(client.53385387.0:23 30.f7 30.bcc140f7 (undecoded) ondisk+retry+read+known_if_redirected e233362) Question: How can I analyse and solve the issue with slow ops? THX

3 2

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

Ceph and centos 8
by fleg＠lpnhe.in2p3.fr 17 Dec '19

17 Dec '19

Hi, We have a ceph+cephfs cluster runing nautilus version 14.2.4 We have debian buster/ubuntu bionic clients mounting cephfs in kernel mode without problems. We now want to mount cephfs from our new centos 8 clients. Unfortunately, ceph-common is needed but there are no packages available for el8 (only el7). And no way to install the el7 packages on centos 8 (missing deps). Thus, despite the fact that centos 8 have a 4.18 kernel (required to use quota, snapshots etc...), it seems impossible to mount in kernel mode (good perfs) and we still have to use the so slow fuse mode. Is it possible to workaround this problem ? Or when is it planned to provides (even as beta) the ceph packages for centos 8 ? Thanks.

4 5

Past_interval start interval mismatch (last_clean_epoch reported)
by Huseyin Cotuk 10 Nov '19

10 Nov '19

Hi all, I also hit the bug #24866 in my test environment. According to the logs, the last_clean_epoch in the specified OSD/PG is 17703, but the interval starts with 17895. So the OSD fails to start. There are some other OSDs in the same status. 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=18430/18431 n=1 ec=295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=1 lpr=18431 pi=[17895,18430)/3 crt=18388'4 lcod 0'0 unknown m=1 mbc={}] 18.51 past_intervals [17895,18430) start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the monitor. I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the reported epoch #? Thanks in advance. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

3 2

Re: Wrong %USED and MAX AVAIL stats for pool
by Yordan Yordanov (Innologica) 09 Nov '19

09 Nov '19

The documentation states: https://docs.ceph.com/docs/mimic/rados/operations/monitoring/ The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section DOES NOT reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots. However in our case we are clearly seeing the USAGE field multiplying the total object sizes to the number of replicas. [root@blackmirror ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 34 TiB 46 TiB 46 TiB 58.10 TOTAL 80 TiB 34 TiB 46 TiB 46 TiB 58.10 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.05M 46 TiB 68.32 7.2 TiB bench 5 250 MiB 67 250 MiB 0 22 TiB [root@blackmirror ~]# rbd du -p one NAME PROVISIONED USED ... <TOTAL> 20 TiB 15 TiB This is causing several apps (including ceph dashboard) to display inaccurate percentages, because they calculate the total pool capacity as USED + MAX AVAIL, which in this case yields 53.2TB, which is way off. 7.2TB is about 13% of that, so we receive alarms and this is bugging us for quite some time now.

4 4

iSCSI write performance
by Ryan 02 Nov '19

02 Nov '19

I'm in the process of testing the iscsi target feature of ceph. The cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12 SSD OSDs per host. Some basic testing moving VMs to a ceph backed datastore is only showing 60MB/s transfers. However moving these back off the datastore is fast at 200-300MB/s. What should I be looking at to track down the write performance issue? In comparison with the Nimble Storage arrays I can see 200-300MB/s in both directions. Thanks, Ryan

8 20

Ceph Health error right after starting balancer
by Thomas Schneider 01 Nov '19

01 Nov '19

Hi, after enabling ceph balancer (with command ceph balancer on) the health status changed to error. This is the current output of ceph health detail: root@ld3955:~# ceph health detail HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests are blocked > 4096 sec; mon ld5505 is low on available space REQUEST_SLOW 1438 slow requests are blocked > 32 sec 683 ops are blocked > 2097.15 sec 436 ops are blocked > 1048.58 sec 191 ops are blocked > 524.288 sec 78 ops are blocked > 262.144 sec 35 ops are blocked > 131.072 sec 11 ops are blocked > 65.536 sec 4 ops are blocked > 32.768 sec osd.62 has blocked requests > 65.536 sec osds 39,72 have blocked requests > 262.144 sec osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec osds 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have blocked requests > 1048.58 sec osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have blocked requests > 2097.15 sec REQUEST_STUCK 861 stuck requests are blocked > 4096 sec 25 ops are blocked > 8388.61 sec 836 ops are blocked > 4194.3 sec osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck requests > 4194.3 sec osds 0,30,70,71,184 have stuck requests > 8388.61 sec I understand that when balancer starts shifting PGs to other OSDs that this caused IO load on the cluster. However I don't understand why this is affecting OSD so heavily. And I don't understand why OSD of specific type (SSD, NVME) suffer although there's no balancing occuring on them. Regards Thomas

3 4

subtrees have overcommitted (target_size_bytes / target_size_ratio)
by Lars Täuber 01 Nov '19

01 Nov '19

Hello everybody! What does this mean? health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away. $ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on $ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] Thanks Lars

3 10

V/v Multiple pool for data in Ceph object
by tuan dung 01 Nov '19

01 Nov '19

hi ceph-users, i have a cluster run ceph object using version 14.2.1. I want to creat 2 pool for bucket data for purposes for security: + one bucket-data pool for public client access from internet (name *zone1.rgw.buckets.data-pub) * + one bucket-data pool for private client access from local network (name *zone1.rgw.buckets.data-pub)* each pool bucket-data has one individual access key: access key public (access pool public) and access key private (access pool private). Can you give me a recomment for this or bestpractice that you've done? what needs to be done? Or give me your best solution for securiy a cluster ceph object with public client access and private client access? Thank you very much Br, ---------------------------------------------- Dương Tuấn Dũng Email: dungdt.aicgroup(a)gmail.com ĐT: 0986153686

2 2

ceph pg dump hangs on mons w/o mgr
by Frank R 31 Oct '19

31 Oct '19

I have a 12.2.12 cluster with 3 mons where mgr will be active on 1. I have noticed that the command "ceph pg dump" hangs on all mons except the one where the mgr is running. "ceph pg dump" also runs fine on osd nodes. Is this expected behavior? thx Frank

1 0

Error in MGR Log: auth: could not find secret_id=<number>
by Thomas Schneider 31 Oct '19

31 Oct '19

Hello, I noticed several error messages in MGR (active) log: 2019-10-31 10:40:16.341 7ff9edd64700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.341 7ff9edd64700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:16.541 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.541 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:16.941 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.941 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:17.741 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:17.741 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 These error message are logged every 5 sec. Can you please advise how to troubleshoot this issue? THX

1 0

Splitting PGs not happening on Nautilus 14.2.2
by Bryan Stillwell 30 Oct '19

30 Oct '19

This morning I noticed that on a new cluster the number of PGs for the default.rgw.buckets.data pool was way too small (just 8 PGs), but when I try to split the PGs the cluster doesn't do anything: # ceph osd pool set default.rgw.buckets.data pg_num 16 set pool 13 pg_num to 16 It seems to set the pg_num_target/pgp_num_target to 16, but pg_num/pgp_num never increase: # ceph osd dump | grep default.rgw.buckets.data pool 13 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 pg_num_target 16 pgp_num_target 16 autoscale_mode warn last_change 43217 flags hashpspool,creating stripe_width 0 compression_mode aggressive application rgw The clusters has 295 OSDs, so I need to do a bit of splitting today. Any ideas why the splits aren't starting? Thanks, Bryan

1 1

Re: rgw recovering shards
by Konstantin Shalygin 30 Oct '19

30 Oct '19

On 10/27/19 6:01 AM, Frank R wrote: > I hate to be a pain but I have one more question. > > After I run > > radosgw-admin reshard stale-instances rm > > if I run > > radosgw-admin reshard stale-instances list > > some new entries appear for a bucket that no longer exists. Is there a > way to cancel the operation on the old bucket? > `radosgw-admin reshard stale-instances rm` should fix all your issues, if this not you should debug this. k

2 3

Dirlisting hangs with cephfs
by Kári Bertilsson 30 Oct '19

30 Oct '19

This seems to happen mostly when listing folders containing 10k+ folders. The dirlisting hangs indefinitely or until i restart the active MDS and then the hanging "ls" command will finish running. Every time restarting the active MDS fixes the problem for a while.

4 14

Lower mem radosgw config?
by Dan van der Ster 30 Oct '19

30 Oct '19

Hi all, Does anyone have a good config for lower memory radosgw machines? We have 16GB VMs and our radosgw's go OOM when we have lots of parallel clients (e.g. I see around 500 objecter_ops via the rgw asok). Maybe lowering rgw_thread_pool_size from 512 would help? (This is running latest luminous). Thanks, Dan

2 1

ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12
by Marc Roos 30 Oct '19

30 Oct '19

I am getting these since Nautilus upgrade [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d9 ffff9117b63e3900 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859da ffff911993e84700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e6 ffff911993e85500 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859db ffff911993e85800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859df ffff9119697ca700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d9 ffff9117b63e3900 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859da ffff911993e84700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e6 ffff911993e85500 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859db ffff911993e85800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859df ffff9119697ca700 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859de ffff9121cf0c3c00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e8 ffff911993e85e00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e7 ffff91203fbdb600 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859e5 ffff9121cf0c2b00 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859dd ffff911cca33b800 fail -12 [Wed Oct 30 01:32:09 2019] ceph: build_snap_context 100020859d2 ffff911d3eef5a00 fail -12

1 0

V/v Log IP clinet in rados gateway log
by tuan dung 30 Oct '19

30 Oct '19

Hi all, I want to log IP client to rados gateway log to check information about loadbalancing and other things. I using LB before rados gateway nodes, what need to be done for configuration in rados gateway? thank you very much. Br, ---------------------------------------------- Dương Tuấn Dũng Email: dungdt.aicgroup(a)gmail.com ĐT: 0986153686

2 1

pg stays in unknown states for a long time
by 潘东元 30 Oct '19

30 Oct '19

I have 104 pg stays in unknown states for a long time [root@node-1 /]# ceph -s cluster: id: 653c6c1a-607e-4a62-bb92-dfe2f0d7afb6 health: HEALTH_ERR 1 osds down Reduced data availability: 104 pgs inactive 24 slow requests are blocked > 32 sec. Implicated osds 0,1,2,8,9,10 14 stuck requests are blocked > 4096 sec. Implicated osds 5,6 services: mon: 3 daemons, quorum node-1,node-2,node-3 mgr: node-1(active), standbys: node-2, node-3 osd: 12 osds: 11 up, 12 in flags nodeep-scrub rbd-mirror: 1 daemon active data: pools: 7 pools, 360 pgs objects: 1.80k objects, 3.91GiB usage: 17.6GiB used, 7.96TiB / 7.98TiB avail pgs: 28.889% pgs unknown 256 active+clean 104 unknown io: client: 1.56MiB/s wr, 0op/s rd, 83op/s wr [root@node-1 /]# ceph health detail HEALTH_ERR Reduced data availability: 104 pgs inactive; 30 slow requests are blocked > 32 sec. Implicated osds 0,1,2,4,8,9,10; 14 stuck requests are blocked > 4096 sec. Implicated osds 5,6 PG_AVAILABILITY Reduced data availability: 104 pgs inactive pg 1.0 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.1 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.2 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.3 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.4 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.5 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.6 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 1.7 is stuck inactive for 2857.069686, current state unknown, last acting [] pg 2.0 is stuck inactive for 2857.069686, current state unknown, last acting [] ...... [root@node-1 /]# ceph pg dump_stuck inactive ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 3.1d unknown [] -1 [] -1 3.1c unknown [] -1 [] -1 3.1b unknown [] -1 [] -1 3.1a unknown [] -1 [] -1 3.19 unknown [] -1 [] -1 ...... my pool size = 3 [root@node-1 /]# ceph pg 3.1d query Error ENOENT: i don't have pgid 3.1d

1 0

Several ceph osd commands hang
by Thomas Schneider 30 Oct '19

30 Oct '19

Hi, in my unhealthy cluster I cannot run several ceph osd command because they hang, e.g. ceph osd df ceph osd pg dump Also, ceph balancer status hangs. How can I fix this issue? THX

3 5

After delete 8.5M Objects in a bucket still 500K left
by EDH - Manuel Rios Fernandez 29 Oct '19

29 Oct '19

Hi Ceph's! We started deteling a bucket several days ago. Total size 47TB / 8.5M objects. Now we see the cli bucket rm stucked and by console drop this messages. [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart uploads 2019-10-28 13:56:24.021 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 2000 incomplete multipart uploads 2019-10-28 13:57:04.726 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 3000 incomplete multipart uploads 2019-10-28 13:57:45.424 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 4000 incomplete multipart uploads 2019-10-28 13:58:25.905 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 5000 incomplete multipart uploads 2019-10-28 13:59:06.898 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 6000 incomplete multipart uploads 2019-10-28 13:59:47.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 7000 incomplete multipart uploads 2019-10-28 14:00:42.102 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 8000 incomplete multipart uploads 2019-10-28 14:01:23.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 9000 incomplete multipart uploads 2019-10-28 14:02:06.028 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 10000 incomplete multipart uploads 2019-10-28 14:02:48.648 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 11000 incomplete multipart uploads 2019-10-28 14:03:29.807 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 12000 incomplete multipart uploads 2019-10-28 14:04:11.180 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 13000 incomplete multipart uploads 2019-10-28 14:04:52.396 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 14000 incomplete multipart uploads 2019-10-28 14:05:33.050 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 15000 incomplete multipart uploads 2019-10-28 14:06:13.652 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 16000 incomplete multipart uploads 2019-10-28 14:06:54.806 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 17000 incomplete multipart uploads 2019-10-28 14:07:35.867 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 18000 incomplete multipart uploads 2019-10-28 14:08:16.886 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 19000 incomplete multipart uploads 2019-10-28 14:08:57.711 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 20000 incomplete multipart uploads 2019-10-28 14:09:38.032 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 21000 incomplete multipart uploads 2019-10-28 14:10:18.377 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 22000 incomplete multipart uploads 2019-10-28 14:10:58.833 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 23000 incomplete multipart uploads 2019-10-28 14:11:39.078 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 24000 incomplete multipart uploads 2019-10-28 14:12:24.731 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 25000 incomplete multipart uploads 2019-10-28 14:13:12.176 7f0dd92c9700 0 abort_bucket_multiparts WARNING : aborted 26000 incomplete multipart uploads Bucket stats show 500K objects left. Looks like bucket rm is trying to abort all incompleted mutipart. But in bucket stats this operation is not reflected removing objects from stats. May be wait to get up 500K or it's a bug? Regards Manuel

2 2

Compression on existing RGW buckets
by Bryan Stillwell 29 Oct '19

29 Oct '19

I'm wondering if it's possible to enable compression on existing RGW buckets? The cluster is running Luminous 12.2.12 with FileStore as the backend (no BlueStore compression then). We have a cluster that recently started to rapidly fill up with compressible content (qcow2 images) and I would like to enable compression for new uploads to slow the growth. The documentation seems to imply that changing zone placement rules can only be done at creation time. Is there something I'm missing that would allow me to enable compression on a per-bucket or even a per-user basis after a cluster has been used for quite a while? Thanks, Bryan

3 3

Bogus Entries in RGW Usage Log / Large omap object in rgw.log pool
by David Monschein 29 Oct '19

29 Oct '19

Hi All, Running an object storage cluster, originally deployed with Nautilus 14.2.1 and now running 14.2.4. Last week I was alerted to a new warning from my object storage cluster: [root@ceph1 ~]# ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. I looked into this and found the object and pool in question (default.rgw.log): [root@ceph1 /var/log/ceph]# grep -R -i 'Large omap object found' . ./ceph.log:2019-10-24 12:21:26.984802 osd.194 (osd.194) 715 : cluster [WRN] Large omap object found. Object: 5:0fbdcb32:usage::usage.17:head Key count: 702330 Size (bytes): 92881228 [root@ceph1 ~]# ceph --format=json pg ls-by-pool default.rgw.log | jq '.[]' | egrep '(pgid|num_large_omap_objects)' | grep -v '"num_large_omap_objects": 0,' | grep -B1 num_large_omap_objects "pgid": "5.70", "num_large_omap_objects": 1, While I was investigating, I noticed an enormous amount of entries in the RGW usage log: [root@ceph ~]# radosgw-admin usage show | grep -c bucket 223326 There are entries for over 223k buckets! This was pretty scary to see, considering we only have maybe 500 legitimate buckets in this fairly new cluster. Almost all of the entries in the usage log are bogus entries from anonymous users. It looks like someone/something was scanning, looking for vulnerabilities, etc. Here are a few example entries, notice none of the operations were successful: <-----SNIP-----> { "bucket": "pk1914.php", "time": "2019-07-26 21:00:00.000000Z", "epoch": 1564174800, "owner": "anonymous", "categories": [ { "category": "post_obj", "bytes_sent": 586, "bytes_received": 0, "ops": 2, "successful_ops": 0 } ] }, { "bucket": "plus", "time": "2019-07-26 21:00:00.000000Z", "epoch": 1564174800, "owner": "anonymous", "categories": [ { "category": "post_obj", "bytes_sent": 6314, "bytes_received": 0, "ops": 22, "successful_ops": 0 } ] }, { "bucket": "pma.php", "time": "2019-07-26 21:00:00.000000Z", "epoch": 1564174800, "owner": "anonymous", "categories": [ { "category": "post_obj", "bytes_sent": 580, "bytes_received": 0, "ops": 2, "successful_ops": 0 } ] <-----SNIP-----> I suspect that the large omap warning from Ceph is related to the 223k+ entries in the RGW usage log. I have two questions: 1) How can I remove these bogus bucket entries from the RGW usage log? When I issue $(radosgw-admin usage trim), it only resets the stats to 0, but does not actually remove the bogus bucket entries. 2) Is it possible to prevent Ceph from logging usage statistics for buckets that do not exist? Some more output if it's useful:## ceph --format=json pg ls-by-pool default.rgw.log | jq '.[]' <-----SNIP-----> { "pgid": "5.70", "version": "4712'5226850", "reported_seq": "5636679", "reported_epoch": "4712", "state": "active+clean", "last_fresh": "2019-10-24 14:40:59.287019", "last_change": "2019-10-24 12:21:26.984997", "last_active": "2019-10-24 14:40:59.287019", "last_peered": "2019-10-24 14:40:59.287019", "last_clean": "2019-10-24 14:40:59.287019", "last_became_active": "2019-10-18 16:02:06.007865", "last_became_peered": "2019-10-18 16:02:06.007865", "last_unstale": "2019-10-24 14:40:59.287019", "last_undegraded": "2019-10-24 14:40:59.287019", "last_fullsized": "2019-10-24 14:40:59.287019", "mapping_epoch": 4672, "log_start": "4712'5223781", "ondisk_log_start": "4712'5223781", "created": 1105, "last_epoch_clean": 4673, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "4712'5223077", "last_scrub_stamp": "2019-10-24 12:21:26.984947", "last_deep_scrub": "4712'5223077", "last_deep_scrub_stamp": "2019-10-24 12:21:26.984947", "last_clean_scrub_stamp": "2019-10-24 12:21:26.984947", "log_size": 3069, "ondisk_log_size": 3069, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 0, "num_objects": 2, "num_object_clones": 0, "num_object_copies": 6, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2, "num_whiteouts": 0, "num_read": 6140544, "num_read_kb": 6769195, "num_write": 10664318, "num_write_kb": 9859326, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 8, "num_bytes_recovered": 0, "num_keys_recovered": 2666590, "num_objects_omap": 1, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 1, "num_objects_manifest": 0, "num_omap_bytes": 92881228, "num_omap_keys": 702330, "num_objects_repaired": 0 }, "up": [ 194, 79, 51 ], "acting": [ 194, 79, 51 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 194, "acting_primary": 194, "purged_snaps": [] }, I appreciate any assistance. Thank you. - Dave Monschein

2 2

Static website hosting with RGW
by Oliver Freyermuth 29 Oct '19

29 Oct '19

Dear Cephers, I have a question concerning static websites with RGW. To my understanding, it is best to run >=1 RGW client for "classic" S3 and in addition operate >=1 RGW client for website serving (potentially with HAProxy or its friends in front) to prevent messup of requests via the different protocols. I'd prefer to avoid "*.example.com" entries in DNS if possible. So my current setup has these settings for the "web" RGW client: rgw_enable_static_website = true rgw_enable_apis = s3website rgw_dns_s3website_name = some_value_unused_when_A_records_are_used_pointing_to_the_IP_but_it_needs_to_be_set and I create simple A records for each website pointing to the IP of this "web" RGW node. I can easily upload content for those websites to the other RGW instances which are serving S3, so S3 and s3website APIs are cleanly separated in separate instances. However, one issue remains: How do I run s3cmd ws-create on each website-bucket once? I can't do that against the "classic" S3-serving RGW nodes. This will give me a 405 (not allowed), since they do not have rgw_enable_static_website enabled. I also can not run it against the "web S3" nodes, since they do not have the S3 API enabled. Of course I could enable that, but then the RGW node can't cleanly disentangle S3 and website requests since I use A records. Does somebody have a good idea on how to solve this issue? Setting "rgw_enable_static_website = true" on the S3-serving RGW nodes would solve it, but does that have any bad side-effects on their S3 operation? Also, if there's an expert on this: Exposing a bucket under a tenant as static website is not possible since the colon (:) can't be encoded in DNS, right? In case somebody also wants to set something like this up, here are the best docs I could find: https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c and of course: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html-s… Cheers, Oliver

5 6

Help
by Sumit Gaur 28 Oct '19

28 Oct '19

Unsubscribe please from all email ids On Tue, 29 Oct 2019 at 7:12 am, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Help (Sumit Gaur) > > > ---------------------------------------------------------------------- > > Date: Tue, 29 Oct 2019 07:06:17 +1100 > From: Sumit Gaur <sumitkgaur(a)gmail.com> > Subject: [ceph-users] Help > To: ceph-users(a)ceph.io > Message-ID: > <CAH_rbop2r7d4BVMi_cxYQgOaEcnpcdyjbsKR= > UdQwc9wBnAh_g(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000000dfea80595fe0971" > > --0000000000000dfea80595fe0971 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > On Tue, 29 Oct 2019 at 1:50 am, <ceph-users-request(a)ceph.io> wrote: > > > Send ceph-users mailing list submissions to > > ceph-users(a)ceph.io > > > > To subscribe or unsubscribe via email, send a message with subject or > > body 'help' to > > ceph-users-request(a)ceph.io > > > > You can reach the person managing the list at > > ceph-users-owner(a)ceph.io > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of ceph-users digest..." > > > > Today's Topics: > > > > 1. Re: subtrees have overcommitted (target_size_bytes / > > target_size_ratio) > > (Lars T=C3=A4uber) > > 2. After delete 8.5M Objects in a bucket still 500K left > > (EDH - Manuel Rios Fernandez) > > 3. Re: Static website hosting with RGW (Casey Bodley) > > > > > > ---------------------------------------------------------------------- > > > > Date: Mon, 28 Oct 2019 11:24:54 +0100 > > From: Lars T=C3=A4uber <taeuber(a)bbaw.de> > > Subject: [ceph-users] Re: subtrees have overcommitted > > (target_size_bytes / target_size_ratio) > > To: ceph-users <ceph-users(a)ceph.io> > > Message-ID: <20191028112454.0362fe66(a)bbaw.de> > > Content-Type: text/plain; charset=3DUTF-8 > > > > Is there a way to get rid of this warnings with activated autoscaler > > besides adding new osds? > > > > Yet I couldn't get a satisfactory answer to the question why this all > > happens. > > > > ceph osd pool autoscale-status : > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET > > RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > cephfs_data 122.2T 1.5 165.4T 1.1085 > > 0.8500 1.0 1024 on > > > > versus > > > > ceph df : > > RAW STORAGE: > > CLASS SIZE AVAIL USED RAW USED %RAW USED > > hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95 > > > > POOLS: > > POOL ID STORED OBJECTS USED %USED > > MAX AVAIL > > cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 > > 12 TiB > > > > > > It seems that the overcommitment is wrongly calculated. Isn't the RATE > > already used to calculate the SIZE? > > > > It seems USED(df) =3D SIZE(autoscale-status) > > Isn't the RATE already taken into account here? > > > > Could someone please explain the numbers to me? > > > > > > Thanks! > > Lars > > > > Fri, 25 Oct 2019 07:42:58 +0200 > > Lars T=C3=A4uber <taeuber(a)bbaw.de> =3D=3D> Nathan Fish > <lordcirth(a)gmail.c= > om> : > > > Hi Nathan, > > > > > > Thu, 24 Oct 2019 10:59:55 -0400 > > > Nathan Fish <lordcirth(a)gmail.com> =3D=3D> Lars T=C3=A4uber > <taeuber@bba= > w.de> : > > > > Ah, I see! The BIAS reflects the number of placement groups it should > > > > create. Since cephfs metadata pools are usually very small, but have > > > > many objects and high IO, the autoscaler gives them 4x the number of > > > > placement groups that it would normally give for that amount of data. > > > > > > > ah ok, I understand. > > > > > > > So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to > > > > 0.3? Are the two pools using entirely different device classes, so > > > > they are not sharing space? > > > > > > Yes, the metadata is on SSDs and the data on HDDs. > > > > > > > Anyway, I see that your overcommit is only "1.031x". So if you set > > > > cephfs_data to 0.85, it should go away. > > > > > > This is not the case. I set the target_ratio to 0.7 and get this: > > > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > cephfs_metadata 15736M 3.0 2454G 0.0188 > > 0.3000 4.0 256 on > > > cephfs_data 122.2T 1.5 165.4T 1.1085 > > 0.7000 1.0 1024 on > > > > > > The ratio seems to have nothing to do with the target_ratio but the > SIZ= > E > > and the RAW_CAPACITY. > > > Because the pool is still getting more data the SIZE increases and > > therefore the RATIO increases. > > > The RATIO seems to be calculated by this formula > > > RATIO =3D SIZE * RATE / RAW_CAPACITY. > > > > > > This is what I don't understand. The data in the cephfs_data pool seems > > to need more space than the raw capacity of the cluster provides. Hence > t= > he > > situation is called "overcommitment". > > > > > > But why is this only the case when the autoscaler is active? > > > > > > Thanks > > > Lars > > > > > > > > > > > On Thu, Oct 24, 2019 at 10:09 AM Lars T=C3=A4uber <taeuber(a)bbaw.de> > > wrote: > > > > > > > > > > Thanks Nathan for your answer, > > > > > > > > > > but I set the the Target Ratio to 0.9. It is the cephfs_data pool > > that makes the troubles. > > > > > > > > > > The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is > > not explained on the page linked below. So I don't know its meaning. > > > > > > > > > > How can be a pool overcommited when it is the only pool on a set of > > OSDs? > > > > > > > > > > Best regards, > > > > > Lars > > > > > > > > > > Thu, 24 Oct 2019 09:39:51 -0400 > > > > > Nathan Fish <lordcirth(a)gmail.com> =3D=3D> Lars T=C3=A4uber > <taeuber= > @bbaw.de> > > : > > > > > > The formatting is mangled on my phone, but if I am reading it > > correctly, > > > > > > you have set Target Ratio to 4.0. This means you have told the > > balancer > > > > > > that this pool will occupy 4x the space of your whole cluster, > an= > d > > to > > > > > > optimize accordingly. This is naturally a problem. Setting it to > = > 0 > > will > > > > > > clear the setting and allow the autobalancer to work. > > > > > > > > > > > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars T=C3=A4uber, <taeuber@bbaw > = > .de> > > wrote: > > > > > > > > > > > > > This question is answered here: > > > > > > > > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning= > / > > > > > > > > > > > > > > But it tells me that there is more data stored in the pool than > > the raw > > > > > > > capacity provides (taking the replication factor RATE into > > account) hence > > > > > > > the RATIO being above 1.0 . > > > > > > > > > > > > > > How comes this is the case? - Data is stored outside of the > poo= > l? > > > > > > > How comes this is only the case when the autoscaler is active? > > > > > > > > > > > > > > Thanks > > > > > > > Lars > > > > > > > > > > > > > > > > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > > > > > > Lars T=C3=A4uber <taeuber(a)bbaw.de> =3D=3D> ceph-users(a)ceph.io > : > > > > > > > > My question requires too complex an answer. > > > > > > > > So let me ask a simple question: > > > > > > > > > > > > > > > > What does the SIZE of "osd pool autoscale-status" > > tell/mean/comes from? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Lars > > > > > > > > > > > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > > > > > > Lars T=C3=A4uber <taeuber(a)bbaw.de> =3D=3D> > ceph-users(a)ceph.io= > : > > > > > > > > > Hello everybody! > > > > > > > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > > > > > > > health: HEALTH_WARN > > > > > > > > > 1 subtrees have overcommitted pool > > target_size_bytes > > > > > > > > > 1 subtrees have overcommitted pool > > target_size_ratio > > > > > > > > > > > > > > > > > > and what does it have to do with the autoscaler? > > > > > > > > > When I deactivate the autoscaler the warning goes away. > > > > > > > > > > > > > > > > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > > > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY > > RATIO > > > > > > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > > > > > > > cephfs_metadata 15106M 3.0 2454G > > 0.0180 > > > > > > > 0.3000 4.0 256 on > > > > > > > > > cephfs_data 113.6T 1.5 165.4T > > 1.0306 > > > > > > > 0.9000 1.0 512 on > > > > > > > > > > > > > > > > > > > > > > > > > > > $ ceph health detail > > > > > > > > > HEALTH_WARN 1 subtrees have overcommitted pool > > target_size_bytes; 1 > > > > > > > subtrees have overcommitted pool target_size_ratio > > > > > > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have > > overcommitted > > > > > > > pool target_size_bytes > > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > > 1.031x due > > > > > > > to target_size_bytes 0 on pools [] > > > > > > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have > > overcommitted > > > > > > > pool target_size_ratio > > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > > 1.031x due > > > > > > > to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Lars > > > > > > > > > _______________________________________________ > > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > > > Date: Mon, 28 Oct 2019 14:18:01 +0100 > > From: "EDH - Manuel Rios Fernandez" <mriosfer(a)easydatahost.com> > > Subject: [ceph-users] After delete 8.5M Objects in a bucket still 500K > > left > > To: <ceph-users(a)ceph.io> > > Message-ID: <02a201d58d92$1fe85880$5fb90980$(a)easydatahost.com> > > Content-Type: multipart/alternative; > > boundary=3D"----=3D_NextPart_000_02A3_01D58D9A.81B17B70" > > > > This is a multipart message in MIME format. > > > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > > Content-Type: text/plain; > > charset=3D"us-ascii" > > Content-Transfer-Encoding: 7bit > > > > Hi Ceph's! > > > > > > > > We started deteling a bucket several days ago. Total size 47TB / 8.5M > > objects. > > > > > > > > Now we see the cli bucket rm stucked and by console drop this messages. > > > > > > > > [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0 > > abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart > uploa= > ds > > > > 2019-10-28 13:56:24.021 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 2000 incomplete multipart uploads > > > > 2019-10-28 13:57:04.726 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 3000 incomplete multipart uploads > > > > 2019-10-28 13:57:45.424 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 4000 incomplete multipart uploads > > > > 2019-10-28 13:58:25.905 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 5000 incomplete multipart uploads > > > > 2019-10-28 13:59:06.898 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 6000 incomplete multipart uploads > > > > 2019-10-28 13:59:47.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 7000 incomplete multipart uploads > > > > 2019-10-28 14:00:42.102 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 8000 incomplete multipart uploads > > > > 2019-10-28 14:01:23.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 9000 incomplete multipart uploads > > > > 2019-10-28 14:02:06.028 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 10000 incomplete multipart uploads > > > > 2019-10-28 14:02:48.648 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 11000 incomplete multipart uploads > > > > 2019-10-28 14:03:29.807 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 12000 incomplete multipart uploads > > > > 2019-10-28 14:04:11.180 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 13000 incomplete multipart uploads > > > > 2019-10-28 14:04:52.396 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 14000 incomplete multipart uploads > > > > 2019-10-28 14:05:33.050 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 15000 incomplete multipart uploads > > > > 2019-10-28 14:06:13.652 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 16000 incomplete multipart uploads > > > > 2019-10-28 14:06:54.806 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 17000 incomplete multipart uploads > > > > 2019-10-28 14:07:35.867 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 18000 incomplete multipart uploads > > > > 2019-10-28 14:08:16.886 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 19000 incomplete multipart uploads > > > > 2019-10-28 14:08:57.711 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 20000 incomplete multipart uploads > > > > 2019-10-28 14:09:38.032 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 21000 incomplete multipart uploads > > > > 2019-10-28 14:10:18.377 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 22000 incomplete multipart uploads > > > > 2019-10-28 14:10:58.833 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 23000 incomplete multipart uploads > > > > 2019-10-28 14:11:39.078 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 24000 incomplete multipart uploads > > > > 2019-10-28 14:12:24.731 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 25000 incomplete multipart uploads > > > > 2019-10-28 14:13:12.176 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > > aborted 26000 incomplete multipart uploads > > > > > > > > > > > > Bucket stats show 500K objects left. Looks like bucket rm is trying to > > abort > > all incompleted mutipart. But in bucket stats this operation is not > > reflected removing objects from stats. > > > > > > > > May be wait to get up 500K or it's a bug? > > > > > > > > Regards > > > > Manuel > > > > > > > > > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > > Content-Type: text/html; > > charset=3D"us-ascii" > > Content-Transfer-Encoding: quoted-printable > > > > <html xmlns:v=3D3D"urn:schemas-microsoft-com:vml" =3D > > xmlns:o=3D3D"urn:schemas-microsoft-com:office:office" =3D > > xmlns:w=3D3D"urn:schemas-microsoft-com:office:word" =3D > > xmlns:m=3D3D"http://schemas.microsoft.com/office/2004/12/omml" =3D > > xmlns=3D3D"http://www.w3.org/TR/REC-html40"><head><meta =3D > > http-equiv=3D3DContent-Type content=3D3D"text/html; =3D > > charset=3D3Dus-ascii"><meta name=3D3DGenerator content=3D3D"Microsoft > Wor= > d 15 =3D > > (filtered medium)"><style></style></head><body lang=3D3DES =3D > > link=3D3D"#0563C1" vlink=3D3D"#954F72"><div class=3D3DWordSection1> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Hi =3D > > Ceph’s!<o:p></o:p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>We started deteling a = > =3D > > bucket several days ago. Total size 47TB / 8.5M =3D > > objects.<o:p></o:p> lang=3D3DEN-US= > =3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Now we see the cli =3D > > bucket rm stucked and by console drop this =3D > > messages.<o:p></o:p> lang=3D3DEN-U= > S =3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>[root@ceph-rgw03 ~]# = > =3D > > 2019-10-28 13:55:43.880 7f0dd92c9700  0 abort_bucket_multiparts =3D > > WARNING : aborted 1000 incomplete multipart =3D > > uploads<o:p></o:p> = > =3D > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:56:24.021= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 2000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:57:04.726= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 3000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:57:45.424= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 4000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:58:25.905= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 5000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:59:06.898= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 6000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:59:47.829= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 7000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:00:42.102= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 8000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:01:23.829= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 9000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:02:06.028= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 10000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:02:48.648= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 11000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:03:29.807= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 12000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:04:11.180= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 13000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:04:52.396= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 14000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:05:33.050= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 15000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:06:13.652= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 16000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:06:54.806= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 17000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:07:35.867= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 18000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:08:16.886= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 19000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:08:57.711= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 20000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:09:38.032= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 21000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:10:18.377= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 22000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:10:58.833= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 23000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:11:39.078= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 24000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:12:24.731= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 25000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:13:12.176= > =3D > > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 26000 =3D > > incomplete multipart uploads<o:p></o:p> > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Bucket stats show 500K > = > =3D > > objects left. Looks like bucket rm is trying to abort all incompleted =3D > > mutipart. But in bucket stats this operation is not reflected removing = > =3D > > objects from stats.<o:p></o:p> > lang=3D3DEN-US =3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>May be wait to get up = > =3D > > 500K or it’s a bug?<o:p></o:p> > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Regards<o:p></o:p></spa= > n>=3D > > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Manuel<o:p></o:p></span= > ><=3D > > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p></span= > ><=3D > > /p></div></body></html> > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70-- > > > > ------------------------------ > > > > Date: Mon, 28 Oct 2019 10:48:44 -0400 > > From: Casey Bodley <cbodley(a)redhat.com> > > Subject: [ceph-users] Re: Static website hosting with RGW > > To: ceph-users(a)ceph.io > > Message-ID: <20834361-445e-1ee5-433b-dd4792f90608(a)redhat.com> > > Content-Type: text/plain; charset=3DUTF-8; format=3Dflowed > > > > > > On 10/24/19 8:38 PM, Oliver Freyermuth wrote: > > > Dear Cephers, > > > > > > I have a question concerning static websites with RGW. > > > To my understanding, it is best to run >=3D1 RGW client for "classic" > S= > 3 > > and in addition operate >=3D1 RGW client for website serving > > > (potentially with HAProxy or its friends in front) to prevent messup of > > requests via the different protocols. > > > > > > I'd prefer to avoid "*.example.com" entries in DNS if possible. > > > So my current setup has these settings for the "web" RGW client: > > > rgw_enable_static_website =3D true > > > rgw_enable_apis =3D s3website > > > rgw_dns_s3website_name =3D > > > some_value_unused_when_A_records_are_used_pointing_to_the_IP_but_it_needs= > _to_be_set > > > and I create simple A records for each website pointing to the IP of > > this "web" RGW node. > > > > > > I can easily upload content for those websites to the other RGW > > instances which are serving S3, > > > so S3 and s3website APIs are cleanly separated in separate instances. > > > > > > However, one issue remains: How do I run > > > s3cmd ws-create > > > on each website-bucket once? > > > I can't do that against the "classic" S3-serving RGW nodes. This will > > give me a 405 (not allowed), > > > since they do not have rgw_enable_static_website enabled. > > > I also can not run it against the "web S3" nodes, since they do not > hav= > e > > the S3 API enabled. > > > Of course I could enable that, but then the RGW node can't cleanly > > disentangle S3 and website requests since I use A records. > > > > > > Does somebody have a good idea on how to solve this issue? > > > Setting "rgw_enable_static_website =3D true" on the S3-serving RGW > node= > s > > would solve it, but does that have any bad side-effects on their S3 > > operation? > > > > Enabling static website on the gateway serving the S3 api does look like > > the right solution. As far as I can tell, it's only used to control > > whether the S3 ops for PutBucketWebsite, GetBucketWebsite, and > > DeleteBucketWebsite are exposed. > > > > > > > > Also, if there's an expert on this: Exposing a bucket under a tenant as > > static website is not possible since the colon (:) can't be encoded in > DN= > S, > > right? > > > > > > > > > In case somebody also wants to set something like this up, here are the > > best docs I could find: > > > https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c > > > and of course: > > > > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html= > > -single/object_gateway_guide_for_red_hat_enterprise_linux/index#configuring= > <https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html=-…> > _gateways_for_static_web_hosting > > > > > > > > > Cheers, > > > Oliver > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > > > > ------------------------------ > > > > End of ceph-users Digest, Vol 81, Issue 79 > > ****************************************** > > > > --0000000000000dfea80595fe0971 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div> </div><div> <div class=3D"gmail_quote"><div dir=3D"ltr" class= > =3D"gmail_attr">On Tue, 29 Oct 2019 at 1:50 am, <<a href=3D"mailto: > ceph-= > users-request(a)ceph.io">ceph-users-request(a)ceph.io</a>> > wrote: </div><= > blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 > .8ex;border-left:1px= > #ccc solid;padding-left:1ex">Send ceph-users mailing list submissions > to<b= > r> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:ceph-users@ceph.io" > target=3D= > "_blank">ceph-users(a)ceph.io</a> > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:ceph-users-request@ceph.io" > t= > arget=3D"_blank">ceph-users-request(a)ceph.io</a> > > You can reach the person managing the list at > =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"mailto:ceph-users-owner@ceph.io" > tar= > get=3D"_blank">ceph-users-owner(a)ceph.io</a> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > =C2=A0 =C2=A01. Re: subtrees have overcommitted (target_size_bytes / > target= > _size_ratio) > =C2=A0 =C2=A0 =C2=A0 (Lars T=C3=A4uber) > =C2=A0 =C2=A02. After delete 8.5M Objects in a bucket still 500K left > =C2=A0 =C2=A0 =C2=A0 (EDH - Manuel Rios Fernandez) > =C2=A0 =C2=A03. Re: Static website hosting with RGW (Casey Bodley) > > > ---------------------------------------------------------------------- > > Date: Mon, 28 Oct 2019 11:24:54 +0100 > From: Lars T=C3=A4uber <<a href=3D"mailto:taeuber@bbaw.de" > target=3D"_bl= > ank">taeuber(a)bbaw.de</a>> > Subject: [ceph-users] Re: subtrees have overcommitted > =C2=A0 =C2=A0 =C2=A0 =C2=A0 (target_size_bytes / target_size_ratio) > To: ceph-users <<a href=3D"mailto:ceph-users@ceph.io" > target=3D"_blank">= > ceph-users(a)ceph.io</a>> > Message-ID: <<a href=3D"mailto:20191028112454.0362fe66@bbaw.de" target= > =3D"_blank">20191028112454.0362fe66(a)bbaw.de</a>> > Content-Type: text/plain; charset=3DUTF-8 > > Is there a way to get rid of this warnings with activated autoscaler > beside= > s adding new osds? > > Yet I couldn't get a satisfactory answer to the question why this all > h= > appens. > > ceph osd pool autoscale-status : > =C2=A0POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2=A0SIZE=C2=A0= > TARGET SIZE=C2=A0 RATE=C2=A0 RAW CAPACITY=C2=A0 =C2=A0RATIO=C2=A0 TARGET > R= > ATIO=C2=A0 BIAS=C2=A0 PG_NUM=C2=A0 NEW PG_NUM=C2=A0 AUTOSCALE > =C2=A0cephfs_data=C2=A0 =C2=A0 =C2=A0 122.2T=C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 1.5=C2=A0 =C2=A0 =C2=A0 =C2=A0 165.4T=C2=A0 > 1.1085= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 0.8500=C2=A0 =C2=A01.0=C2=A0 =C2=A0 1024=C2=A0 > = > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on > > versus > > =C2=A0ceph df=C2=A0 : > RAW STORAGE: > =C2=A0 =C2=A0 CLASS=C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 =C2=A0 =C2=A0 =C2=A0 > AVAI= > L=C2=A0 =C2=A0 =C2=A0 =C2=A0USED=C2=A0 =C2=A0 =C2=A0 =C2=A0 RAW USED=C2=A0 > = > =C2=A0 =C2=A0%RAW USED > =C2=A0 =C2=A0 hdd=C2=A0 =C2=A0 =C2=A0 =C2=A0165 TiB=C2=A0 =C2=A0 =C2=A0 41 > = > TiB=C2=A0 =C2=A0 =C2=A0124 TiB=C2=A0 =C2=A0 =C2=A0 124 TiB=C2=A0 =C2=A0 > =C2= > =A0 =C2=A0 =C2=A074.95 > > POOLS: > =C2=A0 =C2=A0 POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > I= > D=C2=A0 =C2=A0 =C2=A0STORED=C2=A0 =C2=A0 =C2=A0OBJECTS=C2=A0 =C2=A0 > =C2=A0U= > SED=C2=A0 =C2=A0 =C2=A0 =C2=A0 %USED=C2=A0 =C2=A0 =C2=A0MAX AVAIL > =C2=A0 =C2=A0 cephfs_data=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2=A0 > = > =C2=A075 TiB=C2=A0 =C2=A0 =C2=A0 49.31M=C2=A0 =C2=A0 =C2=A0122 TiB=C2=A0 = > =C2=A0 =C2=A087.16=C2=A0 =C2=A0 =C2=A0 =C2=A0 12 TiB > > > It seems that the overcommitment is wrongly calculated. Isn't the RATE > = > already used to calculate the SIZE? > > It seems USED(df) =3D SIZE(autoscale-status) > Isn't the RATE already taken into account here? > > Could someone please explain the numbers to me? > > > Thanks! > Lars > > Fri, 25 Oct 2019 07:42:58 +0200 > Lars T=C3=A4uber <<a href=3D"mailto:taeuber@bbaw.de" > target=3D"_blank">t= > aeuber(a)bbaw.de</a>> =3D=3D> Nathan Fish <<a href=3D"mailto: > lordcir= > th(a)gmail.com" target=3D"_blank">lordcirth(a)gmail.com</a>> : > > Hi Nathan, > > > > Thu, 24 Oct 2019 10:59:55 -0400 > > Nathan Fish <<a href=3D"mailto:lordcirth@gmail.com" > target=3D"_blan= > k">lordcirth(a)gmail.com</a>> =3D=3D> Lars T=C3=A4uber <<a > href=3D"m= > ailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> : > > > Ah, I see! The BIAS reflects the number of placement groups it > sh= > ould > > > create. Since cephfs metadata pools are usually very small, but > h= > ave > > > many objects and high IO, the autoscaler gives them 4x the > number= > of > > > placement groups that it would normally give for that amount of > d= > ata. > > >=C2=A0 =C2=A0 > > ah ok, I understand. > > > > > So, your cephfs_data is set to a ratio of 0.9, and > cephfs_metadat= > a to > > > 0.3? Are the two pools using entirely different device classes, > s= > o > > > they are not sharing space?=C2=A0 > > > > Yes, the metadata is on SSDs and the data on HDDs. > > > > > Anyway, I see that your overcommit is only "1.031x". > So= > if you set > > > cephfs_data to 0.85, it should go away.=C2=A0 > > > > This is not the case. I set the target_ratio to 0.7 and get this: > > > >=C2=A0 POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE= > =C2=A0 TARGET SIZE=C2=A0 RATE=C2=A0 RAW CAPACITY=C2=A0 =C2=A0RATIO=C2=A0 > TA= > RGET RATIO=C2=A0 BIAS=C2=A0 PG_NUM=C2=A0 NEW PG_NUM=C2=A0 AUTOSCALE > >=C2=A0 cephfs_metadata=C2=A0 15736M=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = > =C2=A0 =C2=A0 =C2=A0 3.0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02454G=C2=A0 > 0.018= > 8=C2=A0 =C2=A0 =C2=A0 =C2=A0 0.3000=C2=A0 =C2=A04.0=C2=A0 =C2=A0 =C2=A0256= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 > =C2= > =A0 > >=C2=A0 cephfs_data=C2=A0 =C2=A0 =C2=A0 122.2T=C2=A0 =C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.5=C2=A0 =C2=A0 =C2=A0 =C2=A0 > 165.4T=C2=A0= > 1.1085=C2=A0 =C2=A0 =C2=A0 =C2=A0 0.7000=C2=A0 =C2=A01.0=C2=A0 =C2=A0 > 1024= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 > =C2= > =A0 > > > > The ratio seems to have nothing to do with the target_ratio but the > SI= > ZE and the RAW_CAPACITY. > > Because the pool is still getting more data the SIZE increases and > the= > refore the RATIO increases. > > The RATIO seems to be calculated by this formula > > RATIO =3D SIZE * RATE / RAW_CAPACITY. > > > > This is what I don't understand. The data in the cephfs_data pool > = > seems to need more space than the raw capacity of the cluster provides. > Hen= > ce the situation is called "overcommitment". > > > > But why is this only the case when the autoscaler is active? > > > > Thanks > > Lars > > > > > > > > On Thu, Oct 24, 2019 at 10:09 AM Lars T=C3=A4uber <<a > href=3D"= > mailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> > wrote:=C2= > =A0 > > > > > > > > Thanks Nathan for your answer, > > > > > > > > but I set the the Target Ratio to 0.9. It is the > cephfs_data= > pool that makes the troubles. > > > > > > > > The 4.0 is the BIAS from the cephfs_metadata pool. This > &quo= > t;BIAS" is not explained on the page linked below. So I don't > know= > its meaning. > > > > > > > > How can be a pool overcommited when it is the only pool on > a= > set of OSDs? > > > > > > > > Best regards, > > > > Lars > > > > > > > > Thu, 24 Oct 2019 09:39:51 -0400 > > > > Nathan Fish <<a href=3D"mailto:lordcirth@gmail.com" > targe= > t=3D"_blank">lordcirth(a)gmail.com</a>> =3D=3D> Lars T=C3=A4uber > <<a= > href=3D"mailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> > = > :=C2=A0 =C2=A0 > > > > > The formatting is mangled on my phone, but if I am > read= > ing it correctly, > > > > > you have set Target Ratio to 4.0. This means you have > t= > old the balancer > > > > > that this pool will occupy 4x the space of your whole > c= > luster, and to > > > > > optimize accordingly. This is naturally a problem. > Sett= > ing it to 0 will > > > > > clear the setting and allow the autobalancer to > work.<b= > r> > > > > > > > > > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars T=C3=A4uber, > &lt= > ;<a href=3D"mailto:taeuber@bbaw.de" target=3D"_blank">taeuber(a)bbaw.de > </a>&g= > t; wrote: > > > > >=C2=A0 =C2=A0 > > > > > > This question is answered here: > > > > > > <a href=3D" > https://ceph.io/rados/new-in-nautilus-p= > g-merging-and-autotuning/ > <https://ceph.io/rados/new-in-nautilus-p=g-merging-and-autotuning/>" > rel=3D"noreferrer" target=3D"_blank">https://cep= > h.io/rados/new-in-nautilus-pg-merging-and-autotuning/</a> > > > > > > > > > > > > But it tells me that there is more data stored in > = > the pool than the raw > > > > > > capacity provides (taking the replication factor > R= > ATE into account) hence > > > > > > the RATIO being above 1.0 . > > > > > > > > > > > > How comes this is the case? - Data is stored > outsi= > de of the pool? > > > > > > How comes this is only the case when the > autoscale= > r is active? > > > > > > > > > > > > Thanks > > > > > > Lars > > > > > > > > > > > > > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > > > > > Lars T=C3=A4uber <<a href=3D"mailto: > taeuber@bba= > w.de" target=3D"_blank">taeuber(a)bbaw.de</a>> =3D=3D> <a > href=3D"mailt= > o:ceph-users@ceph.io" target=3D"_blank">ceph-users(a)ceph.io</a> :=C2=A0 > =C2= > =A0 > > > > > > > My question requires too complex an > answer.<b= > r> > > > > > > > So let me ask a simple question: > > > > > > > > > > > > > > What does the SIZE of "osd pool > autoscal= > e-status" tell/mean/comes from? > > > > > > > > > > > > > > Thanks > > > > > > > Lars > > > > > > > > > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > > > > > Lars T=C3=A4uber <<a href=3D"mailto: > taeube= > r(a)bbaw.de" target=3D"_blank">taeuber(a)bbaw.de</a>> =3D=3D> <a > href=3D"= > mailto:ceph-users@ceph.io" target=3D"_blank">ceph-users(a)ceph.io</a> > :=C2=A0= > =C2=A0 > > > > > > > > Hello everybody! > > > > > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > > > > >=C2=A0 =C2=A0 =C2=A0health: > HEALTH_WARN<b= > r> > > > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2=A0= > =C2=A01 subtrees have overcommitted pool target_size_bytes > > > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > =C2=A0= > =C2=A01 subtrees have overcommitted pool target_size_ratio > > > > > > > > > > > > > > > > and what does it have to do with the > aut= > oscaler? > > > > > > > > When I deactivate the autoscaler the > war= > ning goes away. > > > > > > > > > > > > > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > > > > >=C2=A0 POOL=C2=A0 =C2=A0 =C2=A0 =C2=A0 = > =C2=A0 =C2=A0 =C2=A0 =C2=A0SIZE=C2=A0 TARGET SIZE=C2=A0 RATE=C2=A0 RAW > CAPA= > CITY=C2=A0 =C2=A0RATIO=C2=A0 =C2=A0 > > > > > > TARGET RATIO=C2=A0 BIAS=C2=A0 PG_NUM=C2=A0 NEW > PG_= > NUM=C2=A0 AUTOSCALE=C2=A0 =C2=A0 > > > > > > > >=C2=A0 cephfs_metadata=C2=A0 > 15106M=C2=A0= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3.0=C2=A0 =C2=A0 =C2=A0 = > =C2=A0 =C2=A02454G=C2=A0 0.0180=C2=A0 =C2=A0 > > > > > >=C2=A0 =C2=A00.3000=C2=A0 =C2=A04.0=C2=A0 =C2=A0 = > =C2=A0256=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 > <= > br> > > > > > > > >=C2=A0 cephfs_data=C2=A0 =C2=A0 =C2=A0 > 11= > 3.6T=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.5=C2=A0 =C2= > =A0 =C2=A0 =C2=A0 165.4T=C2=A0 1.0306=C2=A0 =C2=A0 > > > > > >=C2=A0 =C2=A00.9000=C2=A0 =C2=A01.0=C2=A0 =C2=A0 = > =C2=A0512=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 > <= > br> > > > > > > > > > > > > > > > > > > > > > > > > $ ceph health detail > > > > > > > > HEALTH_WARN 1 subtrees have > overcommitte= > d pool target_size_bytes; 1=C2=A0 =C2=A0 > > > > > > subtrees have overcommitted pool > target_size_ratio= > =C2=A0 =C2=A0 > > > > > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 > s= > ubtrees have overcommitted=C2=A0 =C2=A0 > > > > > > pool target_size_bytes=C2=A0 =C2=A0 > > > > > > > >=C2=A0 =C2=A0 =C2=A0Pools > ['cephfs_da= > ta'] overcommit available storage by 1.031x due=C2=A0 =C2=A0 > > > > > > to target_size_bytes=C2=A0 =C2=A0 0=C2=A0 on > pools= > []=C2=A0 =C2=A0 > > > > > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 > s= > ubtrees have overcommitted=C2=A0 =C2=A0 > > > > > > pool target_size_ratio=C2=A0 =C2=A0 > > > > > > > >=C2=A0 =C2=A0 =C2=A0Pools > ['cephfs_da= > ta'] overcommit available storage by 1.031x due=C2=A0 =C2=A0 > > > > > > to target_size_ratio 0.900 on pools > ['cephfs_d= > ata']=C2=A0 =C2=A0 > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > Lars > > > > > > > > > ________________________________________= > _______ > > > > > > > > ceph-users mailing list -- <a > href=3D"ma= > ilto:ceph-users@ceph.io" target=3D"_blank">ceph-users(a)ceph.io</a> > > > > > > > > To unsubscribe send an email to <a href= > =3D"mailto:ceph-users-leave@ceph.io" > target=3D"_blank">ceph-users-leave@cep= > h.io</a>=C2=A0 =C2=A0 > > ------------------------------ > > Date: Mon, 28 Oct 2019 14:18:01 +0100 > From: "EDH - Manuel Rios Fernandez" <<a href=3D"mailto: > mriosfe= > r(a)easydatahost.com" target=3D"_blank">mriosfer(a)easydatahost.com > </a>> > Subject: [ceph-users] After delete 8.5M Objects in a bucket still 500K > =C2=A0 =C2=A0 =C2=A0 =C2=A0 left > To: <<a href=3D"mailto:ceph-users@ceph.io" > target=3D"_blank">ceph-users@= > ceph.io</a>> > Message-ID: <02a201d58d92$1fe85880$5fb90980$@<a href=3D" > http://easydatah= > ost.com" rel=3D"noreferrer" target=3D"_blank">easydatahost.com</a>> > Content-Type: multipart/alternative; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 > boundary=3D"----=3D_NextPart_000_02A3_01D5= > 8D9A.81B17B70" > > This is a multipart message in MIME format. > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/plain; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 charset=3D"us-ascii" > Content-Transfer-Encoding: 7bit > > Hi Ceph's! > > > > We started deteling a bucket several days ago. Total size 47TB / 8.5M > objects. > > > > Now we see the cli bucket rm stucked and by console drop this messages. > > > > [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700=C2=A0 0 > abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart > uploads= > > > 2019-10-28 13:56:24.021 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 2000 incomplete multipart uploads > > 2019-10-28 13:57:04.726 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 3000 incomplete multipart uploads > > 2019-10-28 13:57:45.424 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 4000 incomplete multipart uploads > > 2019-10-28 13:58:25.905 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 5000 incomplete multipart uploads > > 2019-10-28 13:59:06.898 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 6000 incomplete multipart uploads > > 2019-10-28 13:59:47.829 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 7000 incomplete multipart uploads > > 2019-10-28 14:00:42.102 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 8000 incomplete multipart uploads > > 2019-10-28 14:01:23.829 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 9000 incomplete multipart uploads > > 2019-10-28 14:02:06.028 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 10000 incomplete multipart uploads > > 2019-10-28 14:02:48.648 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 11000 incomplete multipart uploads > > 2019-10-28 14:03:29.807 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 12000 incomplete multipart uploads > > 2019-10-28 14:04:11.180 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 13000 incomplete multipart uploads > > 2019-10-28 14:04:52.396 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 14000 incomplete multipart uploads > > 2019-10-28 14:05:33.050 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 15000 incomplete multipart uploads > > 2019-10-28 14:06:13.652 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 16000 incomplete multipart uploads > > 2019-10-28 14:06:54.806 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 17000 incomplete multipart uploads > > 2019-10-28 14:07:35.867 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 18000 incomplete multipart uploads > > 2019-10-28 14:08:16.886 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 19000 incomplete multipart uploads > > 2019-10-28 14:08:57.711 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 20000 incomplete multipart uploads > > 2019-10-28 14:09:38.032 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 21000 incomplete multipart uploads > > 2019-10-28 14:10:18.377 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 22000 incomplete multipart uploads > > 2019-10-28 14:10:58.833 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 23000 incomplete multipart uploads > > 2019-10-28 14:11:39.078 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 24000 incomplete multipart uploads > > 2019-10-28 14:12:24.731 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 25000 incomplete multipart uploads > > 2019-10-28 14:13:12.176 7f0dd92c9700=C2=A0 0 abort_bucket_multiparts > WARNIN= > G : > aborted 26000 incomplete multipart uploads > > > > > > Bucket stats show 500K objects left. Looks like bucket rm is trying to > abor= > t > all incompleted mutipart. But in bucket stats this operation is not > reflected removing objects from stats. > > > > May be wait to get up 500K or it's a bug? > > > > Regards > > Manuel > > > > > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/html; > =C2=A0 =C2=A0 =C2=A0 =C2=A0 charset=3D"us-ascii" > Content-Transfer-Encoding: quoted-printable > > <html xmlns:v=3D3D"urn:schemas-microsoft-com:vml" =3D > xmlns:o=3D3D"urn:schemas-microsoft-com:office:office" =3D > xmlns:w=3D3D"urn:schemas-microsoft-com:office:word" =3D > xmlns:m=3D3D"<a href=3D" > http://schemas.microsoft.com/office/2004/12/om= > ml" rel=3D"noreferrer" target=3D"_blank"> > http://schemas.microsoft.com/offic= > e/2004/12/omml <http://schemas.microsoft.com/offic=e/2004/12/omml></a>" > =3D > xmlns=3D3D"<a href=3D"http://www.w3.org/TR/REC-html40" > rel=3D"noreferr= > er" target=3D"_blank">http://www.w3.org/TR/REC-html40 > </a>"><head= > ><meta =3D > http-equiv=3D3DContent-Type content=3D3D"text/html; =3D > charset=3D3Dus-ascii"><meta name=3D3DGenerator > content=3D3D&quot= > ;Microsoft Word 15 =3D > (filtered medium)"><style></style></head><body > l= > ang=3D3DES =3D > link=3D3D"#0563C1" vlink=3D3D"#954F72"><div > class= > =3D3DWordSection1> > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Hi =3D > Ceph&#8217;s!<o:p></o:p> =3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>We started > det= > eling a =3D > bucket several days ago. Total size 47TB / 8.5M =3D > objects.<o:p></o:p> class=3D3DMsoNo= > rmal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Now we see > the= > cli =3D > bucket rm stucked and by console drop this =3D > messages.<o:p></o:p> class=3D3DMsoN= > ormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>[root@ceph-rgw > = > 03 ~]# =3D > 2019-10-28 13:55:43.880 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts = > =3D > WARNING : aborted 1000 incomplete multipart =3D > uploads<o:p></o:p> class=3D3DMsoNor= > mal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 56:24.021 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 2000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 57:04.726 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 3000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 57:45.424 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 4000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 58:25.905 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 5000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 59:06.898 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 6000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 13:= > 59:47.829 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 7000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 00:42.102 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 8000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 01:23.829 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 9000 > =3D= > > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 02:06.028 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 10000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 02:48.648 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 11000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 03:29.807 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 12000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 04:11.180 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 13000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 04:52.396 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 14000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 05:33.050 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 15000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 06:13.652 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 16000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 06:54.806 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 17000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 07:35.867 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 18000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 08:16.886 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 19000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 08:57.711 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 20000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 09:38.032 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 21000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 10:18.377 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 22000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 10:58.833 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 23000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 11:39.078 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 24000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 12:24.731 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 25000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 > 14:= > 13:12.176 =3D > 7f0dd92c9700&nbsp; 0 abort_bucket_multiparts WARNING : aborted 26000 = > =3D > incomplete multipart > uploads<o:p></o:p>&= > lt;p =3D > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Bucket stats > s= > how 500K =3D > objects left. Looks like bucket rm is trying to abort all incompleted > =3D<b= > r> > mutipart. But in bucket stats this operation is not reflected removing > =3D<= > br> > objects from stats.<o:p></o:p> clas= > s=3D3DMsoNormal> > lang=3D3DEN-US =3D > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > style=3D3D'color:#1F497D;mso-fareast-language:ES'>May be wait > to= > get up =3D > 500K or it&#8217;s a > bug?<o:p></o:p>= > > class=3D3DMsoNormal> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Regards<o:p= > ></o:p>=3D > > > style=3D3D'color:#1F497D;mso-fareast-language:ES'>Manuel<o:p&= > gt;</o:p><=3D > /p> > > style=3D3D'color:#1F497D;mso-fareast-language:ES'><o:p>&am= > p;nbsp;</o:p><=3D > /p></div></body></html> > ------=3D_NextPart_000_02A3_01D58D9A.81B17B70-- > > ------------------------------ > > Date: Mon, 28 Oct 2019 10:48:44 -0400 > From: Casey Bodley <<a href=3D"mailto:cbodley@redhat.com" > target=3D"_bla= > nk">cbodley(a)redhat.com</a>> > Subject: [ceph-users] Re: Static website hosting with RGW > To: <a href=3D"mailto:ceph-users@ceph.io" > target=3D"_blank">ceph-users@ceph= > .io</a> > Message-ID: <<a href=3D"mailto: > 20834361-445e-1ee5-433b-dd4792f90608@redh= > at.com" target=3D"_blank">20834361-445e-1ee5-433b-dd4792f90608(a)redhat.com > </= > a>> > Content-Type: text/plain; charset=3DUTF-8; format=3Dflowed > > > On 10/24/19 8:38 PM, Oliver Freyermuth wrote: > > Dear Cephers, > > > > I have a question concerning static websites with RGW. > > To my understanding, it is best to run >=3D1 RGW client for > "c= > lassic" S3 and in addition operate >=3D1 RGW client for website > ser= > ving > > (potentially with HAProxy or its friends in front) to prevent messup > o= > f requests via the different protocols. > > > > I'd prefer to avoid "*.<a href=3D"http://example.com" > rel=3D"= > noreferrer" target=3D"_blank">example.com</a>" entries in DNS if > possi= > ble. > > So my current setup has these settings for the "web" RGW > cli= > ent: > >=C2=A0 =C2=A0rgw_enable_static_website =3D true > >=C2=A0 =C2=A0rgw_enable_apis =3D s3website > >=C2=A0 =C2=A0rgw_dns_s3website_name =3D > some_value_unused_when_A_record= > s_are_used_pointing_to_the_IP_but_it_needs_to_be_set > > and I create simple A records for each website pointing to the IP of > t= > his "web" RGW node. > > > > I can easily upload content for those websites to the other RGW > instan= > ces which are serving S3, > > so S3 and s3website APIs are cleanly separated in separate > instances.<= > br> > > > > However, one issue remains: How do I run > >=C2=A0 =C2=A0s3cmd ws-create > > on each website

1 0

Help
by Sumit Gaur 28 Oct '19

28 Oct '19

On Tue, 29 Oct 2019 at 1:50 am, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: subtrees have overcommitted (target_size_bytes / > target_size_ratio) > (Lars Täuber) > 2. After delete 8.5M Objects in a bucket still 500K left > (EDH - Manuel Rios Fernandez) > 3. Re: Static website hosting with RGW (Casey Bodley) > > > ---------------------------------------------------------------------- > > Date: Mon, 28 Oct 2019 11:24:54 +0100 > From: Lars Täuber <taeuber(a)bbaw.de> > Subject: [ceph-users] Re: subtrees have overcommitted > (target_size_bytes / target_size_ratio) > To: ceph-users <ceph-users(a)ceph.io> > Message-ID: <20191028112454.0362fe66(a)bbaw.de> > Content-Type: text/plain; charset=UTF-8 > > Is there a way to get rid of this warnings with activated autoscaler > besides adding new osds? > > Yet I couldn't get a satisfactory answer to the question why this all > happens. > > ceph osd pool autoscale-status : > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET > RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > cephfs_data 122.2T 1.5 165.4T 1.1085 > 0.8500 1.0 1024 on > > versus > > ceph df : > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 165 TiB 41 TiB 124 TiB 124 TiB 74.95 > > POOLS: > POOL ID STORED OBJECTS USED %USED > MAX AVAIL > cephfs_data 1 75 TiB 49.31M 122 TiB 87.16 > 12 TiB > > > It seems that the overcommitment is wrongly calculated. Isn't the RATE > already used to calculate the SIZE? > > It seems USED(df) = SIZE(autoscale-status) > Isn't the RATE already taken into account here? > > Could someone please explain the numbers to me? > > > Thanks! > Lars > > Fri, 25 Oct 2019 07:42:58 +0200 > Lars Täuber <taeuber(a)bbaw.de> ==> Nathan Fish <lordcirth(a)gmail.com> : > > Hi Nathan, > > > > Thu, 24 Oct 2019 10:59:55 -0400 > > Nathan Fish <lordcirth(a)gmail.com> ==> Lars Täuber <taeuber(a)bbaw.de> : > > > Ah, I see! The BIAS reflects the number of placement groups it should > > > create. Since cephfs metadata pools are usually very small, but have > > > many objects and high IO, the autoscaler gives them 4x the number of > > > placement groups that it would normally give for that amount of data. > > > > > ah ok, I understand. > > > > > So, your cephfs_data is set to a ratio of 0.9, and cephfs_metadata to > > > 0.3? Are the two pools using entirely different device classes, so > > > they are not sharing space? > > > > Yes, the metadata is on SSDs and the data on HDDs. > > > > > Anyway, I see that your overcommit is only "1.031x". So if you set > > > cephfs_data to 0.85, it should go away. > > > > This is not the case. I set the target_ratio to 0.7 and get this: > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > cephfs_metadata 15736M 3.0 2454G 0.0188 > 0.3000 4.0 256 on > > cephfs_data 122.2T 1.5 165.4T 1.1085 > 0.7000 1.0 1024 on > > > > The ratio seems to have nothing to do with the target_ratio but the SIZE > and the RAW_CAPACITY. > > Because the pool is still getting more data the SIZE increases and > therefore the RATIO increases. > > The RATIO seems to be calculated by this formula > > RATIO = SIZE * RATE / RAW_CAPACITY. > > > > This is what I don't understand. The data in the cephfs_data pool seems > to need more space than the raw capacity of the cluster provides. Hence the > situation is called "overcommitment". > > > > But why is this only the case when the autoscaler is active? > > > > Thanks > > Lars > > > > > > > > On Thu, Oct 24, 2019 at 10:09 AM Lars Täuber <taeuber(a)bbaw.de> > wrote: > > > > > > > > Thanks Nathan for your answer, > > > > > > > > but I set the the Target Ratio to 0.9. It is the cephfs_data pool > that makes the troubles. > > > > > > > > The 4.0 is the BIAS from the cephfs_metadata pool. This "BIAS" is > not explained on the page linked below. So I don't know its meaning. > > > > > > > > How can be a pool overcommited when it is the only pool on a set of > OSDs? > > > > > > > > Best regards, > > > > Lars > > > > > > > > Thu, 24 Oct 2019 09:39:51 -0400 > > > > Nathan Fish <lordcirth(a)gmail.com> ==> Lars Täuber <taeuber(a)bbaw.de> > : > > > > > The formatting is mangled on my phone, but if I am reading it > correctly, > > > > > you have set Target Ratio to 4.0. This means you have told the > balancer > > > > > that this pool will occupy 4x the space of your whole cluster, and > to > > > > > optimize accordingly. This is naturally a problem. Setting it to 0 > will > > > > > clear the setting and allow the autobalancer to work. > > > > > > > > > > On Thu., Oct. 24, 2019, 5:18 a.m. Lars Täuber, <taeuber(a)bbaw.de> > wrote: > > > > > > > > > > > This question is answered here: > > > > > > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ > > > > > > > > > > > > But it tells me that there is more data stored in the pool than > the raw > > > > > > capacity provides (taking the replication factor RATE into > account) hence > > > > > > the RATIO being above 1.0 . > > > > > > > > > > > > How comes this is the case? - Data is stored outside of the pool? > > > > > > How comes this is only the case when the autoscaler is active? > > > > > > > > > > > > Thanks > > > > > > Lars > > > > > > > > > > > > > > > > > > Thu, 24 Oct 2019 10:36:52 +0200 > > > > > > Lars Täuber <taeuber(a)bbaw.de> ==> ceph-users(a)ceph.io : > > > > > > > My question requires too complex an answer. > > > > > > > So let me ask a simple question: > > > > > > > > > > > > > > What does the SIZE of "osd pool autoscale-status" > tell/mean/comes from? > > > > > > > > > > > > > > Thanks > > > > > > > Lars > > > > > > > > > > > > > > Wed, 23 Oct 2019 14:28:10 +0200 > > > > > > > Lars Täuber <taeuber(a)bbaw.de> ==> ceph-users(a)ceph.io : > > > > > > > > Hello everybody! > > > > > > > > > > > > > > > > What does this mean? > > > > > > > > > > > > > > > > health: HEALTH_WARN > > > > > > > > 1 subtrees have overcommitted pool > target_size_bytes > > > > > > > > 1 subtrees have overcommitted pool > target_size_ratio > > > > > > > > > > > > > > > > and what does it have to do with the autoscaler? > > > > > > > > When I deactivate the autoscaler the warning goes away. > > > > > > > > > > > > > > > > > > > > > > > > $ ceph osd pool autoscale-status > > > > > > > > POOL SIZE TARGET SIZE RATE RAW CAPACITY > RATIO > > > > > > TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE > > > > > > > > cephfs_metadata 15106M 3.0 2454G > 0.0180 > > > > > > 0.3000 4.0 256 on > > > > > > > > cephfs_data 113.6T 1.5 165.4T > 1.0306 > > > > > > 0.9000 1.0 512 on > > > > > > > > > > > > > > > > > > > > > > > > $ ceph health detail > > > > > > > > HEALTH_WARN 1 subtrees have overcommitted pool > target_size_bytes; 1 > > > > > > subtrees have overcommitted pool target_size_ratio > > > > > > > > POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have > overcommitted > > > > > > pool target_size_bytes > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > 1.031x due > > > > > > to target_size_bytes 0 on pools [] > > > > > > > > POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have > overcommitted > > > > > > pool target_size_ratio > > > > > > > > Pools ['cephfs_data'] overcommit available storage by > 1.031x due > > > > > > to target_size_ratio 0.900 on pools ['cephfs_data'] > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > Lars > > > > > > > > _______________________________________________ > > > > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 28 Oct 2019 14:18:01 +0100 > From: "EDH - Manuel Rios Fernandez" <mriosfer(a)easydatahost.com> > Subject: [ceph-users] After delete 8.5M Objects in a bucket still 500K > left > To: <ceph-users(a)ceph.io> > Message-ID: <02a201d58d92$1fe85880$5fb90980$(a)easydatahost.com> > Content-Type: multipart/alternative; > boundary="----=_NextPart_000_02A3_01D58D9A.81B17B70" > > This is a multipart message in MIME format. > > ------=_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/plain; > charset="us-ascii" > Content-Transfer-Encoding: 7bit > > Hi Ceph's! > > > > We started deteling a bucket several days ago. Total size 47TB / 8.5M > objects. > > > > Now we see the cli bucket rm stucked and by console drop this messages. > > > > [root@ceph-rgw03 ~]# 2019-10-28 13:55:43.880 7f0dd92c9700 0 > abort_bucket_multiparts WARNING : aborted 1000 incomplete multipart uploads > > 2019-10-28 13:56:24.021 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 2000 incomplete multipart uploads > > 2019-10-28 13:57:04.726 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 3000 incomplete multipart uploads > > 2019-10-28 13:57:45.424 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 4000 incomplete multipart uploads > > 2019-10-28 13:58:25.905 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 5000 incomplete multipart uploads > > 2019-10-28 13:59:06.898 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 6000 incomplete multipart uploads > > 2019-10-28 13:59:47.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 7000 incomplete multipart uploads > > 2019-10-28 14:00:42.102 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 8000 incomplete multipart uploads > > 2019-10-28 14:01:23.829 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 9000 incomplete multipart uploads > > 2019-10-28 14:02:06.028 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 10000 incomplete multipart uploads > > 2019-10-28 14:02:48.648 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 11000 incomplete multipart uploads > > 2019-10-28 14:03:29.807 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 12000 incomplete multipart uploads > > 2019-10-28 14:04:11.180 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 13000 incomplete multipart uploads > > 2019-10-28 14:04:52.396 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 14000 incomplete multipart uploads > > 2019-10-28 14:05:33.050 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 15000 incomplete multipart uploads > > 2019-10-28 14:06:13.652 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 16000 incomplete multipart uploads > > 2019-10-28 14:06:54.806 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 17000 incomplete multipart uploads > > 2019-10-28 14:07:35.867 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 18000 incomplete multipart uploads > > 2019-10-28 14:08:16.886 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 19000 incomplete multipart uploads > > 2019-10-28 14:08:57.711 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 20000 incomplete multipart uploads > > 2019-10-28 14:09:38.032 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 21000 incomplete multipart uploads > > 2019-10-28 14:10:18.377 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 22000 incomplete multipart uploads > > 2019-10-28 14:10:58.833 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 23000 incomplete multipart uploads > > 2019-10-28 14:11:39.078 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 24000 incomplete multipart uploads > > 2019-10-28 14:12:24.731 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 25000 incomplete multipart uploads > > 2019-10-28 14:13:12.176 7f0dd92c9700 0 abort_bucket_multiparts WARNING : > aborted 26000 incomplete multipart uploads > > > > > > Bucket stats show 500K objects left. Looks like bucket rm is trying to > abort > all incompleted mutipart. But in bucket stats this operation is not > reflected removing objects from stats. > > > > May be wait to get up 500K or it's a bug? > > > > Regards > > Manuel > > > > > ------=_NextPart_000_02A3_01D58D9A.81B17B70 > Content-Type: text/html; > charset="us-ascii" > Content-Transfer-Encoding: quoted-printable > > <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = > xmlns:o=3D"urn:schemas-microsoft-com:office:office" = > xmlns:w=3D"urn:schemas-microsoft-com:office:word" = > xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = > xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = > http-equiv=3DContent-Type content=3D"text/html; = > charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 = > (filtered medium)"><style></style></head><body lang=3DES = > link=3D"#0563C1" vlink=3D"#954F72"><div class=3DWordSection1> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>Hi = > Ceph’s!<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>We started deteling a = > bucket several days ago. Total size 47TB / 8.5M = > objects.<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>Now we see the cli = > bucket rm stucked and by console drop this = > messages.<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>[root@ceph-rgw03 ~]# = > 2019-10-28 13:55:43.880 7f0dd92c9700  0 abort_bucket_multiparts = > WARNING : aborted 1000 incomplete multipart = > uploads<o:p></o:p> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:56:24.021 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 2000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:57:04.726 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 3000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:57:45.424 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 4000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:58:25.905 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 5000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:59:06.898 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 6000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 13:59:47.829 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 7000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:00:42.102 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 8000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:01:23.829 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 9000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:02:06.028 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 10000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:02:48.648 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 11000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:03:29.807 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 12000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:04:11.180 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 13000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:04:52.396 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 14000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:05:33.050 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 15000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:06:13.652 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 16000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:06:54.806 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 17000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:07:35.867 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 18000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:08:16.886 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 19000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:08:57.711 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 20000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:09:38.032 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 21000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:10:18.377 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 22000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:10:58.833 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 23000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:11:39.078 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 24000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:12:24.731 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 25000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'>2019-10-28 14:13:12.176 = > 7f0dd92c9700  0 abort_bucket_multiparts WARNING : aborted 26000 = > incomplete multipart uploads<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>Bucket stats show 500K = > objects left. Looks like bucket rm is trying to abort all incompleted = > mutipart. But in bucket stats this operation is not reflected removing = > objects from stats.<o:p></o:p> lang=3DEN-US = > style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>May be wait to get up = > 500K or it’s a bug?<o:p></o:p> class=3DMsoNormal> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'>Regards<o:p></o:p>= > style=3D'color:#1F497D;mso-fareast-language:ES'>Manuel<o:p></o:p><= > /p> style=3D'color:#1F497D;mso-fareast-language:ES'><o:p> </o:p><= > /p></div></body></html> > ------=_NextPart_000_02A3_01D58D9A.81B17B70-- > > ------------------------------ > > Date: Mon, 28 Oct 2019 10:48:44 -0400 > From: Casey Bodley <cbodley(a)redhat.com> > Subject: [ceph-users] Re: Static website hosting with RGW > To: ceph-users(a)ceph.io > Message-ID: <20834361-445e-1ee5-433b-dd4792f90608(a)redhat.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > > On 10/24/19 8:38 PM, Oliver Freyermuth wrote: > > Dear Cephers, > > > > I have a question concerning static websites with RGW. > > To my understanding, it is best to run >=1 RGW client for "classic" S3 > and in addition operate >=1 RGW client for website serving > > (potentially with HAProxy or its friends in front) to prevent messup of > requests via the different protocols. > > > > I'd prefer to avoid "*.example.com" entries in DNS if possible. > > So my current setup has these settings for the "web" RGW client: > > rgw_enable_static_website = true > > rgw_enable_apis = s3website > > rgw_dns_s3website_name = > some_value_unused_when_A_records_are_used_pointing_to_the_IP_but_it_needs_to_be_set > > and I create simple A records for each website pointing to the IP of > this "web" RGW node. > > > > I can easily upload content for those websites to the other RGW > instances which are serving S3, > > so S3 and s3website APIs are cleanly separated in separate instances. > > > > However, one issue remains: How do I run > > s3cmd ws-create > > on each website-bucket once? > > I can't do that against the "classic" S3-serving RGW nodes. This will > give me a 405 (not allowed), > > since they do not have rgw_enable_static_website enabled. > > I also can not run it against the "web S3" nodes, since they do not have > the S3 API enabled. > > Of course I could enable that, but then the RGW node can't cleanly > disentangle S3 and website requests since I use A records. > > > > Does somebody have a good idea on how to solve this issue? > > Setting "rgw_enable_static_website = true" on the S3-serving RGW nodes > would solve it, but does that have any bad side-effects on their S3 > operation? > > Enabling static website on the gateway serving the S3 api does look like > the right solution. As far as I can tell, it's only used to control > whether the S3 ops for PutBucketWebsite, GetBucketWebsite, and > DeleteBucketWebsite are exposed. > > > > > Also, if there's an expert on this: Exposing a bucket under a tenant as > static website is not possible since the colon (:) can't be encoded in DNS, > right? > > > > > > In case somebody also wants to set something like this up, here are the > best docs I could find: > > https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c > > and of course: > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html-s… > > > > > > Cheers, > > Oliver > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 79 > ****************************************** >

1 0

Ceph monitor start error: monitor data filesystem reached concerning levels of available storage space
by Thomas Schneider 28 Oct '19

28 Oct '19

Hi, I'm facing an issue with starting 1 monitor. In the relevant error log the issue seems to be clear: root@ld5506:~# tail -n 30 /var/log/ceph/ceph-mon.ld5506.log 2019-10-28 17:14:46.471 7f157b097440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:14:46.471 7f157b097440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 421378 2019-10-28 17:14:46.471 7f157b097440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:14:56.563 7fb006b4c440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:14:56.563 7fb006b4c440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 421600 2019-10-28 17:14:56.563 7fb006b4c440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:15:06.823 7fb2c7084440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:15:06.823 7fb2c7084440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 421824 2019-10-28 17:15:06.823 7fb2c7084440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:15:17.067 7f722b0c4440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:15:17.067 7f722b0c4440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 422009 2019-10-28 17:15:17.067 7f722b0c4440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) 2019-10-28 17:15:27.323 7f34e37df440 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-28 17:15:27.323 7f34e37df440 0 ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable), process ceph-mon, pid 422197 2019-10-28 17:15:27.323 7f34e37df440 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 5% 11 GiB) you may adjust 'mon data avail crit' to a lower value to make this go away (default: 5%) However there is now sufficient disk space available on this node after completing some file cleanup: root@ld5506:~# btrfs filesystem usage / Overall: Device size: 192.54GiB Device allocated: 183.54GiB Device unallocated: 9.00GiB Device missing: 0.00B Used: 110.76GiB Free (estimated): 75.01GiB (min: 70.51GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 237.86MiB (used: 0.00B) Data,single: Size:175.53GiB, Used:109.51GiB /dev/sdbq3 175.53GiB Metadata,DUP: Size:4.00GiB, Used:639.77MiB /dev/sdbq3 8.00GiB System,DUP: Size:8.00MiB, Used:48.00KiB /dev/sdbq3 16.00MiB Unallocated: /dev/sdbq3 9.00GiB I would expect that +70GB disk space must be sufficient for MON to start. Please advise how to fix this error. THX

1 0

Strage RBD images created
by Randall Smith 28 Oct '19

28 Oct '19

Greetings, I am running a mimic cluster. I noticed that I suddenly have over 200 rbd images that have seemingly random 64 character names. They were all created within a short time period. rbd info on one of the rbds looks like this: rbd image 'ff8f0ad0b5323fc3a8e930d96d3021d4c4dfc18c62ef9a1305b49456b8560904': size 16 GiB in 4096 objects order 22 (4 MiB objects) id: cb09e16b8b4567 block_name_prefix: rbd_data.cb09e16b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Fri Mar 1 11:19:14 2019 `rbd lock ls` shows that these images are not being used. I can certainly delete them but I found it odd that they showed up at all. Has anyone seen anything like this? -- Randall Smith Computing Services Adams State University http://www.adams.edu/ 719-587-7741

2 2

radosgw recovering shards
by Frank R 28 Oct '19

28 Oct '19

Hi all, Apologies for all the messages to the list over the past few days. After an upgrade from 12.2.7 to 12.2.12 (inherited cluster) for an RGW multisite active/active setup I am almost constantly seeing 1-10 "recovering shards" when running "radosgw-admin sync status", ie: ---------- # radosgw-admin sync status realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod) zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod) zone 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2) metadata sync no sync (zone is master) data sync source: ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shards behind shards: [102] 8 shards are recovering recovering shards: [31,37,56,60,83,92,95,127] ---------- Every once in a while it will go to full sync. This is seen on both the master and secondary side. There were a bunch of stale reshard instances (on both ends) that I was able to remove with: ----- radosgw-admin reshard stale-instances rm ----- What exactly is a "recovering shard". What can be performed to troubleshoot/fix this condition? I have verified that rgw_num_rados_handles is 1. Additionally, what exactly do: # radosgw-admin metadata sync init # radosgw-admin data sync init do and do they tend to take a long time to perform. thx Frank

1 0

RMDA Bug?
by Mason-Williams, Gabryel (DLSLtd,RAL,LSCI) 28 Oct '19

28 Oct '19

I am currently trying to run Ceph on RDMA, either RoCE 1 or 2. However, I am experiencing issues with this. When using Ceph on RDMA I experience issues where OSD’s will randomly become unreachable even if the cluster is left alone alone, it also is not properly talking over RDMA and using Ethernet when the config states it should as shown by the same results in the bench marking of the two setups. After reloading the cluster [cid:36020940-0085-40fc-bb5b-d91de6ace453] After 5m 9s the cluster went from being healthy to down. [cid:ed084bcc-0b97-44bd-9648-ce2e06859cd5] This problem even happens when running a bench mark test on the cluster, OSD’s will just fall over. Another curious issue is that it is not properly talking over RDMA as well and instead using the Ethernet. [cid:05e9dc68-075e-425d-b76b-ce7fa1d2f7a8] Next test: [cid:4183557e-b1da-41f3-afc3-f081b9fb4034] The config used for the RDMA is a so: [global] fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa mon_initial_members = node1, node2, node3 mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx auth_cluster_required = cephx auth_service_required =cephx auth_client_required = cephx public_network = xxx.xxx.xxx.xxx/24 cluster_network = yyy.yyy.yyy.yyy/16 ms_cluster_type =async+rdma ms_public_type = async+posix ms_async_rdma_device_name = mlx4_0 [osd.0] ms_async_rdma_local_gid = xxxx [osd.1] ms_async_rdma_local_gid = xxxx [osd.2] ms_async_rdma_local_gid =xxxx Tests to check the system is using RDMA sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep ms_cluster OUTPUT "ms_cluster_type": "async+rdma", sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1 OUTPUT { "AsyncMessenger::RDMAWorker-1": { "tx_no_mem": 0, "tx_parital_mem": 0, "tx_failed_post": 0, "rx_no_registered_mem": 0, "tx_chunks": 9, "tx_bytes": 2529, "rx_chunks": 0, "rx_bytes": 0, "pending_sent_conns": 0 } } When running over Ethernet I have a completely stable system with the current benchmarks as so [cid:544ecbbc-10d9-43e6-ab2f-aa7c2bcd88c0] Config setup when using Ethernet is The Config setup when using Ethernet is [global] fsid = aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa mon_initial_members = node1, node2, node3 mon_host =xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx auth_cluster_required = cephx auth_service_required =cephx auth_client_required = cephx public_network = xxx.xxx.xxx.xxx/24 cluster_network = yyy.yyy.yyy.yyy/16 ms_cluster_type =async+posix ms_public_type = async+posix ms_async_rdma_device_name = mlx4_0 [osd.0] ms_async_rdma_local_gid = xxxx [osd.1] ms_async_rdma_local_gid = xxxx [osd.2] ms_async_rdma_local_gid =xxxx Tests to check the system is using async+posix sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | grep ms_cluster OUTPUT "ms_cluster_type": "async+posix" sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMAWorker-1 OUTPUT {} This clearly a issue with RDMA and not with the OSD's shown by the fact the system is completely fine over Ethernet and not with RDMA. Any guidance or ideas on how to approach this problem to make Ceph work with RDMA would be greatly appreciated. Regards Gabryel Mason-Williams, Placement Student Address: Diamond Light Source Ltd., Diamond House, Harwell Science & Innovation Campus, Didcot, Oxfordshire OX11 0DE Email: gabryel.mason-williams(a)diamond.ac.uk -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Issue with cephfs
by LeeQ ＠ BitBahn.io 26 Oct '19

26 Oct '19

We have been using a cephfs pool to store machine data to, the data is not overly critical at this time but. Its got to around 8TB and we started to see kernel panics with the hosts that had the mounts in place. Now when try to start the MDS's they cycle through, Active, Replay, ClientReplay about 10 times and then just fail in a active(laggy)state. So I delete the MDS's (docker-croit)@us-croit-enc-deploy01 ~ $ ceph fs dump dumped fsmap epoch 5307 e5307 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 5307 flags 12 created 2019-10-26 20:43:02.087584 modified 2019-10-26 21:35:17.285598 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client -1 (unspecified) last_failure 0 last_failure_osd_epoch 2122066 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=267576193} failed damaged stopped data_pools [5,14] metadata_pool 3 inline_data disabled balancer standby_count_wanted 1 267576193: v1:100.129.255.186:6800/1355970155 'us-ceph-enc-svc02' mds.0.5301 up:active seq 16 laggy since 2019-10-26 21:12:08.027863 Looks ok. Then run (docker-croit)@us-croit-enc-deploy01 ~ $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data us_enc_datarefuge_001a ] (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-journal-tool --rank cephfs:all journal export lee.bak journal is 523855986688~99768 wrote 99768 bytes at offset 523855986688 to lee.bak NOTE: this is a _sparse_ file; you can $ tar cSzf lee.bak.tgz lee.bak to efficiently compress it while preserving sparseness. (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-journal-tool --rank cephfs:all event recover_dentries summary Events by type: RESETJOURNAL: 1 SESSION: 363 SESSIONS: 17 UPDATE: 14 Errors: 0 (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-journal-tool --rank cephfs:all journal reset old journal was 523855986688~99768 new journal start will be 523860180992 (4094536 bytes past old end) writing journal head writing EResetJournal entry done (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-table-tool all reset session { "0": { "data": {}, "result": 0 } } (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-table-tool all reset snap { "result": 0 } (docker-croit)@us-croit-enc-deploy01 ~ $ cephfs-table-tool all reset inode { "0": { "data": {}, "result": 0 } } Re add the MDS's and we go back round in a circle. Am I missing something? do I need to drop the metadata and recreate it maybe? If it comes to it I can drop all the data and start over, but don't really want to.

1 0

ceph pg commands hang forever
by Frank R 26 Oct '19

26 Oct '19

Hi all, I have an issue where all "ceph pg" command hang forever, ie: ceph pg dump ceph pg ls-by-pool etc All pgs report as active+clean. Any pointers on correcting this? thx Frank

1 1

0B OSDs
by DHilsbos＠performair.com 25 Oct '19

25 Oct '19

All; We're setting up our second cluster, using version 14.2.4, and we've run into a weird issue: all of our OSDs are created with a size of 0 B. Weights are appropriate for the size of the underlying drives, but ceph -s shows this: cluster: id: <id> health: HEALTH_WARN Reduced data availability: 256 pgs inactive too few PGs per OSD (28 < min 30) services: mon: 3 daemons, quorum s700041,s700042,s700043 (age 4d) mgr: s700041(active, since 3d), standbys: s700042, s700043 osd: 9 osds: 9 up (since 21m), 9 in (since 44m) data: pools: 1 pools, 256 pgs objects: 0 objects, 0 B -->usage: 0 B used, 0 B / 0 B avail<-- (emphasis added) pgs: 100.000% pgs unknown 256 unknown Thoughts? I have ceph-volumne.log, and the log from one of the OSD daemons, though it looks like the auth keys get printed to the ceph-volume.log. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com 300 S. Hamilton Pl. Gilbert, AZ 85233 Phone: (480) 610-3500 Fax: (480) 610-3501 www.PerformAir.com

2 1

0B OSDs?
by DHilsbos＠performair.com 25 Oct '19

25 Oct '19

All; We're setting up our second cluster, using version 14.2.4, and we've run into a weird issue: all of our OSDs are created with a size of 0 B. Weights are appropriate for the size of the underlying drives, but ceph -s shows this: cluster: id: <id> health: HEALTH_WARN Reduced data availability: 256 pgs inactive too few PGs per OSD (28 < min 30) services: mon: 3 daemons, quorum s700041,s700042,s700043 (age 4d) mgr: s700041(active, since 3d), standbys: s700042, s700043 osd: 9 osds: 9 up (since 21m), 9 in (since 44m) data: pools: 1 pools, 256 pgs objects: 0 objects, 0 B -->usage: 0 B used, 0 B / 0 B avail<-- (emphasis added) pgs: 100.000% pgs unknown 256 unknown Thoughts? I have ceph-volumne.log, and the log from one of the OSD daemons, though it looks like the auth keys get printed to the ceph-volume.log. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International inc. DHilsbos(a)PerformAir.com www.PerformAir.com

1 0

minimum osd size?
by Frank R 25 Oct '19

25 Oct '19

Hi all, I have 40 nvme drives with about 20G free space each. Would creating a 10GB partition/lvm on each of the nvmes for an rgw index pool be a bad idea? RGW has about about 5 million objects I don't think space will be an issue but I am worried about the 10G size, is it just too small for a bluestore OSD? thx Frank

3 2

Re: Authentication failure at radosgw for presigned urls
by Biswajeet Patra 25 Oct '19

25 Oct '19

++ceph-users On Fri, Sep 20, 2019 at 4:48 PM Biswajeet Patra < biswajeet.patra(a)flipkart.com> wrote: > Hi, > We recently faced an issue with radosgw authentication of presigned urls. > The presigned url generated by client to download object fails at radosgw > with http error 403 i.e SignatureDoesNotMatch. > > The radosgw computes the signature (*v2 signature in this case*) using > the s3 specification > https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html. > In this process, a string_to_sign is created by concatenating selected > elements from requests which is then authenticated using hmac to create the > final signature. If this signature matches with the client signature, the > authentication is successful or else it fails with SignatureDoesNotMatch > error. As part of the StringToSign, the CanonicalizedAmzHeaders should only > include headers that start with "x-amz-" and ignore other headers. But in > the radosgw code, there are other meta prefixes that are checked against > the http request headers and if matched are included in the > CanonicalizedAmzHeaders to compute the final signature. For e.g, if a > request header contains "HTTP_X_ACCOUNT" its selected by radosgw to include > in amz_headers but the same will be ignored by AWS SDK as it does not start > with "x-amz-". This will result in different signature computed by client > and radosgw. > > Code Snippet: rgw_common.cc > struct str_len meta_prefixes[] = { STR_LEN_ENTRY("HTTP_X_AMZ"), > STR_LEN_ENTRY("HTTP_X_GOOG"), > STR_LEN_ENTRY("HTTP_X_DHO"), > STR_LEN_ENTRY("HTTP_X_RGW"), > STR_LEN_ENTRY("HTTP_X_OBJECT"), > STR_LEN_ENTRY("HTTP_X_CONTAINER"), > STR_LEN_ENTRY("HTTP_X_ACCOUNT"), > {NULL, 0} }; > > The method init_meta_info() which matches the above prefixes is called > from RGWREST::preprocess() which is invoked for all s3 requests. It will be > helpful to know as to why these prefixes that are not specified by AWS S3 > comes in the path of s3 authentication. Was it added for swift use-cases > only? If so, then why its included in rgw_common.cc? > As a proposed fix, we can remove the highlighted meta prefixes that are > not specified by aws from the s3 authentication path signature calculation. > Let me know if you have any queries or solutions. > > Regards, > Biswajeet > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 1

Unbalanced data distribution
by Thomas 25 Oct '19

25 Oct '19

Hi, in my 7 OSD node cluster I have the following disks: Node 1 48x 1.6TB Node 2 48x 1.6TB Node 3 48x 1.6TB Node 4 48x 1.6TB Node 5 48x 7.2TB Node 6 48x 7.2TB Node 7 48x 7.2TB The disk sizes are represented in CRUSH map accordingly. For these disks only I created a pool "hdb_backup" with size 3. Based on output of rados df this pool is using 247TB currently. root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR cephfs_data 345 GiB 99092 0 297276 0 0 0 1638 811 MiB 109235 365 GiB 0 B 0 B cephfs_metadata 102 MiB 48 0 144 0 0 0 8 8 KiB 8588 106 MiB 0 B 0 B hdb_backup 247 TiB 64671398 0 194014194 0 0 0 12902005 4.3 TiB 323647757 601 TiB 0 B 0 B hdd 2.4 TiB 635457 0 1270914 0 0 0 13237278 321 GiB 21526953 3.0 TiB 0 B 0 B nvme 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B ssd 251 GiB 64307 0 128614 0 0 0 615475 29 GiB 885085 55 GiB 0 B 0 B total_objects 65470302 total_used 747 TiB total_avail 784 TiB total_space 1.5 PiB In order to rebalance the data in this pool I have configured balance mode upmap: root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Unfortunately the data distribution is not balanced at all on 1.6TB disks, means the range is between 53.37% and 83.04%. root@ld3955:~# ceph osd df | awk '{ print "osd."$1, "size: "$5, "usage: " $17, "reweight: "$4 }' | sort -nk5 osd.ID size: SIZE usage: reweight: REWEIGHT osd.MIN/MAX size: 26.45 usage: reweight: STDDEV: osd.TOTAL size: TiB usage: reweight: 747 osd.265 size: 1.6 usage: 53.37 reweight: 1.00000 osd.248 size: 1.6 usage: 53.41 reweight: 1.00000 osd.111 size: 1.6 usage: 53.43 reweight: 1.00000 osd.161 size: 1.6 usage: 53.46 reweight: 1.00000 osd.85 size: 1.6 usage: 53.46 reweight: 1.00000 osd.241 size: 1.6 usage: 53.49 reweight: 1.00000 osd.238 size: 1.6 usage: 53.51 reweight: 1.00000 osd.259 size: 1.6 usage: 53.56 reweight: 1.00000 osd.88 size: 1.6 usage: 53.57 reweight: 1.00000 osd.204 size: 1.6 usage: 53.58 reweight: 1.00000 osd.159 size: 1.6 usage: 55.16 reweight: 1.00000 osd.81 size: 1.6 usage: 55.16 reweight: 1.00000 osd.116 size: 1.6 usage: 55.20 reweight: 1.00000 osd.195 size: 1.6 usage: 55.25 reweight: 1.00000 osd.169 size: 1.6 usage: 55.33 reweight: 1.00000 osd.158 size: 1.6 usage: 55.34 reweight: 1.00000 [...] osd.146 size: 1.6 usage: 79.31 reweight: 1.00000 osd.140 size: 1.6 usage: 79.34 reweight: 0.89999 osd.262 size: 1.6 usage: 79.38 reweight: 0.89999 osd.217 size: 1.6 usage: 79.48 reweight: 1.00000 osd.83 size: 1.6 usage: 79.50 reweight: 1.00000 osd.239 size: 1.6 usage: 79.52 reweight: 0.79999 osd.190 size: 1.6 usage: 80.87 reweight: 1.00000 osd.97 size: 1.6 usage: 80.95 reweight: 1.00000 osd.216 size: 1.6 usage: 80.97 reweight: 1.00000 osd.160 size: 1.6 usage: 81.03 reweight: 1.00000 osd.145 size: 1.6 usage: 81.19 reweight: 1.00000 osd.137 size: 1.6 usage: 81.20 reweight: 0.89999 osd.136 size: 1.6 usage: 81.21 reweight: 0.89999 osd.54 size: 1.6 usage: 82.88 reweight: 1.00000 osd.252 size: 1.6 usage: 83.04 reweight: 0.89999 Question: Why is the data distribution on the 1.6TB disks unequal? How can I correct this? THX

4 10

rgw recovering shards
by Frank R 25 Oct '19

25 Oct '19

Hi all, After an RGW upgrade from 12.2.7 to 12.2.12 for RGW multisite a few days ago the "sync status" has constantly shown a few "recovering shards", ie: ----- # radosgw-admin sync status realm 8f7fd3fd-f72d-411d-b06b-7b4b579f5f2f (prod) zonegroup 60a2cb75-6978-46a3-b830-061c8be9dc75 (prod) zone ffce148e-3b24-462d-98bf-8c212de31de5 (us-east-1) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 7fe96e52-d6f7-4ad6-b66e-ecbbbffbc18e (us-east-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 1 shards behind shards: [48] oldest incremental change not applied: 2019-10-21 22:34:11.0.293798s 5 shards are recovering recovering shards: [11,37,48,110,117] ----- This is the secondary zone. I am worried about the "oldest incremental change not applied" being from the 21st. Is there a way to have RGW just stop trying to recover these shards and just sync them from this point in time? thx Frank

2 1

Change device class in EC profile
by Frank Schilder 24 Oct '19

24 Oct '19

I recently moved an EC pool from HDD to SSD by changing the device class in the crush rule. I would like to complete this operation by cleaning up a dirty trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself contains the device class as well: crush-device-class=hdd [...] I can already see the confusion this will cause in the future. Is this situation one of the few instances where using the --force option is warranted to change the device class of a profile, as in: osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force If not, how can I change the device class of the profile? Many thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

Re: Don't know how to use bucket notification
by Yuval Lifshitz 24 Oct '19

24 Oct '19

the endpoint is not the RGW endpoint, it is the server to which you want to send the bucket notifications to. E.g. if you have a rabbitmq server running at address: 1.2.3.4, you should use: push-endpoint=amqp://1.2.3.4 note that in such a case the: amqp-exchange parameter must be set as well. assuming you have an http server, on the same address, listening on port 8080, and you want your notifications to get to it, you should use: push-endpoint=http://1.2.3.4:8080 Yuval > Subject: [ceph-users] Don't know how to use bucket notification > Date: Thu, 24 Oct 2019 16:26:54 +0800 > From: 柯名澤 <mingze.ke(a)gmail.com> > To: ceph-users(a)ceph.io > > > > Hi, all. > Does anyone know where the endpoint of CREATE TOPIC is? (for bucket > notification) > https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic > Is that the same with the normal S3 API? I tried but failed. > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

[ceph-user] Ceph mimic support FIPS
by Amit Ghadge 24 Oct '19

24 Oct '19

Hi all, We have FIPS enable cluster where it is running on ceph-12.2.12, after upgrading to mimic 13.2.6 can't serve any requests. and not able to get/put objects, buckets. Is there Mimic support FIPS? Thanks, Amit G

1 0

Don't know how to use bucket notification
by 柯名澤 24 Oct '19

24 Oct '19

Hi, all. Does anyone know where the endpoint of CREATE TOPIC is? (for bucket notification) https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic Is that the same with the normal S3 API? I tried but failed. Thanks.

1 0

PG badly corrupted after merging PGs on mixed FileStore/BlueStore setup
by Paul Emmerich 23 Oct '19

23 Oct '19

Hi, I'm working on a curious case that looks like a bug in PG merging maybe related to FileStore. Setup is 14.2.1 that is half BlueStore half FileStore (being migrated), and the number of PGs on an RGW index pool were reduced, now one of the PGs (3 FileStore OSDs) seems to be corrupted. There are some (29) objects that are affected (~20% of the PG), the issue looks like this for one of the affected objects which I'll call dir.A here # object seems to exist according to rados rados -p default.rgw.buckets.index ls | grep .dir.A .dir.A # or doesn't it? rados -p default.rgw.buckets.index get .dir.A - error getting default.rgw.buckets.index/.dir.A: (2) No such file or directory Running deep-scrub reports that everything is okay with the affected PG This is what the OSD logs when trying to access it, nothing really relevant with debug 20: 10 osd.57 pg_epoch: 1149030 pg[18.2( v 1148996'1422066 (1144429'1418988,1148996'1422066] local-lis/les=1149021/1149022 n=135 ec=49611/596 lis/c 1149021/1149021 les/c/f 1149022/1149022/0 1149015/1149021/1149021) [57,0,31] r=0 lpr=1149021 crt=1148996'1422066 lcod 1148996'1422065 mlcod 0'0 active+clean] get_object_context: no obc for soid 18:764060e4:::.dir.A:head and !can_create So going one level deeper with ceph-objectstore-tool: # --op list (29 messages like this) error getting default.rgw.buckets.index/.dir.A: (2) No such file or directory followed by a complete autoput of the json for the objects including the broken ones # .dir.A dump dump Error stat on : 18.2_head,#18:73996afb:::.dir.A:head#, (2) No such file or directory Error getting snapset on : 18.2_head,#18:73996afb:::.dir.A:head#, (2) No such file or directory { "id": { "oid": ".dir.A", "key": "", "snapid": -2, "hash": 3746994638, "max": 0, "pool": 18, "namespace": "", "max": 0 } } # --op export stops after encountering a bad object with 'export_files error -2' This is the same for all 3 OSDs in that PG. Has anyone encountered something similar? I'll probably just nuke the affected bucket indices tomorrow and re-create them. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

2 2

Radosgw sync incomplete bucket indexes
by Benjamin.Zieglmeier 23 Oct '19

23 Oct '19

Hello, We are having an issue with bucket index consistency between two zones in a multisite environment. The master zone (originally a single zone implementation) is running 12.2.5, and the secondary zone is running 12.2.11. We implemented a multisite configuration to migrate to new hardware (among other tasks which justified the use of multisite vs. simply introducing the new nodes). We have one bucket in particular that has ~40k obj. missing (of 330k total) in the bucket index on the secondary cluster after sync completed. The objects are present in the data pool, and can be accessed directly, but cannot be listed. We’ve attempted `bucket check –bucket=bucket1 –fix` and also `--check-objects`. None of the missing objects are listed in output of –fix. We’ve tried using the ceph documentation to reshard the bucket in master after purging the bucket in secondary. Same result. All obj sync, but none of the ~40k are listed in the index. We’ve searched the user list history and see occurrences of this issue with other users but those threads seem to end before a resolution is met. We are simply looking to get our bucket indexes in sync so we can cutover master to the newer cluster and destroy the old cluster. Any help would be greatly appreciated. Thanks, Ben

1 0

Re: Fwd: large concurrent rbd operations block for over 15 mins!
by Frank Schilder 23 Oct '19

23 Oct '19

Getting decent RBD performance is not a trivial exercise. While at a first glance 61 SSDs for 245 clients sounds more or less OK, it does come down to a bit more than that. The first thing is, how to get SSD performance out of SSDs with ceph. This post will provide very good clues and might already point out the bottleneck: https://yourcmc.ru/wiki/index.php?title=Ceph_performance . Do you have good enterprise SSDs? Next thing to look at, what kind of data pool, replicated or erasure coded? If erasure coded, has the profile been benchmarked? There are very poor choices. Good ones are 4+m, 8+m. 4+m better IOps, 8+m better throughput. m>=2. More complications: do you need to deploy more than one OSD per SSD to boost performance? This is indicated by the iodepth required in an fio benchmark to get full IOPs. Good SSDs deliver already spec performance with 1 OSD. More common ones require 2-4 OSDs per disk. Are you using ceph-volume already, its default is 2 OSDs per SSD (batch mode). To give a base line, after extensive testing and working through all the required tuning steps, I could run about 250 VMs on a 6+2 EC data pool on 33 enterprise SAS SSDs with 1 OSD per disk, each VM getting 50IOPs write performance. This is probably what you would like to see as well. If you use replicated data pool, this should be relatively easy. With EC data pool, this is a bit of a battle. Good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of Void Star Nill <void.star.nill(a)gmail.com> Sent: 22 October 2019 03:00 To: ceph-users Subject: [ceph-users] Fwd: large concurrent rbd operations block for over 15 mins! Apparently the graph is too big, so my last post is stuck. Resending without the graph. Thanks ---------- Forwarded message --------- From: Void Star Nill <void.star.nill(a)gmail.com<mailto:void.star.nill@gmail.com>> Date: Mon, Oct 21, 2019 at 4:41 PM Subject: large concurrent rbd operations block for over 15 mins! To: ceph-users <ceph-users(a)lists.ceph.com<mailto:ceph-users@lists.ceph.com>> Hello, I have been running some benchmark tests with a mid-size cluster and I am seeing some issues. Wanted to know if this is a bug or something that can be tuned. Appreciate any help on this. - I have a 15 node Ceph cluster, with 3 monitors and 12 data nodes with total 61 OSDs on SSDs running 14.2.4 nautilus (stable) version. Each node has 100G link. - I have 245 client machines from which I am triggering rbd operations. Each client has 25G link - rbd operations include, creating an RBD image of 50G size and layering feature, mapping the image to the client machine, formatting the device in ext4 format, mounting it, running dd to write to the full disk and cleaning up (unmount, unmap and remove). If I run these RBD operations concurrently on a small number of machines (say 16-20), they run very well and I see good throughput. All image operations (except for dd) take less than 2 seconds. However, when I scale it up to 245 clients, each running these operations concurrently, I see lot of operations getting hung for a long time and the overall throughput reduces drastically. For example, some of the format operations take over 10-15 mins!!! Note that, all operations do complete - so its most likely not a deadlock kind of situation. I dont see any errors in ceph.log on the monitor nodes. However, the clients do report "hung_task_timeout" in dmesg logs. As you can see in the below image, half the format operations are completing in less than a second time, while the other half is over 10mins (y axis is in seconds) [11117.113618] INFO: task umount:9902 blocked for more than 120 seconds. [11117.113677] Tainted: G OE 4.15.0-51-generic #55~16.04.1-Ubuntu [11117.113731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [11117.113787] umount D 0 9902 9901 0x00000000 [11117.113793] Call Trace: [11117.113804] __schedule+0x3d6/0x8b0 [11117.113810] ? _raw_spin_unlock_bh+0x1e/0x20 [11117.113814] schedule+0x36/0x80 [11117.113821] wb_wait_for_completion+0x64/0x90 [11117.113828] ? wait_woken+0x80/0x80 [11117.113831] __writeback_inodes_sb_nr+0x8e/0xb0 [11117.113835] writeback_inodes_sb+0x27/0x30 [11117.113840] __sync_filesystem+0x51/0x60 [11117.113844] sync_filesystem+0x26/0x40 [11117.113850] generic_shutdown_super+0x27/0x120 [11117.113854] kill_block_super+0x2c/0x80 [11117.113858] deactivate_locked_super+0x48/0x80 [11117.113862] deactivate_super+0x5a/0x60 [11117.113866] cleanup_mnt+0x3f/0x80 [11117.113868] __cleanup_mnt+0x12/0x20 [11117.113874] task_work_run+0x8a/0xb0 [11117.113881] exit_to_usermode_loop+0xc4/0xd0 [11117.113885] do_syscall_64+0x100/0x130 [11117.113887] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [11117.113891] RIP: 0033:0x7f0094384487 [11117.113893] RSP: 002b:00007fff4199efc8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [11117.113897] RAX: 0000000000000000 RBX: 0000000000944030 RCX: 00007f0094384487 [11117.113899] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000944210 [11117.113900] RBP: 0000000000944210 R08: 0000000000000000 R09: 0000000000000014 [11117.113902] R10: 00000000000006b2 R11: 0000000000000246 R12: 00007f009488d83c [11117.113903] R13: 0000000000000000 R14: 0000000000000000 R15: 00007fff4199f250

2 3

Since nautilus upgrade(?) getting ceph: build_snap_context fail -12
by Marc Roos 23 Oct '19

23 Oct '19

Getting these since the upgrade to nautilus [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d5c ffff911d8b648900 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d18 ffff9115f344ac00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d15 ffff9121cbf0ea00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d14 ffff9119bfa95600 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d0d ffff911b643ba200 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d0c ffff9115f7a8b300 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d0b ffff9121c3290d00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf9 ffff911b06b6be00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf8 ffff9117c6c32100 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf7 ffff911993788a00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf6 ffff9118fabfc600 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d61 ffff9121c8978000 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d60 ffff911ce2c08700 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d5f ffff9118d4edfb00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d5e ffff9119382a1c00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d5d ffff91182d107f00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d5c ffff911d8b648900 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d18 ffff9115f344ac00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d15 ffff9121cbf0ea00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d14 ffff9119bfa95600 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d0d ffff911b643ba200 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d0c ffff9115f7a8b300 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085d0b ffff9121c3290d00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf9 ffff911b06b6be00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf8 ffff9117c6c32100 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf7 ffff911993788a00 fail -12 [Wed Oct 23 01:59:12 2019] ceph: build_snap_context 10002085cf6 ffff9118fabfc600 fail -12

1 0

mix ceph-disk and ceph-volume
by Frank R 23 Oct '19

23 Oct '19

Is it ok to create a new OSD using ceph-volume on a server where the other OSDs were created with ceph-disk? thx Frank

2 1

Help unsubscribe please
by Sumit Gaur 22 Oct '19

22 Oct '19

On Wed, 23 Oct 2019 at 3:12 am, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: Replace ceph osd in a container (Sasha Litvak) > 2. Re: Fwd: large concurrent rbd operations block for over 15 mins! > (Mark Nelson) > 3. Re: rgw multisite failover (Ed Fisher) > > > ---------------------------------------------------------------------- > > Date: Tue, 22 Oct 2019 08:52:54 -0500 > From: Sasha Litvak <alexander.v.litvak(a)gmail.com> > Subject: [ceph-users] Re: Replace ceph osd in a container > To: Frank Schilder <frans(a)dtu.dk> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > < > CALi_L49RxWcBx_ZivRHHWgYg8Ea_UrH-0YKGMY4+b20KhXu6UQ(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="000000000000a0731a0595801ea4" > > --000000000000a0731a0595801ea4 > Content-Type: text/plain; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > Frank, > > Thank you for your suggestion. It sounds very promising. I will > definitely try it. > > Best, > > On Tue, Oct 22, 2019, 2:44 AM Frank Schilder <frans(a)dtu.dk> wrote: > > > > I am suspecting that mon or mgr have no access to /dev or /var/lib > whil= > e > > osd containers do. > > > Cluster configured originally by ceph-ansible (nautilus 14.2.2) > > > > They don't, because they don't need to. > > > > > The question is if I want to replace all disks on a single node, and I > > have 6 nodes with pools > > > replication 3, is it safe to restart mgr mounting /dev and > /var/lib/cep= > h > > volumes (not configured right now). > > > > Restarting mons is safe in the sense that data will not get lost. > However= > , > > access might get lost temporarily. > > > > The question is, how many mons do you have? If you have only 1 or 2, it > > will mean downtime. If you can bear the downtime, it doesn't matter. If > y= > ou > > have at least 3, you can restart one after the other. > > > > However, I would not do that. Having to restart a mon container every > tim= > e > > some minor container config changes for reasons that have nothing to do > > with a mon sounds like calling for trouble. > > > > I also use containers and would recommend a different approach. I created > > an additional type of container (ceph-adm) that I use for all admin > tasks= > . > > Its the same image and the entry point simply executes a sleep infinity. > = > In > > this container I make all relevant hardware visible. You might also want > = > to > > expose /var/run/ceph to be able to use admin sockets without hassle. This > > way, I separated admin operations from actual storage daemons and can > > modify and restart the admin container as I like. > > > > Best regards, > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Frank Schilder > > AIT Ris=C3=B8 Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of Alex > > Litvak <alexander.v.litvak(a)gmail.com> > > Sent: 22 October 2019 08:04 > > To: ceph-users(a)lists.ceph.com > > Subject: [ceph-users] Replace ceph osd in a container > > > > Hello cephers, > > > > So I am having trouble with a new hardware systems with strange OSD > > behavior and I want to replace a disk with a brand new one to test the > > theory. > > > > I run all daemons in containers and on one of the nodes I have mon, mgr, > > and 6 osds. So following > > > https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/#replac= > ing-an-osd > <https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/#replac=i…> > > > > I stopped container with osd.23, waited until it is down and out, ran > > safe-to-destroy loop and then destroyed the osd all using the monitor > fro= > m > > the container on this node. All good. > > > > Then I swapped the SSDs and started running additional steps (from step > 3= > ) > > using the same mon container. I have no ceph packages installed on the > > bare metal box. It looks like mon container doesn't > > see the disk. > > > > podman exec -it ceph-mon-storage2n2-la ceph-volume lvm zap /dev/sdh > > stderr: lsblk: /dev/sdh: not a block device > > stderr: error: /dev/sdh: No such file or directory > > stderr: Unknown device, --name=3D, --path=3D, or absolute path in > /dev/= > or > > /sys expected. > > usage: ceph-volume lvm zap [-h] [--destroy] [--osd-id OSD_ID] > > [--osd-fsid OSD_FSID] > > [DEVICES [DEVICES ...]] > > ceph-volume lvm zap: error: Unable to proceed with non-existing device: > > /dev/sdh > > Error: exit status 2 > > root@storage2n2-la:~# ls -l /dev/sd > > sda sdc sdd sde sdf sdg sdg1 sdg2 sdg5 sdh > > root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la ceph-volume > > lvm zap sdh > > stderr: lsblk: sdh: not a block device > > stderr: error: sdh: No such file or directory > > stderr: Unknown device, --name=3D, --path=3D, or absolute path in > /dev/= > or > > /sys expected. > > usage: ceph-volume lvm zap [-h] [--destroy] [--osd-id OSD_ID] > > [--osd-fsid OSD_FSID] > > [DEVICES [DEVICES ...]] > > ceph-volume lvm zap: error: Unable to proceed with non-existing device: > s= > dh > > Error: exit status 2 > > > > I execute lsblk and it sees device sdh > > root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la lsblk > > lsblk: dm-1: failed to get device path > > lsblk: dm-2: failed to get device path > > lsblk: dm-4: failed to get device path > > lsblk: dm-6: failed to get device path > > lsblk: dm-4: failed to get device path > > lsblk: dm-2: failed to get device path > > lsblk: dm-1: failed to get device path > > lsblk: dm-0: failed to get device path > > lsblk: dm-0: failed to get device path > > lsblk: dm-7: failed to get device path > > lsblk: dm-5: failed to get device path > > lsblk: dm-7: failed to get device path > > lsblk: dm-6: failed to get device path > > lsblk: dm-5: failed to get device path > > lsblk: dm-3: failed to get device path > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > sdf 8:80 0 1.8T 0 disk > > sdd 8:48 0 1.8T 0 disk > > sdg 8:96 0 223.5G 0 disk > > |-sdg5 8:101 0 223G 0 part > > |-sdg1 8:97 487M 0 part > > `-sdg2 8:98 1K 0 part > > sde 8:64 0 1.8T 0 disk > > sdc 8:32 0 3.5T 0 disk > > sda 8:0 0 3.5T 0 disk > > sdh 8:112 0 3.5T 0 disk > > > > So I use a fellow osd container (osd.5) on the same node and run all of > > the operations (zap and prepare) successfully. > > > > I am suspecting that mon or mgr have no access to /dev or /var/lib while > > osd containers do. Cluster configured originally by ceph-ansible > (nautil= > us > > 14.2.2) > > > > The question is if I want to replace all disks on a single node, and I > > have 6 nodes with pools replication 3, is it safe to restart mgr mounting > > /dev and /var/lib/ceph volumes (not configured right now). > > > > I cannot use other osd containers on the same box because my controller > > reverts from raid to non-raid mode with all disks lost and not just a > > single one. So I need to replace all 6 osds to run back > > in containers and the only things will remain operational on node are mon > > and mgr containers. > > > > I prefer not to install a full cluster or client on the bare metal node > i= > f > > possible. > > > > Thank you for your help, > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users(a)lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > --000000000000a0731a0595801ea4 > Content-Type: text/html; charset="UTF-8" > Content-Transfer-Encoding: quoted-printable > > <div dir=3D"auto">Frank,<div dir=3D"auto"> </div><div > dir=3D"auto">Thank= > you=C2=A0for your suggestion.=C2=A0 It sounds very promising.=C2=A0 I > will= > definitely try it.</div><div dir=3D"auto"> </div><div > dir=3D"auto">Best= > ,</div></div> <div class=3D"gmail_quote"><div dir=3D"ltr" > class=3D"gmail= > _attr">On Tue, Oct 22, 2019, 2:44 AM Frank Schilder <<a href=3D"mailto: > f= > rans(a)dtu.dk">frans(a)dtu.dk</a>> wrote: </div><blockquote > class=3D"gmai= > l_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc > solid;padding-left= > :1ex">> I am suspecting that mon or mgr have no access to /dev or > /var/l= > ib while osd containers do. > > Cluster configured originally by ceph-ansible (nautilus 14.2.2) > > They don't, because they don't need to. > > > The question is if I want to replace all disks on a single node, and > I= > have 6 nodes with pools > > replication 3, is it safe to restart mgr mounting /dev and > /var/lib/ce= > ph volumes (not configured right now). > > Restarting mons is safe in the sense that data will not get lost. However, > = > access might get lost temporarily. > > The question is, how many mons do you have? If you have only 1 or 2, it > wil= > l mean downtime. If you can bear the downtime, it doesn't matter. If > yo= > u have at least 3, you can restart one after the other. > > However, I would not do that. Having to restart a mon container every time > = > some minor container config changes for reasons that have nothing to do > wit= > h a mon sounds like calling for trouble. > > I also use containers and would recommend a different approach. I created > a= > n additional type of container (ceph-adm) that I use for all admin tasks. > I= > ts the same image and the entry point simply executes a sleep infinity. In > = > this container I make all relevant hardware visible. You might also want > to= > expose /var/run/ceph to be able to use admin sockets without hassle. This > = > way, I separated admin operations from actual storage daemons and can > modif= > y and restart the admin container as I like. > > Best regards, > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Frank Schilder > AIT Ris=C3=B8 Campus > Bygning 109, rum S14 > > ________________________________________ > From: ceph-users <<a href=3D"mailto:ceph-users-bounces@lists.ceph.com" > t= > arget=3D"_blank" rel=3D"noreferrer">ceph-users-bounces(a)lists.ceph.com > </a>&g= > t; on behalf of Alex Litvak <<a href=3D"mailto: > alexander.v.litvak(a)gmail.= > com" target=3D"_blank" rel=3D"noreferrer">alexander.v.litvak(a)gmail.com > </a>&= > gt; > Sent: 22 October 2019 08:04 > To: <a href=3D"mailto:ceph-users@lists.ceph.com" target=3D"_blank" > rel=3D"n= > oreferrer">ceph-users(a)lists.ceph.com</a> > Subject: [ceph-users] Replace ceph osd in a container > > Hello cephers, > > So I am having trouble with a new hardware systems with strange OSD > behavio= > r and I want to replace a disk with a brand new one to test the theory. > > I run all daemons in containers and on one of the nodes I have mon, mgr, > an= > d 6 osds.=C2=A0 So following <a href=3D" > https://docs.ceph.com/docs/master/r= > ados/operations/add-or-rm-osds/#replacing-an-osd > <https://docs.ceph.com/docs/master/r=ados/operations/add-or-rm-osds/#replaci…>" > rel=3D"noreferrer norefer= > rer" target=3D"_blank"> > https://docs.ceph.com/docs/master/rados/operations/a= > dd-or-rm-osds/#replacing-an-osd > <https://docs.ceph.com/docs/master/rados/operations/a=dd-or-rm-osds/#replaci…> > </a> > > I stopped container with osd.23, waited until it is down and out, ran > safe-= > to-destroy loop and then destroyed the osd all using the monitor from the > c= > ontainer on this node.=C2=A0 All good. > > Then I swapped the SSDs and started running additional steps (from step 3) > = > using the same mon container.=C2=A0 I have no ceph packages installed on > th= > e bare metal box. It looks like mon container doesn't > see the disk. > > =C2=A0 =C2=A0 =C2=A0podman exec -it ceph-mon-storage2n2-la ceph-volume lvm > = > zap /dev/sdh > =C2=A0 stderr: lsblk: /dev/sdh: not a block device > =C2=A0 stderr: error: /dev/sdh: No such file or directory > =C2=A0 stderr: Unknown device, --name=3D, --path=3D, or absolute path in > /d= > ev/ or /sys expected. > usage: ceph-volume lvm zap [-h] [--destroy] [--osd-id OSD_ID] > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 [--osd-fsid OSD_FSID] > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 [DEVICES [DEVICES ...]] > ceph-volume lvm zap: error: Unable to proceed with non-existing device: > /de= > v/sdh > Error: exit status 2 > root@storage2n2-la:~# ls -l /dev/sd > sda=C2=A0 =C2=A0sdc=C2=A0 =C2=A0sdd=C2=A0 =C2=A0sde=C2=A0 =C2=A0sdf=C2=A0 = > =C2=A0sdg=C2=A0 =C2=A0sdg1=C2=A0 sdg2=C2=A0 sdg5=C2=A0 sdh > root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la ceph-volume > lv= > m zap sdh > =C2=A0 stderr: lsblk: sdh: not a block device > =C2=A0 stderr: error: sdh: No such file or directory > =C2=A0 stderr: Unknown device, --name=3D, --path=3D, or absolute path in > /d= > ev/ or /sys expected. > usage: ceph-volume lvm zap [-h] [--destroy] [--osd-id OSD_ID] > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 [--osd-fsid OSD_FSID] > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 =C2=A0 [DEVICES [DEVICES ...]] > ceph-volume lvm zap: error: Unable to proceed with non-existing device: > sdh= > > Error: exit status 2 > > I execute lsblk and it sees device sdh > root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la lsblk > lsblk: dm-1: failed to get device path > lsblk: dm-2: failed to get device path > lsblk: dm-4: failed to get device path > lsblk: dm-6: failed to get device path > lsblk: dm-4: failed to get device path > lsblk: dm-2: failed to get device path > lsblk: dm-1: failed to get device path > lsblk: dm-0: failed to get device path > lsblk: dm-0: failed to get device path > lsblk: dm-7: failed to get device path > lsblk: dm-5: failed to get device path > lsblk: dm-7: failed to get device path > lsblk: dm-6: failed to get device path > lsblk: dm-5: failed to get device path > lsblk: dm-3: failed to get device path > NAME=C2=A0 =C2=A0MAJ:MIN RM=C2=A0 =C2=A0SIZE RO TYPE MOUNTPOINT > sdf=C2=A0 =C2=A0 =C2=A0 8:80=C2=A0 =C2=A00=C2=A0 =C2=A01.8T=C2=A0 0 > disk<br= > > > sdd=C2=A0 =C2=A0 =C2=A0 8:48=C2=A0 =C2=A00=C2=A0 =C2=A01.8T=C2=A0 0 > disk<br= > > > sdg=C2=A0 =C2=A0 =C2=A0 8:96=C2=A0 =C2=A00 223.5G=C2=A0 0 disk > |-sdg5=C2=A0 =C2=A08:101=C2=A0 0=C2=A0 =C2=A0223G=C2=A0 0 part > |-sdg1=C2=A0 =C2=A08:97=C2=A0 =C2=A0 =C2=A0 =C2=A0487M=C2=A0 0 part > `-sdg2=C2=A0 =C2=A08:98=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01K=C2=A0 0 > part<br= > > > sde=C2=A0 =C2=A0 =C2=A0 8:64=C2=A0 =C2=A00=C2=A0 =C2=A01.8T=C2=A0 0 > disk<br= > > > sdc=C2=A0 =C2=A0 =C2=A0 8:32=C2=A0 =C2=A00=C2=A0 =C2=A03.5T=C2=A0 0 > disk<br= > > > sda=C2=A0 =C2=A0 =C2=A0 8:0=C2=A0 =C2=A0 0=C2=A0 =C2=A03.5T=C2=A0 0 > disk<br= > > > sdh=C2=A0 =C2=A0 =C2=A0 8:112=C2=A0 0=C2=A0 =C2=A03.5T=C2=A0 0 disk > > So I use a fellow osd container (osd.5) on the same node and run all of > the= > operations (zap and prepare) successfully. > > I am suspecting that mon or mgr have no access to /dev or /var/lib while > os= > d containers do.=C2=A0 Cluster configured originally by ceph-ansible > (nauti= > lus 14.2.2) > > The question is if I want to replace all disks on a single node, and I > have= > 6 nodes with pools replication 3, is it safe to restart mgr mounting /dev > = > and /var/lib/ceph volumes (not configured right now). > > I cannot use other osd containers on the same box because my controller > rev= > erts from raid to non-raid mode with all disks lost and not just a single > o= > ne.=C2=A0 So I need to replace all 6 osds to run back > in containers and the only things will remain operational on node are mon > a= > nd mgr containers. > > I prefer not to install a full cluster or client on the bare metal node if > = > possible. > > Thank you for your help, > > _______________________________________________ > ceph-users mailing list > <a href=3D"mailto:ceph-users@lists.ceph.com" target=3D"_blank" > rel=3D"noref= > errer">ceph-users(a)lists.ceph.com</a> > <a href=3D"http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com" > rel=3D"n= > oreferrer noreferrer" target=3D"_blank"> > http://lists.ceph.com/listinfo.cgi/= > ceph-users-ceph.com > <http://lists.ceph.com/listinfo.cgi/=ceph-users-ceph.com></a> > </blockquote></div> > > --000000000000a0731a0595801ea4-- > > ------------------------------ > > Date: Tue, 22 Oct 2019 08:59:21 -0500 > From: Mark Nelson <mnelson(a)redhat.com> > Subject: [ceph-users] Re: Fwd: large concurrent rbd operations block > for over 15 mins! > To: ceph-users(a)ceph.io > Message-ID: <362e3930-8c30-d3e0-d0b0-30187c8551e4(a)redhat.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > Out of curiosity, when you chose EC over replication how did you weigh > IOPS vs space amplification in your decision making process? I'm > wondering if we should prioritize EC latency vs other tasks in future > tuning efforts (it's always a tradeoff deciding what to focus on). > > > Thanks, > > Mark > > On 10/22/19 2:35 AM, Frank Schilder wrote: > > Getting decent RBD performance is not a trivial exercise. While at a > first glance 61 SSDs for 245 clients sounds more or less OK, it does come > down to a bit more than that. > > > > The first thing is, how to get SSD performance out of SSDs with ceph. > This post will provide very good clues and might already point out the > bottleneck: https://yourcmc.ru/wiki/index.php?title=Ceph_performance . Do > you have good enterprise SSDs? > > > > Next thing to look at, what kind of data pool, replicated or erasure > coded? If erasure coded, has the profile been benchmarked? There are very > poor choices. Good ones are 4+m, 8+m. 4+m better IOps, 8+m better > throughput. m>=2. > > > > More complications: do you need to deploy more than one OSD per SSD to > boost performance? This is indicated by the iodepth required in an fio > benchmark to get full IOPs. Good SSDs deliver already spec performance with > 1 OSD. More common ones require 2-4 OSDs per disk. Are you using > ceph-volume already, its default is 2 OSDs per SSD (batch mode). > > > > To give a base line, after extensive testing and working through all the > required tuning steps, I could run about 250 VMs on a 6+2 EC data pool on > 33 enterprise SAS SSDs with 1 OSD per disk, each VM getting 50IOPs write > performance. This is probably what you would like to see as well. > > > > If you use replicated data pool, this should be relatively easy. With EC > data pool, this is a bit of a battle. > > > > Good luck, > > > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of Void > Star Nill <void.star.nill(a)gmail.com> > > Sent: 22 October 2019 03:00 > > To: ceph-users > > Subject: [ceph-users] Fwd: large concurrent rbd operations block for > over 15 mins! > > > > Apparently the graph is too big, so my last post is stuck. Resending > without the graph. > > > > Thanks > > > > > > ---------- Forwarded message --------- > > From: Void Star Nill <void.star.nill(a)gmail.com<mailto: > void.star.nill(a)gmail.com>> > > Date: Mon, Oct 21, 2019 at 4:41 PM > > Subject: large concurrent rbd operations block for over 15 mins! > > To: ceph-users <ceph-users(a)lists.ceph.com<mailto: > ceph-users(a)lists.ceph.com>> > > > > > > Hello, > > > > I have been running some benchmark tests with a mid-size cluster and I > am seeing some issues. Wanted to know if this is a bug or something that > can be tuned. Appreciate any help on this. > > > > - I have a 15 node Ceph cluster, with 3 monitors and 12 data nodes with > total 61 OSDs on SSDs running 14.2.4 nautilus (stable) version. Each node > has 100G link. > > - I have 245 client machines from which I am triggering rbd operations. > Each client has 25G link > > - rbd operations include, creating an RBD image of 50G size and layering > feature, mapping the image to the client machine, formatting the device in > ext4 format, mounting it, running dd to write to the full disk and cleaning > up (unmount, unmap and remove). > > > > If I run these RBD operations concurrently on a small number of machines > (say 16-20), they run very well and I see good throughput. All image > operations (except for dd) take less than 2 seconds. > > > > However, when I scale it up to 245 clients, each running these > operations concurrently, I see lot of operations getting hung for a long > time and the overall throughput reduces drastically. > > > > For example, some of the format operations take over 10-15 mins!!! > > > > Note that, all operations do complete - so its most likely not a > deadlock kind of situation. > > > > I dont see any errors in ceph.log on the monitor nodes. However, the > clients do report "hung_task_timeout" in dmesg logs. > > > > As you can see in the below image, half the format operations are > completing in less than a second time, while the other half is over 10mins > (y axis is in seconds) > > > > > > > > [11117.113618] INFO: task umount:9902 blocked for more than 120 seconds. > > [11117.113677] Tainted: G OE 4.15.0-51-generic > #55~16.04.1-Ubuntu > > [11117.113731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables this message. > > [11117.113787] umount D 0 9902 9901 0x00000000 > > [11117.113793] Call Trace: > > [11117.113804] __schedule+0x3d6/0x8b0 > > [11117.113810] ? _raw_spin_unlock_bh+0x1e/0x20 > > [11117.113814] schedule+0x36/0x80 > > [11117.113821] wb_wait_for_completion+0x64/0x90 > > [11117.113828] ? wait_woken+0x80/0x80 > > [11117.113831] __writeback_inodes_sb_nr+0x8e/0xb0 > > [11117.113835] writeback_inodes_sb+0x27/0x30 > > [11117.113840] __sync_filesystem+0x51/0x60 > > [11117.113844] sync_filesystem+0x26/0x40 > > [11117.113850] generic_shutdown_super+0x27/0x120 > > [11117.113854] kill_block_super+0x2c/0x80 > > [11117.113858] deactivate_locked_super+0x48/0x80 > > [11117.113862] deactivate_super+0x5a/0x60 > > [11117.113866] cleanup_mnt+0x3f/0x80 > > [11117.113868] __cleanup_mnt+0x12/0x20 > > [11117.113874] task_work_run+0x8a/0xb0 > > [11117.113881] exit_to_usermode_loop+0xc4/0xd0 > > [11117.113885] do_syscall_64+0x100/0x130 > > [11117.113887] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 > > [11117.113891] RIP: 0033:0x7f0094384487 > > [11117.113893] RSP: 002b:00007fff4199efc8 EFLAGS: 00000246 ORIG_RAX: > 00000000000000a6 > > [11117.113897] RAX: 0000000000000000 RBX: 0000000000944030 RCX: > 00007f0094384487 > > [11117.113899] RDX: 0000000000000001 RSI: 0000000000000000 RDI: > 0000000000944210 > > [11117.113900] RBP: 0000000000944210 R08: 0000000000000000 R09: > 0000000000000014 > > [11117.113902] R10: 00000000000006b2 R11: 0000000000000246 R12: > 00007f009488d83c > > [11117.113903] R13: 0000000000000000 R14: 0000000000000000 R15: > 00007fff4199f250 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 22 Oct 2019 11:10:38 -0500 > From: Ed Fisher <ed(a)debacle.org> > Subject: [ceph-users] Re: rgw multisite failover > To: Frank R <frankaritchie(a)gmail.com> > Cc: ceph-users <ceph-users(a)ceph.com> > Message-ID: <084F4293-88CC-456A-B8A4-2E36ACA24B65(a)debacle.org> > Content-Type: multipart/alternative; > boundary="Apple-Mail=_AE5E92AF-C94B-43D4-8A65-947B2DE6F04A" > > > --Apple-Mail=_AE5E92AF-C94B-43D4-8A65-947B2DE6F04A > Content-Transfer-Encoding: quoted-printable > Content-Type: text/plain; > charset=us-ascii > > > > > On Oct 18, 2019, at 10:40 PM, Frank R <frankaritchie(a)gmail.com> wrote: > >=20 > > I am looking to change an RGW multisite deployment so that the = > secondary will become master. This is meant to be a permanent change. > >=20 > > Per: > > https://docs.ceph.com/docs/mimic/radosgw/multisite/ = > <https://docs.ceph.com/docs/mimic/radosgw/multisite/> > >=20 > > I need to: > >=20 > > 1. Stop RGW daemons on the current master end. > >=20 > > On a secondary RGW node: > > 2. radosgw-admin zone modify --rgw-zone=3D{zone-name} --master = > --default > > 3. radosgw-admin period update --commit > > 4. systemctl restart ceph-radosgw(a)rgw.`hostname -s` > >=20 > > Since I want the former master to be secondary permanently do I need = > to do anything after restarting the RGW daemons on the old master end? > > > Before you restart the RGW daemons on the old master you want to make = > sure you pull the current realm from the new master. Beyond that there = > should be no changes needed.=20= > > --Apple-Mail=_AE5E92AF-C94B-43D4-8A65-947B2DE6F04A > Content-Transfer-Encoding: quoted-printable > Content-Type: text/html; > charset=us-ascii > > <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = > charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = > -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""> class=3D""><div> <blockquote type=3D"cite" class=3D""><div = > class=3D"">On Oct 18, 2019, at 10:40 PM, Frank R <<a = > href=3D"mailto:frankaritchie@gmail.com" = > class=3D"">frankaritchie(a)gmail.com</a>> wrote:</div> class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = > class=3D"">I am looking to change an RGW multisite deployment so that = > the secondary will become master. This is meant to be a permanent = > change.<div class=3D""> </div><div class=3D"">Per:</div><div= > class=3D""><a = > href=3D"https://docs.ceph.com/docs/mimic/radosgw/multisite/" = > class=3D"">https://docs.ceph.com/docs/mimic/radosgw/multisite/</a> class=3D""></div><div class=3D""> </div><div class=3D"">I = > need to:</div><div class=3D""> </div><div class=3D"">1. = > Stop RGW daemons on the current master end.</div><div class=3D""> class=3D""></div><div class=3D"">On a secondary RGW node:</div><div = > class=3D"">2. radosgw-admin zone modify --rgw-zone=3D{zone-name} = > --master --default</div><div class=3D"">3. radosgw-admin period = > update --commit</div><div class=3D"">4. systemctl restart = > ceph-radosgw(a)rgw.`hostname -s`</div><div class=3D""> class=3D""></div><div class=3D"">Since I want the former master to be = > secondary permanently do I need to do anything after restarting the RGW = > daemons on the old master end?</div></div></div></blockquote><div> class=3D""></div><div> </div>Before you restart the RGW = > daemons on the old master you want to make sure you pull the current = > realm from the new master. Beyond that there should be no changes = > needed. </div></body></html>= > > --Apple-Mail=_AE5E92AF-C94B-43D4-8A65-947B2DE6F04A-- > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 56 > ****************************************** >

1 0

Nautilus power outage - 2/3 mons and mgrs dead and no cephfs
by Alex L 22 Oct '19

22 Oct '19

Hi list, Had a power outage killing the whole cluster. Cephfs will not start at all, but RBD works just fine. I did have 4 unfound objects that I eventually had to rollback or delete which I don't really understand as I should've had a copy of the those pbjects on the other drives? 2/3 mons and mgrs are damaged but without any errors. I have loads stored on cephfs so would very much like to get that running as a first priority. Thanks! Alex Info about the home cluster: I run 23 osds on 3 hosts. 6 of these are a SSD cache layer for the spinning rust, as well as the metadata portion for cephfs which in retrospect might have to be put back on the spinning rust. # ceph -v ceph version 14.2.4 (65249672c6e6d843510e7e01f8a4b976dcac3db1) nautilus (stable) # head ceph-mgr.pve21.log.7 2019-10-04 00:00:00.397 7fee56df3700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 2019-10-04 00:00:00.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855f6b7500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 2019-10-04 00:00:00.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory 2019-10-04 00:00:01.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2680: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 4 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) 2019-10-04 00:00:01.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e486380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 2019-10-04 00:00:01.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory 2019-10-04 00:00:02.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e4b5500 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 2019-10-04 00:00:02.573 7fee545ee700 1 mgr finish mon failed to return metadata for mds.pve21: (2) No such file or directory 2019-10-04 00:00:03.553 7fee43aef700 0 log_channel(cluster) log [DBG] : pgmap v2681: 1088 pgs: 1 active+clean+inconsistent, 4 active+recovery_unfound+undersized+degraded+remapped, 1083 active+clean; 4.2 TiB data, 13 TiB used, 15 TiB / 28 TiB avail; 4.7 KiB/s rd, 33 KiB/s wr, 2 op/s; 12/3843345 objects degraded (0.000%); 4/1281115 objects unfound (0.000%) 2019-10-04 00:00:03.573 7fee44af1700 0 ms_deliver_dispatch: unhandled message 0x55855e3b0380 mgrreport(mds.pve21 +110-0 packed 1366) v7 from mds.0 v2:192.168.1.21:6800/3783320901 # head ceph-mon.pve21.log.7 2019-10-04 00:00:00.389 7f7c25b52700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 2019-10-04 00:00:00.397 7f7c25b52700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw (PID: 193052) UID: 0 2019-10-04 00:00:00.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 2019-10-04 00:00:00.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch 2019-10-04 00:00:01.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 2019-10-04 00:00:01.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch 2019-10-04 00:00:02.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 2019-10-04 00:00:02.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch 2019-10-04 00:00:03.573 7f7c1f345700 0 mon.pve21@0(leader) e20 handle_command mon_command({"prefix": "mds metadata", "who": "pve21"} v 0) v1 2019-10-04 00:00:03.573 7f7c1f345700 0 log_channel(audit) log [DBG] : from='mgr.137464844 192.168.1.21:0/2201' entity='mgr.pve21' cmd=[{"prefix": "mds metadata", "who": "pve21"}]: dispatch # head ceph-mds.pve21.log.7 2019-10-04 00:00:00.389 7f1b2f1b5700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw (PID: 193051) UID: 0 2019-10-04 00:00:00.397 7f1b2f1b5700 -1 received signal: Hangup from (PID: 193052) UID: 0 2019-10-04 00:00:04.881 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839bb200 0x556f838d4000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0) 2019-10-04 00:00:06.157 7f1b321bb700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2770609702 conn(0x556f839e0000 0x556f83807800 :6801 s=READ_FOOTER_AND_DISPATCH pgs=6 cs=4 l=0).handle_message_footer missed message? skipped from seq 0 to 2 2019-10-04 00:01:19.167 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby 2019-10-04 00:01:23.555 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.22:0/2875552603 conn(0x556f839bda80 0x556f837f9000 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby 2019-10-04 00:02:08.768 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.23:0/2427365808 conn(0x556f839bd180 0x556f83e9d800 :6801 s=OPENED pgs=2 cs=1 l=0).fault server, going to standby 2019-10-04 00:02:20.140 7f1b311b9700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839c2900 0x556f837f8000 :6801 s=OPENED pgs=5 cs=3 l=0).fault server, going to standby 2019-10-04 00:02:21.420 7f1b319ba700 0 --1- [v2:192.168.1.21:6800/3783320901,v1:192.168.1.21:6801/3783320901] >> v1:192.168.1.21:0/3200878088 conn(0x556f839e0480 0x556f83d7f000 :6801 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept replacing existing (lossy) channel (new one lossy=0)

2 4

rgw multisite failover
by Frank R 22 Oct '19

22 Oct '19

I am looking to change an RGW multisite deployment so that the secondary will become master. This is meant to be a permanent change. Per: https://docs.ceph.com/docs/mimic/radosgw/multisite/ I need to: 1. Stop RGW daemons on the current master end. On a secondary RGW node: 2. radosgw-admin zone modify --rgw-zone={zone-name} --master --default 3. radosgw-admin period update --commit 4. systemctl restart ceph-radosgw(a)rgw.`hostname -s` Since I want the former master to be secondary permanently do I need to do anything after restarting the RGW daemons on the old master end?

2 1

Re: Replace ceph osd in a container
by Frank Schilder 22 Oct '19

22 Oct '19

> I am suspecting that mon or mgr have no access to /dev or /var/lib while osd containers do. > Cluster configured originally by ceph-ansible (nautilus 14.2.2) They don't, because they don't need to. > The question is if I want to replace all disks on a single node, and I have 6 nodes with pools > replication 3, is it safe to restart mgr mounting /dev and /var/lib/ceph volumes (not configured right now). Restarting mons is safe in the sense that data will not get lost. However, access might get lost temporarily. The question is, how many mons do you have? If you have only 1 or 2, it will mean downtime. If you can bear the downtime, it doesn't matter. If you have at least 3, you can restart one after the other. However, I would not do that. Having to restart a mon container every time some minor container config changes for reasons that have nothing to do with a mon sounds like calling for trouble. I also use containers and would recommend a different approach. I created an additional type of container (ceph-adm) that I use for all admin tasks. Its the same image and the entry point simply executes a sleep infinity. In this container I make all relevant hardware visible. You might also want to expose /var/run/ceph to be able to use admin sockets without hassle. This way, I separated admin operations from actual storage daemons and can modify and restart the admin container as I like. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: ceph-users <ceph-users-bounces(a)lists.ceph.com> on behalf of Alex Litvak <alexander.v.litvak(a)gmail.com> Sent: 22 October 2019 08:04 To: ceph-users(a)lists.ceph.com Subject: [ceph-users] Replace ceph osd in a container Hello cephers, So I am having trouble with a new hardware systems with strange OSD behavior and I want to replace a disk with a brand new one to test the theory. I run all daemons in containers and on one of the nodes I have mon, mgr, and 6 osds. So following https://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/#replacin… I stopped container with osd.23, waited until it is down and out, ran safe-to-destroy loop and then destroyed the osd all using the monitor from the container on this node. All good. Then I swapped the SSDs and started running additional steps (from step 3) using the same mon container. I have no ceph packages installed on the bare metal box. It looks like mon container doesn't see the disk. podman exec -it ceph-mon-storage2n2-la ceph-volume lvm zap /dev/sdh stderr: lsblk: /dev/sdh: not a block device stderr: error: /dev/sdh: No such file or directory stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. usage: ceph-volume lvm zap [-h] [--destroy] [--osd-id OSD_ID] [--osd-fsid OSD_FSID] [DEVICES [DEVICES ...]] ceph-volume lvm zap: error: Unable to proceed with non-existing device: /dev/sdh Error: exit status 2 root@storage2n2-la:~# ls -l /dev/sd sda sdc sdd sde sdf sdg sdg1 sdg2 sdg5 sdh root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la ceph-volume lvm zap sdh stderr: lsblk: sdh: not a block device stderr: error: sdh: No such file or directory stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. usage: ceph-volume lvm zap [-h] [--destroy] [--osd-id OSD_ID] [--osd-fsid OSD_FSID] [DEVICES [DEVICES ...]] ceph-volume lvm zap: error: Unable to proceed with non-existing device: sdh Error: exit status 2 I execute lsblk and it sees device sdh root@storage2n2-la:~# podman exec -it ceph-mon-storage2n2-la lsblk lsblk: dm-1: failed to get device path lsblk: dm-2: failed to get device path lsblk: dm-4: failed to get device path lsblk: dm-6: failed to get device path lsblk: dm-4: failed to get device path lsblk: dm-2: failed to get device path lsblk: dm-1: failed to get device path lsblk: dm-0: failed to get device path lsblk: dm-0: failed to get device path lsblk: dm-7: failed to get device path lsblk: dm-5: failed to get device path lsblk: dm-7: failed to get device path lsblk: dm-6: failed to get device path lsblk: dm-5: failed to get device path lsblk: dm-3: failed to get device path NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdf 8:80 0 1.8T 0 disk sdd 8:48 0 1.8T 0 disk sdg 8:96 0 223.5G 0 disk |-sdg5 8:101 0 223G 0 part |-sdg1 8:97 487M 0 part `-sdg2 8:98 1K 0 part sde 8:64 0 1.8T 0 disk sdc 8:32 0 3.5T 0 disk sda 8:0 0 3.5T 0 disk sdh 8:112 0 3.5T 0 disk So I use a fellow osd container (osd.5) on the same node and run all of the operations (zap and prepare) successfully. I am suspecting that mon or mgr have no access to /dev or /var/lib while osd containers do. Cluster configured originally by ceph-ansible (nautilus 14.2.2) The question is if I want to replace all disks on a single node, and I have 6 nodes with pools replication 3, is it safe to restart mgr mounting /dev and /var/lib/ceph volumes (not configured right now). I cannot use other osd containers on the same box because my controller reverts from raid to non-raid mode with all disks lost and not just a single one. So I need to replace all 6 osds to run back in containers and the only things will remain operational on node are mon and mgr containers. I prefer not to install a full cluster or client on the bare metal node if possible. Thank you for your help, _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

2 1

ceph mon failed to start
by 徐蕴 22 Oct '19

22 Oct '19

Hi, Our cluster got an unexpected power outage. Ceph mon cannot start after that. The log shows: Running command: '/usr/bin/ceph-mon -f -i 10.10.198.11 --public-addr 10.10.198.11:6789' Corruption: 15 missing files; e.g.: /var/lib/ceph/mon/ceph-10.10.198.11/store.db/2676107.sst Is there any way to fix this problem? Thank you very much! We are running ceph 10.2.10. Br, Xu Yun

2 3

mds log showing msg with HANGUP
by Amudhan P 22 Oct '19

22 Oct '19

Hi, I am getting below error msg in ceph nautilus cluster, do I need to worry about this? Oct 14 06:25:02 mon01 ceph-mds[35067]: 2019-10-14 06:25:02.209 7f55a4c48700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse Oct 14 06:25:02 mon01 ceph-mds[35067]: 2019-10-14 06:25:02.253 7f55a4c48700 -1 received signal: Hangup from (PID: 244322) UID: 0 Oct 15 06:25:01 mon01 ceph-mds[35067]: 2019-10-15 06:25:01.988 7f55a4c48700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse Oct 15 06:25:02 mon01 ceph-mds[35067]: 2019-10-15 06:25:02.040 7f55a4c48700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|r Oct 16 06:25:02 mon01 ceph-mds[35067]: 2019-10-16 06:25:02.646 7f55a4c48700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse Oct 16 06:25:02 mon01 ceph-mds[35067]: 2019-10-16 06:25:02.678 7f55a4c48700 -1 received signal: Hangup from (PID: 305528) UID: 0 Oct 17 06:25:02 mon01 ceph-mds[35067]: 2019-10-17 06:25:02.337 7f55a4c48700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse Oct 17 06:25:02 mon01 ceph-mds[35067]: 2019-10-17 06:25:02.381 7f55a4c48700 -1 received signal: Hangup from (PID: 374957) UID: 0 Oct 18 06:25:01 mon01 ceph-mds[35067]: 2019-10-18 06:25:01.947 7f55a4c48700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse Oct 18 06:25:02 mon01 ceph-mds[35067]: 2019-10-18 06:25:02.015 7f55a4c48700 -1 Fail to open '/proc/436318/cmdline' error = (2) No such file or directory Oct 18 06:25:02 mon01 ceph-mds[35067]: 2019-10-18 06:25:02.015 7f55a4c48700 -1 received signal: Hangup from <unknown> (PID: 436318) UID: 0

2 2

multiple nvme per osd
by Frank R 22 Oct '19

22 Oct '19

Hi all, Has anyone successfully created multiple partitions on an NVME device using ceph-disk? If so, which commands were used?

4 3

Ubuntu Disco with most recent Kernel 5.0.0-32 seems to be instable
by Ranjan Ghosh 21 Oct '19

21 Oct '19

Hi all, it seems Ceph on Ubuntu Disco (19.04) with the most recent kernel 5.0.0-32 is instable. It crashes sometimes after a few hours, sometimes even after a few minutes. I found this bug here on CoreOS: https://github.com/coreos/bugs/issues/2616 Which is exactly also the error message I get ("cache_from_obj: Wrong slab cache. inode_cache but object is from ceph_inode_info") and the problem seems fairly recent. The problem vanished immediately after downgrading the kernel again. Could it be that this isn't exclusively a CoreOS issue? Where should I file this? Ubuntu Launchpad? Seems quite serious issue to me... Thank you / BR Ranjan

2 4

Occasionally ceph.dir.rctime is incorrect (14.2.4 nautilus)
by Toby Darling 21 Oct '19

21 Oct '19

Hi Occasionally, a directory's ceph.dir.rctime isn't updated when it is copied (rsync'd) to ceph, it reads the same as from stat, not the time it was created on ceph. The cluster is all Scientific Linux release 7.7 with ceph 14.2.4 nautilus (stable), the problem happened with kernel client running on one of the mon nodes. The tree that was rsync'd in was about 20TB, 340k files, 3000 directories. In all, I've found just 5 directories suffering this - possibly of interest, they are all empty (although there were 472 empty dirs in total). * dir1 is reported correctly: [root@ceph ~]# ls -ld path/to/dir1 drwxr-xr-x 1 thisuser thisuser 0 Jun 18 2018 path/to/dir1 [root@ceph ~]# getfattr --only-values -n ceph.dir.rctime path/to/dir1 1571035073.09398156844 [root@ceph ~]# stat path/to/dir1 File: ‘path/to/dir1’ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 0h/0d Inode: 1099637142596 Links: 1 Access: (0755/drwxr-xr-x) Uid: (26089/thisuser) Gid: (26089/thisuser) Access: 2019-10-14 07:37:53.397156853 +0100 Modify: 2018-06-18 18:19:12.001114388 +0100 Change: 2019-10-14 07:37:53.398156844 +0100 Birth: - * But dir2's rctime is wrong: [root@ceph ~]# ls -ld path/to/dir1/dir2 drwxr-xr-x 1 thisuser thisuser 0 Jun 18 2018 path/to/dir1/dir2 [root@ceph ~]# getfattr --only-values -n ceph.dir.rctime path/to/dir1/dir2 1529342352.091114388 [root@ceph ~]# stat path/to/dir1/dir2 File: ‘path/to/dir1/dir2’ Size: 0 Blocks: 0 IO Block: 65536 directory Device: 0h/0d Inode: 1099637142960 Links: 1 Access: (0755/drwxr-xr-x) Uid: (26089/thisuser) Gid: (26089/thisuser) Access: 2019-10-14 07:37:53.398156844 +0100 Modify: 2018-06-18 18:19:12.001114388 +0100 Change: 2019-10-14 07:37:53.398156844 +0100 Birth: - Cheers Toby

1 1

RBD Mirror, Clone non primary Image
by yveskretzschmar＠web.de 21 Oct '19

21 Oct '19

Hi, I have a working RBD Mirror Setup using ceph version 14.2.4 on both sides. I want to have a clone of a non primay image. I do it this way: 1. create snapshot of primary image 2. wait for the snapshot to appear on the backup cluster 3. create a clone in backup cluster (using Simplified RBD Image Cloning - no snap protect) 4. delete the snapshot of the primary image - this is to save space because the clone will last a long time 5. if I do rbd --cluster backup snap ls --all IMAGENAME it presents the deleted snapshot in the namespace trash 6. delete the clone 7. I expected that deleting the clone would purge the snapshot from trash, but it doesn't 8. trash purge or trash remove is impossible (read only filesystem) Only possbility for me to purge the trash is to disable und reenable the image mirror. Is there something wrong in this order. Essential for me is the snaphot removal of the primary image. Flatten the clone is no option because of space. Regards, Yves

2 1

Dashboard doesn't respond after failover
by Matthew Stroud 21 Oct '19

21 Oct '19

For some reason the active MGR process just resets the connection after failover. Nothing really sticks out in the logs to explain this. However if I restart the MGR process, it will start responding just fine. Thoughts? Thanks, Matt ________________________________ CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you.

2 2

Ceph BlueFS Superblock Lost
by Winger Cheng 21 Oct '19

21 Oct '19

Hello Everyone， My osd is broken recently, the first 8M size block have been clean. Since I use ceph-volume create bluestore with all wal, db, slow on one disk, I lost the superblock. Thanks to the lvm backup, I save the superblock of Bluestore, but I can't get the bluefs superblock back. What can I do to save the bluefs superblock?

1 0

rgw index large omap
by Frank R 21 Oct '19

21 Oct '19

I have an rgw index pool that is alerting as "large" in 2 of the 3 osds on the PG. The primary has a large omap. The index is definitely in use by the bucket. Any opinions on the best way to solve this? 1. Remove the 2 osds with large index from cluster and rebalance? 2. Delete 2 of the 3 and deep-scrub? 3. something else?

1 0

Ceph Tech Talk October 2019: Ceph at Nasa
by Mike Perez 21 Oct '19

21 Oct '19

Hi Cephers, I'm pleased to announce that we're starting Ceph Tech Talks back up (unfortunately probably the last one for this year due to holidays). On October 24th at 17:00 UTC Kevin Hrpcek will be presenting on Ceph at Nasa, and why they use librados instead of higher-level features. For information to join the live recorded stream, see our tech talk page: https://ceph.com/ceph-tech-talks/ If you're interested in giving your own tech talk, please let me know. -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Re: RadosGW cant list objects when there are too many of them
by Paul Emmerich 21 Oct '19

21 Oct '19

On Mon, Oct 21, 2019 at 11:20 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > Yes listing v2 is not supported yet. I checked metadata osds and all of them are 600gb 10k hdd I dont think this was the issue. > I will test the --allow-unordered 5 million objects in a single bucket and metadata on HDD is a disaster waiting to happen if this continues to grow. rgw setups with large buckets need SSDs (or better NVMe) for metadata if you value availability. Recovering after a node failure will be horrible if you keep this on HDDs. Paul > > > Regards > ________________________________ > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Thursday, October 17, 2019 10:00 AM > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: Re: [ceph-users] RadosGW cant list objects when there are too many of them > > Listing large buckets is slow due to S3 ordering requirements, it's > approximately O(n^2). > However, I wouldn't consider 5M to be a large bucket, it should go to > only ~50 shards which should still perform reasonable. How fast are > your metadata OSDs? > > Try --allow-unordered in radosgw-admin to get an unordered result > which is only O(n) as you'd expect. > > For boto3: I'm not sure if v2 object listing is available yet (I think > it has only been merged into master but has not yet made it into a > release?). It doesn't support unordered listing but there has been > some work to implement it there, not sure about the current state. > > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > > Dear All > > > > I have a bucket with 5 million Objects and I cant list objects with > > radosgw-admin bucket list --bucket=bucket | jq .[].name > > or listing files using boto3 > > > > s3 = boto3.client('s3', > > endpoint_url=credentials['endpoint_url'], > > aws_access_key_id=credentials['access_key'], > > aws_secret_access_key=credentials['secret_key']) > > > > response = s3.list_objects_v2(Bucket=bucket_name) > > for item in response['Contents']: > > print(item['Key']) > > > > what is the solution ? how can I find list of my objects ? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Install error
by masud parvez 21 Oct '19

21 Oct '19

I am trying to install ceph in ubuntu 16.04 by this link https://www.supportsages.com/ceph-part-5-ceph-configuration-on-ubuntu/ but when I am run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 I am facing this error. [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw $ sudo apt-get install ceph-common Reading package lists... Done Building dependency tree Reading state information... Done ceph-common is already the newest version (12.2.12-1xenial). ceph-common set to manually installed. 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 8 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] y Setting up ceph-common (12.2.12-1xenial) ... Setting system user ceph properties..usermod: user ceph is currently used by process 1648 dpkg: error processing package ceph-common (--configure): subprocess installed post-installation script returned error exit status 8 dpkg: dependency problems prevent configuration of ceph-base: ceph-base depends on ceph-common (= 12.2.12-1xenial); however: Package ceph-common is not configured yet. dpkg: error processing package ceph-base (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of ceph-mds: ceph-mds depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-mds (--configure): dependency problems - leaving unconfigured No apport report written because the error message indicates its a followup error from a previous failure. No apport report written because the error message indicates its a followup error from a previous failure. dpkg: dependency problems prevent configuration of ceph-osd: ceph-osd depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-osd (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of ceph-mon: ceph-mon depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-mon (--configure): dependency problems - leaving unconfigured No apport report written because MaxReports is reached already No apport report written because MaxReports is reached already dpkg: dependency problems prevent configuration of ceph-mgr: ceph-mgr depends on ceph-base (= 12.2.12-1xenial); however: Package ceph-base is not configured yet. dpkg: error processing package ceph-mgr (--configure): dependency problems - leaving unconfigured No apport report written because MaxReports is reached already dpkg: dependency problems prevent configuration of ceph: ceph depends on ceph-mgr (= 12.2.12-1xenial); however: Package ceph-mgr is not configured yet. ceph depends on ceph-mon (= 12.2.12-1xenial); however: Package ceph-mon is not configured yet. ceph depends on ceph-osd (= 12.2.12-1xenial); however: Package ceph-osd is not configured yet. dpkg: error processing package ceph (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of radosgw: radosgw depends on ceph-common (= 12.2.12-1xenial); however: Package ceph-common is not configured yet. dpkg: error processing package radosgw (--configure): dependency problems - leaving unconfigured No apport report written because MaxReports is reached already No apport report written because MaxReports is reached already Processing triggers for libc-bin (2.23-0ubuntu11) ... Errors were encountered while processing: ceph-common ceph-base ceph-mds ceph-osd ceph-mon ceph-mgr ceph radosgw E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https I have wasted a lot of time with this error. Could anyone give me a solution?

2 1

Re: RadosGW cant list objects when there are too many of them
by Paul Emmerich 21 Oct '19

21 Oct '19

Listing large buckets is slow due to S3 ordering requirements, it's approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs? Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect. For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > Dear All > > I have a bucket with 5 million Objects and I cant list objects with > radosgw-admin bucket list --bucket=bucket | jq .[].name > or listing files using boto3 > > s3 = boto3.client('s3', > endpoint_url=credentials['endpoint_url'], > aws_access_key_id=credentials['access_key'], > aws_secret_access_key=credentials['secret_key']) > > response = s3.list_objects_v2(Bucket=bucket_name) > for item in response['Contents']: > print(item['Key']) > > what is the solution ? how can I find list of my objects ? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Getting after upgrade to nautilus every few seconds: cluster [DBG] pgmap
by Marc Roos 20 Oct '19

20 Oct '19

Is it really necessary to get these every few seconds? mgr.b (mgr.3714636) 264 : cluster [DBG] pgmap v265: 384

1 0

Re: Module 'rbd_support' has failed: Not found or unloadable
by Jason Dillaman 20 Oct '19

20 Oct '19

What are the caps for your mgr user? Does it have permissions to talk to the OSDs? # ceph auth get mgr.<XYZ> On Sat, Oct 19, 2019 at 6:58 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > 2019-10-19 22:26:12.606 7fe16448cb80 1 mgr[py] Loading python module > 'rbd_support' > 2019-10-19 22:26:14.627 7fe13b494700 -1 mgr load Failed to construct > class in 'rbd_support' > File "/usr/share/ceph/mgr/rbd_support/module.py", line 1326, in > __init__ > File "/usr/share/ceph/mgr/rbd_support/module.py", line 610, in > __init__ > File "/usr/share/ceph/mgr/rbd_support/module.py", line 674, in > init_task_queue > File "/usr/share/ceph/mgr/rbd_support/module.py", line 708, in > load_task_queue > 2019-10-19 22:26:14.628 7fe13b494700 -1 mgr operator() Failed to run > module in active mode ('rbd_support') > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] Module 'rbd_support' has failed: Not found or > unloadable > > > After luminous -> nautilus upgrade I have this error, how to resolve? > > Tried enabling and disabling it. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Jason

3 3

Module 'rbd_support' has failed: Not found or unloadable
by Marc Roos 19 Oct '19

19 Oct '19

After luminous -> nautilus upgrade I have this error, how to resolve? Tried enabling and disabling it.

1 1

Luminous -> nautilus upgrade on centos7 lots of Unknown lvalue logs
by Marc Roos 19 Oct '19

19 Oct '19

Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' I am getting lots these in the log files after updating the rpms Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:16] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:19] Unknown lvalue 'ProtectControlGroups' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:21] Unknown lvalue 'ProtectKernelModules' in section 'Service' Oct 19 22:16:49 c03 systemd: [/usr/lib/systemd/system/ceph-osd@.service:23] Unknown lvalue 'ProtectKernelTunables' in section 'Service'

1 0

ceph balancer do not start
by Jan Peters 18 Oct '19

18 Oct '19

Hello, i use ceph 12.2.12 and would like to activate the ceph balancer. unfortunately no redistribution of the PGs is started: ceph balancer status { "active": true, "plans": [], "mode": "crush-compat" } ceph balancer eval current cluster score 0.023776 (lower is better) ceph config-key dump { "initial_mon_keyring": "AQBLchlbAAAAABAA+5CuVU+8MB69xfc3xAXkjQ==", "mgr/balancer/active": "1", "mgr/balancer/max_misplaced:": "0.01", "mgr/balancer/mode": "crush-compat" } What am I not doing correctly? best regards

1 0

CephFS and 32-bit Inode Numbers
by Dan van der Ster 18 Oct '19

18 Oct '19

Hi all, One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3. I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3. Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?) Thanks! Dan

7 8

OSD node suddenly slow to responding cmd
by Amudhan P 18 Oct '19

18 Oct '19

Hi, I am using Ceph nautilus cluster and i found one of my OSD node running 3 OSD's service suddenly went down and it was very slow typing command. I killed ceph-osd process and system become normal and started all OSD service. After that it becomes normal I figure out that due low memory the system was slow but there is no OOM in any of the logs. I can see heartbeat_check and suicide time out in below msg in OSD logs, but why did this happened? what could be the issue? *osd.1* 2019-10-16 20:49:38.433 7f246f302700 0 bluestore(/var/lib/ceph/osd/ceph-1) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6.20965s, txc = 0x55 a2541f1500 2019-10-16 20:50:05.233 7f247bf7d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15 2019-10-16 20:50:06.721 7f247af7b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15 2019-10-16 20:53:57.881 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.711088 front 2019-10-16 20:4 9:47.684146 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 20:55:26.997 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:44.711031 front 2019-10-16 20:4 9:44.711495 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 20:56:25.513 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019-10-16 20:49:44.484031 front 2019-10-16 20:4 9:44.711337 (oldest deadline 2019-10-16 20:50:03.162338) 2019-10-16 20:57:52.557 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:44.484064 front 2019-10-16 20:4 9:44.711504 (oldest deadline 2019-10-16 20:50:03.162338) 2019-10-16 20:59:22.837 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:44.711483 front 2019-10-16 20:4 9:44.483337 (oldest deadline 2019-10-16 20:50:03.162338) 2019-10-16 21:01:18.761 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019-10-16 20:49:44.711275 front 2019-10-16 20:4 9:44.711352 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 21:02:34.021 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:44.711517 front 2019-10-16 20:4 9:44.711470 (oldest deadline 2019-10-16 20:50:07.937387) 2019-10-16 21:04:26.361 7f247b77c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245d4e4700' had timed out after 60 2019-10-16 21:04:46.225 7f247b77c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had timed out after 15 2019-10-16 21:05:01.453 7f247b77c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f245e4e6700' had suicide timed out after 150 2019-10-16 21:34:06.365 7f0a6105cf80 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-10-16 21:34:06.365 7f0a6105cf80 0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process ceph-osd, pid 203416 2019-10-16 21:34:06.365 7f0a6105cf80 0 pidfile_write: ignore empty --pid-file *osd.4* 2019-10-16 20:53:57.881 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019-10-16 20:49:41.268550 front 2019-10-16 20:49:41.268546 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:55:26.997 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.484184 front 2019-10-16 20:49:44.483961 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:56:25.513 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:41.192151 front 2019-10-16 20:49:41.191856 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:57:52.557 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:44.484278 front 2019-10-16 20:49:47.920080 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 20:59:22.837 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:41.268846 front 2019-10-16 20:49:41.192139 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 21:01:16.561 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019-10-16 20:49:41.380152 front 2019-10-16 20:49:41.380178 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 21:02:15.621 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:43.862767 front 2019-10-16 20:49:44.484153 (oldest deadline 2019-10-16 20:50:07.683860) 2019-10-16 21:04:14.997 7f7efba77700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60 2019-10-16 21:04:31.689 7f7efba77700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7eddfe0700' had timed out after 60 2019-10-16 21:04:56.025 7f7efba77700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7ede7e1700' had timed out after 60 2019-10-16 21:05:10.409 7f7efc278700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60 2019-10-16 21:05:20.813 7f7efc278700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7eddfe0700' had timed out after 60 2019-10-16 21:05:34.149 7f7efc278700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7ede7e1700' had timed out after 60 2019-10-16 21:05:48.465 7f7efb276700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7edd7df700' had timed out after 60 2019-10-16 21:06:15.409 7f7efb276700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7eddfe0700' had timed out after 60 2019-10-16 21:06:32.173 7f7efb276700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f7ede7e1700' had timed out after 60 2019-10-16 21:21:53.125 7f7ee87f5700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2019-10-16 21:22:15.137 7f7ee87f5700 4 rocksdb: [db/db_impl.cc:778] *osd.7* 2019-10-16 20:53:57.881 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019-10-16 20:49:41.380241 front 2019-10-16 20:49:41.380043 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:55:26.997 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019-10-16 20:49:44.484063 front 2019-10-16 20:49:47.920022 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:56:25.513 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019-10-16 20:49:41.379978 front 2019-10-16 20:49:41.380217 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:57:49.685 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019-10-16 20:49:41.380202 front 2019-10-16 20:49:41.380119 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 20:59:18.389 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019-10-16 20:49:43.862826 front 2019-10-16 20:49:43.723800 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 21:01:16.561 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019-10-16 20:49:41.380229 front 2019-10-16 20:49:41.380023 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 21:02:19.197 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019-10-16 20:49:44.484017 front 2019-10-16 20:49:44.484078 (oldest deadline 2019-10-16 20:50:07.683838) 2019-10-16 21:21:53.125 7f959cbe1700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2019-10-16 21:22:20.221 7f959cbe1700 4 rocksdb: [db/db_impl.cc:778] *Journalctl - output* Oct 16 20:53:38 test-node2 systemd-resolved[1005]: Using degraded feature set (TCP) for DNS server 192.168.4.12. Oct 16 20:56:02 test-node2 ceph-osd[1710]: 2019-10-16 20:53:57.881 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019- Oct 16 20:56:29 test-node2 ceph-osd[91197]: 2019-10-16 20:53:57.881 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019 Oct 16 20:53:40 test-node2 systemd-resolved[1005]: Using degraded feature set (TCP) for DNS server 192.168.4.7. Oct 16 20:57:17 test-node2 ceph-osd[1713]: 2019-10-16 20:53:57.881 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6813 osd.1 since back 2019- Oct 16 20:57:17 test-node2 ceph-osd[1713]: 2019-10-16 20:55:26.997 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019- Oct 16 20:57:59 test-node2 ceph-osd[1710]: 2019-10-16 20:55:26.997 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019- Oct 16 20:57:59 test-node2 ceph-osd[1710]: 2019-10-16 20:56:25.513 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019- Oct 16 20:58:43 test-node2 ceph-osd[91197]: 2019-10-16 20:55:26.997 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6805 osd.2 since back 2019 Oct 16 20:58:43 test-node2 ceph-osd[91197]: 2019-10-16 20:56:25.513 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019 Oct 16 20:53:51 test-node2 systemd-networkd[907]: enp5s0: Failed to save LLDP data to /run/systemd/netif/lldp/2: No space left on device Oct 16 21:00:13 test-node2 ceph-osd[1713]: 2019-10-16 20:56:25.513 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6821 osd.3 since back 2019- Oct 16 21:00:13 test-node2 ceph-osd[1713]: 2019-10-16 20:57:52.557 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019- Oct 16 21:01:42 test-node2 ceph-osd[1710]: 2019-10-16 20:57:52.557 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019- Oct 16 21:01:42 test-node2 ceph-osd[1710]: 2019-10-16 20:59:22.837 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019- Oct 16 21:02:33 test-node2 ceph-osd[91197]: 2019-10-16 20:57:49.685 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.112:6805 osd.4 since back 2019 Oct 16 21:02:33 test-node2 ceph-osd[91197]: 2019-10-16 20:59:18.389 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6813 osd.5 since back 2019 Oct 16 20:54:24 test-node2 systemd-networkd[907]: enp5s0: Failed to save LLDP data to /run/systemd/netif/lldp/2: No space left on device Oct 16 21:04:08 test-node2 ceph-osd[1713]: 2019-10-16 20:59:22.837 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019- Oct 16 21:04:08 test-node2 ceph-osd[1713]: 2019-10-16 21:01:16.561 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019- Oct 16 21:04:54 test-node2 ceph-osd[1710]: 2019-10-16 21:01:18.761 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.112:6821 osd.7 since back 2019- Oct 16 21:04:54 test-node2 ceph-osd[1710]: 2019-10-16 21:02:34.021 7f2475d15700 -1 osd.1 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019- Oct 16 21:05:55 test-node2 ceph-osd[91197]: 2019-10-16 21:01:16.561 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.111:6805 osd.6 since back 2019 Oct 16 21:05:55 test-node2 ceph-osd[91197]: 2019-10-16 21:02:19.197 7f95aa3fc700 -1 osd.7 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019 Oct 16 20:54:56 test-node2 systemd-networkd[907]: enp5s0: Failed to save LLDP data to /run/systemd/netif/lldp/2: No space left on device Oct 16 21:07:37 test-node2 ceph-osd[1713]: 2019-10-16 21:02:15.621 7f7ef6010700 -1 osd.4 2035 heartbeat_check: no reply from 192.168.100.113:6821 osd.8 since back 2019- Oct 16 21:08:12 test-node2 ceph-osd[1710]: *** Caught signal (Aborted) ** Oct 16 21:08:12 test-node2 ceph-osd[1710]: in thread 7f245e4e6700 thread_name:tp_osd_tp Oct 16 21:11:49 test-node2 ceph-osd[1710]: /build/ceph-14.2.4/src/common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, Oct 16 21:11:49 test-node2 ceph-osd[1710]: /build/ceph-14.2.4/src/common/HeartbeatMap.cc: 82: ceph_abort_msg("hit suicide timeout") Oct 16 21:12:36 test-node2 ceph-osd[1710]: ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) Oct 16 21:12:36 test-node2 ceph-osd[1710]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocato Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Failed with result 'watchdog'. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Main process exited, code=killed, status=6/ABRT Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Failed with result 'signal'. Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart. Oct 16 21:34:05 test-node2 systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)4.service: Main process exited, code=killed, status=9/KILL Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)4.service: Failed with result 'signal'. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Service hold-off time over, scheduling restart. Oct 16 21:34:05 test-node2 systemd[1]: ceph-osd(a)1.service: Scheduled restart job, restart counter is at 1. Oct 16 21:34:05 test-node2 systemd[1]: Started Ceph object storage daemon osd.4. Oct 16 21:34:05 test-node2 systemd[1]: Started Ceph object storage daemon osd.1. Oct 16 21:34:05 test-node2 systemd[1]: Started Ceph object storage daemon osd.7. Oct 17 06:25:03 test-node2 ceph-osd[203415]: 2019-10-17 06:25:03.652 7f91880ca700 -1 Fail to open '/proc/204405/cmdline' error = (2) No such file or directory Oct 17 06:25:03 test-node2 ceph-osd[203415]: 2019-10-17 06:25:03.680 7f91880ca700 -1 received signal: Hangup from <unknown> (PID: 204405) UID: 0 Oct 17 06:25:03 test-node2 ceph-osd[203415]: 2019-10-17 06:25:03.736 7f91880ca700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph- Oct 17 06:25:04 test-node2 ceph-osd[203417]: 2019-10-17 06:25:03.652 7f2ce27ac700 -1 Fail to open '/proc/204405/cmdline' error = (2) No such file or directory Oct 17 06:25:04 test-node2 ceph-osd[203417]: 2019-10-17 06:25:03.692 7f2ce27ac700 -1 received signal: Hangup from <unknown> (PID: 204405) UID: 0 Oct 17 06:25:04 test-node2 ceph-osd[203417]: 2019-10-17 06:25:03.736 7f2ce27ac700 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph- Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 Fail to open '/proc/204406/cmdline' error = (2) No such file or directory Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 received signal: Hangup from <unknown> (PID: 204406) UID: 0 Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 Fail to open '/proc/204406/cmdline' error = (2) No such file or directory Oct 17 06:25:04 test-node2 ceph-osd[203416]: 2019-10-17 06:25:03.960 7f0a577ab700 -1 received signal: Hangup from <unknown> (PID: 204406) UID: 0

1 0

Re: Change device class in EC profile
by Frank Schilder 18 Oct '19

18 Oct '19

Hi Maks, thanks for looking at this. Unfortunately, this does not answer the question. After steps 1-3 you are exactly in the same situation as I am, namely that the profile attached to the pool is outdated and, therefore, contains invalid information that will be confusing. To check, execute ceph osd pool get POOL_NAME erasure_code_profile and then do ceph osd erasure-code-profile get PROFILE_NAME_RETURNED_ABOVE This will not match the new crush rule unless there was no change. Note that you cannot *set* the EC profile on a pool to match the profile you created your new crush rule from. Hence, the pool info will drag outdated information along. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Maks Kowalik <maks_kowalik(a)poczta.fm> Sent: 18 October 2019 14:01 To: Frank Schilder Subject: Re: [ceph-users] Change device class in EC profile I managed to achieve a similar goal (old EC profile didn't use classes at all) by: 1. Creating a new EC profile. 2. Creating new crush rule based on the EC profile. 3. Changing crush rule for a pool to a newly created one. Kind regards, Maks pt., 18 paź 2019 o 13:07 Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> napisał(a): I recently moved an EC pool from HDD to SSD by changing the device class in the crush rule. I would like to complete this operation by cleaning up a dirty trail. The EC profile attached to this pool is called sr-ec-6-2-hdd and it is easy enough to rename that to sr-ec-6-2-ssd. However, the profile itself contains the device class as well: crush-device-class=hdd [...] I can already see the confusion this will cause in the future. Is this situation one of the few instances where using the --force option is warranted to change the device class of a profile, as in: osd erasure-code-profile set sr-ec-6-2-ssd crush-device-class=ssd --force If not, how can I change the device class of the profile? Many thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

iscsi gate install
by Fyodor Ustinov 18 Oct '19

18 Oct '19

Hi! CEPH documentation requre "tcmu-runner-1.4.0 or newer package", but I can not find this package for Centos. Maybe someone knows where to download this package? WBR, Fyodor.

2 2

Monitor unable to join existing cluster, stuck at probing
by msmit＠smit-it.info 18 Oct '19

18 Oct '19

Hi, I'm currently working on upgrading my existing monitors within my cluster. During the first deployment of this production cluster I made some choices that in hindsight where not the best. But, it worked, I learned and now I wish to mediate my previous bad choices. The cluster exists of three monitors that are currently in quorum and I wish to upgrade each of them by fully removing them from the cluster and rejoining them after a complete reinstall of the os (new hostname, new ip). Therefore I want to maintain quorum by temporary adding a monitor but this won't go as planned as the monitor will join, with `ceph-deploy add monitor mon4` but never leave the probing state (see log below). I have verified all networking and firewall settings and don't notice any connection errors, neither do I see any weird hostnames or ip-addresses in the existing monmap on all the hosts. Also manually confirmed that all the keys on the cluster are the same, so don't suspect a authentication error. Hope someone has any guidance. Thx. Log from mon4 > /var/log/ceph/ceph-mon.mon4.log 2019-10-16 11:21:51.960 7fc709c73a00 0 mon.mon4 does not exist in monmap, will attempt to join an existing cluster 2019-10-16 11:21:51.962 7fc709c73a00 0 using public_addr 10.200.1.104:0/0 -> 10.200.1.104:6789/0 2019-10-16 11:21:51.963 7fc709c73a00 0 starting mon.mon4 rank -1 at public addr 10.200.1.104:6789/0 at bind addr 10.200.1.104:6789/0 mon_data /var/lib/ceph/mon/ceph-mon4 fsid aaf1547b-8944-4f48-b354-93659202c6fe 2019-10-16 11:21:51.964 7fc709c73a00 0 starting mon.mon4 rank -1 at 10.200.1.104:6789/0 mon_data /var/lib/ceph/mon/ceph-mon4 fsid aaf1547b-8944-4f48-b354-93659202c6fe 2019-10-16 11:21:51.965 7fc709c73a00 1 mon.mon4@-1(probing) e0 preinit fsid aaf1547b-8944-4f48-b354-93659202c6fe 2019-10-16 11:21:51.965 7fc709c73a00 1 mon.mon4@-1(probing) e0 initial_members mon1,mon2,mon3,mon4, filtering seed monmap 2019-10-16 11:21:51.965 7fc709c73a00 1 mon.mon4(a)-1(probing).mds e0 Unable to load 'last_metadata' 2019-10-16 11:21:51.967 7fc709c73a00 0 mon.mon4@-1(probing) e0 my rank is now 3 (was -1) 2019-10-16 11:21:54.054 7fc6f934b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch 2019-10-16 11:21:54.054 7fc6f934b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished 2019-10-16 11:21:54.300 7fc6f934b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch 2019-10-16 11:21:54.300 7fc6f934b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished 2019-10-16 11:22:26.967 7fc6f5ad5700 -1 mon.mon4@3(probing) e0 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2019-10-16 11:21:54.055387) 2019-10-16 11:22:31.967 7fc6f5ad5700 -1 mon.mon4@3(probing) e0 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2019-10-16 11:21:54.055387) 2019-10-16 11:22:36.967 7fc6f5ad5700 -1 mon.mon4@3(probing) e0 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2019-10-16 11:21:54.055387) 2019-10-16 11:22:37.478 7fc6f934b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch 2019-10-16 11:22:37.478 7fc6f934b700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished 2019-10-16 11:22:41.968 7fc6f5ad5700 -1 mon.mon4@3(probing) e0 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2019-10-16 11:21:54.055387) 2019-10-16 11:22:46.968 7fc6f5ad5700 -1 mon.mon4@3(probing) e0 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2019-10-16 11:21:54.055387)

3 2

RGW blocking on large objects
by Robert LeBlanc 17 Oct '19

17 Oct '19

We set up a new Nautilus cluster and only have RGW on it. While we had a job doing 200k IOPs of really small objects, I noticed that HAProxy was kicking out RGW backends because they were taking more than 2 seconds to return. We GET a large ~4GB file each minute and use that as a health check to determine if the system is taking too long to service requests. It seems that other IO is being blocked by this large transfer. This seems to be the case with both civetweb and beast. But I'm double checking beast at the moment because I'm not 100% sure we were using it at the start. Any ideas how to mitigate this? It seems that IOs are being scheduled on a thread and if they get unlucky enough to be scheduled behind a big IO, they are just stuck, in this case HAProxy could kick out the backend before the IO is returned and it has to re-request it. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

4 16

RDMA
by gabryel.mason-williams＠diamond.ac.uk 17 Oct '19

17 Oct '19

Hello, I was wondering what user experience was with using Ceph over RDMA? - How you set it up? - Documentation used to set it up? - Known issues when using it? - If you still use it? Kind regards Gabryel Mason-Williams

7 7

Re: ceph-users Digest, Vol 81, Issue 39 Re:RadosGW cant list objects when there are too many of them
by Romit Misra 17 Oct '19

17 Oct '19

Hi Arash, If the number of objects in a bucket are too large in the order of millions, a paginated listing approach works better. There are also ceratin RGW configs, that controls on how big a RGW response (in terms of number objects can be, by default I believe this is 1000) The code for Paginated Listing (Snippet can be modified):- *"* *try: buckethandle = s3_conn_src.get_bucket(bucket_name)* * while True:* * keys = buckethandle.get_all_keys(max_keys=1000,marker = marker) for k in keys:* * #do operation on keys (which are the objects)* * print k.name <http://k.name>* * #update marker* * marker = k.name <http://k.name>* * if keys.is_truncated is False: print "Breaking" break* *except Exception, e:* * print e* *"* Thanks Romit Misra Thanks Romit On Thu, Oct 17, 2019 at 4:18 PM <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. RadosGW cant list objects when there are too many of them > (Arash Shams) > 2. Re: Recovering from a Failed Disk (replication 1) (Burkhard Linke) > 3. Re: RGW blocking on large objects (Paul Emmerich) > 4. Re: RadosGW cant list objects when there are too many of them > (Paul Emmerich) > 5. Re: Recovering from a Failed Disk (replication 1) (Frank Schilder) > > > ---------------------------------------------------------------------- > > Date: Thu, 17 Oct 2019 07:19:12 +0000 > From: Arash Shams <ara4sh(a)hotmail.com> > Subject: [ceph-users] RadosGW cant list objects when there are too > many of them > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <LNXP265MB0508FF1F47CB5EA9C29219FA926D0(a)LNXP265MB0508.GBR > P265.PROD.OUTLOOK.COM> > Content-Type: multipart/alternative; boundary="_000_LNXP265MB0508FF > 1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_" > > --_000_LNXP265MB0508FF1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_ > Content-Type: text/plain; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > Dear All > > I have a bucket with 5 million Objects and I cant list objects with > radosgw-admin bucket list --bucket=3Dbucket | jq .[].name > or listing files using boto3 > > s3 =3D boto3.client('s3', > endpoint_url=3Dcredentials['endpoint_url'], > aws_access_key_id=3Dcredentials['access_key'], > aws_secret_access_key=3Dcredentials['secret_key']) > > response =3D s3.list_objects_v2(Bucket=3Dbucket_name) > for item in response['Contents']: > print(item['Key']) > > what is the solution ? how can I find list of my objects ? > > > > > --_000_LNXP265MB0508FF1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_ > Content-Type: text/html; charset="iso-8859-1" > Content-Transfer-Encoding: quoted-printable > > <html> > <head> > <meta http-equiv=3D"Content-Type" content=3D"text/html; > charset=3Diso-8859-= > 1"> > <style type=3D"text/css" style=3D"display:none;"> P > {margin-top:0;margin-bo= > ttom:0;} </style> > </head> > <body dir=3D"ltr"> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > Dear All > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > I have a bucket with 5 million Objects and I cant list objects with > radosgw-admin bucket list --bucket=3Dbucket | jq .[].name > or listing files using boto3 > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> >     s3 =3D boto3.client('s3', > > <div>                    > = >   endpoint_url=3Dcredentials['endpoint_url'], > </div> > <div>                    > = >   aws_access_key_id=3Dcredentials['access_key'], > </div> > <div>                    > = >   aws_secret_access_key=3Dcredentials['secret_key']) > </div> > <div> > </div> > <div>    response =3D > s3.list_objects_v2(Bucket=3Dbucket_name)<br= > > > </div> > <div>    for item in response['Contents']: > </div> >         print(item['Key']) > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > what is the solution ? how can I find list of my objects ?</div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > > </div> > <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: > 12pt;= > color: rgb(0, 0, 0);"> > > </div> > </body> > </html> > > --_000_LNXP265MB0508FF1F47CB5EA9C29219FA926D0LNXP265MB0508GBRP_-- > > ------------------------------ > > Date: Thu, 17 Oct 2019 10:18:11 +0200 > From: Burkhard Linke <Burkhard.Linke(a)computational.bio.uni-giessen.de> > Subject: [ceph-users] Re: Recovering from a Failed Disk (replication > 1) > To: ceph-users(a)ceph.io > Message-ID: <f341533b-cc7a-865d-0440-79084e0c5707(a)computational.bio.un > i-giessen.de> > Content-Type: multipart/alternative; > boundary="------------71A0D501B0D56489A2F673CA" > > This is a multi-part message in MIME format. > --------------71A0D501B0D56489A2F673CA > Content-Type: text/plain; charset=utf-8; format=flowed > Content-Transfer-Encoding: 7bit > > Hi, > > > On 10/17/19 5:56 AM, Ashley Merrick wrote: > > I think your better off doing the DD method, you can export and import > > a PG at a time (ceph-objectstore-tool) > > > > But if the disk is failing a DD is probably your best method. > > > In case of hardware problems or broken sectors, I would recommend > 'dd_rescue' instead of dd. It can handle broken sectors, automatic > retries, skipping etc. > > > You will definitely need a second disk to rescue to. > > > Regards, > > Burkhard > > > > --------------71A0D501B0D56489A2F673CA > Content-Type: text/html; charset=utf-8 > Content-Transfer-Encoding: 7bit > > <html> > <head> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> > </head> > <body text="#000000" bgcolor="#FFFFFF"> > Hi, > > > <div class="moz-cite-prefix">On 10/17/19 5:56 AM, Ashley Merrick > wrote: > </div> > <blockquote type="cite" > cite=" > mid:16dd7dc0f88.e12acd11469078.7469588663842930162@amerrick.co.uk"> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> > <div style="font-family: Verdana, Arial, Helvetica, sans-serif; > font-size: 10pt;">I think your better off doing the DD method, > you can export and import a PG at a time > (ceph-objectstore-tool) > > But if the disk is failing a DD is probably your best method. > </div> > </blockquote> > > > In case of hardware problems or broken sectors, I would recommend > 'dd_rescue' instead of dd. It can handle broken sectors, automatic > retries, skipping etc. > > > You will definitely need a second disk to rescue to. > > > Regards, > Burkhard > > </body> > </html> > > --------------71A0D501B0D56489A2F673CA-- > > ------------------------------ > > Date: Thu, 17 Oct 2019 11:50:37 +0200 > From: Paul Emmerich <paul.emmerich(a)croit.io> > Subject: [ceph-users] Re: RGW blocking on large objects > To: Robert LeBlanc <robert(a)leblancnet.us> > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > < > CAD9yTbFSpxo1cMAAZ56YcwKxY2dcv0KRPcWYapT2fLmYhmrLkg(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > On Thu, Oct 17, 2019 at 12:17 AM Robert LeBlanc <robert(a)leblancnet.us> > wrote: > > > > On Wed, Oct 16, 2019 at 2:50 PM Paul Emmerich <paul.emmerich(a)croit.io> > wrote: > > > > > > On Wed, Oct 16, 2019 at 11:23 PM Robert LeBlanc <robert(a)leblancnet.us> > wrote: > > > > > > > > On Tue, Oct 15, 2019 at 8:05 AM Robert LeBlanc <robert(a)leblancnet.us> > wrote: > > > > > > > > > > On Mon, Oct 14, 2019 at 2:58 PM Paul Emmerich < > paul.emmerich(a)croit.io> wrote: > > > > > > > > > > > > Could the 4 GB GET limit saturate the connection from rgw to > Ceph? > > > > > > Simple to test: just rate-limit the health check GET > > > > > > > > > > I don't think so, we have dual 25Gbp in a LAG, so Ceph to RGW has > > > > > multiple paths, but we aren't balancing on port yet, so RGW to > HAProxy > > > > > is probably limited to one link. > > > > > > > > > > > Did you increase "objecter inflight ops" and "objecter inflight > op bytes"? > > > > > > You absolutely should adjust these settings for large RGW setups, > > > > > > defaults of 1024 and 100 MB are way too low for many RGW setups, > we > > > > > > default to 8192 and 800MB > > > > > > > > On Nautilus the defaults already seem to be: > > > > objecter_inflight_op_bytes 104857600 > > > > default > > > = 100MiB > > > > > > > objecter_inflight_ops 24576 > > > > default > > > > > > not sure where you got this from, but the default is still 1024 even > > > in master: > https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/… > > > > Looks like it is overridden in > > > https://github.com/ceph/ceph/blob/4774808cb2923f65f6919fe8be5f98917075cdd7/… > > you are right, this is new in Nautilus. Last time I had to play around > with these settings was indeed on a Mimic deployment. > > > I'm just not > > understanding how your suggestions would help, the problem doesn't > > seem to be on the RADOS side (which it appears your tweaks target), > > but on the HTTP side as an HTTP health check takes a long time to come > > back when a big transfer is going on. > > I was guessing a bottleneck on the RADOS side because you mentioned > that you tried both civetweb and beast, somewhat unlikely to run into > the exact same problem with both > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > ------------------------------ > > Date: Thu, 17 Oct 2019 12:00:20 +0200 > From: Paul Emmerich <paul.emmerich(a)croit.io> > Subject: [ceph-users] Re: RadosGW cant list objects when there are too > many of them > To: Arash Shams <ara4sh(a)hotmail.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > <CAD9yTbGoCGPh= > Ba5tQqBvCLu9uUKo0KmjR9E3ayo6DkR-E2bxQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Listing large buckets is slow due to S3 ordering requirements, it's > approximately O(n^2). > However, I wouldn't consider 5M to be a large bucket, it should go to > only ~50 shards which should still perform reasonable. How fast are > your metadata OSDs? > > Try --allow-unordered in radosgw-admin to get an unordered result > which is only O(n) as you'd expect. > > For boto3: I'm not sure if v2 object listing is available yet (I think > it has only been merged into master but has not yet made it into a > release?). It doesn't support unordered listing but there has been > some work to implement it there, not sure about the current state. > > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh(a)hotmail.com> wrote: > > > > Dear All > > > > I have a bucket with 5 million Objects and I cant list objects with > > radosgw-admin bucket list --bucket=bucket | jq .[].name > > or listing files using boto3 > > > > s3 = boto3.client('s3', > > endpoint_url=credentials['endpoint_url'], > > aws_access_key_id=credentials['access_key'], > > aws_secret_access_key=credentials['secret_key']) > > > > response = s3.list_objects_v2(Bucket=bucket_name) > > for item in response['Contents']: > > print(item['Key']) > > > > what is the solution ? how can I find list of my objects ? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Thu, 17 Oct 2019 10:46:53 +0000 > From: Frank Schilder <frans(a)dtu.dk> > Subject: [ceph-users] Re: Recovering from a Failed Disk (replication > 1) > To: vladimir franciz blando <vladimir.blando(a)gmail.com>, > "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <58e22bc6345b48718dbce7238a06e35d(a)dtu.dk> > Content-Type: text/plain; charset="utf-8" > > You probably need to attempt a physical data rescue. Data access will be > lost until done. > > First thing is shut down the OSD to avoid any further damage to the disk. > Second thing is to try ddrescue, repair data on a copy if possible and > then create a clone on a new disk from the copy. > If this doesn't help and you really need that last bit of data, you might > need support from one of those companies that restore disk data with > electron microscopy. > > I successfully transferred OSDs between disks using ddrescue. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: vladimir franciz blando <vladimir.blando(a)gmail.com> > Sent: 17 October 2019 05:29:13 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Recovering from a Failed Disk (replication 1) > > Hi, > > I have a not ideal setup on one of my cluster, 3 ceph nodes but using > replication 1 on all pools (don't ask me why replication 1, it's a long > story). > > So it has come to this situation that a disk keeps on crashing, possible a > hardware failure and I need to recover from that. > > What's my best option for me to recover the data from the failed disk and > transfer it to the other healthy disks? > > This cluster is using Firefly > > - Vlad > [ > https://mailfoogae.appspot.com/t?sender=admxhZGltaXIuYmxhbmRvQGdtYWlsLmNvbQ… > <https://mailfoogae.appspot.com/t?sender=admxhZGltaXIuYmxhbmRvQGdtYWlsLmNvbQ…> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 81, Issue 39 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

Recovering from a Failed Disk (replication 1)
by vladimir franciz blando 17 Oct '19

17 Oct '19

Hi, I have a not ideal setup on one of my cluster, 3 ceph nodes but using replication 1 on all pools (don't ask me why replication 1, it's a long story). So it has come to this situation that a disk keeps on crashing, possible a hardware failure and I need to recover from that. What's my best option for me to recover the data from the failed disk and transfer it to the other healthy disks? This cluster is using Firefly - Vlad ᐧ

6 8

RadosGW cant list objects when there are too many of them
by Arash Shams 17 Oct '19

17 Oct '19

Dear All I have a bucket with 5 million Objects and I cant list objects with radosgw-admin bucket list --bucket=bucket | jq .[].name or listing files using boto3 s3 = boto3.client('s3', endpoint_url=credentials['endpoint_url'], aws_access_key_id=credentials['access_key'], aws_secret_access_key=credentials['secret_key']) response = s3.list_objects_v2(Bucket=bucket_name) for item in response['Contents']: print(item['Key']) what is the solution ? how can I find list of my objects ?

1 0

Dealing with changing EC Rules with drive classifications
by Jeremi Avenant 16 Oct '19

16 Oct '19

Good day I'm currently administrating a Ceph cluster that consists out of HDDs & SSDs. The rule for cephfs_data (ec) is to write to both these drive classifications (HDD+SSD). I would like to change it so that cephfs_metadata (non-ec) writes to SSD & cephfs_data (erasure encoded "ec") writes to HDD since we're experiencing high disk latency. 1) The first option to come to mind would be to migrate each pool to a new rule but this would mean moving a tonne of data around. (How is disk space calculated on this, if I use 600 TB in an EC pool, do I need another 600 TB pool to move it over, or does it shrink the existing pool as it inflates the new pool while moving?) 2) I would like to know if the alternative is possible: i.e. Delete the SSDs from the default host bucket (leave everything as it is) and move the metadata pool to the SSD based crush rule. However I'm not sure if this is possible as it will be deleting a leaf from a bucket in our default root. Which means when you add a new SSD osd where does it end up? crush map - http://pastefile.fr/6f37e7e594a61d0edd9dc947349c756b ceph osd pool ls detail - http://pastefile.fr/0f215e1252ec58c144d9abfe1688adc8 osd tree - http://pastefile.fr/2acdd377a2db021b6af2996929b85082 If anyone has any input it would be greatly appreciated. Regards -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600

3 4

Please help me understand this large omap object found message.
by Robert LeBlanc 16 Oct '19

16 Oct '19

I've been searching around trying to learn about this, but it doesn't seem to be in index sharding problem, so I'm not sure how to approach it and I'm still new to RGW. This is what is in the cluster logs: Large omap object found. Object: 7:f2aa93d5:::.dir.b614530a-c47b-4109-a817-f90c1165bd50.204287.1:head Key count: 3626157 Size (bytes): 912540654 We primarily have one bucket and it currently has 64 shards, so the auto sharding seems to be working right. The '.dir' component of the object seems to indicate that it's not an index. Please help me understand what is going on here and what should be the best course of action. Thank you, ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

1 0

mix sata/sas same pool
by Frank R 16 Oct '19

16 Oct '19

I have inherited a cluster where about 30% of the osds in a pool are 7200 SAS. The other 70% are 7200 SATA. Should I look into creating 2 pools or will this likely not be a huge deal?

3 2

File listing with browser
by Rok Jaklič 16 Oct '19

16 Oct '19

Hi, I installed ceph object gateway and I have put one test object onto storage. I can see it with rados -p mytest ls How do I setup ceph that users can access (download,upload) files to this pool? Kind regards, Rok

4 3

MDS Crashes at “ceph fs volume v011”
by Guilherme Geronimo 16 Oct '19

16 Oct '19

Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3 ==== osd_op_reply(21 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3715233608 0 0) 0x2776340 con 0x18bd500 Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544 Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents [1,head] Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head] ~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17 15:29:27.000000 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800] Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482 Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205 7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1: (()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]#012 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d) [0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels --- Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 5 none Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 1 lockdep Cluster status information: cluster: id: b8205875-e56f-4280-9e52-6aab9c758586 health: HEALTH_WARN 1 filesystem is degraded 1 nearfull osd(s) 11 pool(s) nearfull services: mon: 3 daemons, quorum mon1,mon2,mon3 mgr: mon1(active), standbys: mon2, mon3 mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)} osd: 90 osds: 90 up, 90 in data: pools: 11 pools, 1984 pgs objects: 75.99 M objects, 285 TiB usage: 457 TiB used, 181 TiB / 639 TiB avail pgs: 1896 active+clean 87 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing io: client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr Has anyone seen anything like this? Regards, Arthur

1 0

CEPH HOLDING : un évènement à organiser ?
by Groupe Partouche 16 Oct '19

16 Oct '19

Se désabonner des communications du groupe Partouche L'email ne s'affiche pas correctement ? Cliquez ici. Offre dédiée aux entreprises   Vous souhaitez organiser un séminaire, un cocktail, une soirée de lancement de produit ou tout autre évènement pour votre entreprise ? découvrez nos lieux Offrez-vous des lieux atypiques dans les plus belles destinations de France Nous serons ravis de vous faire bénéficier de 15% de remise* sur votre évènement avec le code OFFREME2019 pour toute nouvelle réservation sur notre site. Confort, détente, divertissement et plaisir des papilles s'allient pour ne faire qu'un, une unité de prestations haut de gamme au service de votre évènement sur-mesure. * 15% de remise pour toute nouvelle réservation entre le 1er novembre 2019 et le 31 mars 2020 sous réserve de disponibilité et pour un minimum de 30 personnes. Ce message vous est adressé par le Groupe Partouche. En application du Règlement du 27 avril 2016 relatif à la protection des données personnelles, vous disposez sur ces données de droits (i) d'accès, (ii) de rectification, (iii) d'effacement, (iv) de limitation et (v) de portabilité de vos données vers un prestataire tiers le cas échéant. Vous pouvez exercer ces droits par courrier postal adressé à : Groupe Partouche / 141 Bis rue de Saussure - 75017 Paris Si vous ne souhaitez plus recevoir d’emails de notre part, vous pouvez vous désinscrire. Conformément aux dispositions de la loi informatique et Libertés du 6 janvier 1978, vous disposez d’un droit d'accès, de rectification et d'opposition. Afin de l'exercer, merci de cliquer sur le lien de désabonnement suivant : Se désabonner Les données de notre Programme proviennent des fichiers des entreprises partenaires et de registres d'entreprises. Vous disposez sur les données personnelles vous concernant de droits d'accès, de rectification, d'effacement, de limitation, d'opposition, de portabilité et du droit de retirer votre consentement à tout moment en suivant ce lien. https://c.evenements-btob.com/unsubscribe?&url=UNSUBSCRIBE&uid=2531…

1 0

Ceph Day Content & Sponsors Needed
by Mike Perez 16 Oct '19

16 Oct '19

Hi everyone, We have some exciting Ceph Days coming up and we're still looking for content and sponsors. Take a look: * Ceph Day Argentina (Spanish speakers only) - October 16: https://ceph.io/cephdays/ceph-day-argentina-2019/ * Ceph Day London - October 24: https://ceph.io/cephdays/ceph-day-london-2019/ * Ceph Day Poland - October 28: https://ceph.io/cephdays/ceph-day-poland-2019/ * Ceph Day San Diego - November 18: https://ceph.io/cephdays/ceph-rook-day-san-diego-2019/ If you or you know someone who can speak at these events please follow the call for proposal links on each page. If you need help working on your title/abstract, please feel free to reach out to me directly. If your company is interested in sponsoring a Ceph Day, please take a look at our brochure: https://ceph.io/wp-content/uploads/2019/10/Ceph-Day-Partner-Sponsorship.pdf Thanks! -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 1

MDS Crashes on “ceph fs volume v011”
by Guilherme Geronimo 15 Oct '19

15 Oct '19

Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3 ==== osd_op_reply(21 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3715233608 0 0) 0x2776340 con 0x18bd500 Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544 Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents [1,head] Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head] ~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17 15:29:27.000000 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800] Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482 Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205 7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1: (()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]#012 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d) [0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels --- Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 5 none Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 1 lockdep Cluster status information: cluster: id: b8205875-e56f-4280-9e52-6aab9c758586 health: HEALTH_WARN 1 filesystem is degraded 1 nearfull osd(s) 11 pool(s) nearfull services: mon: 3 daemons, quorum mon1,mon2,mon3 mgr: mon1(active), standbys: mon2, mon3 mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)} osd: 90 osds: 90 up, 90 in data: pools: 11 pools, 1984 pgs objects: 75.99 M objects, 285 TiB usage: 457 TiB used, 181 TiB / 639 TiB avail pgs: 1896 active+clean 87 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing io: client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr Has anyone seen anything like this? Regards, []'s Arthur Enviado via iPhone

1 0

Crashed MDS (segfault)
by Gustavo Tonini 15 Oct '19

15 Oct '19

Dear ceph users, we're experiencing a segfault during MDS startup (replay process) which is making our FS inaccessible. MDS log messages: Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c08f49700 1 -- 192.168.8.195:6800/3181891717 <== osd.26 192.168.8.209:6821/2419345 3 ==== osd_op_reply(21 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (3715233608 0 0) 0x2776340 con 0x18bd500 Oct 15 03:41:39.894584 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched got 0 and 544 Oct 15 03:41:39.894658 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.snaprealm(0x100 seq 1 0x1799c00) open_parents [1,head] Oct 15 03:41:39.894735 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x100) _fetched [inode 0x100 [...2,head] ~mds0/ auth v275131 snaprealm=0x1799c00 f(v0 1=1+0) n(v76166 rc2020-07-17 15:29:27.000000 b41838692297 -3184=-3168+-16)/n() (iversion lock) 0x18bf800] Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 MDSIOContextBase::complete: 18C_IO_Inode_Fetched Oct 15 03:41:39.894821 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) _fetched got 0 and 482 Oct 15 03:41:39.894891 mds1 ceph-mds: -472> 2019-10-15 00:40:30.201 7f3c00589700 10 mds.0.cache.ino(0x1) magic is 'ceph fs volume v011' (expecting 'ceph fs volume v011') Oct 15 03:41:39.894958 mds1 ceph-mds: -472> 2019-10-15 00:40:30.205 7f3c00589700 -1 *** Caught signal (Segmentation fault) **#012 in thread 7f3c00589700 thread_name:fn_anonymous#012#012 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)#012 1: (()+0x11390) [0x7f3c0e48a390]#012 2: (operator<<(std::ostream&, SnapRealm const&)+0x42) [0x72cb92]#012 3: (SnapRealm::merge_to(SnapRealm*)+0x308) [0x72f488]#012 4: (CInode::decode_snap_blob(ceph::buffer::list&)+0x53) [0x6e1f63]#012 5: (CInode::decode_store(ceph::buffer::list::iterator&)+0x76) [0x702b86]#012 6: (CInode::_fetched(ceph::buffer::list&, ceph::buffer::list&, Context*)+0x1b2) [0x702da2]#012 7: (MDSIOContextBase::complete(int)+0x119) [0x74fcc9]#012 8: (Finisher::finisher_thread_entry()+0x12e) [0x7f3c0ebffece]#012 9: (()+0x76ba) [0x7f3c0e4806ba]#012 10: (clone()+0x6d) [0x7f3c0dca941d]#012 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 15 03:41:39.895400 mds1 ceph-mds: --- logging levels --- Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 5 none Oct 15 03:41:39.895473 mds1 ceph-mds: 0/ 1 lockdep Cluster status information: cluster: id: b8205875-e56f-4280-9e52-6aab9c758586 health: HEALTH_WARN 1 filesystem is degraded 1 nearfull osd(s) 11 pool(s) nearfull services: mon: 3 daemons, quorum mon1,mon2,mon3 mgr: mon1(active), standbys: mon2, mon3 mds: fs_padrao-1/1/1 up {0=mds1=up:replay(laggy or crashed)} osd: 90 osds: 90 up, 90 in data: pools: 11 pools, 1984 pgs objects: 75.99 M objects, 285 TiB usage: 457 TiB used, 181 TiB / 639 TiB avail pgs: 1896 active+clean 87 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing io: client: 89 KiB/s wr, 0 op/s rd, 3 op/s wr Has anyone seen anything like this? Regards, Gustavo.

1 0

run-s3tests.sh against Nautilus
by Francisco Londono 15 Oct '19

15 Oct '19

I ran the s3 test (run-s3tests.sh) in vstart mode against Nautilus. Is there a better guide out there than this one? https://docs.ceph.com/ceph-prs/17381/dev/#testing-how-to-run-s3-tests-local… Thus far I ran into a ceph.conf parse issue, a keyring permission issue and a radosgw crash. + radosgw-admin user create --uid=s3test1 --display-name=tester1 --access-key=access1 --secret=secret1 --email=tester1(a)ceph.com server name not found: [v2:xx.x.x.xx:40466 (Name or service not known) unable to parse addrs in '[v2:xx.x.x.xx:40466,v1:xx.x.x.xx:40467]' couldn't init storage provider The ceph.conf under the global header shows: mon host = [v2:xx.x.x.xx:40466,v1:xx.x.x.xx:40467] In Luminous, the ceph.conf created used to be in this format: host = (hostname) mon addr = xx.x.x.xx:6789 Thanks.

1 0

Librados in openstack
by solarflow99 15 Oct '19

15 Oct '19

I was wondering if this is provided somehow? All I see is rbd and radosgw mentioned. If you have applications built with librados surely openstack must have a way to provide it?

1 0

Recurring issue: PG is inconsistent, but lists no inconsistent objects
by Florian Haas 15 Oct '19

15 Oct '19

Hello, I am running into an "interesting" issue with a PG that is being flagged as inconsistent during scrub (causing the cluster to go to HEALTH_ERR), but doesn't actually appear to contain any inconsistent objects. $ ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent OSD_SCRUB_ERRORS 1 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 10.10d is active+clean+inconsistent, acting [15,13] $ rados list-inconsistent-obj 10.10d {"epoch":12138,"inconsistents":[]} "ceph pg query" (see below) on that PG does report num_scrub_errors=1, num_shallow_scrub_errors=1, and num_objects_dirty=1. "osd scrub auto repair = true" is set on all OSDs, but the PG never auto-repairs. (This is a test cluster, the pool size is 2 — this may preclude auto repair from ever kicking in; I'm not sure on that one.) "ceph pg repair" does repair, but the issue reappears on the next scheduled scrub. This issue was first discovered while the cluster was on Jewel/Filestore. In an event like this I would normally suspect either a problem with an individual OSD, or a bug in the FileStore code. But the cluster has had *all* of it's OSDs replaced since, as part of a full Jewel→Luminous→Nautilus upgrade and a FileStore→BlueStore conversion. The issue still persists. A full "ceph pg 10.10d query" result is below. If anyone has ideas on how to permanently fix this issue, I'd be most grateful. Thanks! Cheers, Florian { "state": "active+clean+inconsistent", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 12143, "up": [ 15, 13 ], "acting": [ 15, 13 ], "acting_recovery_backfill": [ "13", "15" ], "info": { "pgid": "10.10d", "last_update": "100'11", "last_complete": "100'11", "log_tail": "0'0", "last_user_version": 11, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 45, "epoch_pool_created": 45, "last_epoch_started": 12139, "last_interval_started": 12138, "last_epoch_clean": 12139, "last_interval_clean": 12138, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 12138, "same_interval_since": 12138, "same_primary_since": 12114, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-14 08:33:57.347097", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946" }, "stats": { "version": "100'11", "reported_seq": "4927", "reported_epoch": "12143", "state": "active+clean+inconsistent", "last_fresh": "2019-10-14 08:33:57.347147", "last_change": "2019-10-14 08:33:57.347147", "last_active": "2019-10-14 08:33:57.347147", "last_peered": "2019-10-14 08:33:57.347147", "last_clean": "2019-10-14 08:33:57.347147", "last_became_active": "2019-10-11 14:44:09.312226", "last_became_peered": "2019-10-11 14:44:09.312226", "last_unstale": "2019-10-14 08:33:57.347147", "last_undegraded": "2019-10-14 08:33:57.347147", "last_fullsized": "2019-10-14 08:33:57.347147", "mapping_epoch": 12138, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 45, "last_epoch_clean": 12139, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-14 08:33:57.347097", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946", "log_size": 11, "ondisk_log_size": 11, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": true, "pin_stats_invalid": true, "manifest_stats_invalid": true, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 11, "num_objects": 1, "num_object_clones": 0, "num_object_copies": 2, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 1, "num_whiteouts": 0, "num_read": 33, "num_read_kb": 22, "num_write": 11, "num_write_kb": 6, "num_scrub_errors": 1, "num_shallow_scrub_errors": 1, "num_deep_scrub_errors": 0, "num_objects_recovered": 2, "num_bytes_recovered": 22, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 15, 13 ], "acting": [ 15, 13 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 15, "acting_primary": 15, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 12139, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "13", "pgid": "10.10d", "last_update": "100'11", "last_complete": "100'11", "log_tail": "0'0", "last_user_version": 11, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [], "history": { "epoch_created": 45, "epoch_pool_created": 45, "last_epoch_started": 12139, "last_interval_started": 12138, "last_epoch_clean": 12139, "last_interval_clean": 12138, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 12138, "same_interval_since": 12138, "same_primary_since": 12114, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-14 08:33:57.347097", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946" }, "stats": { "version": "100'11", "reported_seq": "36", "reported_epoch": "12113", "state": "active+undersized+degraded", "last_fresh": "2019-10-11 14:39:58.946532", "last_change": "2019-10-11 14:39:58.924989", "last_active": "2019-10-11 14:39:58.946532", "last_peered": "2019-10-11 14:39:58.946532", "last_clean": "2014-11-05 15:48:35.131248", "last_became_active": "2019-10-11 14:39:58.924989", "last_became_peered": "2019-10-11 14:39:58.924989", "last_unstale": "2019-10-11 14:39:58.946532", "last_undegraded": "2019-10-11 14:39:58.892352", "last_fullsized": "2019-10-11 14:39:58.892331", "mapping_epoch": 12138, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 45, "last_epoch_clean": 12103, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "100'11", "last_scrub_stamp": "2019-10-11 14:09:29.016946", "last_deep_scrub": "100'11", "last_deep_scrub_stamp": "2019-10-11 14:09:29.016946", "last_clean_scrub_stamp": "2019-10-11 14:09:29.016946", "log_size": 11, "ondisk_log_size": 11, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": true, "pin_stats_invalid": true, "manifest_stats_invalid": true, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 11, "num_objects": 1, "num_object_clones": 0, "num_object_copies": 2, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 1, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 1, "num_whiteouts": 0, "num_read": 33, "num_read_kb": 22, "num_write": 11, "num_write_kb": 6, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 2, "num_bytes_recovered": 22, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 15, 13 ], "acting": [ 15, 13 ], "avail_no_missing": [ "13" ], "object_location_counts": [ { "shards": "13", "objects": 1 } ], "blocked_by": [], "up_primary": 15, "acting_primary": 15, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 12139, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2019-10-11 14:44:09.175574", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "pull_from_peer": [], "pushing": [] } }, "scrub": { "scrubber.epoch_start": "12138", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2019-10-11 14:44:08.833757" } ], "agent_state": {} }

3 8

Corrupted block.db for osd. How to extract particular PG from that osd?
by Alexey Kalinkin 15 Oct '19

15 Oct '19

Hello cephers! We're lost block.db file for one of our osd. This results in down osd and incomplete PG. Block file from osd, which symlinks to particular /dev folder is live and correct. My main question is there any thereoretical possibility to extract particular PG from osd which block.db device has been lost? Is there any way I can make it using ceph-objectstore-tool or ceph-bluestore-tool? We're running ceph version 12.2.12 -- Very truly yours, Kalinkin Alexey Research fellow of Epigenetic lab, RCMG.

2 1